Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.lang.python > #78183

Re: program to generate data helpful in finding duplicate large files

Path csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.xs4all.nl!newsfeed4.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path <ckaynor@zindagigames.com>
X-Original-To python-list@python.org
Delivered-To python-list@mail.python.org
X-Spam-Status OK 0.001
X-Spam-Evidence '*H*': 1.00; '*S*': 0.00; 'python,': 0.02; 'explicitly': 0.05; 'binary': 0.07; 'modified': 0.07; 'utf-8': 0.07; '22,': 0.09; 'converted': 0.09; 'differently.': 0.09; 'subject:files': 0.09; 'windows,': 0.09; 'python': 0.11; '"\\r\\n"': 0.16; "'r'": 0.16; "'rb')": 0.16; 'file).': 0.16; 'newlines': 0.16; 'subject:program': 0.16; 'files.': 0.16; 'wrote:': 0.18; 'looked': 0.18; 'thu,': 0.19; 'not,': 0.20; 'seems': 0.21; 'platforms': 0.22; 'to:name:python- list@python.org': 0.22; '(such': 0.24; 'bytes': 0.24; 'certainly': 0.24; 'specify': 0.24; 'mon,': 0.24; '(or': 0.24; '&gt;': 0.26; 'received:74.125.82.174': 0.27; 'header:In-Reply-To:1': 0.27; 'chris': 0.29; 'mode': 0.30; 'especially': 0.30; 'message- id:@mail.gmail.com': 0.30; 'url:mailman': 0.30; 'went': 0.31; 'sep': 0.31; 'stands': 0.31; 'universal': 0.31; 'file': 0.32; 'text': 0.33; 'url:python': 0.33; 'raw': 0.33; 'sense': 0.34; 'received:74.125.82': 0.34; "can't": 0.35; 'convert': 0.35; 'received:google.com': 0.35; 'subject:data': 0.36; 'url:listinfo': 0.36; 'doing': 0.36; 'url:org': 0.36; 'should': 0.36; 'skip:& 10': 0.38; 'to:addr:python-list': 0.38; 'files': 0.38; 'pm,': 0.38; 'received:74.125': 0.39; 'does': 0.39; 'bad': 0.39; 'to:addr:python.org': 0.39; 'skip:p 20': 0.39; 'url:mail': 0.40; 'ensure': 0.60; 'even': 0.60; 'read': 0.60; 'truly': 0.60; 'most': 0.60; 'due': 0.66; 'importantly,': 0.68; 'results': 0.69; 'therefore': 0.72; '2014,': 0.84; 'treating': 0.84
X-Google-DKIM-Signature v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:content-type; bh=ls8dgF0p/p92PvaLmAvFc96i72sBeIEjvWDngdIcibo=; b=izBlaYj2WLgYTGxnwUhJB6dvLar7XsmN/PCHw4x+jw1gXVtLYl89aNW34VK/Xlq3LH wML/sRLRzSFw+N8tWHCzLZXMhPc+YyvgUmPV11pSwVWle0uteSILxrcaJAjoeQV1+JcK TGqjrqnX9bTYnm3x/NZbgqtVckRBC8GQVE2uUPj8xasSOL9RxZ1EjIkGl6xHIBd5ZgTL D4Rq/jSPe6XCQRGOv5IL9b1cqd96s8WWYKTyfnSPy0y9kaFHm1fCFsJfax5qpGwHn/sm b2Zo0pycsc1vLD+VcyHDQg/qkVSv17r1Dp953skbVLv8kVDolxB7rr0h47zGM+04llGd ANtw==
X-Gm-Message-State ALoCoQmJM/k20XN/gfjjBPT/nl//zxcixDhkIsFdK1LPGCjNnsUbJvJ3ECHavNKRBF7y9XXd9iCv
X-Received by 10.180.91.70 with SMTP id cc6mr17710713wib.66.1411415266638; Mon, 22 Sep 2014 12:47:46 -0700 (PDT)
MIME-Version 1.0
In-Reply-To <1411414443.4182630.170466465.768582FE@webmail.messagingengine.com>
References <CALDD_==AYbQNPu29jRoLFp8WPZaZ9mMs79334-m_z3dgdxZRJw@mail.gmail.com> <CALvWhxvvtyH0Us8+9eg+JBcooUvq1kiTVqmfU0rg1XTtcJGmVg@mail.gmail.com> <1411414443.4182630.170466465.768582FE@webmail.messagingengine.com>
From Chris Kaynor <ckaynor@zindagigames.com>
Date Mon, 22 Sep 2014 12:47:26 -0700
Subject Re: program to generate data helpful in finding duplicate large files
To "python-list@python.org" <python-list@python.org>
Content-Type multipart/alternative; boundary=f46d0438edd79958050503acb91a
X-BeenThere python-list@python.org
X-Mailman-Version 2.1.15
Precedence list
List-Id General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive <http://mail.python.org/pipermail/python-list/>
List-Post <mailto:python-list@python.org>
List-Help <mailto:python-list-request@python.org?subject=help>
List-Subscribe <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups comp.lang.python
Message-ID <mailman.14235.1411415617.18130.python-list@python.org> (permalink)
Lines 76
NNTP-Posting-Host 2001:888:2000:d::a6
X-Trace 1411415617 news.xs4all.nl 2941 [2001:888:2000:d::a6]:53885
X-Complaints-To abuse@xs4all.nl
Xref csiph.com comp.lang.python:78183

Show key headers only | View raw


[Multipart message — attachments visible in raw view] - view raw

I went and looked up the PEPs regarding universal new-lines, and it seems
it would be platform-independent - all of "\r\n", "\r", and "\n" will
always be converted to "\n" in Python, unless explicitly modified on the
file object (or Universal newlines are disabled).

It still stands that for platform independence, "rb" should be used due to
some platforms treating binary files differently.

Additionally, and most importantly, for what the OP was doing (determining
if files are the same), you almost certainly want to read the raw bytes -
especially if some of the files are truly binary (such as *.exe on Windows).

Chris

On Mon, Sep 22, 2014 at 12:34 PM, <random832@fastmail.us> wrote:

> On Thu, Sep 18, 2014, at 14:45, Chris Kaynor wrote:
> > Additionally, you may want to specify binary mode by using
> > open(file_path,
> > 'rb') to ensure platform-independence ('r' uses Universal newlines, which
> > means on Windows, Python will convert "\r\n" to "\n" while reading the
> > file). Additionally, some platforms will treat binary files differently.
>
> Does 'r' not use universal newlines on unix? If not, why not? The only
> purpose seems to be to allow people to write bad programs and have them
> work on unix. It makes even less sense in python 3 where opening a file
> in text mode results in a TextIOWrapper with utf-8 encoding, and
> therefore can't be used on arbitrary binary files.
> --
> https://mail.python.org/mailman/listinfo/python-list
>

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread


Thread

Re: program to generate data helpful in finding duplicate large files Chris Kaynor <ckaynor@zindagigames.com> - 2014-09-22 12:47 -0700

csiph-web