Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.xs4all.nl!newsfeed4.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.001 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'python,': 0.02; 'explicitly': 0.05; 'binary': 0.07; 'modified': 0.07; 'utf-8': 0.07; '22,': 0.09; 'converted': 0.09; 'differently.': 0.09; 'subject:files': 0.09; 'windows,': 0.09; 'python': 0.11; '"\\r\\n"': 0.16; "'r'": 0.16; "'rb')": 0.16; 'file).': 0.16; 'newlines': 0.16; 'subject:program': 0.16; 'files.': 0.16; 'wrote:': 0.18; 'looked': 0.18; 'thu,': 0.19; 'not,': 0.20; 'seems': 0.21; 'platforms': 0.22; 'to:name:python- list@python.org': 0.22; '(such': 0.24; 'bytes': 0.24; 'certainly': 0.24; 'specify': 0.24; 'mon,': 0.24; '(or': 0.24; '>': 0.26; 'received:74.125.82.174': 0.27; 'header:In-Reply-To:1': 0.27; 'chris': 0.29; 'mode': 0.30; 'especially': 0.30; 'message- id:@mail.gmail.com': 0.30; 'url:mailman': 0.30; 'went': 0.31; 'sep': 0.31; 'stands': 0.31; 'universal': 0.31; 'file': 0.32; 'text': 0.33; 'url:python': 0.33; 'raw': 0.33; 'sense': 0.34; 'received:74.125.82': 0.34; "can't": 0.35; 'convert': 0.35; 'received:google.com': 0.35; 'subject:data': 0.36; 'url:listinfo': 0.36; 'doing': 0.36; 'url:org': 0.36; 'should': 0.36; 'skip:& 10': 0.38; 'to:addr:python-list': 0.38; 'files': 0.38; 'pm,': 0.38; 'received:74.125': 0.39; 'does': 0.39; 'bad': 0.39; 'to:addr:python.org': 0.39; 'skip:p 20': 0.39; 'url:mail': 0.40; 'ensure': 0.60; 'even': 0.60; 'read': 0.60; 'truly': 0.60; 'most': 0.60; 'due': 0.66; 'importantly,': 0.68; 'results': 0.69; 'therefore': 0.72; '2014,': 0.84; 'treating': 0.84 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:content-type; bh=ls8dgF0p/p92PvaLmAvFc96i72sBeIEjvWDngdIcibo=; b=izBlaYj2WLgYTGxnwUhJB6dvLar7XsmN/PCHw4x+jw1gXVtLYl89aNW34VK/Xlq3LH wML/sRLRzSFw+N8tWHCzLZXMhPc+YyvgUmPV11pSwVWle0uteSILxrcaJAjoeQV1+JcK TGqjrqnX9bTYnm3x/NZbgqtVckRBC8GQVE2uUPj8xasSOL9RxZ1EjIkGl6xHIBd5ZgTL D4Rq/jSPe6XCQRGOv5IL9b1cqd96s8WWYKTyfnSPy0y9kaFHm1fCFsJfax5qpGwHn/sm b2Zo0pycsc1vLD+VcyHDQg/qkVSv17r1Dp953skbVLv8kVDolxB7rr0h47zGM+04llGd ANtw== X-Gm-Message-State: ALoCoQmJM/k20XN/gfjjBPT/nl//zxcixDhkIsFdK1LPGCjNnsUbJvJ3ECHavNKRBF7y9XXd9iCv X-Received: by 10.180.91.70 with SMTP id cc6mr17710713wib.66.1411415266638; Mon, 22 Sep 2014 12:47:46 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <1411414443.4182630.170466465.768582FE@webmail.messagingengine.com> References: <1411414443.4182630.170466465.768582FE@webmail.messagingengine.com> From: Chris Kaynor Date: Mon, 22 Sep 2014 12:47:26 -0700 Subject: Re: program to generate data helpful in finding duplicate large files To: "python-list@python.org" Content-Type: multipart/alternative; boundary=f46d0438edd79958050503acb91a X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 76 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1411415617 news.xs4all.nl 2941 [2001:888:2000:d::a6]:53885 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:78183 --f46d0438edd79958050503acb91a Content-Type: text/plain; charset=UTF-8 I went and looked up the PEPs regarding universal new-lines, and it seems it would be platform-independent - all of "\r\n", "\r", and "\n" will always be converted to "\n" in Python, unless explicitly modified on the file object (or Universal newlines are disabled). It still stands that for platform independence, "rb" should be used due to some platforms treating binary files differently. Additionally, and most importantly, for what the OP was doing (determining if files are the same), you almost certainly want to read the raw bytes - especially if some of the files are truly binary (such as *.exe on Windows). Chris On Mon, Sep 22, 2014 at 12:34 PM, wrote: > On Thu, Sep 18, 2014, at 14:45, Chris Kaynor wrote: > > Additionally, you may want to specify binary mode by using > > open(file_path, > > 'rb') to ensure platform-independence ('r' uses Universal newlines, which > > means on Windows, Python will convert "\r\n" to "\n" while reading the > > file). Additionally, some platforms will treat binary files differently. > > Does 'r' not use universal newlines on unix? If not, why not? The only > purpose seems to be to allow people to write bad programs and have them > work on unix. It makes even less sense in python 3 where opening a file > in text mode results in a TextIOWrapper with utf-8 encoding, and > therefore can't be used on arbitrary binary files. > -- > https://mail.python.org/mailman/listinfo/python-list > --f46d0438edd79958050503acb91a Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
I went and looked up the PEPs regarding universal new-line= s, and it seems it would be platform-independent - all of "\r\n",= "\r", and "\n" will always be converted to "\n&qu= ot; in Python, unless explicitly modified on the file object (or Universal = newlines are disabled).

It still stands that for platfor= m independence, "rb" should be used due to some platforms treatin= g binary files differently.

Additionally, and most= importantly, for what the OP was doing (determining if files are the same)= , you almost certainly want to read the raw bytes - especially if some of t= he files are truly binary (such as *.exe on Windows).

Chris

On Mon, Sep 22, 2014 at 12:34 PM, <ran= dom832@fastmail.us> wrote:
= On Thu, Sep 18, 2014, at 14:45, Chris Kaynor wrote:
> Additionally, you may want to specify binary mode by using
> open(file_path,
> 'rb') to ensure platform-independence ('r' uses Univer= sal newlines, which
> means on Windows, Python will convert "\r\n" to "\n&quo= t; while reading the
> file). Additionally, some platforms will treat binary files differentl= y.

Does 'r' not use universal newlines on unix? If not, why not= ? The only
purpose seems to be to allow people to write bad programs and have them
work on unix. It makes even less sense in python 3 where opening a file
in text mode results in a TextIOWrapper with utf-8 encoding, and
therefore can't be used on arbitrary binary files.
--
https://mail.python.org/mailman/listinfo/python-list

--f46d0438edd79958050503acb91a--