Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!border3.nntp.ams.giganews.com!backlog3.nntp.ams.giganews.com!border1.nntp.ams.giganews.com!nntp.giganews.com!newsfeed.xs4all.nl!newsfeed3a.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
Sender: extasia@gmail.com
Date: Fri, 19 Sep 2014 06:32:02 -0700
Subject: Fwd: program to generate data helpful in finding duplicate large files
From: David Alban <extasia@extasia.org>
To: python-list@python.org
Content-Type: multipart/alternative; boundary=047d7b87375c56023705036b2095
Precedence: list
Reply-To: extasia@extasia.org
Newsgroups: comp.lang.python
Message-ID: <mailman.14146.1411133530.18130.python-list@python.org>
Lines: 264
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:78078

--047d7b87375c56023705036b2095
Content-Type: text/plain; charset=UTF-8

here is my reworked code in a plain text email.

---------- Forwarded message ----------
From: <bizcor@gmail.com>
Date: Thu, Sep 18, 2014 at 3:58 PM
Subject: Re: program to generate data helpful in finding duplicate large
files
To: python-list@python.org


thanks for the responses.   i'm having quite a good time learning python.

On Thu, Sep 18, 2014 at 11:45 AM, Chris Kaynor <ckaynor@zindagigames.com>
wrote:
>
> Additionally, you may want to specify binary mode by using
open(file_path, 'rb') to ensure platform-independence ('r' uses Universal
newlines, which means on Windows, Python will convert "\r\n" to "\n" while
reading the file). Additionally, some platforms will treat binary files
differently.


would it be good to use 'rb' all the time?

On Thu, Sep 18, 2014 at 11:48 AM, Chris Angelico <rosuav@gmail.com> wrote:
>
> On Fri, Sep 19, 2014 at 4:11 AM, David Alban <extasia@extasia.org> wrote:
> > exit( 0 )
>
> Unnecessary - if you omit this, you'll exit 0 implicitly at the end of
> the script.


aha.  i've been doing this for years even with perl, and apparently it's
not necessary in perl either.  i was influenced by shell.

this shell code:

     if [[ -n $report_mode ]] ; then
        do_report
     fi

     exit 0

is an example of why you want the last normally executed shell statement to
be "exit 0".  if you omit the exit statement it in this example, and
$report_mode is not set, your shell program will give a non-zero return
code and appear to have terminated with an error.  in shell the last
expression evaluated determines the return code to the os.

ok, i don't need to do this in python.

On Thu, Sep 18, 2014 at 1:23 PM, Peter Otten <__peter__@web.de> wrote:
>
> file_path may contain newlines, therefore you should probably use "\0" to
> separate the records.


i chose to stick with ascii nul as the default field separator, but i added
a --field-separator option in case someone wants human readable output.

style question:  if there is only one, possibly short statement in a block,
do folks usually move it up to the line starting the block?

  if not S_ISREG( mode ) or S_ISLNK( mode ):
    return

vs.

  if not S_ISREG( mode ) or S_ISLNK( mode ): return

or even:

  with open( file_path, 'rb' ) as f: md5sum = md5_for_file( file_path )



fyi, here are my changes:


usage: dupscan [-h] [--start-directory START_DIRECTORY]
               [--field-separator FIELD_SEPARATOR]

scan files in a tree and print a line of information about each regular file

optional arguments:
  -h, --help            show this help message and exit
  --start-directory START_DIRECTORY, -d START_DIRECTORY
                        Specify the root of the filesystem tree to be
                        processed. The default is '.'
  --field-separator FIELD_SEPARATOR, -s FIELD_SEPARATOR
                        Specify the string to use as a field separator in
                        output. The default is the ascii nul character.



#!/usr/bin/python

import argparse
import hashlib
import os

from platform import node
from stat import S_ISREG, S_ISLNK

ASCII_NUL = chr(0)

     # from:
http://stackoverflow.com/questions/1131220/get-md5-hash-of-big-files-in-python
     # except that i use hexdigest() rather than digest()
def md5_for_file( path, block_size=2**20 ):
  md5 = hashlib.md5()
  with open( path, 'rb' ) as f:
    while True:
      data = f.read(block_size)
      if not data:
        break
      md5.update(data)
  return md5.hexdigest()

def file_info( directory, basename, field_separator=ASCII_NUL ):
  file_path = os.path.join( directory, basename )
  st = os.lstat( file_path )

  mode = st.st_mode
  if not S_ISREG( mode ) or S_ISLNK( mode ):
    return

  with open( file_path, 'rb' ) as f:
    md5sum = md5_for_file( file_path )

  return field_separator.join( [ thishost, md5sum, str( st.st_dev ), str(
st.st_ino ), str( st.st_nlink ), str( st.st_size ), file_path ] )

if __name__ == "__main__":
  parser = argparse.ArgumentParser(description='scan files in a tree and
print a line of information about each regular file')
  parser.add_argument('--start-directory', '-d', default='.',
help='''Specify the root of the filesystem tree to be processed.  The
default is '.' ''')
  parser.add_argument('--field-separator', '-s', default=ASCII_NUL,
help='Specify the string to use as a field separator in output.  The
default is the ascii nul character.')
  args = parser.parse_args()

  start_directory = args.start_directory.rstrip('/')
  field_separator = args.field_separator

  thishost = node()
  if thishost == '':
    thishost='[UNKNOWN]'

  for directory_path, directory_names, file_names in os.walk(
start_directory ):
    for file_name in file_names:
      print file_info( directory_path, file_name, field_separator )

-- 
Our decisions are the most important things in our lives.
***
Live in a world of your own, but always welcome visitors.

--047d7b87375c56023705036b2095
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>here is my reworked code in a plain text email.</div>=
<br>---------- Forwarded message ----------<br>From: &lt;<a href=3D"mailto:=
bizcor@gmail.com">bizcor@gmail.com</a>&gt;<br>Date: Thu, Sep 18, 2014 at 3:=
58 PM<br>Subject: Re: program to generate data helpful in finding duplicate=
 large files<br>To: <a href=3D"mailto:python-list@python.org">python-list@p=
ython.org</a><br><br><br>thanks for the responses. =C2=A0 i&#39;m having qu=
ite a good time learning python.<br><br>On Thu, Sep 18, 2014 at 11:45 AM, C=
hris Kaynor &lt;<a href=3D"mailto:ckaynor@zindagigames.com">ckaynor@zindagi=
games.com</a>&gt; wrote: <br>&gt;<br>&gt; Additionally, you may want to spe=
cify binary mode by using open(file_path, &#39;rb&#39;) to ensure platform-=
independence (&#39;r&#39; uses Universal newlines, which means on Windows, =
Python will convert &quot;\r\n&quot; to &quot;\n&quot; while reading the fi=
le). Additionally, some platforms will treat binary files differently.<br><=
br><br>would it be good to use &#39;rb&#39; all the time?<br><br>On Thu, Se=
p 18, 2014 at 11:48 AM, Chris Angelico &lt;<a href=3D"mailto:rosuav@gmail.c=
om">rosuav@gmail.com</a>&gt; wrote:<br>&gt;<br>&gt; On Fri, Sep 19, 2014 at=
 4:11 AM, David Alban &lt;<a href=3D"mailto:extasia@extasia.org">extasia@ex=
tasia.org</a>&gt; wrote:<br>&gt; &gt; exit( 0 )<br>&gt;<br>&gt; Unnecessary=
 - if you omit this, you&#39;ll exit 0 implicitly at the end of<br>&gt; the=
 script.<br><br><br>aha. =C2=A0i&#39;ve been doing this for years even with=
 perl, and apparently it&#39;s not necessary in perl either. =C2=A0i was in=
fluenced by shell.<br><br>this shell code:<br><br>=C2=A0 =C2=A0 =C2=A0if [[=
 -n $report_mode ]] ; then<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 do_report<br>=C2=
=A0 =C2=A0 =C2=A0fi<br><br>=C2=A0 =C2=A0 =C2=A0exit 0<br><br>is an example =
of why you want the last normally executed shell statement to be &quot;exit=
 0&quot;. =C2=A0if you omit the exit statement it in this example, and $rep=
ort_mode is not set, your shell program will give a non-zero return code an=
d appear to have terminated with an error. =C2=A0in shell the last expressi=
on evaluated determines the return code to the os.<br><br>ok, i don&#39;t n=
eed to do this in python.<br><br>On Thu, Sep 18, 2014 at 1:23 PM, Peter Ott=
en &lt;__<a href=3D"mailto:peter__@web.de">peter__@web.de</a>&gt; wrote:<br=
>&gt;<br>&gt; file_path may contain newlines, therefore you should probably=
 use &quot;\0&quot; to<br>&gt; separate the records.<br><br><br>i chose to =
stick with ascii nul as the default field separator, but i added a --field-=
separator option in case someone wants human readable output.<br><br>style =
question: =C2=A0if there is only one, possibly short statement in a block, =
do folks usually move it up to the line starting the block?<br><br>=C2=A0 i=
f not S_ISREG( mode ) or S_ISLNK( mode ):<br>=C2=A0 =C2=A0 return<br><br>vs=
.<br><br>=C2=A0 if not S_ISREG( mode ) or S_ISLNK( mode ): return<br><br>or=
 even:<br><br><div>=C2=A0 with open( file_path, &#39;rb&#39; ) as f: md5sum=
 =3D md5_for_file( file_path )<br><br><br><br>fyi, here are my changes:<br>=
<br><br>usage: dupscan [-h] [--start-directory START_DIRECTORY]<br>=C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0[--field-separator FIELD_SE=
PARATOR]<br><br>scan files in a tree and print a line of information about =
each regular file<br><br>optional arguments:<br>=C2=A0 -h, --help =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0show this help message and exit<br>=C2=A0=
 --start-directory START_DIRECTORY, -d START_DIRECTORY<br>=C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Specify =
the root of the filesystem tree to be<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 processed. The default is=
 &#39;.&#39;<br>=C2=A0 --field-separator FIELD_SEPARATOR, -s FIELD_SEPARATO=
R<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 Specify the string to use as a field separator in<br>=C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 output. The default is the ascii nul character.<br><br><br><br>#!/usr/b=
in/python<br><br>import argparse<br>import hashlib<br>import os<br><br>from=
 platform import node<br>from stat import S_ISREG, S_ISLNK<br><br>ASCII_NUL=
 =3D chr(0)<br><br>=C2=A0 =C2=A0 =C2=A0# from: <a href=3D"http://stackoverf=
low.com/questions/1131220/get-md5-hash-of-big-files-in-python">http://stack=
overflow.com/questions/1131220/get-md5-hash-of-big-files-in-python</a><br>=
=C2=A0 =C2=A0 =C2=A0# except that i use hexdigest() rather than digest()<br=
>def md5_for_file( path, block_size=3D2**20 ):<br>=C2=A0 md5 =3D hashlib.md=
5()<br>=C2=A0 with open( path, &#39;rb&#39; ) as f:<br>=C2=A0 =C2=A0 while =
True:<br>=C2=A0 =C2=A0 =C2=A0 data =3D f.read(block_size)<br>=C2=A0 =C2=A0 =
=C2=A0 if not data:<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 break<br>=C2=A0 =C2=A0 =
=C2=A0 md5.update(data)<br>=C2=A0 return md5.hexdigest()<br><br>def file_in=
fo( directory, basename, field_separator=3DASCII_NUL ):<br>=C2=A0 file_path=
 =3D os.path.join( directory, basename )<br>=C2=A0 st =3D os.lstat( file_pa=
th )<br><br>=C2=A0 mode =3D st.st_mode<br>=C2=A0 if not S_ISREG( mode ) or =
S_ISLNK( mode ): <br>=C2=A0 =C2=A0 return<br><br>=C2=A0 with open( file_pat=
h, &#39;rb&#39; ) as f:<br>=C2=A0 =C2=A0 md5sum =3D md5_for_file( file_path=
 )<br><br>=C2=A0 return field_separator.join( [ thishost, md5sum, str( st.s=
t_dev ), str( st.st_ino ), str( st.st_nlink ), str( st.st_size ), file_path=
 ] )<br><br>if __name__ =3D=3D &quot;__main__&quot;:<br>=C2=A0 parser =3D a=
rgparse.ArgumentParser(description=3D&#39;scan files in a tree and print a =
line of information about each regular file&#39;)<br>=C2=A0 parser.add_argu=
ment(&#39;--start-directory&#39;, &#39;-d&#39;, default=3D&#39;.&#39;, help=
=3D&#39;&#39;&#39;Specify the root of the filesystem tree to be processed. =
=C2=A0The default is &#39;.&#39; &#39;&#39;&#39;)<br>=C2=A0 parser.add_argu=
ment(&#39;--field-separator&#39;, &#39;-s&#39;, default=3DASCII_NUL, help=
=3D&#39;Specify the string to use as a field separator in output. =C2=A0The=
 default is the ascii nul character.&#39;)<br>=C2=A0 args =3D parser.parse_=
args()<br><br>=C2=A0 start_directory =3D args.start_directory.rstrip(&#39;/=
&#39;)<br>=C2=A0 field_separator =3D args.field_separator<br><br>=C2=A0 thi=
shost =3D node()<br>=C2=A0 if thishost =3D=3D &#39;&#39;:<br>=C2=A0 =C2=A0 =
thishost=3D&#39;[UNKNOWN]&#39;<br><br>=C2=A0 for directory_path, directory_=
names, file_names in os.walk( start_directory ):<br>=C2=A0 =C2=A0 for file_=
name in file_names:<br>=C2=A0 =C2=A0 =C2=A0 print file_info( directory_path=
, file_name, field_separator )<br clear=3D"all"><div><br></div>-- <br><div =
dir=3D"ltr">Our decisions are the most important things in our lives.<div>*=
**</div><div>Live in a world of your own, but always welcome visitors.</div=
></div>
<br><br></div></div>

--047d7b87375c56023705036b2095--