Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #8902 > unrolled thread

Does hashlib support a file mode?

Back to article view | Back to comp.lang.python

Contents

  Does hashlib support a file mode? Phlip <phlip2005@gmail.com> - 2011-07-05 22:54 -0700
    Re: Does hashlib support a file mode? Thomas Rachel <nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de> - 2011-07-06 08:37 +0200
    Re: Does hashlib support a file mode? Chris Rebert <clp2@rebertia.com> - 2011-07-05 23:44 -0700
    Re: Does hashlib support a file mode? Anssi Saari <as@sci.fi> - 2011-07-06 12:47 +0300
      Re: Does hashlib support a file mode? Phlip <phlip2005@gmail.com> - 2011-07-06 06:47 -0700
    Re: Does hashlib support a file mode? Adam Tauno Williams <awilliam@whitemice.org> - 2011-07-06 06:55 -0400

#8902 — Does hashlib support a file mode?

Pythonistas:

Consider this hashing code:

  import hashlib
  file = open(path)
  m = hashlib.md5()
  m.update(file.read())
  digest = m.hexdigest()
  file.close()

If the file were huge, the file.read() would allocate a big string and
thrash memory. (Yes, in 2011 that's still a problem, because these
files could be movies and whatnot.)

So if I do the stream trick - read one byte, update one byte, in a
loop, then I'm essentially dragging that movie thru 8 bits of a 64 bit
CPU. So that's the same problem; it would still be slow.

So now I try this:

  sum = os.popen('sha256sum %r' % path).read()

Those of you who like to lie awake at night thinking of new ways to
flame abusers of 'eval()' may have a good vent, there.

Does hashlib have a file-ready mode, to hide the streaming inside some
clever DMA operations?

Prematurely optimizingly y'rs

--
  Phlip
  http://bit.ly/ZeekLand

[toc] | [next] | [standalone]

#8906

Am 06.07.2011 07:54 schrieb Phlip:
> Pythonistas:
>
> Consider this hashing code:
>
>    import hashlib
>    file = open(path)
>    m = hashlib.md5()
>    m.update(file.read())
>    digest = m.hexdigest()
>    file.close()
>
> If the file were huge, the file.read() would allocate a big string and
> thrash memory. (Yes, in 2011 that's still a problem, because these
> files could be movies and whatnot.)
>
> So if I do the stream trick - read one byte, update one byte, in a
> loop, then I'm essentially dragging that movie thru 8 bits of a 64 bit
> CPU. So that's the same problem; it would still be slow.

Yes. That is why you should read with a reasonable block size. Not too 
small and not too big.

def filechunks(f, size=8192):
     while True:
         s = f.read(size)
         if not s: break
         yield s
#    f.close() # maybe...

import hashlib
file = open(path)
m = hashlib.md5()
fc = filechunks(file)
for chunk in fc:
     m.update(chunk)
digest = m.hexdigest()
file.close()

So you are reading in 8 kiB chunks. Feel free to modify this - maybe use 
os.stat(file).st_blksize instead (which is AFAIK the recommended 
minimum), or a value of about 1 MiB...


> So now I try this:
>
>    sum = os.popen('sha256sum %r' % path).read()

This is not as nice as the above, especially not with a path containing 
strange characters. What about, at least,

def shellquote(*strs):
	return " ".join([
		"'"+st.replace("'","'\\''")+"'"
		for st in strs
	])

sum = os.popen('sha256sum %r' % shellquote(path)).read()


or, even better,

import subprocess
sp = subprocess.Popen(['sha256sum', path'],
     stdin=subprocess.PIPE, stdout=subprocess.PIPE)
sp.stdin.close() # generate EOF
sum = sp.stdout.read()
sp.wait()

?


> Does hashlib have a file-ready mode, to hide the streaming inside some
> clever DMA operations?

AFAIK not.


Thomas

[toc] | [prev] | [next] | [standalone]

#8907

On Tue, Jul 5, 2011 at 10:54 PM, Phlip <phlip2005@gmail.com> wrote:
> Pythonistas:
>
> Consider this hashing code:
>
>  import hashlib
>  file = open(path)
>  m = hashlib.md5()
>  m.update(file.read())
>  digest = m.hexdigest()
>  file.close()
>
> If the file were huge, the file.read() would allocate a big string and
> thrash memory. (Yes, in 2011 that's still a problem, because these
> files could be movies and whatnot.)
>
> So if I do the stream trick - read one byte, update one byte, in a
> loop, then I'm essentially dragging that movie thru 8 bits of a 64 bit
> CPU. So that's the same problem; it would still be slow.
>
> So now I try this:
>
>  sum = os.popen('sha256sum %r' % path).read()
>
> Those of you who like to lie awake at night thinking of new ways to
> flame abusers of 'eval()' may have a good vent, there.

Indeed (*eyelid twitch*). That one-liner is arguably better written as:
sum = subprocess.check_output(['sha256sum', path])

> Does hashlib have a file-ready mode, to hide the streaming inside some
> clever DMA operations?

Barring undocumented voodoo, no, it doesn't appear to. You could
always read from the file in suitably large chunks instead (rather
than byte-by-byte, which is indeed ridiculous); see
io.DEFAULT_BUFFER_SIZE and/or the os.stat() trick referenced therein
and/or the block_size attribute of hash objects.
http://docs.python.org/library/io.html#io.DEFAULT_BUFFER_SIZE
http://docs.python.org/library/hashlib.html#hashlib.hash.block_size

Cheers,
Chris
--
http://rebertia.com

[toc] | [prev] | [next] | [standalone]

#8924

Phlip <phlip2005@gmail.com> writes:

> If the file were huge, the file.read() would allocate a big string and
> thrash memory. (Yes, in 2011 that's still a problem, because these
> files could be movies and whatnot.)

I did a crc32 calculator like that and actually ran into some kind of
string length limit with large files. So I switched to 4k blocks and
the speed is about the same as a C implementation in the program
cksfv. Well, of course crc32 is usually done with a table lookup, so
it's always fast.

I just picked 4k, since it's the page size in x86 systems and also a
common block size for file systems. Seems to be big enough.
io.DEFAULT_BUFFER_SIZE is 8k here. I suppose using that would be the
proper way.

[toc] | [prev] | [next] | [standalone]

#8937

wow, tx y'all!

I forgot to mention that hashlib itself is not required; I could also
use Brand X. But y'all agree that blocking up the file in python adds
no overhead to hashing each block in C, so hashlib in a loop it is!

[toc] | [prev] | [next] | [standalone]

#8927

On Tue, 2011-07-05 at 22:54 -0700, Phlip wrote:
> Pythonistas
> Consider this hashing code:
>   import hashlib
>   file = open(path)
>   m = hashlib.md5()
>   m.update(file.read())
>   digest = m.hexdigest()
>   file.close()
> If the file were huge, the file.read() would allocate a big string and
> thrash memory. (Yes, in 2011 that's still a problem, because these
> files could be movies and whatnot.)

Yes, the simple rule is do not *ever* file.read().  No matter what the
year this will never be OK.  Always chunk reading a file into reasonable
I/O blocks.

For example I use this function to copy a stream and return a SHA512 and
the output streams size:

    def write(self, in_handle, out_handle):
        m = hashlib.sha512()
        data = in_handle.read(4096)
        while True:
            if not data:
                break
            m.update(data)
            out_handle.write(data)
            data = in_handle.read(4096)
        out_handle.flush()
        return (m.hexdigest(), in_handle.tell())

> Does hashlib have a file-ready mode, to hide the streaming inside some
> clever DMA operations?

Chunk it to something close to the block size of your underlying
filesystem.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web