Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #8924

Re: Does hashlib support a file mode?

Show all headers | View raw

Phlip <phlip2005@gmail.com> writes:

> If the file were huge, the file.read() would allocate a big string and
> thrash memory. (Yes, in 2011 that's still a problem, because these
> files could be movies and whatnot.)

I did a crc32 calculator like that and actually ran into some kind of
string length limit with large files. So I switched to 4k blocks and
the speed is about the same as a C implementation in the program
cksfv. Well, of course crc32 is usually done with a table lookup, so
it's always fast.

I just picked 4k, since it's the page size in x86 systems and also a
common block size for file systems. Seems to be big enough.
io.DEFAULT_BUFFER_SIZE is 8k here. I suppose using that would be the
proper way.

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Does hashlib support a file mode? Phlip <phlip2005@gmail.com> - 2011-07-05 22:54 -0700
  Re: Does hashlib support a file mode? Thomas Rachel <nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de> - 2011-07-06 08:37 +0200
  Re: Does hashlib support a file mode? Chris Rebert <clp2@rebertia.com> - 2011-07-05 23:44 -0700
  Re: Does hashlib support a file mode? Anssi Saari <as@sci.fi> - 2011-07-06 12:47 +0300
    Re: Does hashlib support a file mode? Phlip <phlip2005@gmail.com> - 2011-07-06 06:47 -0700
  Re: Does hashlib support a file mode? Adam Tauno Williams <awilliam@whitemice.org> - 2011-07-06 06:55 -0400

csiph-web