Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #8948 > unrolled thread

Re: Does hashlib support a file mode?

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Does hashlib support a file mode? Phlip <phlip2005@gmail.com> - 2011-07-06 08:59 -0700
    Re: Does hashlib support a file mode? Peter Otten <__peter__@web.de> - 2011-07-06 18:26 +0200
      Re: Does hashlib support a file mode? Phlip <phlip2005@gmail.com> - 2011-07-06 09:49 -0700
        Re: Does hashlib support a file mode? Peter Otten <__peter__@web.de> - 2011-07-06 19:06 +0200
          Re: Does hashlib support a file mode? Phlip <phlip2005@gmail.com> - 2011-07-06 10:38 -0700
            Re: Does hashlib support a file mode? Andrew Berg <bahamutzero8825@gmail.com> - 2011-07-06 13:42 -0500
        Re: Does hashlib support a file mode? Chris Torek <nospam@torek.net> - 2011-07-06 17:54 +0000

#8948 — Re: Does hashlib support a file mode?

Tx, all!. But...

> For example I use this function to copy a stream and return a SHA512 and
> the output streams size:
>
>     def write(self, in_handle, out_handle):
>         m = hashlib.sha512()
>         data = in_handle.read(4096)
>         while True:
>             if not data:
>                 break
>             m.update(data)
>             out_handle.write(data)
>             data = in_handle.read(4096)
>         out_handle.flush()
>         return (m.hexdigest(), in_handle.tell())

The operation was a success but the patient died.

My version of that did not return the same hex digest as the md5sum
version:


def file_to_hash(path, m = hashlib.md5()):

    with open(path, 'r') as f:

        s = f.read(8192)

        while s:
            m.update(s)
            s = f.read(8192)

    return m.hexdigest()

You'll notice it has the same control flow as yours.

That number must eventually match an iPad's internal MD5 opinion of
that file, after it copies up, so I naturally cannot continue working
this problem until we see which of the two numbers the iPad likes!

[toc] | [next] | [standalone]

#8951

Phlip wrote:

> Tx, all!. But...
> 
>> For example I use this function to copy a stream and return a SHA512 and
>> the output streams size:
>>
>> def write(self, in_handle, out_handle):
>> m = hashlib.sha512()
>> data = in_handle.read(4096)
>> while True:
>> if not data:
>> break
>> m.update(data)
>> out_handle.write(data)
>> data = in_handle.read(4096)
>> out_handle.flush()
>> return (m.hexdigest(), in_handle.tell())
> 
> The operation was a success but the patient died.
> 
> My version of that did not return the same hex digest as the md5sum
> version:
> 
> 
> def file_to_hash(path, m = hashlib.md5()):
> 
>     with open(path, 'r') as f:
> 
>         s = f.read(8192)
> 
>         while s:
>             m.update(s)
>             s = f.read(8192)
> 
>     return m.hexdigest()
> 
> You'll notice it has the same control flow as yours.
> 
> That number must eventually match an iPad's internal MD5 opinion of
> that file, after it copies up, so I naturally cannot continue working
> this problem until we see which of the two numbers the iPad likes!

- Open the file in binary mode.
- Do the usual dance for default arguments:
    def file_to_hash(path, m=None):
        if m is None:
            m = hashlib.md5()

[toc] | [prev] | [next] | [standalone]

#8955

> - Open the file in binary mode.

I had tried open(path, 'rb') and it didn't change the "wrong" number.

And I added --binary to my evil md5sum version, and it didn't change
the "right" number!

Gods bless those legacy hacks that will never die, huh? But I'm using
Ubuntu (inside VMWare, on Win7, on a Core i7, because I rule), so that
might explain why "binary mode" is a no-op.

> - Do the usual dance for default arguments:
>     def file_to_hash(path, m=None):
>         if m is None:
>             m = hashlib.md5()

Not sure why if that's what the defaulter does? I did indeed get an
MD5-style string of what casually appeared to be the right length, so
that implies the defaulter is not to blame...

[toc] | [prev] | [next] | [standalone]

#8961

Phlip wrote:

>> - Open the file in binary mode.
> 
> I had tried open(path, 'rb') and it didn't change the "wrong" number.
> 
> And I added --binary to my evil md5sum version, and it didn't change
> the "right" number!
> 
> Gods bless those legacy hacks that will never die, huh? But I'm using
> Ubuntu (inside VMWare, on Win7, on a Core i7, because I rule), so that
> might explain why "binary mode" is a no-op.

Indeed. That part was a defensive measure mostly meant to make your function 
Windows-proof.

>> - Do the usual dance for default arguments:
>> def file_to_hash(path, m=None):
>> if m is None:
>> m = hashlib.md5()
> 
> Not sure why if that's what the defaulter does? I did indeed get an
> MD5-style string of what casually appeared to be the right length, so
> that implies the defaulter is not to blame...

The first call will give you the correct checksum, the second: not. As the 
default md5 instance remembers the state from the previous function call 
you'll get the checksum of both files combined. 

[toc] | [prev] | [next] | [standalone]

#8964

> >> def file_to_hash(path, m=None):
> >> if m is None:
> >> m = hashlib.md5()

> The first call will give you the correct checksum, the second: not. As the
> default md5 instance remembers the state from the previous function call
> you'll get the checksum of both files combined.

Ouch. That was it.

Python sucks. m = md5() looks like an initial assignment, not a
special magic storage mode. Principle of least surprise fail, and
principle of most helpful default behavior fail.

[toc] | [prev] | [next] | [standalone]

#8971

On 2011.07.06 12:38 PM, Phlip wrote:
> Python sucks. m = md5() looks like an initial assignment, not a
> special magic storage mode. Principle of least surprise fail, and
> principle of most helpful default behavior fail.
func() = whatever the function returns
func = the function object itself (in Python, everything's an object)

Maybe you have Python confused with another language (I don't know what
exactly you mean by initial assignment). Typically one does not need
more than one name for a function/method. When a function/method is
defined, it gets created as a function object and occupies the namespace
in which it's defined.

[toc] | [prev] | [next] | [standalone]

#8966

>> - Do the usual dance for default arguments:
>>     def file_to_hash(path, m=None):
>>         if m is None:
>>             m = hashlib.md5()

[instead of

    def file_to_hash(path, m = hashlib.md5()):

]

In article <b317226a-8008-4177-aaa6-3fdc30125eea@e20g2000prf.googlegroups.com>
Phlip  <phlip2005@gmail.com> wrote:
>Not sure why if that's what the defaulter does?

For the same reason that:

    def spam(somelist, so_far = []):
        for i in somelist:
            if has_eggs(i):
                so_far.append(i)
        return munch(so_far)

is probably wrong.  Most beginners appear to expect this to take
a list of "things that pass my has_eggs test", add more things
to that list, and return whatever munch(adjusted_list) returns ...
which it does.  But then they *also* expect:

    result1_on_clean_list = spam(list1)
    result2_on_clean_list = spam(list2)
    result3_on_partly_filled_list = spam(list3, prefilled3)

to run with a "clean" so_far list for *each* of the first two
calls ... but it does not; the first call starts with a clean
list, and the second one starts with "so_far" containing all
the results accumulated from list1.

(The third call, of course, starts with the prefilled3 list and
adjusts that list.)

>I did indeed get an MD5-style string of what casually appeared
>to be the right length, so that implies the defaulter is not to
>blame...

In this case, if you do:

    print('big1:', file_to_hash('big1'))
    print('big2:', file_to_hash('big2'))

you will get two md5sum values for your two files, but the
md5sum value for big2 will not be the equivalent of "md5sum big2"
but rather that of "cat big1 big2 | md5sum".  The reason is
that you are re-using the md5-sum-so-far on the second call
(for file 'big2'), so you have the accumulated sum from file
'big1', which you then update via the contents of 'big2'.
-- 
In-Real-Life: Chris Torek, Wind River Systems
Intel require I note that my opinions are not those of WRS or Intel
Salt Lake City, UT, USA (40°39.22'N, 111°50.29'W)  +1 801 277 2603
email: gmail (figure it out)      http://web.torek.net/torek/index.html

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web