Re: cPickle.load vs. file.read+cPickle.loads on large binary files

From	Peter Otten <__peter__@web.de>
Newsgroups	comp.lang.python
Subject	Re: cPickle.load vs. file.read+cPickle.loads on large binary files
Date	2015-11-17 16:57 +0100
Organization	None
Message-ID	<mailman.395.1447775861.16136.python-list@python.org> (permalink)
References	<463ad93c-0186-4911-9cd1-92d97b9dc87b@googlegroups.com> <mailman.387.1447769670.16136.python-list@python.org> <54330891-6568-4469-93ae-7a7825961500@googlegroups.com> <mailman.392.1447773612.16136.python-list@python.org> <420ec4e9-6af6-49bd-a9f4-8b47ef1f136e@googlegroups.com>

Show all headers | View raw

andrea.gavana@gmail.com wrote:

> Hi Chris,
> 
> On Tuesday, November 17, 2015 at 4:20:34 PM UTC+1, Chris Angelico wrote:
>> On Wed, Nov 18, 2015 at 1:20 AM,  Andrea Gavana wrote:
>> > Thank you for your answer. I do get similar timings when I swap the two
>> > functions, and specifically still 15 seconds to read the file via
>> > file.read() and 2.4 seconds (more or less as before) via
>> > cPickle.load(fid).
>> >
>> > I thought that the order of operations might be an issue but apparently
>> > that was not the case...
>> 
>> What if you call one of them twice and then the other? Just trying to
>> rule out any possibility that it's a caching problem.
>> 
>> On my Linux box, running 2.7.9 64-bit, the two operations take roughly
>> the same amount of time (1.8 seconds for load vs 1s to read and 0.8 to
>> loads). Are you able to run this off a RAM disk or something?
>> 
>> Most curious.
> 
> 
> Thank you for taking the time to run my little script. I have now run it
> with multiple combinations of calls (twice the first then the other, then
> viceversa, then alternate between the two functions multiple times, then
> three times the second and once the first, ...) with no luck at all.
> 
> The file.read() line of code takes always at minimum 14 seconds (in all
> the trials I have done), while the cPickle.load call ranges between 2.3
> and 2.5 seconds.
> 
> I am puzzled with no end... Might there be something funny with my C
> libraries that use fread? I'm just shooting in the dark. I have a standard
> Python installation on Windows, nothing fancy :-(

Perhaps there is a size threshold? You could experiment with different block 
sizes in the following f.read() replacement:

def read_chunked(f, size=2**20):
    read = functools.partial(f.read, size)
    return "".join(iter(read, ""))

Thread

cPickle.load vs. file.read+cPickle.loads on large binary files andrea.gavana@gmail.com - 2015-11-17 05:26 -0800
  Re: cPickle.load vs. file.read+cPickle.loads on large binary files Peter Otten <__peter__@web.de> - 2015-11-17 15:14 +0100
    Re: cPickle.load vs. file.read+cPickle.loads on large binary files andrea.gavana@gmail.com - 2015-11-17 06:20 -0800
      Re: cPickle.load vs. file.read+cPickle.loads on large binary files Chris Angelico <rosuav@gmail.com> - 2015-11-18 02:20 +1100
        Re: cPickle.load vs. file.read+cPickle.loads on large binary files andrea.gavana@gmail.com - 2015-11-17 07:31 -0800
          Re: cPickle.load vs. file.read+cPickle.loads on large binary files Peter Otten <__peter__@web.de> - 2015-11-17 16:57 +0100
            Re: cPickle.load vs. file.read+cPickle.loads on large binary files andrea.gavana@gmail.com - 2015-11-17 08:31 -0800
              Re: cPickle.load vs. file.read+cPickle.loads on large binary files Peter Otten <__peter__@web.de> - 2015-11-17 18:20 +0100
          Re: cPickle.load vs. file.read+cPickle.loads on large binary files Nagy László Zsolt <gandalf@shopzeus.com> - 2015-11-18 10:00 +0100
            Re: cPickle.load vs. file.read+cPickle.loads on large binary files andrea.gavana@gmail.com - 2015-11-18 02:31 -0800

csiph-web