Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From: Peter Otten <__peter__@web.de>
Newsgroups: comp.lang.python
Subject: Re: cPickle.load vs. file.read+cPickle.loads on large binary files
Date: Tue, 17 Nov 2015 18:20:44 +0100
Organization: None
Lines: 62
Message-ID: <mailman.399.1447780855.16136.python-list@python.org>
References: <463ad93c-0186-4911-9cd1-92d97b9dc87b@googlegroups.com> <mailman.387.1447769670.16136.python-list@python.org> <54330891-6568-4469-93ae-7a7825961500@googlegroups.com> <mailman.392.1447773612.16136.python-list@python.org> <420ec4e9-6af6-49bd-a9f4-8b47ef1f136e@googlegroups.com> <mailman.395.1447775861.16136.python-list@python.org> <f9a98231-e6df-4557-8ca1-20d9644825ca@googlegroups.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 7Bit
User-Agent: KNode/4.13.3
Precedence: list
Xref: csiph.com comp.lang.python:98938

andrea.gavana@gmail.com wrote:

>> > I am puzzled with no end... Might there be something funny with my C
>> > libraries that use fread? I'm just shooting in the dark. I have a
>> > standard Python installation on Windows, nothing fancy :-(
>> 
>> Perhaps there is a size threshold? You could experiment with different
>> block sizes in the following f.read() replacement:
>> 
>> def read_chunked(f, size=2**20):
>>     read = functools.partial(f.read, size)
>>     return "".join(iter(read, ""))
> 
> 
> Thank you for the suggestion. I have used the read_chunked function in my
> experiments now and I can report a nice improvements - I have tried
> various chunk sizes, from 2**10 to 2**31-1, and in general the optimum
> lies around size=2**22, although it is essentially flat from 2**20 up to
> 2**30 - with some interesting spikes at 45 seconds for 2**14 and 2**15
> (see table below).
> 
> Using your suggestion, I got it down to 3.4 seconds (on average). Still at
> least twice slower than cPickle.load, but better.
> 
> What I find most puzzling is that a pure file.read() (or your read_chunked
> variation) should normally be much faster than a cPickle.load (which does
> so many more things than just reading a file), shouldn't it?

That would have been my expectation, too. 

I had a quick look into the fileobject.c source and didn't see anything that 
struck me as suspicious.

I think you should file a bug report so that an expert can check if there is 
an underlying problem in Python or if it is a matter of the OS. 

> Timing table:
> 
> Size (power of 2)	Read Time (seconds)
> 10	9.14
> 11	8.59
> 12	7.67
> 13	5.70
> 14	46.06
> 15	45.00
> 16	24.80
> 17	14.23
> 18	8.95
> 19	5.58
> 20	3.41
> 21	3.39
> 22	3.34
> 23	3.39
> 24	3.39
> 25	3.42
> 26	3.43
> 27	3.44
> 28	3.48
> 29	3.59
> 30	3.72