Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail From: Peter Otten <__peter__@web.de> Newsgroups: comp.lang.python Subject: Re: cPickle.load vs. file.read+cPickle.loads on large binary files Date: Tue, 17 Nov 2015 16:57:28 +0100 Organization: None Lines: 44 Message-ID: References: <463ad93c-0186-4911-9cd1-92d97b9dc87b@googlegroups.com> <54330891-6568-4469-93ae-7a7825961500@googlegroups.com> <420ec4e9-6af6-49bd-a9f4-8b47ef1f136e@googlegroups.com> Mime-Version: 1.0 Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: 7Bit X-Trace: news.uni-berlin.de BzTYeB9o0B7GAmaL36yKyAOSWg1FkYCG+RhmYfIm91gg== Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.000 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; ':-(': 0.07; 'subject:skip:c 10': 0.07; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'size)': 0.09; 'subject:files': 0.09; 'python': 0.10; 'times,': 0.13; 'def': 0.13; 'wed,': 0.15; '(more': 0.16; '...)': 0.16; '64-bit,': 0.16; 'caching': 0.16; 'case...': 0.16; 'chris,': 0.16; 'combinations': 0.16; 'curious.': 0.16; 'dark.': 0.16; 'done),': 0.16; 'f.read()': 0.16; 'fancy': 0.16; 'file.read()': 0.16; 'received:80.91.229.3': 0.16; 'received:dip0.t-ipconnect.de': 0.16; 'received:io': 0.16; 'received:plane.gmane.org': 0.16; 'received:psf.io': 0.16; 'received:t-ipconnect.de': 0.16; 'seconds.': 0.16; 'shooting': 0.16; 'wrote:': 0.16; 'alternate': 0.18; 'script.': 0.18; '(in': 0.18; 'load': 0.20; '2015': 0.20; 'first,': 0.20; 'libraries': 0.22; 'trying': 0.22; 'am,': 0.23; 'installation': 0.23; 'second': 0.24; 'all.': 0.24; 'header:User-Agent:1': 0.26; 'header:X -Complaints-To:1': 0.26; 'linux': 0.26; 'chris': 0.26; 'possibility': 0.27; '2.3': 0.27; 'disk': 0.27; 'specifically': 0.28; 'other,': 0.29; 'windows,': 0.29; "i'm": 0.30; 'code': 0.30; 'operations': 0.31; 'seconds': 0.31; 'run': 0.33; 'ram': 0.33; 'rule': 0.33; 'similar': 0.33; 'file': 0.34; 'running': 0.34; 'could': 0.35; 'nov': 0.35; 'something': 0.35; 'problem.': 0.35; 'but': 0.36; 'there': 0.36; 'to:addr:python-list': 0.36; 'subject:: ': 0.37; 'two': 0.37; 'received:org': 0.37; 'thought': 0.37; 'minimum': 0.38; 'thank': 0.38; 'takes': 0.39; 'to:addr:python.org': 0.40; 'still': 0.40; 'received:de': 0.40; 'your': 0.60; 'email addr:gmail.com': 0.62; 'different': 0.63; 'times': 0.63; 'between': 0.65; 'box,': 0.67; 'subject:. ': 0.67; 'answer.': 0.72; 'ranges': 0.76; 'funny': 0.83; '0.8': 0.84; 'andrea': 0.84; 'experiment': 0.84; 'timings': 0.84; 'subject:+': 0.91; 'luck': 0.95 X-Injected-Via-Gmane: http://gmane.org/ X-Gmane-NNTP-Posting-Host: p57bd9e92.dip0.t-ipconnect.de User-Agent: KNode/4.13.3 X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.20+ Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Xref: csiph.com comp.lang.python:98932 andrea.gavana@gmail.com wrote: > Hi Chris, > > On Tuesday, November 17, 2015 at 4:20:34 PM UTC+1, Chris Angelico wrote: >> On Wed, Nov 18, 2015 at 1:20 AM, Andrea Gavana wrote: >> > Thank you for your answer. I do get similar timings when I swap the two >> > functions, and specifically still 15 seconds to read the file via >> > file.read() and 2.4 seconds (more or less as before) via >> > cPickle.load(fid). >> > >> > I thought that the order of operations might be an issue but apparently >> > that was not the case... >> >> What if you call one of them twice and then the other? Just trying to >> rule out any possibility that it's a caching problem. >> >> On my Linux box, running 2.7.9 64-bit, the two operations take roughly >> the same amount of time (1.8 seconds for load vs 1s to read and 0.8 to >> loads). Are you able to run this off a RAM disk or something? >> >> Most curious. > > > Thank you for taking the time to run my little script. I have now run it > with multiple combinations of calls (twice the first then the other, then > viceversa, then alternate between the two functions multiple times, then > three times the second and once the first, ...) with no luck at all. > > The file.read() line of code takes always at minimum 14 seconds (in all > the trials I have done), while the cPickle.load call ranges between 2.3 > and 2.5 seconds. > > I am puzzled with no end... Might there be something funny with my C > libraries that use fread? I'm just shooting in the dark. I have a standard > Python installation on Windows, nothing fancy :-( Perhaps there is a size threshold? You could experiment with different block sizes in the following f.read() replacement: def read_chunked(f, size=2**20): read = functools.partial(f.read, size) return "".join(iter(read, ""))