Re: cPickle.load vs. file.read+cPickle.loads on large binary files

Path	csiph.com!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From	Peter Otten <__peter__@web.de>
Newsgroups	comp.lang.python
Subject	Re: cPickle.load vs. file.read+cPickle.loads on large binary files
Date	Tue, 17 Nov 2015 15:14:18 +0100
Organization	None
Lines	45
Message-ID	<mailman.387.1447769670.16136.python-list@python.org> (permalink)
References	<463ad93c-0186-4911-9cd1-92d97b9dc87b@googlegroups.com>
Mime-Version	1.0
Content-Type	text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding	7Bit
X-Trace	news.uni-berlin.de EHZFet8mPCiNZvyM0qVUAgebwO0I3IlWOphkAVX3ol9w==
Return-Path	<python-python-list@m.gmane.org>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.000
X-Spam-Evidence	'H': 1.00; 'S': 0.00; 'resulting': 0.04; 'binary': 0.05; '(so': 0.07; '__name__': 0.07; 'subject:skip:c 10': 0.07; 'files:': 0.09; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'snippet': 0.09; 'subject:files': 0.09; 'unexpected': 0.09; 'advance': 0.10; 'python': 0.10; ':-)': 0.12; '2.7': 0.13; 'output': 0.13; 'file,': 0.15; "'__main__':": 0.16; '(created': 0.16; '1).': 0.16; 'dummy': 0.16; 'file.read()': 0.16; 'numpy': 0.16; 'problem).': 0.16; 'received:80.91.229.3': 0.16; 'received:dip0.t-ipconnect.de': 0.16; 'received:io': 0.16; 'received:plane.gmane.org': 0.16; 'received:psf.io': 0.16; 'received:t-ipconnect.de': 0.16; 'seconds.': 0.16; 'wrote:': 0.16; 'memory': 0.17; 'string': 0.17; 'directory.': 0.18; 'load': 0.20; 'windows': 0.20; 'arrays': 0.22; 'header:User-Agent:1': 0.26; 'header:X-Complaints-To:1': 0.26; '2.3': 0.27; 'function': 0.28; "i'd": 0.31; 'seconds': 0.31; 'option': 0.31; 'ram': 0.33; '(for': 0.34; 'structure': 0.34; 'file': 0.34; 'done': 0.35; 'but': 0.36; 'list,': 0.36; 'should': 0.36; 'there': 0.36; 'to:addr:python- list': 0.36; 'subject:: ': 0.37; 'two': 0.37; 'received:org': 0.37; 'things': 0.38; 'thank': 0.38; 'files': 0.38; 'data': 0.39; 'takes': 0.39; 'to:addr:python.org': 0.40; 'received:de': 0.40; 'some': 0.40; 'email addr:gmail.com': 0.62; 'relatively': 0.63; 'more': 0.63; 'times': 0.63; 'between': 0.65; 'differences': 0.66; 'subject:. ': 0.67; 'approaches': 0.72; '100': 0.79; 'pc,': 0.91; 'subject:+': 0.91
X-Injected-Via-Gmane	http://gmane.org/
X-Gmane-NNTP-Posting-Host	p57bd942b.dip0.t-ipconnect.de
User-Agent	KNode/4.13.3
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.20+
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Xref	csiph.com comp.lang.python:98923

Show key headers only | View raw

andrea.gavana@gmail.com wrote:

> Hello List,
> 
>      I am working with relatively humongous binary files (created via
>      cPickle), and I stumbled across some unexpected (for me) performance
>      differences between two approaches I use to load those files:
> 
> 1. Simply use cPickle.load(fid)
> 
> 2. Read the file as binary using file.read() and then use cPickle.loads on
> the resulting output
> 
> In the snippet below, the MakePickle function is a dummy function that
> generates a relatively big binary file with cPickle (WARNING: around 3 GB)
> in the current directory. I am using NumPy arrays to make the file big but
> my original data structure is much more complicated, and things like HDF5
> or databases are currently not an option - I'd like to stay with pickles.
> 
> The ReadPickle function simply uses cPickle.load(fid) on the opened binary
> file, and on my PC it takes about 2.3 seconds (approach 1).
> 
> The ReadPlusLoads function reads the file using file.read() and then use
> cPickle.loads on the resulting output (approach 2). On my PC, the
> file.read() process takes 15 seconds (!!!) and the cPickle.loads only 1.5
> seconds.
> 
> What baffles me is the time it takes to read the file using file.read():
> is there any way to slurp it all in one go (somehow) into a string ready
> for cPickle.loads without that much of an overhead?
> 
> Note that all of this has been done on Windows 7 64bit with Python 2.7
> 64bit, with 16 cores and 100 GB RAM (so memory should not be a problem).
> 
> Thank you in advance for all suggestions :-) .
> 
> Andrea.
> 
> if __name__ == '__main__':
>     ReadPickle()
>     ReadPlusLoads()

Do you get roughly the same times when you execute ReadPlusLoads() before 
ReadPIckle()?

Thread

cPickle.load vs. file.read+cPickle.loads on large binary files andrea.gavana@gmail.com - 2015-11-17 05:26 -0800
  Re: cPickle.load vs. file.read+cPickle.loads on large binary files Peter Otten <__peter__@web.de> - 2015-11-17 15:14 +0100
    Re: cPickle.load vs. file.read+cPickle.loads on large binary files andrea.gavana@gmail.com - 2015-11-17 06:20 -0800
      Re: cPickle.load vs. file.read+cPickle.loads on large binary files Chris Angelico <rosuav@gmail.com> - 2015-11-18 02:20 +1100
        Re: cPickle.load vs. file.read+cPickle.loads on large binary files andrea.gavana@gmail.com - 2015-11-17 07:31 -0800
          Re: cPickle.load vs. file.read+cPickle.loads on large binary files Peter Otten <__peter__@web.de> - 2015-11-17 16:57 +0100
            Re: cPickle.load vs. file.read+cPickle.loads on large binary files andrea.gavana@gmail.com - 2015-11-17 08:31 -0800
              Re: cPickle.load vs. file.read+cPickle.loads on large binary files Peter Otten <__peter__@web.de> - 2015-11-17 18:20 +0100
          Re: cPickle.load vs. file.read+cPickle.loads on large binary files Nagy László Zsolt <gandalf@shopzeus.com> - 2015-11-18 10:00 +0100
            Re: cPickle.load vs. file.read+cPickle.loads on large binary files andrea.gavana@gmail.com - 2015-11-18 02:31 -0800

csiph-web