Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #98952

Re: cPickle.load vs. file.read+cPickle.loads on large binary files

From Nagy László Zsolt <gandalf@shopzeus.com>
Newsgroups comp.lang.python
Subject Re: cPickle.load vs. file.read+cPickle.loads on large binary files
Date 2015-11-18 10:00 +0100
Message-ID <mailman.406.1447837229.16136.python-list@python.org> (permalink)
References (1 earlier) <mailman.387.1447769670.16136.python-list@python.org> <54330891-6568-4469-93ae-7a7825961500@googlegroups.com> <mailman.392.1447773612.16136.python-list@python.org> <420ec4e9-6af6-49bd-a9f4-8b47ef1f136e@googlegroups.com> <n2fip9$ajg$1@ger.gmane.org>

Show all headers | View raw

> Perhaps there is a size threshold? You could experiment with different block 
> sizes in the following f.read() replacement:
>
> def read_chunked(f, size=2**20):
>     read = functools.partial(f.read, size)
>     return "".join(iter(read, ""))
>
Under win32 platform, my experience is that the fastest way to read
binary file from disk is the mmap module. You should try that too.

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

cPickle.load vs. file.read+cPickle.loads on large binary files andrea.gavana@gmail.com - 2015-11-17 05:26 -0800
  Re: cPickle.load vs. file.read+cPickle.loads on large binary files Peter Otten <__peter__@web.de> - 2015-11-17 15:14 +0100
    Re: cPickle.load vs. file.read+cPickle.loads on large binary files andrea.gavana@gmail.com - 2015-11-17 06:20 -0800
      Re: cPickle.load vs. file.read+cPickle.loads on large binary files Chris Angelico <rosuav@gmail.com> - 2015-11-18 02:20 +1100
        Re: cPickle.load vs. file.read+cPickle.loads on large binary files andrea.gavana@gmail.com - 2015-11-17 07:31 -0800
          Re: cPickle.load vs. file.read+cPickle.loads on large binary files Peter Otten <__peter__@web.de> - 2015-11-17 16:57 +0100
            Re: cPickle.load vs. file.read+cPickle.loads on large binary files andrea.gavana@gmail.com - 2015-11-17 08:31 -0800
              Re: cPickle.load vs. file.read+cPickle.loads on large binary files Peter Otten <__peter__@web.de> - 2015-11-17 18:20 +0100
          Re: cPickle.load vs. file.read+cPickle.loads on large binary files Nagy László Zsolt <gandalf@shopzeus.com> - 2015-11-18 10:00 +0100
            Re: cPickle.load vs. file.read+cPickle.loads on large binary files andrea.gavana@gmail.com - 2015-11-18 02:31 -0800

csiph-web