Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #98952

Re: cPickle.load vs. file.read+cPickle.loads on large binary files

Path csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From Nagy László Zsolt <gandalf@shopzeus.com>
Newsgroups comp.lang.python
Subject Re: cPickle.load vs. file.read+cPickle.loads on large binary files
Date Wed, 18 Nov 2015 10:00:21 +0100
Lines 10
Message-ID <mailman.406.1447837229.16136.python-list@python.org> (permalink)
References <463ad93c-0186-4911-9cd1-92d97b9dc87b@googlegroups.com> <mailman.387.1447769670.16136.python-list@python.org> <54330891-6568-4469-93ae-7a7825961500@googlegroups.com> <mailman.392.1447773612.16136.python-list@python.org> <420ec4e9-6af6-49bd-a9f4-8b47ef1f136e@googlegroups.com> <n2fip9$ajg$1@ger.gmane.org>
Mime-Version 1.0
Content-Type text/plain; charset=windows-1252
Content-Transfer-Encoding 7bit
X-Trace news.uni-berlin.de NCDgpPXmqDnQomc47HLoxQ4T6fKhHANXF5ZKhtBajl8Q==
Return-Path <gandalf@shopzeus.com>
X-Original-To python-list@python.org
Delivered-To python-list@mail.python.org
X-Spam-Status OK 0.010
X-Spam-Evidence '*H*': 0.98; '*S*': 0.00; 'win32': 0.03; 'binary': 0.05; 'subject:skip:c 10': 0.07; 'size)': 0.09; 'subject:files': 0.09; 'def': 0.13; 'f.read()': 0.16; 'mmap': 0.16; 'received:io': 0.16; 'received:psf.io': 0.16; 'platform,': 0.22; 'header:In- Reply-To:1': 0.24; 'disk': 0.27; 'fastest': 0.27; 'module.': 0.27; 'too.': 0.30; 'file': 0.34; 'could': 0.35; 'should': 0.36; 'there': 0.36; 'to:addr:python-list': 0.36; 'subject:: ': 0.37; 'to:addr:python.org': 0.40; 'charset:windows-1252': 0.62; 'different': 0.63; 'subject:. ': 0.67; 'experiment': 0.84; 'subject:+': 0.91
DKIM-Signature v=1; a=rsa-sha256; c=relaxed/relaxed; d=shopzeus.com; s=shopzeus_com; t=1447837221; bh=D6Kr8BbdN/KcHi5u9QBCrjcovT4Vl/veP4JG4mTLCPk=; h=Subject:To:References:From:Date:In-Reply-To:From; b=Q71l1FbG9hxDc7klLOezpbEW9TaA+b4C9XwI1+1ITlUbgZuG0QxKDbh4WUcQGSLuy vBSKPSVvwI6r6mNggb30r5NaOQ/PHn5NdNxTAmgrnCT8SJE6D2C0XCZh4kp0UQMHbc lWXGlQXbWIX3TBUMdQrX1hHIoqIOetFjVv7DWrWkbhy5QI5CPZjot/9Htmz2CN9zQ1 ECHXhJcVe27+yCQtgrsqITlKFWKcim9vQgcri6Rc5/gSz88MC/NdqPZId+15AZuoHK tpJpK+AKghVaVvBxaglt5qwKCMCjNqkFsZ3rg63pe6jmXIM8cZqHgnNl3QdnvODSgr KYm2fXJr7A3mw==
In-Reply-To <n2fip9$ajg$1@ger.gmane.org>
X-BeenThere python-list@python.org
X-Mailman-Version 2.1.20+
Precedence list
List-Id General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive <http://mail.python.org/pipermail/python-list/>
List-Post <mailto:python-list@python.org>
List-Help <mailto:python-list-request@python.org?subject=help>
List-Subscribe <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Xref csiph.com comp.lang.python:98952

Show key headers only | View raw

> Perhaps there is a size threshold? You could experiment with different block 
> sizes in the following f.read() replacement:
>
> def read_chunked(f, size=2**20):
>     read = functools.partial(f.read, size)
>     return "".join(iter(read, ""))
>
Under win32 platform, my experience is that the fastest way to read
binary file from disk is the mmap module. You should try that too.

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

cPickle.load vs. file.read+cPickle.loads on large binary files andrea.gavana@gmail.com - 2015-11-17 05:26 -0800
  Re: cPickle.load vs. file.read+cPickle.loads on large binary files Peter Otten <__peter__@web.de> - 2015-11-17 15:14 +0100
    Re: cPickle.load vs. file.read+cPickle.loads on large binary files andrea.gavana@gmail.com - 2015-11-17 06:20 -0800
      Re: cPickle.load vs. file.read+cPickle.loads on large binary files Chris Angelico <rosuav@gmail.com> - 2015-11-18 02:20 +1100
        Re: cPickle.load vs. file.read+cPickle.loads on large binary files andrea.gavana@gmail.com - 2015-11-17 07:31 -0800
          Re: cPickle.load vs. file.read+cPickle.loads on large binary files Peter Otten <__peter__@web.de> - 2015-11-17 16:57 +0100
            Re: cPickle.load vs. file.read+cPickle.loads on large binary files andrea.gavana@gmail.com - 2015-11-17 08:31 -0800
              Re: cPickle.load vs. file.read+cPickle.loads on large binary files Peter Otten <__peter__@web.de> - 2015-11-17 18:20 +0100
          Re: cPickle.load vs. file.read+cPickle.loads on large binary files Nagy László Zsolt <gandalf@shopzeus.com> - 2015-11-18 10:00 +0100
            Re: cPickle.load vs. file.read+cPickle.loads on large binary files andrea.gavana@gmail.com - 2015-11-18 02:31 -0800

csiph-web