Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From: =?UTF-8?Q?Nagy_L=c3=a1szl=c3=b3_Zsolt?= <gandalf@shopzeus.com>
Newsgroups: comp.lang.python
Subject: Re: cPickle.load vs. file.read+cPickle.loads on large binary files
Date: Wed, 18 Nov 2015 10:00:21 +0100
Lines: 10
Message-ID: <mailman.406.1447837229.16136.python-list@python.org>
References: <463ad93c-0186-4911-9cd1-92d97b9dc87b@googlegroups.com> <mailman.387.1447769670.16136.python-list@python.org> <54330891-6568-4469-93ae-7a7825961500@googlegroups.com> <mailman.392.1447773612.16136.python-list@python.org> <420ec4e9-6af6-49bd-a9f4-8b47ef1f136e@googlegroups.com> <n2fip9$ajg$1@ger.gmane.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-Trace: news.uni-berlin.de NCDgpPXmqDnQomc47HLoxQ4T6fKhHANXF5ZKhtBajl8Q==
Return-Path: <gandalf@shopzeus.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.010
X-Spam-Evidence: '*H*': 0.98; '*S*': 0.00; 'win32': 0.03; 'binary': 0.05; 'subject:skip:c 10': 0.07; 'size)': 0.09; 'subject:files': 0.09; 'def': 0.13; 'f.read()': 0.16; 'mmap': 0.16; 'received:io': 0.16; 'received:psf.io': 0.16; 'platform,': 0.22; 'header:In- Reply-To:1': 0.24; 'disk': 0.27; 'fastest': 0.27; 'module.': 0.27; 'too.': 0.30; 'file': 0.34; 'could': 0.35; 'should': 0.36; 'there': 0.36; 'to:addr:python-list': 0.36; 'subject:: ': 0.37; 'to:addr:python.org': 0.40; 'charset:windows-1252': 0.62; 'different': 0.63; 'subject:. ': 0.67; 'experiment': 0.84; 'subject:+': 0.91
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shopzeus.com; s=shopzeus_com; t=1447837221; bh=D6Kr8BbdN/KcHi5u9QBCrjcovT4Vl/veP4JG4mTLCPk=; h=Subject:To:References:From:Date:In-Reply-To:From; b=Q71l1FbG9hxDc7klLOezpbEW9TaA+b4C9XwI1+1ITlUbgZuG0QxKDbh4WUcQGSLuy vBSKPSVvwI6r6mNggb30r5NaOQ/PHn5NdNxTAmgrnCT8SJE6D2C0XCZh4kp0UQMHbc lWXGlQXbWIX3TBUMdQrX1hHIoqIOetFjVv7DWrWkbhy5QI5CPZjot/9Htmz2CN9zQ1 ECHXhJcVe27+yCQtgrsqITlKFWKcim9vQgcri6Rc5/gSz88MC/NdqPZId+15AZuoHK tpJpK+AKghVaVvBxaglt5qwKCMCjNqkFsZ3rg63pe6jmXIM8cZqHgnNl3QdnvODSgr KYm2fXJr7A3mw==
In-Reply-To: <n2fip9$ajg$1@ger.gmane.org>
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.20+
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Xref: csiph.com comp.lang.python:98952


> Perhaps there is a size threshold? You could experiment with different block 
> sizes in the following f.read() replacement:
>
> def read_chunked(f, size=2**20):
>     read = functools.partial(f.read, size)
>     return "".join(iter(read, ""))
>
Under win32 platform, my experience is that the fastest way to read
binary file from disk is the mmap module. You should try that too.