Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #98928

Re: cPickle.load vs. file.read+cPickle.loads on large binary files

Path csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From Chris Angelico <rosuav@gmail.com>
Newsgroups comp.lang.python
Subject Re: cPickle.load vs. file.read+cPickle.loads on large binary files
Date Wed, 18 Nov 2015 02:20:04 +1100
Lines 15
Message-ID <mailman.392.1447773612.16136.python-list@python.org> (permalink)
References <463ad93c-0186-4911-9cd1-92d97b9dc87b@googlegroups.com> <mailman.387.1447769670.16136.python-list@python.org> <54330891-6568-4469-93ae-7a7825961500@googlegroups.com>
Mime-Version 1.0
Content-Type text/plain; charset=UTF-8
X-Trace news.uni-berlin.de H0YlXskkXpy7m2i7hwcqcgPwVENPiobaMDdYUtt64yBg==
Return-Path <rosuav@gmail.com>
X-Original-To python-list@python.org
Delivered-To python-list@mail.python.org
X-Spam-Status OK 0.009
X-Spam-Evidence '*H*': 0.98; '*S*': 0.00; 'received:209.85.223': 0.03; 'subject:skip:c 10': 0.07; 'cc:addr:python-list': 0.09; 'subject:files': 0.09; 'wed,': 0.15; '(more': 0.16; '64-bit,': 0.16; 'caching': 0.16; 'case...': 0.16; 'curious.': 0.16; 'file.read()': 0.16; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'received:io': 0.16; 'received:psf.io': 0.16; 'wrote:': 0.16; 'load': 0.20; '2015': 0.20; 'cc:2**0': 0.20; 'cc:addr:python.org': 0.20; 'trying': 0.22; 'am,': 0.23; 'header :In-Reply-To:1': 0.24; 'linux': 0.26; 'possibility': 0.27; 'message-id:@mail.gmail.com': 0.27; 'disk': 0.27; 'specifically': 0.28; 'operations': 0.31; 'seconds': 0.31; 'run': 0.33; 'ram': 0.33; 'rule': 0.33; 'similar': 0.33; 'file': 0.34; 'running': 0.34; 'received:google.com': 0.35; 'nov': 0.35; 'problem.': 0.35; 'but': 0.36; 'received:209.85': 0.36; 'subject:: ': 0.37; 'two': 0.37; 'thought': 0.37; 'received:209': 0.38; 'thank': 0.38; 'still': 0.40; 'your': 0.60; 'box,': 0.67; 'subject:. ': 0.67; 'answer.': 0.72; '0.8': 0.84; 'chrisa': 0.84; 'timings': 0.84; 'to:none': 0.91; 'subject:+': 0.91
DKIM-Signature v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:cc :content-type; bh=f5brrjeZNfxdqt52Z1nu3H5qJlie12u63K5NnZtvRDo=; b=v2ZEuy+bQLhuvJ+wQoeJQGaxx9n/Ty4OM/bffY14jtxb5dBzVByGUEMiO8LYmBNJmi 1+Xao9WcQotjMmmfMltaHGm7HDz8u1mKfTJgU248wyLBF8xqWmvIwrdBKrtQG2iaHLdW iDvnwA1iGD5E4ZbC9P2gDB4pBsvLlyzdJDkKuMdtaHLZoJ2lnkut7pT+4MHrUZi9zf8j wKK1T5/grf3sB0Nk+QQtPxBWkBPtVQ+yo7VaUOgdeHXvX8SbI+qIEivbf+dX6albcWsp 7lBNxK9Rvo+F2sAdycwUmBWmgSMO9NXpIlDQhRkBH/XgNqxC6oi5T0pV8UR29DTE/kxL Fv4Q==
X-Received by 10.107.16.84 with SMTP id y81mr35884759ioi.19.1447773604199; Tue, 17 Nov 2015 07:20:04 -0800 (PST)
In-Reply-To <54330891-6568-4469-93ae-7a7825961500@googlegroups.com>
X-BeenThere python-list@python.org
X-Mailman-Version 2.1.20+
Precedence list
List-Id General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive <http://mail.python.org/pipermail/python-list/>
List-Post <mailto:python-list@python.org>
List-Help <mailto:python-list-request@python.org?subject=help>
List-Subscribe <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Xref csiph.com comp.lang.python:98928

Show key headers only | View raw

On Wed, Nov 18, 2015 at 1:20 AM,  <andrea.gavana@gmail.com> wrote:
> Thank you for your answer. I do get similar timings when I swap the two functions, and specifically still 15 seconds to read the file via file.read() and 2.4 seconds (more or less as before) via cPickle.load(fid).
>
> I thought that the order of operations might be an issue but apparently that was not the case...

What if you call one of them twice and then the other? Just trying to
rule out any possibility that it's a caching problem.

On my Linux box, running 2.7.9 64-bit, the two operations take roughly
the same amount of time (1.8 seconds for load vs 1s to read and 0.8 to
loads). Are you able to run this off a RAM disk or something?

Most curious.

ChrisA

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

cPickle.load vs. file.read+cPickle.loads on large binary files andrea.gavana@gmail.com - 2015-11-17 05:26 -0800
  Re: cPickle.load vs. file.read+cPickle.loads on large binary files Peter Otten <__peter__@web.de> - 2015-11-17 15:14 +0100
    Re: cPickle.load vs. file.read+cPickle.loads on large binary files andrea.gavana@gmail.com - 2015-11-17 06:20 -0800
      Re: cPickle.load vs. file.read+cPickle.loads on large binary files Chris Angelico <rosuav@gmail.com> - 2015-11-18 02:20 +1100
        Re: cPickle.load vs. file.read+cPickle.loads on large binary files andrea.gavana@gmail.com - 2015-11-17 07:31 -0800
          Re: cPickle.load vs. file.read+cPickle.loads on large binary files Peter Otten <__peter__@web.de> - 2015-11-17 16:57 +0100
            Re: cPickle.load vs. file.read+cPickle.loads on large binary files andrea.gavana@gmail.com - 2015-11-17 08:31 -0800
              Re: cPickle.load vs. file.read+cPickle.loads on large binary files Peter Otten <__peter__@web.de> - 2015-11-17 18:20 +0100
          Re: cPickle.load vs. file.read+cPickle.loads on large binary files Nagy László Zsolt <gandalf@shopzeus.com> - 2015-11-18 10:00 +0100
            Re: cPickle.load vs. file.read+cPickle.loads on large binary files andrea.gavana@gmail.com - 2015-11-18 02:31 -0800

csiph-web