Path: csiph.com!usenet.pasdenom.info!dedibox.gegeweb.org!gegeweb.eu!nntpfeed.proxad.net!proxad.net!feeder1-2.proxad.net!usenet-fr.net!nerim.net!novso.com!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path: <python.list@tim.thechases.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.001
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'binary': 0.05; 'cache': 0.05; 'processing.': 0.07; 'python': 0.09; '(it': 0.09; 'backwards': 0.09; 'chunks': 0.09; 'pointers': 0.09; 'times,': 0.13; '(like': 0.15; "hasn't": 0.15; '-tkc': 0.16; 'decimal.': 0.16; 'disk.': 0.16; 'dump': 0.16; 'from:addr:python.list': 0.16; 'from:addr:tim.thechases.com': 0.16; 'from:name:tim chase': 0.16; 'message-id:@tim.thechases.com': 0.16; 'received:70.251': 0.16; 'received:dsl.rcsntx.swbell.net': 0.16; 'received:rcsntx.swbell.net': 0.16; 'received:swbell.net': 0.16; 'run.': 0.16; 'simplest': 0.16; 'written.': 0.16; 'wrote:': 0.17; 'variables': 0.17; 'trying': 0.21; 'finally,': 0.22; 'external': 0.24; 'paul': 0.24; 'header:In-Reply-To:1': 0.25; 'header:User- Agent:1': 0.26; '---': 0.26; 'question': 0.27; 'converting': 0.27; 'actual': 0.28; '(unless': 0.29; 'decimal': 0.29; 'i/o': 0.29; 'writes:': 0.29; 'usually': 0.30; '(and': 0.32; 'file': 0.32; 'running': 0.32; 'to:addr:python-list': 0.33; 'agree': 0.34; 'something': 0.35; 'there': 0.35; 'really': 0.36; 'but': 0.36; 'should': 0.36; 'subject: (': 0.36; 'enough': 0.36; 'passed': 0.37; 'rather': 0.37; 'data': 0.37; 'subject:: ': 0.38; 'files': 0.38; 'performance': 0.39; 'to:addr:python.org': 0.39; 'subject:-': 0.40; 'within': 0.64; 'great': 0.64; 'taking': 0.65; 'reverse': 0.65; 'due': 0.66; 'records': 0.68; 'received:50.22': 0.84; 'subject:Fast': 0.84; 'subject:read': 0.84; 'subject:write': 0.84; 'thing,': 0.84; 'to/from': 0.84; 'approach.': 0.91
Date: Tue, 23 Oct 2012 11:53:37 -0500
From: Tim Chase <python.list@tim.thechases.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.24) Gecko/20111120 Icedove/3.1.16
MIME-Version: 1.0
To: python-list@python.org
Subject: Re: Fast forward-backward (write-read)
References: <mailman.2667.1351003942.27098.python-list@python.org> <7xehkpxiw0.fsf@ruckus.brouhaha.com>
In-Reply-To: <7xehkpxiw0.fsf@ruckus.brouhaha.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - boston.accountservergroup.com
X-AntiAbuse: Original Domain - python.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - tim.thechases.com
X-Source: 
X-Source-Args: 
X-Source-Dir: 
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.2676.1351011158.27098.python-list@python.org>
Lines: 35
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1351011158 news.xs4all.nl 6919 [2001:888:2000:d::a6]:47066
X-Complaints-To: abuse@xs4all.nl
Xref: csiph.com comp.lang.python:31943

On 10/23/12 11:17, Paul Rubin wrote:
> Virgil Stokes <vs@it.uu.se> writes:
>> Finally, to my question --- What is a fast way to write these
>> variables to an external file and then read them in backwards?
> 
> Seeking backwards in files works, but the performance hit is
> significant.  There is also a performance hit to scanning pointers
> backwards in memory, due to cache misprediction.  If it's something
> you're just running a few times, seeking backwards the simplest
> approach.  If you're really trying to optimize the thing, you might
> buffer up large chunks (like 1 MB) before writing.  If you're writing
> once and reading multiple times, you might reverse the order of records
> within the chunks during the writing phase.

I agree with Paul here, it's been a while since I did it, and my
dataset was small enough (and passed through once) so I just let it
run.  Writing larger chunks is definitely a good way to go.

> You're of course taking a performance bath from writing the program in
> Python to begin with (unless using scipy/numpy or the like), enough that
> it might dominate any effects of how the files are written.

I usually find that the I/O almost always overwhelms the actual
processing.

> Of course (it should go without saying) that you want to dump in a
> binary format rather than converting to decimal.

Again, the conversion to/from decimal hasn't been a great cost in my
experience, as it's overwhelmed by the I/O cost of shoveling the
data to/from disk.

-tkc