Re: working with a large file

Path	csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!aioe.org!news.stack.nl!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path	<python-python-list@m.gmane.org>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.003
X-Spam-Evidence	'H': 0.99; 'S': 0.00; 'memory.': 0.05; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:80.91.229.12': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'received:lo.gmane.org': 0.09; 'subject:file': 0.13; 'from:addr:behnel.de': 0.16; 'from:addr:stefan_ml': 0.16; 'from:name:stefan behnel': 0.16; 'mmap': 0.16; 'this?': 0.21; 'file,': 0.21; 'header:In-Reply- To:1': 0.22; 'preferred': 0.25; 'random': 0.28; 'depends': 0.28; 'server': 0.29; 'module.': 0.29; 'stefan': 0.29; '(and': 0.29; 'arguments.': 0.30; 'subject:working': 0.30; 'to:addr:python- list': 0.33; 'operating': 0.33; 'header:User-Agent:1': 0.34; 'header:X-Complaints-To:1': 0.35; 'file': 0.36; 'several': 0.37; 'but': 0.37; 'received:org': 0.38; 'subject:: ': 0.39; 'header :Mime-Version:1': 0.39; 'subject:with': 0.39; 'to:addr:python.org': 0.39; 'might': 0.40; 'received:188': 0.68; 'system),': 0.84
X-Injected-Via-Gmane	http://gmane.org/
To	python-list@python.org
From	Stefan Behnel <stefan_ml@behnel.de>
Subject	Re: working with a large file
Date	Mon, 12 Sep 2011 16:10:44 +0200
References	<CAOF-KfhdAx=HppPy5SFrWRrgsaYxWAkO0YH=WjpBSY3_wBAXqw@mail.gmail.com>
Mime-Version	1.0
Content-Type	text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding	7bit
X-Gmane-NNTP-Posting-Host	ppp-188-174-36-144.dynamic.mnet-online.de
User-Agent	Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.21) Gecko/20110831 Lightning/1.0b2 Thunderbird/3.1.13
In-Reply-To	<CAOF-KfhdAx=HppPy5SFrWRrgsaYxWAkO0YH=WjpBSY3_wBAXqw@mail.gmail.com>
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.12
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups	comp.lang.python
Message-ID	<mailman.1035.1315836659.27778.python-list@python.org> (permalink)
Lines	11
NNTP-Posting-Host	2001:888:2000:d::a6
X-Trace	1315836659 news.xs4all.nl 2486 [2001:888:2000:d::a6]:43510
X-Complaints-To	abuse@xs4all.nl
Xref	x330-a1.tempe.blueboxinc.net comp.lang.python:13180

Show key headers only | View raw

Rita, 12.09.2011 12:34:
> I have a large file, 18gb uncompressed, and I would like to know what is the
> preferred method to read this file for random access. I have several
> processes reading the file which different calculate arguments. My server
> has 64gb of memory. Not sure what is the preferred way to do this?

It depends on the content (and likely also the operating system), but you 
might want to take a look at the mmap module.

Stefan

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

Re: working with a large file Stefan Behnel <stefan_ml@behnel.de> - 2011-09-12 16:10 +0200

csiph-web