Path: csiph.com!usenet.pasdenom.info!aioe.org!news.stack.nl!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path: <rosuav@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.010
X-Spam-Evidence: '*H*': 0.98; '*S*': 0.00; '16,': 0.03; 'wednesday,': 0.07; "people's": 0.09; 'subject:files': 0.09; 'thats': 0.09; 'subject:How': 0.10; 'enough.': 0.16; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'wrote:': 0.18; 'wed,': 0.18; 'solution.': 0.20; 'fit': 0.20; 'work,': 0.20; 'memory': 0.22; 'manual': 0.22; 'header:In-Reply-To:1': 0.27; 'idea': 0.28; 'generally': 0.29; 'bigger': 0.30; 'message-id:@mail.gmail.com': 0.30; 'lines': 0.31; 'file': 0.32; 'says': 0.33; 'but': 0.35; 'received:google.com': 0.35; 'really': 0.36; "he's": 0.36; 'subject:?': 0.36; 'easily': 0.37; 'depends': 0.38; 'to:addr :python-list': 0.38; 'pm,': 0.38; 'to:addr:python.org': 0.39; 'even': 0.60; 'most': 0.60; 'course': 0.61; "it'd": 0.84; 'rusi': 0.91; '2013': 0.98
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; bh=lJMxccrwhHzfM8iS5NGBjL4nOJWbBqg4AFWvRzlYsGg=; b=on9gONsfJ8rj0RD3WVy8ipGr96dXRAc0KRbrkyyN9vzbKWtVFbawQjZ0vb5/ITiPPc 5TGJTQQGyQZzjyKhk95qnq09xzyeTXkmaAQ+A7BHnqXWurQRDJn3mKRD6+ym0RLp7fi1 Oi3CkDGce1s3i8Y+FgfG06wLKF1hhN1HRTzHamajK2QvE7sXk2/FVNF6zQ/e0V5xreXU tZRk/aPkVekeRCxcIH77SjZnnrNv51mMrf+HaQHz4LMJu4P52UVzU4FzVsds35OFHHy9 Kq0mQxQOunhu8kRbA1RIePu+Ax3g7D6qc0NUn37680jcHFzdh/pzk1c53M7gL3pBI2O8 z5fQ==
MIME-Version: 1.0
X-Received: by 10.66.158.72 with SMTP id ws8mr2430120pab.39.1381914555952; Wed, 16 Oct 2013 02:09:15 -0700 (PDT)
In-Reply-To: <81e53ed7-cc3e-437d-966d-9c1d79dc8c9f@googlegroups.com>
References: <0044bfd0-f07f-4f7b-b976-5df034b6fec6@googlegroups.com> <mailman.1107.1381907510.18130.python-list@python.org> <81e53ed7-cc3e-437d-966d-9c1d79dc8c9f@googlegroups.com>
Date: Wed, 16 Oct 2013 20:09:15 +1100
Subject: Re: How pickle helps in reading huge files?
From: Chris Angelico <rosuav@gmail.com>
To: python-list@python.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.1109.1381914565.18130.python-list@python.org>
Lines: 17
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1381914566 news.xs4all.nl 15872 [2001:888:2000:d::a6]:38692
X-Complaints-To: abuse@xs4all.nl
Xref: csiph.com comp.lang.python:56875

On Wed, Oct 16, 2013 at 7:51 PM, rusi <rustompmody@gmail.com> wrote:
> On Wednesday, October 16, 2013 12:35:42 PM UTC+5:30, St=E9phane Wirtel wr=
ote:
>> Keep it in memory
>
> Thats a strange answer given that the OP says his file is huge.
> Of course 'huge' may not really be huge -- that really depends on the h/w=
 he's using.

Most people's idea of a big file is one that has a few thousand lines
in it. That may be pretty huge in terms of manual work, but it'd fit
inside memory easily enough. And even if it really is bigger than
memory, chances are you can use your page file and still keep it in
"memory" - and that's generally the easiest, if perhaps not the most
efficient, solution.

ChrisA