Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #13030

Re: Processing a file using multithreads

Path csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!aioe.org!feeder.news-service.com!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path <abhishek.vit@gmail.com>
X-Original-To python-list@python.org
Delivered-To python-list@mail.python.org
X-Spam-Status OK 0.006
X-Spam-Evidence '*H*': 0.99; '*S*': 0.00; 'subject:using': 0.04; 'assumed': 0.09; 'command-line': 0.09; 'output': 0.10; 'am,': 0.12; 'skip:f 30': 0.13; 'subject:file': 0.13; '"end"': 0.16; 'abhishek': 0.16; 'cores': 0.16; 'input.': 0.16; 'pratap': 0.16; 'roy': 0.16; 'subject:Processing': 0.16; '\xa0in': 0.16; 'cc:addr :python-list': 0.16; 'wrote:': 0.16; 'processed': 0.18; 'cc:no real name:2**0': 0.20; "haven't": 0.20; 'sound': 0.21; 'cc:2**0': 0.22; 'header:In-Reply-To:1': 0.22; 'gil': 0.23; 'pieces': 0.23; 'sep': 0.23; 'input': 0.24; "python's": 0.24; 'handles': 0.25; 'tried': 0.26; 'function': 0.27; 'somebody': 0.28; 'url:mailman': 0.28; 'correct': 0.28; 'bound': 0.29; 'message- id:@mail.gmail.com': 0.29; 'cc:addr:python.org': 0.30; 'module': 0.30; 'i/o': 0.30; 'logic': 0.30; 'parent': 0.30; 'thanks': 0.30; 'this.': 0.32; 'actual': 0.32; 'handling': 0.32; 'agree': 0.33; 'it.': 0.33; 'too': 0.33; 'there': 0.33; 'instead': 0.33; 'url:listinfo': 0.33; 'wondering': 0.33; 'done': 0.34; 'character': 0.34; 'received:209.85.212': 0.34; 'file.': 0.34; 'running': 0.35; 'unless': 0.36; 'file': 0.36; 'url:python': 0.36; 'fri,': 0.36; 'skip:" 10': 0.36; 'thread': 0.37; 'using': 0.37; 'but': 0.37; 'open': 0.37; 'could': 0.38; 'think': 0.38; 'intensive': 0.38; 'some': 0.38; 'received:google.com': 0.38; 'url:org': 0.38; 'received:209.85': 0.38; 'subject:: ': 0.39; 'under': 0.39; 'relatively': 0.39; 'getting': 0.39; 'might': 0.40; 'processing': 0.40; 'your': 0.61; 'total': 0.61; 'address': 0.61; 'kind': 0.61; 'other.': 0.64; 'article': 0.76; 'bottle': 0.84; 'neck.': 0.84; 'parts,': 0.84
DKIM-Signature v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type:content-transfer-encoding; bh=o42qxaNwvQmOcltKlL3ZjZT2mvXqBFeOgqCPBUGlkpA=; b=BG73wvxoGa4fvRYVDKrE8MY1fQAHE8vjR9loUaEng7Ehi6L1R+7s5rs3iMcikTh8Fc 4WFzPZhOi18eYTapxjOwLVEAcqmfvbg5zyTTUHDZXlJeOxe0+SaZrEEA5XdfE3gBt8ye f4Wqt9VbzIv+1d+JfunC05kDfI2SOEOeotm9I=
MIME-Version 1.0
In-Reply-To <roy-77E2CD.09190709092011@news.panix.com>
References <mailman.885.1315522214.27778.python-list@python.org> <c6cbd486-7e5e-4d26-93b9-088d48a25dea@g9g2000yqb.googlegroups.com> <roy-77E2CD.09190709092011@news.panix.com>
From Abhishek Pratap <abhishek.vit@gmail.com>
Date Fri, 9 Sep 2011 10:07:41 -0700
Subject Re: Processing a file using multithreads
To Roy Smith <roy@panix.com>
Content-Type text/plain; charset=ISO-8859-1
Content-Transfer-Encoding quoted-printable
Cc python-list@python.org
X-BeenThere python-list@python.org
X-Mailman-Version 2.1.12
Precedence list
List-Id General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive <http://mail.python.org/pipermail/python-list>
List-Post <mailto:python-list@python.org>
List-Help <mailto:python-list-request@python.org?subject=help>
List-Subscribe <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups comp.lang.python
Message-ID <mailman.908.1315588083.27778.python-list@python.org> (permalink)
Lines 58
NNTP-Posting-Host 2001:888:2000:d::a6
X-Trace 1315588083 news.xs4all.nl 2541 [2001:888:2000:d::a6]:53149
X-Complaints-To abuse@xs4all.nl
Xref x330-a1.tempe.blueboxinc.net comp.lang.python:13030

Show key headers only | View raw

Hi All

@Roy : split in unix sounds good but will it be as efficient as
opening 10 different file handles on a file.  I haven't tried it so
just wondering if you have any experience with it.

Thanks for your input. Also I was not aware of the python's GIL limitation.

My application is not I/O bound as far as I can understand it. Each
line is read and then processed independently of each other. May be
this might sound I/O intensive as #N files will be read but I think if
I have 10 processes running under a parent then it might not be a
bottle neck.

Best,
-Abhi


On Fri, Sep 9, 2011 at 6:19 AM, Roy Smith <roy@panix.com> wrote:
> In article
> <c6cbd486-7e5e-4d26-93b9-088d48a25dea@g9g2000yqb.googlegroups.com>,
>  aspineux <aspineux@gmail.com> wrote:
>
>> On Sep 9, 12:49 am, Abhishek Pratap <abhishek....@gmail.com> wrote:
>> > 1. My input file is 10 GB.
>> > 2. I want to open 10 file handles each handling 1 GB of the file
>> > 3. Each file handle is processed in by an individual thread using the
>> > same function ( so total 10 cores are assumed to be available on the
>> > machine)
>> > 4. There will be 10 different output files
>> > 5. once the 10 jobs are complete a reduce kind of function will
>> > combine the output.
>> >
>> > Could you give some ideas ?
>>
>> You can use "multiprocessing" module instead of thread to bypass the
>> GIL limitation.
>
> I agree with this.
>
>> First cut your file in 10 "equal" parts. If it is line based search
>> for the first line close to the cut. Be sure to have "start" and
>> "end" for each parts, start is the address of the first character of
>> the first line and end is one line too much (== start of the next
>> block)
>
> How much of the total time will be I/O and how much actual processing?
> Unless your processing is trivial, the I/O time will be relatively
> small.  In that case, you might do well to just use the unix
> command-line "split" utility to split the file into pieces first, then
> process the pieces in parallel.  Why waste effort getting the
> file-splitting-at-line-boundaries logic correct when somebody has done
> it for you?
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
>

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Processing a file using multithreads Abhishek Pratap <abhishek.vit@gmail.com> - 2011-09-08 15:49 -0700
  Re: Processing a file using multithreads Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2011-09-09 12:03 +1200
  Re: Processing a file using multithreads aspineux <aspineux@gmail.com> - 2011-09-08 21:44 -0700
    Re: Processing a file using multithreads Roy Smith <roy@panix.com> - 2011-09-09 09:19 -0400
      Re: Processing a file using multithreads Abhishek Pratap <abhishek.vit@gmail.com> - 2011-09-09 10:07 -0700
        Re: Processing a file using multithreads Tim Roberts <timr@probo.com> - 2011-09-09 22:43 -0700

csiph-web