Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!aioe.org!feeder.news-service.com!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.006 X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; 'subject:using': 0.04; 'assumed': 0.09; 'command-line': 0.09; 'output': 0.10; 'am,': 0.12; 'skip:f 30': 0.13; 'subject:file': 0.13; '"end"': 0.16; 'abhishek': 0.16; 'cores': 0.16; 'input.': 0.16; 'pratap': 0.16; 'roy': 0.16; 'subject:Processing': 0.16; '\xa0in': 0.16; 'cc:addr :python-list': 0.16; 'wrote:': 0.16; 'processed': 0.18; 'cc:no real name:2**0': 0.20; "haven't": 0.20; 'sound': 0.21; 'cc:2**0': 0.22; 'header:In-Reply-To:1': 0.22; 'gil': 0.23; 'pieces': 0.23; 'sep': 0.23; 'input': 0.24; "python's": 0.24; 'handles': 0.25; 'tried': 0.26; 'function': 0.27; 'somebody': 0.28; 'url:mailman': 0.28; 'correct': 0.28; 'bound': 0.29; 'message- id:@mail.gmail.com': 0.29; 'cc:addr:python.org': 0.30; 'module': 0.30; 'i/o': 0.30; 'logic': 0.30; 'parent': 0.30; 'thanks': 0.30; 'this.': 0.32; 'actual': 0.32; 'handling': 0.32; 'agree': 0.33; 'it.': 0.33; 'too': 0.33; 'there': 0.33; 'instead': 0.33; 'url:listinfo': 0.33; 'wondering': 0.33; 'done': 0.34; 'character': 0.34; 'received:209.85.212': 0.34; 'file.': 0.34; 'running': 0.35; 'unless': 0.36; 'file': 0.36; 'url:python': 0.36; 'fri,': 0.36; 'skip:" 10': 0.36; 'thread': 0.37; 'using': 0.37; 'but': 0.37; 'open': 0.37; 'could': 0.38; 'think': 0.38; 'intensive': 0.38; 'some': 0.38; 'received:google.com': 0.38; 'url:org': 0.38; 'received:209.85': 0.38; 'subject:: ': 0.39; 'under': 0.39; 'relatively': 0.39; 'getting': 0.39; 'might': 0.40; 'processing': 0.40; 'your': 0.61; 'total': 0.61; 'address': 0.61; 'kind': 0.61; 'other.': 0.64; 'article': 0.76; 'bottle': 0.84; 'neck.': 0.84; 'parts,': 0.84 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type:content-transfer-encoding; bh=o42qxaNwvQmOcltKlL3ZjZT2mvXqBFeOgqCPBUGlkpA=; b=BG73wvxoGa4fvRYVDKrE8MY1fQAHE8vjR9loUaEng7Ehi6L1R+7s5rs3iMcikTh8Fc 4WFzPZhOi18eYTapxjOwLVEAcqmfvbg5zyTTUHDZXlJeOxe0+SaZrEEA5XdfE3gBt8ye f4Wqt9VbzIv+1d+JfunC05kDfI2SOEOeotm9I= MIME-Version: 1.0 In-Reply-To: References: From: Abhishek Pratap Date: Fri, 9 Sep 2011 10:07:41 -0700 Subject: Re: Processing a file using multithreads To: Roy Smith Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: python-list@python.org X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 58 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1315588083 news.xs4all.nl 2541 [2001:888:2000:d::a6]:53149 X-Complaints-To: abuse@xs4all.nl Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:13030 Hi All @Roy : split in unix sounds good but will it be as efficient as opening 10 different file handles on a file. I haven't tried it so just wondering if you have any experience with it. Thanks for your input. Also I was not aware of the python's GIL limitation. My application is not I/O bound as far as I can understand it. Each line is read and then processed independently of each other. May be this might sound I/O intensive as #N files will be read but I think if I have 10 processes running under a parent then it might not be a bottle neck. Best, -Abhi On Fri, Sep 9, 2011 at 6:19 AM, Roy Smith wrote: > In article > , > =A0aspineux wrote: > >> On Sep 9, 12:49=A0am, Abhishek Pratap wrote: >> > 1. My input file is 10 GB. >> > 2. I want to open 10 file handles each handling 1 GB of the file >> > 3. Each file handle is processed in by an individual thread using the >> > same function ( so total 10 cores are assumed to be available on the >> > machine) >> > 4. There will be 10 different output files >> > 5. once the 10 jobs are complete a reduce kind of function will >> > combine the output. >> > >> > Could you give some ideas ? >> >> You can use "multiprocessing" module instead of thread to bypass the >> GIL limitation. > > I agree with this. > >> First cut your file in 10 "equal" parts. If it is line based search >> for the first line close to the cut. Be sure to have "start" and >> "end" for each parts, start is the address of the first character of >> the first line and end is one line too much (=3D=3D start of the next >> block) > > How much of the total time will be I/O and how much actual processing? > Unless your processing is trivial, the I/O time will be relatively > small. =A0In that case, you might do well to just use the unix > command-line "split" utility to split the file into pieces first, then > process the pieces in parallel. =A0Why waste effort getting the > file-splitting-at-line-boundaries logic correct when somebody has done > it for you? > > -- > http://mail.python.org/mailman/listinfo/python-list > >