Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #12988

Re: Processing a file using multithreads

Path csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!gegeweb.org!de-l.enfer-du-nord.net!feeder1.enfer-du-nord.net!feeder.news-service.com!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path <SRS0=1Aec=32=ieee.org=davea@srs.perfora.net>
X-Original-To python-list@python.org
Delivered-To python-list@mail.python.org
X-Spam-Status OK 0.010
X-Spam-Evidence '*H*': 0.98; '*S*': 0.00; 'subject:using': 0.04; 'threads,': 0.07; 'python': 0.08; 'assumed': 0.09; 'output': 0.10; 'subject:file': 0.13; 'abhishek': 0.16; 'chunks': 0.16; 'cores': 0.16; 'pratap': 0.16; 'received:192.168.1.104': 0.16; 'subject:Processing': 0.16; 'cc:addr:python-list': 0.16; 'wrote:': 0.16; 'processed': 0.18; 'cc:no real name:2**0': 0.20; 'cc:2**0': 0.22; 'header:In-Reply-To:1': 0.22; 'pm,': 0.24; 'input': 0.24; 'handles': 0.25; 'function': 0.27; 'separate': 0.28; 'cc:addr:python.org': 0.30; 'separately.': 0.30; 'threads.': 0.30; 'handling': 0.32; 'probably': 0.33; 'there': 0.33; 'header:User- Agent:1': 0.34; 'file.': 0.34; 'test': 0.34; 'similar': 0.35; 'file': 0.36; 'thread': 0.37; 'using': 0.37; 'open': 0.37; 'could': 0.38; 'somewhat': 0.38; 'some': 0.38; 'guys': 0.38; 'should': 0.38; 'subject:: ': 0.39; 'received:192': 0.39; "there's": 0.39; 'case': 0.39; 'total': 0.61; 'kind': 0.61; 'here': 0.65; 'share': 0.65; 'processes,': 0.67; 'header:Reply- To:1': 0.71; 'reply-to:no real name:2**0': 0.71; '02:59': 0.84
Date Thu, 08 Sep 2011 21:34:16 -0400
From Dave Angel <davea@ieee.org>
User-Agent Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.18) Gecko/20110617 Thunderbird/3.1.11
MIME-Version 1.0
To Abhishek Pratap <abhishek.vit@gmail.com>
Subject Re: Processing a file using multithreads
References <CAJbA1KCJ4484icWmThLYUewWk6BGf=HZemVtfeOdOva7=c3YyA@mail.gmail.com>
In-Reply-To <CAJbA1KCJ4484icWmThLYUewWk6BGf=HZemVtfeOdOva7=c3YyA@mail.gmail.com>
Content-Type text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding 7bit
X-Provags-ID V02:K0:A3bd+WDO+AHb+XmgBly1NQRgb89wfrEYwnif3ehxrXP +kGrLX2tAgV1v3V+CoM3b3GNd4L2AsCFckLEyDul2fr0WaoN01 EyhRI+iwwAO3+prTSA2jHKCq6RC+ls+3YMGOTq3lK8hYGe9iFs m6ZgYCTdYvi8dPW2MYtcEMl31nnSHvalVKMpbeihttxwgE6jRi elIsT2w0D1FafQ34VT89AgFXwBNnMav6XN/sGpSm3tvy7rAuoj mNhkJUpn2K3YCVONqUOWXpUU3xQee9v/U7OS+QuN2CgckbVlmT TijLeGDpxH6xoy2f+lkC3RbrasnFYwEik5NgI0lVbR8LwdD4g= =
Cc python-list@python.org
X-BeenThere python-list@python.org
X-Mailman-Version 2.1.12
Precedence list
Reply-To davea@ieee.org
List-Id General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive <http://mail.python.org/pipermail/python-list>
List-Post <mailto:python-list@python.org>
List-Help <mailto:python-list-request@python.org?subject=help>
List-Subscribe <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups comp.lang.python
Message-ID <mailman.889.1315532083.27778.python-list@python.org> (permalink)
Lines 31
NNTP-Posting-Host 2001:888:2000:d::a6
X-Trace 1315532083 news.xs4all.nl 2422 [2001:888:2000:d::a6]:55403
X-Complaints-To abuse@xs4all.nl
Xref x330-a1.tempe.blueboxinc.net comp.lang.python:12988

Show key headers only | View raw

On 01/-10/-28163 02:59 PM, Abhishek Pratap wrote:
> Hi Guys
>
> My experience with python is 2 days and I am looking for a slick way
> to use multi-threading to process a file. Here is what I would like to
> do which is somewhat similar to MapReduce in concept.
>
> # test case
>
> 1. My input file is 10 GB.
> 2. I want to open 10 file handles each handling 1 GB of the file
> 3. Each file handle is processed in by an individual thread using the
> same function ( so total 10 cores are assumed to be available on the
> machine)
> 4. There will be 10 different output files
> 5. once the 10 jobs are complete a reduce kind of function will
> combine the output.
>
> Could you give some ideas ?
>
> So given a file I would like to read it in #N chunks through #N file
> handles and process each of them separately.
>
> Best,
> -Abhi
>
You should probably forget threads, and simply do them as 10 separate 
processes, all launched by a single parent.  Since they don't share any 
state, there's no need to get the inefficiency of threads.

DaveA

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

Re: Processing a file using multithreads Dave Angel <davea@ieee.org> - 2011-09-08 21:34 -0400

csiph-web