Processing a file using multithreads

From	Abhishek Pratap <abhishek.vit@gmail.com>
Date	2011-09-08 15:49 -0700
Subject	Processing a file using multithreads
Newsgroups	comp.lang.python
Message-ID	<mailman.885.1315522214.27778.python-list@python.org> (permalink)

Show all headers | View raw

Hi Guys

My experience with python is 2 days and I am looking for a slick way
to use multi-threading to process a file. Here is what I would like to
do which is somewhat similar to MapReduce in concept.

# test case

1. My input file is 10 GB.
2. I want to open 10 file handles each handling 1 GB of the file
3. Each file handle is processed in by an individual thread using the
same function ( so total 10 cores are assumed to be available on the
machine)
4. There will be 10 different output files
5. once the 10 jobs are complete a reduce kind of function will
combine the output.

Could you give some ideas ?

So given a file I would like to read it in #N chunks through #N file
handles and process each of them separately.

Best,
-Abhi

Back to comp.lang.python | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

Processing a file using multithreads Abhishek Pratap <abhishek.vit@gmail.com> - 2011-09-08 15:49 -0700
  Re: Processing a file using multithreads Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2011-09-09 12:03 +1200
  Re: Processing a file using multithreads aspineux <aspineux@gmail.com> - 2011-09-08 21:44 -0700
    Re: Processing a file using multithreads Roy Smith <roy@panix.com> - 2011-09-09 09:19 -0400
      Re: Processing a file using multithreads Abhishek Pratap <abhishek.vit@gmail.com> - 2011-09-09 10:07 -0700
        Re: Processing a file using multithreads Tim Roberts <timr@probo.com> - 2011-09-09 22:43 -0700

csiph-web