Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #13030

Re: Processing a file using multithreads

Show all headers | View raw

Hi All

@Roy : split in unix sounds good but will it be as efficient as
opening 10 different file handles on a file.  I haven't tried it so
just wondering if you have any experience with it.

Thanks for your input. Also I was not aware of the python's GIL limitation.

My application is not I/O bound as far as I can understand it. Each
line is read and then processed independently of each other. May be
this might sound I/O intensive as #N files will be read but I think if
I have 10 processes running under a parent then it might not be a
bottle neck.

Best,
-Abhi


On Fri, Sep 9, 2011 at 6:19 AM, Roy Smith <roy@panix.com> wrote:
> In article
> <c6cbd486-7e5e-4d26-93b9-088d48a25dea@g9g2000yqb.googlegroups.com>,
>  aspineux <aspineux@gmail.com> wrote:
>
>> On Sep 9, 12:49 am, Abhishek Pratap <abhishek....@gmail.com> wrote:
>> > 1. My input file is 10 GB.
>> > 2. I want to open 10 file handles each handling 1 GB of the file
>> > 3. Each file handle is processed in by an individual thread using the
>> > same function ( so total 10 cores are assumed to be available on the
>> > machine)
>> > 4. There will be 10 different output files
>> > 5. once the 10 jobs are complete a reduce kind of function will
>> > combine the output.
>> >
>> > Could you give some ideas ?
>>
>> You can use "multiprocessing" module instead of thread to bypass the
>> GIL limitation.
>
> I agree with this.
>
>> First cut your file in 10 "equal" parts. If it is line based search
>> for the first line close to the cut. Be sure to have "start" and
>> "end" for each parts, start is the address of the first character of
>> the first line and end is one line too much (== start of the next
>> block)
>
> How much of the total time will be I/O and how much actual processing?
> Unless your processing is trivial, the I/O time will be relatively
> small.  In that case, you might do well to just use the unix
> command-line "split" utility to split the file into pieces first, then
> process the pieces in parallel.  Why waste effort getting the
> file-splitting-at-line-boundaries logic correct when somebody has done
> it for you?
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
>

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Processing a file using multithreads Abhishek Pratap <abhishek.vit@gmail.com> - 2011-09-08 15:49 -0700
  Re: Processing a file using multithreads Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2011-09-09 12:03 +1200
  Re: Processing a file using multithreads aspineux <aspineux@gmail.com> - 2011-09-08 21:44 -0700
    Re: Processing a file using multithreads Roy Smith <roy@panix.com> - 2011-09-09 09:19 -0400
      Re: Processing a file using multithreads Abhishek Pratap <abhishek.vit@gmail.com> - 2011-09-09 10:07 -0700
        Re: Processing a file using multithreads Tim Roberts <timr@probo.com> - 2011-09-09 22:43 -0700

csiph-web