Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #12983

Re: Processing a file using multithreads

Show all headers | View raw

Abhishek Pratap wrote:

> 3. Each file handle is processed in by an individual thread using the
> same function ( so total 10 cores are assumed to be available on the
> machine)

Are you expecting the processing to be CPU bound or
I/O bound?

If it's I/O bound, multiple cores won't help you, and
neither will threading, because it's the disk doing the
work, not the CPU.

If it's CPU bound, multiple threads in one Python process
won't help, because of the GIL. You'll have to fork
multiple OS processes in order to get Python code running
in parallel on different cores.

-- 
Greg

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Processing a file using multithreads Abhishek Pratap <abhishek.vit@gmail.com> - 2011-09-08 15:49 -0700
  Re: Processing a file using multithreads Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2011-09-09 12:03 +1200
  Re: Processing a file using multithreads aspineux <aspineux@gmail.com> - 2011-09-08 21:44 -0700
    Re: Processing a file using multithreads Roy Smith <roy@panix.com> - 2011-09-09 09:19 -0400
      Re: Processing a file using multithreads Abhishek Pratap <abhishek.vit@gmail.com> - 2011-09-09 10:07 -0700
        Re: Processing a file using multithreads Tim Roberts <timr@probo.com> - 2011-09-09 22:43 -0700

csiph-web