Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #55519 > unrolled thread

Re: Idea for removing the GIL...

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Idea for removing the GIL... Adam Tauno Williams <awilliam@whitemice.org> - 2011-02-08 05:05 -0500
    Re: Idea for removing the GIL... Vishal <vsapre80@gmail.com> - 2011-02-08 04:34 -0800
      Re: Idea for removing the GIL... Stefan Behnel <stefan_ml@behnel.de> - 2011-02-08 18:05 +0100
      Re: Idea for removing the GIL... Roy Smith <roy@panix.com> - 2011-02-08 11:52 -0500
      Re: Idea for removing the GIL... Jean-Paul Calderone <calderone.jeanpaul@gmail.com> - 2011-02-08 04:53 -0800

#55519 — Re: Idea for removing the GIL...

On Tue, 2011-02-08 at 01:39 -0800, Vishal wrote: 
> Is it possible that the Python process, creates copies of the
> interpreter for each thread that is launched, and some how the thread
> is bound to its own interpreter ?
> and it "may" also allow the two threads to run in parallel, assuming
> the processors of today can send independent instructions from the
> same process to multiple cores?
> Comments, suggestions, brush offs  are welcome :))

Yes, it is possible, and done.  See the multiprocessing module.  It
works very well.
<http://docs.python.org/library/multiprocessing.html>

It isn't exactly the same as threads, but provides many similar
constructs.

[toc] | [next] | [standalone]

#55579

On Feb 8, 3:05 pm, Adam Tauno Williams <awill...@whitemice.org> wrote:
> On Tue, 2011-02-08 at 01:39 -0800, Vishal wrote:
> > Is it possible that the Python process, creates copies of the
> > interpreter for each thread that is launched, and some how the thread
> > is bound to its own interpreter ?
> > and it "may" also allow the two threads to run in parallel, assuming
> > the processors of today can send independent instructions from the
> > same process to multiple cores?
> > Comments, suggestions, brush offs  are welcome :))
>
> Yes, it is possible, and done.  See the multiprocessing module.  It
> works very well.
> <http://docs.python.org/library/multiprocessing.html>
>
> It isn't exactly the same as threads, but provides many similar
> constructs.

Hi,

Pardon me for my ignorance here, but 'multiprocessing' creates actual
processes using fork() or CreateProcess().
I was talking of a single process, running multiple instances of the
interpreter. Each thread, bound with its own interpreter.
so the GIL wont be an issue anymore...each interpreter has only one
thing to do, and that one thing holds the lock on its own interpreter.
Since its still the same process, data sharing should happen just like
in Threads.

Also, multiprocessing has issues on Windows (most probably because of
the way CreateProcess() functions...)

Thanks and best regards,
Vishal

[toc] | [prev] | [next] | [standalone]

#55718

Roy Smith, 08.02.2011 17:52:
> Robert Kern wrote:
>
>> Unlike a UNIX fork, CreateProcess() does not have the same copy-on-write
>> semantics for initializing the memory of the new process. If you want to pass
>> data to the children, the data must be pickled and sent across the process
>> boundary. He's not saying that multiprocessing isn't useful at all on
>> Windows, just less useful for the scenarios he is considering here.
>
> Amen, brother!  I used to work on a project that had a build system
> which was very fork() intensive (lots of little perl and shell scripts
> driven by make).  A full system build on a linux box took 30-60 minutes.
> Building the same code on windows/cygwin took about 12 hours.  Identical
> hardware (8-core, 16 gig Dell server, or something like that).
>
> As far as we could tell, it was entirely due to how bad Windows was at
> process creation.

Unlikely. Since you mention cygwin, it was likely due to the heavy lifting 
cygwin does in order to emulate fork() on Windows.

http://www.cygwin.com/faq/faq-nochunks.html#faq.api.fork

Stefan

[toc] | [prev] | [next] | [standalone]

#55775

In article <mailman.13.1297183120.1633.python-list@python.org>,
 Robert Kern <robert.kern@gmail.com> wrote:

> Unlike a UNIX fork, CreateProcess() does not have the same copy-on-write 
> semantics for initializing the memory of the new process. If you want to pass 
> data to the children, the data must be pickled and sent across the process 
> boundary. He's not saying that multiprocessing isn't useful at all on 
> Windows, just less useful for the scenarios he is considering here.

Amen, brother!  I used to work on a project that had a build system 
which was very fork() intensive (lots of little perl and shell scripts 
driven by make).  A full system build on a linux box took 30-60 minutes.  
Building the same code on windows/cygwin took about 12 hours.  Identical 
hardware (8-core, 16 gig Dell server, or something like that).

As far as we could tell, it was entirely due to how bad Windows was at 
process creation.

[toc] | [prev] | [next] | [standalone]

#56071

On Feb 8, 7:34 am, Vishal <vsapr...@gmail.com> wrote:
> On Feb 8, 3:05 pm, Adam Tauno Williams <awill...@whitemice.org> wrote:
>
> > On Tue, 2011-02-08 at 01:39 -0800, Vishal wrote:
> > > Is it possible that the Python process, creates copies of the
> > > interpreter for each thread that is launched, and some how the thread
> > > is bound to its own interpreter ?
> > > and it "may" also allow the two threads to run in parallel, assuming
> > > the processors of today can send independent instructions from the
> > > same process to multiple cores?
> > > Comments, suggestions, brush offs  are welcome :))
>
> > Yes, it is possible, and done.  See the multiprocessing module.  It
> > works very well.
> > <http://docs.python.org/library/multiprocessing.html>
>
> > It isn't exactly the same as threads, but provides many similar
> > constructs.
>
> Hi,
>
> Pardon me for my ignorance here, but 'multiprocessing' creates actual
> processes using fork() or CreateProcess().
> I was talking of a single process, running multiple instances of the
> interpreter. Each thread, bound with its own interpreter.
> so the GIL wont be an issue anymore...each interpreter has only one
> thing to do, and that one thing holds the lock on its own interpreter.
> Since its still the same process, data sharing should happen just like
> in Threads.

CPython does support multiple interpreters in a single process.
However,
you cannot have your cake and eat it too.  If you create multiple
interpreters,
then why do you think you'll be able to share objects between them for
free?

In what sense would you have *multiple* interpreters in that scenario?

You will need some sort of locking between the interpreters.  Then
you're either
back to the GIL or to some more limited form of sharing - such as you
might
get with the multiprocessing module.

Jean-Paul

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web