Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder4.news.weretis.net!rt.uk.eu.org!newsfeed.xs4all.nl!newsfeed4.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Date: Fri, 06 Dec 2013 13:04:27 +1030
From: Garthy <garthy_nhtyp@entropicsoftware.com>
User-Agent: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:8.0) Gecko/20111105 Thunderbird/8.0
MIME-Version: 1.0
To: python-list@python.org
Subject: Embedding multiple interpreters
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.3620.1386298086.18130.python-list@python.org>
Lines: 146
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:61110


Hi!

I hope I've got the right list here- there were a few to choose from. :}

I am trying to embed Python with multiple interpreters into an existing 
application. I have things working fine with a single interpreter thus 
far. I am running into problems when using multiple interpreters [1] and 
I am presently trying to track down these issues. Can anyone familiar 
with the process of embedding multiple interpreters have a skim of the 
details below and let me know of any obvious problems? If I can get the 
essentials right, then presumably it's just a matter of my tracking down 
any problems with my code.

I am presently using Python 3.3.3.

What I am after:

- Each sub-interpreter will have its own dedicated thread. Each thread 
will have no more than one sub-interpreter. Basically, there is a 
one-to-one mapping between threads and interpreters (some threads are 
unrelated to Python though).

- The default interpreter in the main thread will never be used, 
although I can explicitly use it if it'll help in some way.

- Each thread is created and managed outside of Python. This can't be 
readily changed.

- I have a single internal module I need to be able to use for each 
interpreter.

- I load scripts into __main__ and create objects from it to bootstrap.

- I understand that for the most part only a single interpreter will be 
running at a time due to the GIL. This is unfortunate but not a major 
problem.

- I don't need to share objects between interpreters (if it is even 
possible- I don't know).

- My fallback if I can't do this is to implement each instance in a 
dedicated *process* rather than per-thread. However, there is a 
significant cost to doing this that I would rather not incur.

Could I confirm:

- There is one GIL in a given process, shared amongst all (sub) 
interpreters. There seems some disagreement on this one online, although 
I'm fairly confident that there is only the one GIL.

- I am using the mod_wsgi source for inspiration. Is there a better 
source for an example of embedding multiple interpreters?

A query re the doco:

http://docs.python.org/3/c-api/init.html#gilstate

"Python supports the creation of additional interpreters (using 
Py_NewInterpreter()), but mixing multiple interpreters and the 
PyGILState_*() API is unsupported."

Is this actually correct? mod_wsgi seems to do it. Have I misunderstood?

I've extracted what I have so far from my code into a form that can be 
followed more easily. Hopefully I have not made any mistakes in doing 
so. The essence of what my code calls should be as follows:

=== Global init, run once:

static PyThreadState *mtstate = NULL;

PyImport_AppendInittab("myinternalmodule", PyInit_myinternalmodule);
Py_SetProgramName((wchar_t *)"foo");
Pu_InitializeEx(0);
PyEval_InitThreads();
mtstate = PyThreadState_Get();
PyEval_ReleaseThread(mtstate);

=== Global shutdown, run once at end:

Py_Finalize();

=== Per-interpreter init in main thread before launching child thread:

(none thus far)

=== Init in dedicated thread for each interpreter:

// NB: Also protected by a single global non-Python mutex to be sure.

PyGILState_STATE gil = PyGILState_Ensure();
PyThreadState *save_tstate = PyThreadState_Swap(NULL);
state = Py_NewInterpreter();
PyThreadState_Swap(save_tstate);

PyObject *mmodule = PyImport_AddModule("__main__");
Py_INCREF(mmodule);

PyImport_ImportModule("myinternalmodule");

PyGILState_Release(gil);

=== Shutdown in dedicated thread for each interpreter:

// NB: Also protected by the same single global non-Python mutex as in 
the init.

PyGILState_STATE gil = PyGILState_Ensure();
PyThreadState *save_tstate = PyThreadState_Swap(state);
Py_EndInterpreter(state);
PyThreadState_Swap(save_tstate);
PyGILState_Release(gil);

=== Placed at top of scope where calls made to Python C API:

SafeLock lock;

=== SafeLock implementation:

class SafeLock
{
   public:
     SafeLock() {gil = PyGILState_Ensure();}
     ~SafeLock() {PyGILState_Release(gil);}

   private:
     PyGILState_STATE gil;
};

===

Does this look roughly right? Have I got the global and per-interpreter 
init and shutdown right? Am I locking correctly in SafeLock- is 
PyGILState_Ensure() and PyGILState_Release() sufficient?

Is there an authoritative summary of the global and per-interpreter init 
and shutdown somewhere that I have missed? Any resource I should be reading?

Cheers,
Garth

[1] It presently crashes in Py_EndInterpreter() after running through a 
series of tests during the shutdown of the 32nd interpreter I create. I 
don't know if this is significant, but the tests pass for the first 31 
interpreters.