Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #61132

Re: Embedding multiple interpreters

Show key headers only | View raw

Hi Chris,

On 06/12/13 19:03, Chris Angelico wrote:
 > On Fri, Dec 6, 2013 at 6:59 PM, Garthy
 > <garthy_nhtyp@entropicsoftware.com>  wrote:
 >> Hi Chris (and Michael),
 >
 > Hehe. People often say that to me IRL, addressing me and my brother.
 > But he isn't on python-list, so you clearly mean Michael Torrie, yet
 > my brain still automatically thought you were addressing Michael
 > Angelico :)

These strange coincidences happen from time to time- it's entertaining 
when they do. :)

 >> To allow each script to run in its own environment, with minimal 
chance of
 >> inadvertent interaction between the environments, whilst allowing each
 >> script the ability to stall on conditions that will be later met by 
another
 >> thread supplying the information, and to fit in with existing
 >> infrastructure.
 >
 > Are the scripts written cooperatively, or must you isolate one from
 > another? If you need to isolate them for trust reasons, then there's
 > only one solution, and that's separate processes with completely
 > separate interpreters. But if you're prepared to accept that one
 > thread of execution is capable of mangling another's state, things are
 > a lot easier. You can protect against *inadvertent* interaction much
 > more easily than malicious interference. It may be that you can get
 > away with simply running multiple threads in one interpreter;
 > obviously that would have problems if you need more than one CPU core
 > between them all (hello GIL), but that would really be your first
 > limit. One thread could fiddle with __builtins__ or a standard module
 > and thus harass another thread, but you would know if that's what's
 > going on.

I think the ideal is completely sandboxed, but it's something that I 
understand I may need to make compromises on. The bare minimum would be 
protection against inadvertent interaction. Better yet would be a setup 
that made such interaction annoyingly difficult, and the ideal would be 
where it was impossible to interfere. My approaching this problem with 
interpreters was based on an assumption that it might provide a 
reasonable level of isolation- perhaps not ideal, but hopefully good enough.

The closest analogy for understanding would be browser plugins: Scripts 
from multiple authors who for the most part aren't looking to create 
deliberate incompatibilities or interference between plugins. The 
isolation is basic, and some effort is made to make sure that one plugin 
can't cripple another trivially, but the protection is not exhaustive.

Strangely enough, the GIL restriction isn't a big one in this case. For 
the application, the common case is actually one script running at a 
time, with other scripts waiting or not running at that time. They do 
sometimes overlap, but this isn't the common case. If it turned out that 
only one script could be progressing at a time, it's an annoyance but 
not a deal-breaker. If it's suboptimal (as seems to be the case), then 
it's actually not a major issue.

With the single interpreter and multiple thread approach suggested, do 
you know if this will work with threads created externally to Python, 
ie. if I can create a thread in my application as normal, and then call 
something like PyGILState_Ensure() to make sure that Python has the 
internals it needs to work with it, and then use the GIL (or similar) to 
ensure that accesses to it remain thread-safe? If the answer is yes I 
can integrate such a thing more easily as an experiment. If it requires 
calling a dedicated "control" script that feeds out threads then it 
would need a fair bit more mucking about to integrate- I'd like to avoid 
this if possible.

Cheers,
Garth

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Embedding multiple interpreters Garthy <garthy_nhtyp@entropicsoftware.com> - 2013-12-06 13:04 +1030
  Re: Embedding multiple interpreters Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2013-12-06 19:58 +1300
    Re: Embedding multiple interpreters Garthy <garthy_nhtyp@entropicsoftware.com> - 2013-12-06 18:29 +1030
      Re: Embedding multiple interpreters Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2013-12-07 11:09 +1300
        Re: Embedding multiple interpreters Garthy <garthy_nhtyp@entropicsoftware.com> - 2013-12-07 11:27 +1030
    Re: Embedding multiple interpreters Garthy <garthy_nhtyp@entropicsoftware.com> - 2013-12-06 18:51 +1030
    Re: Embedding multiple interpreters Chris Angelico <rosuav@gmail.com> - 2013-12-06 19:33 +1100
    Re: Embedding multiple interpreters Chris Angelico <rosuav@gmail.com> - 2013-12-06 20:27 +1100
    Re: Embedding multiple interpreters Garthy <garthy_nhtyp@entropicsoftware.com> - 2013-12-06 20:05 +1030
      Re: Embedding multiple interpreters Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2013-12-07 11:23 +1300
        Re: Embedding multiple interpreters Garthy <garthy_nhtyp@entropicsoftware.com> - 2013-12-07 11:01 +1030
    Re: Embedding multiple interpreters Garthy <garthy_nhtyp@entropicsoftware.com> - 2013-12-06 20:14 +1030
    Re: Embedding multiple interpreters Tim Golden <mail@timgolden.me.uk> - 2013-12-06 10:17 +0000
    Re: Embedding multiple interpreters Chris Angelico <rosuav@gmail.com> - 2013-12-06 22:57 +1100
    Re: Embedding multiple interpreters Garthy <garthy_nhtyp@entropicsoftware.com> - 2013-12-07 00:26 +1030
    Re: Embedding multiple interpreters Garthy <garthy_nhtyp@entropicsoftware.com> - 2013-12-07 00:34 +1030

csiph-web