Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #103937 > unrolled thread

Inception

Back to article view | Back to comp.lang.python

Contents

  Inception Denis Akhiyarov <denis.akhiyarov@gmail.com> - 2016-03-02 21:01 -0800
    Re: Inception Ben Finney <ben+python@benfinney.id.au> - 2016-03-03 16:32 +1100
      Re: Inception Denis Akhiyarov <denis.akhiyarov@gmail.com> - 2016-03-02 21:43 -0800
        Re: Inception Ben Finney <ben+python@benfinney.id.au> - 2016-03-03 17:06 +1100
          Re: Inception Paul Rubin <no.email@nospam.invalid> - 2016-03-02 22:30 -0800
        Re: Inception Rustom Mody <rustompmody@gmail.com> - 2016-03-02 22:22 -0800
    Re: Inception Ben Finney <ben+python@benfinney.id.au> - 2016-03-03 16:37 +1100
    Re: Inception Christian Gollwitzer <auriocus@gmx.de> - 2016-03-03 09:09 +0100
      Re: Inception Denis Akhiyarov <denis.akhiyarov@gmail.com> - 2016-03-03 15:04 -0800

#103937 — Inception

Is it possible to embed CPython in CPython, e.g. using ctypes, cffi, or cython?

[toc] | [next] | [standalone]

#103938

Denis Akhiyarov <denis.akhiyarov@gmail.com> writes:

> Is it possible to embed CPython in CPython, e.g. using ctypes, cffi,
> or cython?

I don't know what you mean by “embed” in this context. What would
qualify as “yes” for you?

-- 
 \        “You don't change the world by placidly finding your bliss — |
  `\        you do it by focusing your discontent in productive ways.” |
_o__)                                       —Paul Z. Myers, 2011-08-31 |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#103940

Embed using C-API from ctypes, cffi or cython, but not using subprocesses. Thanks for asking! 

Here is link to official documentation about embedding:

https://docs.python.org/3.6/extending/index.html

[toc] | [prev] | [next] | [standalone]

#103941

Denis Akhiyarov <denis.akhiyarov@gmail.com> writes:

> Embed using C-API from ctypes, cffi or cython, but not using
> subprocesses. Thanks for asking!

I'm glad you appreciate my asking, but I am no closer to an answer. What
would meet your requirements, what would be a “yes” for you?

> Here is link to official documentation about embedding:
> https://docs.python.org/3.6/extending/index.html

I'm familiar with that document. It doesn't help me understand what you
mean by “embed CPython in CPython”.

-- 
 \        “I went to the hardware store and bought some used paint. It |
  `\                      was in the shape of a house.” —Steven Wright |
_o__)                                                                  |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#103943

Ben Finney <ben+python@benfinney.id.au> writes:
> I'm familiar with that document. It doesn't help me understand what you
> mean by “embed CPython in CPython”.

It seems straightforward enough to me.  Lots of Python programs load C
extensions.  Denis is asking whether one of those C extensions could
itself embed CPython through the embedding interface.  My answer: I
don't know, but I'm dubious about the prospect, based on how the
embedding interface works.  One of Armin Ronacher's old blog posts
discusses the interface and some of its drawbacks:

http://lucumr.pocoo.org/2014/8/16/the-python-i-would-like-to-see/#the-damn-interpreter

It might be that MicroPython or PyPy is easier to embed in that way.

[toc] | [prev] | [next] | [standalone]

#103942

On Thursday, March 3, 2016 at 11:13:58 AM UTC+5:30, Denis Akhiyarov wrote:
> Embed using C-API from ctypes, cffi or cython, but not using subprocesses. Thanks for asking! 
> 
> Here is link to official documentation about embedding:
> 
> https://docs.python.org/3.6/extending/index.html

Nowadays its fashionable to talk of the "xy problem"
http://code.tutsplus.com/tutorials/what-is-the-xy-problem-and-why-is-it-bad--cms-21023

ie you really need y but only know how to ask for x.

Maybe you need to back off from python... docs/apis/jargon etc
And tell us at a higher level WHAT you want and leave it to others to come
up with HOWs

[Disclaimer: Dont know that much about the extending API but if I remember right
full-scale re-entrancy was considered and rejected]

[toc] | [prev] | [next] | [standalone]

#103939

Ben Finney <ben+python@benfinney.id.au> writes:

> Denis Akhiyarov <denis.akhiyarov@gmail.com> writes:
>
> > Is it possible to embed CPython in CPython, e.g. using ctypes, cffi,
> > or cython?
>
> I don't know what you mean by “embed” in this context. What would
> qualify as “yes” for you?

Here is CPython embedded in CPython::

    >>> import subprocess
    >>> subprocess.check_call(["python3", "--version"])
    Python 3.5.1+

If that doesn't impress you, then you'll need to be more specific about
what you're expecting.

-- 
 \        “If the arguments in favor of atheism upset you, explain why |
  `\        they’re wrong. If you can’t do that, that’s your problem.” |
_o__)                                     —Amanda Marcotte, 2015-02-13 |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#103948

Hi Denis,

Am 03.03.16 um 06:01 schrieb Denis Akhiyarov:
> Is it possible to embed CPython in CPython, e.g. using ctypes, cffi, or cython?
>

since you titled your question with a famous movie title, I take it more 
as a "philosophical" question (musing about the CPython world) than an 
actual demand. Technically, I think it cannot work with the current 
state of CPython. Embedding CPython into another application requires 
you to load libpython3.5.so (or similar depending on OS and version) and 
to call the functino sfrom within there. When a Python program runs in 
the standalone interpreter, then this DLL is already loaded, obviously, 
so the question is if the Python C API functions can be called from a 
Python script.

Possibly  ctypes or cffi can do that for you, but you would end up with 
the same interpreter that executes your main script. A DLL is loaded by 
the OS only once, which makes it possible to create plugin interfaces 
etc. using shared symbols in C.

Now, the CPython interpreter saves its state in global variables. 
Therefore only one interpreter can exist in a single process, and this 
is also the reason that multithreading is complicated and does not 
really work (GIL &c).

How useful could it be to have more than a single interpreter? For 
instance, Tcl does it in a different way. All of the interpreter state 
is stored within a Tcl_Interp struct which is passed around to every API 
function. You can create more than a single interpreter, and that even 
works from the script level using the "interp create" command. This is 
useful to shield user script evaluation from the main script. Such 
interpreters can be made "safe", i.e. to not allow file operations, or 
to limit the CPU time for script evaluation, which has its primary use 
case in things like web browser plugins or simple servers. A pythonized 
version of the API would look like this:

from pyembedding import Interp

i=Interp()
i2=Interp(safe=True)

i.eval('print("Hello world")')
# prints Hello world

i2.eval('print("Hello world")')
# raises a SafeInterpException
# I/O  is unsafe and disabled here

def greetme():
	print("Hello world")

i2.alias(greet=greetme)
i2.eval('greet')
# calls back to greetme in main interpreter

This mechanism would also allow true multithreading. If you run another 
interpreter in the second thread, it will have it's own GIL.

I'm not familiar with the internals of other Python implementation, but 
I can imagine that maybe in Jython such an extension could be written.

	Christian

[toc] | [prev] | [next] | [standalone]

#103998

Thank you Paul and Christian for your reply and understanding my question. I was actually inspired by this example for IronPython and thought that this is pretty useful for testing and exploring of C-API:

http://www.voidspace.org.uk/ironpython/ip_in_ip.shtml

So does it all boil down to GIL restriction in CPython?



On Thursday, March 3, 2016 at 2:10:15 AM UTC-6, Christian Gollwitzer wrote:
> Hi Denis,
> 
> Am 03.03.16 um 06:01 schrieb Denis Akhiyarov:
> > Is it possible to embed CPython in CPython, e.g. using ctypes, cffi, or cython?
> >
> 
> since you titled your question with a famous movie title, I take it more 
> as a "philosophical" question (musing about the CPython world) than an 
> actual demand. Technically, I think it cannot work with the current 
> state of CPython. Embedding CPython into another application requires 
> you to load libpython3.5.so (or similar depending on OS and version) and 
> to call the functino sfrom within there. When a Python program runs in 
> the standalone interpreter, then this DLL is already loaded, obviously, 
> so the question is if the Python C API functions can be called from a 
> Python script.
> 
> Possibly  ctypes or cffi can do that for you, but you would end up with 
> the same interpreter that executes your main script. A DLL is loaded by 
> the OS only once, which makes it possible to create plugin interfaces 
> etc. using shared symbols in C.
> 
> Now, the CPython interpreter saves its state in global variables. 
> Therefore only one interpreter can exist in a single process, and this 
> is also the reason that multithreading is complicated and does not 
> really work (GIL &c).
> 
> How useful could it be to have more than a single interpreter? For 
> instance, Tcl does it in a different way. All of the interpreter state 
> is stored within a Tcl_Interp struct which is passed around to every API 
> function. You can create more than a single interpreter, and that even 
> works from the script level using the "interp create" command. This is 
> useful to shield user script evaluation from the main script. Such 
> interpreters can be made "safe", i.e. to not allow file operations, or 
> to limit the CPU time for script evaluation, which has its primary use 
> case in things like web browser plugins or simple servers. A pythonized 
> version of the API would look like this:
> 
> from pyembedding import Interp
> 
> i=Interp()
> i2=Interp(safe=True)
> 
> i.eval('print("Hello world")')
> # prints Hello world
> 
> i2.eval('print("Hello world")')
> # raises a SafeInterpException
> # I/O  is unsafe and disabled here
> 
> def greetme():
> 	print("Hello world")
> 
> i2.alias(greet=greetme)
> i2.eval('greet')
> # calls back to greetme in main interpreter
> 
> This mechanism would also allow true multithreading. If you run another 
> interpreter in the second thread, it will have it's own GIL.
> 
> I'm not familiar with the internals of other Python implementation, but 
> I can imagine that maybe in Jython such an extension could be written.
> 
> 	Christian

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web