Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!newsreader4.netcologne.de!news.netcologne.de!xlned.com!feeder3.xlned.com!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.000 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'python.': 0.02; 'scripts': 0.03; 'programmer': 0.03; 'string.': 0.05; '(using': 0.07; 'binary': 0.07; 'context': 0.07; 'sys': 0.07; 'string': 0.09; 'attributes': 0.09; 'badly': 0.09; 'builtin': 0.09; 'exec': 0.09; 'lookup': 0.09; 'newly': 0.09; 'obj': 0.09; 'pep': 0.09; 'snippet': 0.09; 'subject:files': 0.09; 'subject:modules': 0.09; 'to)': 0.09; 'used.': 0.09; 'yeah,': 0.09; 'python': 0.11; 'def': 0.12; 'assume': 0.14; 'question.': 0.14; "(i'm": 0.16; '04:01': 0.16; 'bit.': 0.16; 'c/c++': 0.16; 'cleaner': 0.16; 'easier.': 0.16; 'executed.': 0.16; 'filename:fname piece:signature': 0.16; 'finder': 0.16; 'first:': 0.16; 'in-memory': 0.16; 'machinery.': 0.16; 'object)': 0.16; 'object).': 0.16; 'received:192.168.1.20': 0.16; 'situation.': 0.16; 'there?': 0.16; 'url:ch': 0.16; 'url:file': 0.16; 'url:peps': 0.16; 'pushed': 0.16; 'skip:= 10': 0.16; 'appropriate': 0.16; 'wrote:': 0.18; 'code.': 0.18; 'module': 0.19; 'implementing': 0.19; 'properly': 0.19; 'stefan': 0.19; '>>>': 0.22; 'code,': 0.22; 'memory': 0.22; 'import': 0.22; '(in': 0.22; 'load': 0.23; 'header:User-Agent:1': 0.23; 'byte': 0.24; 'script.': 0.24; 'simpler': 0.24; 'url:dev': 0.24; 'looks': 0.24; '(or': 0.24; "i've": 0.25; 'source': 0.25; 'script': 0.25; 'extension': 0.26; 'skip:v 30': 0.26; 'this:': 0.26; 'least': 0.26; 'skip:" 20': 0.27; 'gets': 0.27; 'header:In-Reply-To:1': 0.27; 'point': 0.28; 'function': 0.29; 'fixed': 0.29; 'rest': 0.29; 'skip:p 30': 0.29; "doesn't": 0.30; '???': 0.30; 'said,': 0.30; 'especially': 0.30; "i'm": 0.30; 'work.': 0.31; 'code': 0.31; 'getting': 0.31; 'url:wiki': 0.31; '>>>>': 0.31; 'loading': 0.31; 'skip:= 40': 0.31; 'though.': 0.31; 'class': 0.32; 'stuff': 0.32; 'supposed': 0.32; 'interface': 0.32; 'regular': 0.32; 'run': 0.32; 'another': 0.32; 'url:python': 0.33; 'running': 0.33; 'totally': 0.33; 'subject:from': 0.34; 'could': 0.34; 'problem': 0.35; "can't": 0.35; 'classes': 0.35; 'created': 0.35; 'anybody': 0.35; 'no,': 0.35; 'but': 0.35; 'add': 0.35; 'there': 0.35; 'version': 0.36; 'c++': 0.36; 'module.': 0.36; 'doing': 0.36; 'entry': 0.36; 'method': 0.36; 'url:org': 0.36; 'should': 0.36; 'integration': 0.37; 'too': 0.37; 'sincerely': 0.63; 'skip:n 10': 0.64; 'map': 0.64; 'provide': 0.64; 'more': 0.64; 'finally': 0.65; 'land': 0.65; 'within': 0.65; 'between': 0.67; 'url:4': 0.69; 'evaluate': 0.72; 'export': 0.74; 'special': 0.74; 'doable': 0.84; 'imp': 0.84; 'injecting': 0.84; 'url:cpython': 0.84; 'years:': 0.84; 'url:php': 0.85; 'yours': 0.88; 'involved.': 0.91; 'step.': 0.91 Date: Sun, 18 May 2014 21:41:03 +0200 From: =?UTF-8?B?Um9sYW5kIFBsw7xzcw==?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.4.0 MIME-Version: 1.0 To: python-list@python.org Subject: Re: Loading modules from files through C++ References: <5376ACE7.8030706@rptd.ch> <53775D68.2010009@rptd.ch> <537768FB.7060303@rptd.ch> <53778005.8030105@rptd.ch> <53778E22.3040701@rptd.ch> In-Reply-To: X-Enigmail-Version: 1.6 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="s8VCnxhERdI8ncB5V5l4a8T59PFdsd0ae" X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 468 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1400442078 news.xs4all.nl 2975 [2001:888:2000:d::a6]:50738 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:71737 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --s8VCnxhERdI8ncB5V5l4a8T59PFdsd0ae Content-Type: multipart/alternative; boundary="------------050905080907060904060306" This is a multi-part message in MIME format. --------------050905080907060904060306 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 05/17/2014 07:05 PM, Stefan Behnel wrote: > Roland Pl=C3=BCss, 17.05.2014 18:28: >> On 05/17/2014 05:49 PM, Stefan Behnel wrote: >>> Roland Pl=C3=BCss, 17.05.2014 17:28: >>>> On 05/17/2014 04:01 PM, Stefan Behnel wrote: >>>>> Roland Pl=C3=BCss, 17.05.2014 15:49: >>>>>> On 05/17/2014 03:26 PM, Stefan Behnel wrote: >>>>>>> Roland Pl=C3=BCss, 17.05.2014 15:00: >>>>>>>> On 05/17/2014 01:58 PM, Stefan Behnel wrote: >>>>>>>>> Roland Pl=C3=BCss, 17.05.2014 02:27: >>>>>>>>>> I'm using Python in an embedded situation. In particular I hav= e to load >>>>>>>>>> python scripts through a memory interface so regular python mo= dule >>>>>>>>>> loading can not be used. I got working so far a module loader = object >>>>>>>>>> I've added using C++ to sys.meta_path . Now I'm totally stuck = at the >>>>>>>>>> finally loading step. >>>>>>>>>> >>>>>>>>>> I've got this a C++ loader method "load_module(fullname)" whic= h does >>>>>>>>>> load the requested module script files into a null-terminated = string. I >>>>>>>>>> know that "load_module" has to return the module PyObject*. Bu= t I can't >>>>>>>>>> get the python source in the c-string into a module PyObject*.= >>>>>>>>>> [...] >>>>>>>>>> Can anybody help how in gods name one is supposed to create a = module >>>>>>>>>> from an in-memory c-string when called from within load_module= (or >>>>>>>>>> anywhere)? >>>>>>>>> Looks like you want to implement a SourceLoader: >>>>>>>>> >>>>>>>>> https://docs.python.org/3.4/library/importlib.html#importlib.ab= c.SourceLoader >>>>>>>>> >>>>>>>>> I recommend implementing this in Python code instead of C code,= though. >>>>>>>>> Much easier. Cython can help with the integration between both.= >>>>>>>> That doesn't work in 2.x, doesn't it? >>>>>>> Is there a reason you have to use Py2? >>>>>>> >>>>>>> Anyway, PEP 302 predates Py3 by a couple of years: >>>>>>> >>>>>>> http://legacy.python.org/dev/peps/pep-0302/ >>>>>> I'm willing to go to Py3 but only if the solution to the problem i= s >>>>>> simpler than getting it fixed in Py2. So some questions first: >>>>>> >>>>>> - does this importlib stuff you showed there apply to C++ land (I = need >>>>>> to fully drive it from C++ not Python code)? >>>>> As I said, implementing this in Python code is much simpler than do= ing it >>>>> in C/C++ code. Basically, stop where you got the C string and do th= e rest >>>>> in Python. All your C code has to do is to take a module lookup req= uest >>>>> from your custom Python module Finder and return a byte string with= the >>>>> code. Then let your Python code wrap that in a Loader and return it= to the >>>>> import machinery. >>>> I don't get how this is supposed to work. I'm running it as fully >>>> embedded Python. There is no main script. The builtin modules are ad= ded >>>> as C++ bound classes and a user made main script is loaded but not r= un >>>> directly (I'm hooking into a create object). For this purpose I load= the >>>> script module using C++ code using PyImport_ImportModule(moduleName)= =2E At >>>> this time the module loading code has to kick in already (I've added= >>>> this one by C++ too before). The problem is now that in this call I = end >>>> up in my C++ loader version where there is no Python script involved= =2E I >>>> came to the conclusion that I can solve this only by having the C++ = end >>>> properly load the module. I could add Python code with >>>> PyRun_SimpleString but then I'm down to the same problem as before: = how >>>> to evaluate code so it is attached to a module or type-class? As I >>>> understand it the problem is the same as before just pushed around a= bit. >>> No, just run some Python code (using PyRun_SimpleString() if you have= to) >>> and let it do whatever you like. Such as, defining a Finder class and= >>> injecting it into the import hook. Just provide it with the entry poi= nt of >>> your C++ loader as a (CFunction) object when you execute it, and then= let >>> it call that function at need whenever the Finder gets executed. >>> >>> Alternatively, compile your Python integration code with Cython and l= ink it >>> into your main program as yet another binary extension module. >> I'm not using Cython so that's out of question. Concerning the injecti= on >> how would this work? From the PEP I assume it would have to look like = this: >> >> # CODE # >> import sys >> class VFSModuleLoader: >> def find_module(fullname, path=3DNone): >> return self if VFS.exists( vfsPathFromFullname(fullname) ) else = None >> def load_module(fullname): >> sourceCode =3D VFS.read( vfsPathFromFullname(fullname)) >> ??? >> # CODE # >> >> How does ??? work? If I use "eval" I end up with the code inside >> VFSModuleLoader.load_modules as context but it should go into the glob= al >> context as an own module. > Yeah, well, the import machinery is rather badly exposed in Py2. This g= ot > much cleaner in Py3, especially 3.3/3.4. > > Here's a hacky way to do it in Py2: > > import imp > module =3D imp.new_module(module_name) > module.__importer__ =3D self > exec source_code in module.__dict__ > return module > > Found here: > > http://hg.python.org/cpython/file/568041fd8090/Lib/imputil.py#l284 > > You may have to set a couple of more special attributes on the module > (__name__? __path__? __package__? others?), but the above should at lea= st work. > > Stefan > > This exec source_code in module.__dict__ , should this not also be doable with PyEval_EvalCode? I've found a snippet like this: # CODE #| PyCodeObject*code =3D(PyCodeObject*)Py_CompileString(s,"test",Py_file_inp= ut); PyObject*main_module =3DPyImport_AddModule("__main__"); PyObject*global_dict =3DPyModule_GetDict(main_module); PyObject*local_dict =3DPyDict_New(); PyObject*obj =3DPyEval_EvalCode(code,global_dict,local_dict);| # CODE # If I would use a newly created module instead of PyImport_AddModule with the appropriate name would this not evaluate the code (in my case a string not a code object) in the right code space? Or did I misunderstand the snippet there? --=20 Yours sincerely Pl=C3=BCss Roland Leader and Head Programmer - Game: Epsylon ( http://www.indiedb.com/games/epsylon ) - Game Engine: Drag[en]gine ( http://www.indiedb.com/engines/dragengine , http://dragengine.rptd.ch/wiki ) - Normal Map Generator: DENormGen ( http://epsylon.rptd.ch/denormgen.php = ) - As well as various Blender export scripts und game tools --------------050905080907060904060306 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
On 05/17/2014 07:05 PM, Stefan Behnel wrote:
Roland Pl=C3=BCss, 17.05.2014 18:28:

On 05/17/2014 05:49 PM, Stefan Behnel wrote:

Roland Pl=C3=BCss, 17.05.2014 17:28:

On 05/17/2014 04:01 PM, Stefan Behnel wrote:

Roland Pl=C3=BCss, 17.05.2014 15:49:

On 05/17/2014 03:26 PM, Stefan Behnel wrot=
e:

Roland Pl=C3=BCss, 17.05.2014 15:00:

On 05/17/2014 01:58 PM, Stefan Behnel =
wrote:

Roland Pl=C3=BCss, 17.05.2014 02:27:=


I'm using Python in an embedded si=
tuation. In particular I have to load
python scripts through a memory interface so regular python module
loading can not be used. I got working so far a module loader object
I've added using C++ to sys.meta_path . Now I'm totally stuck at the
finally loading step.

I've got this a C++ loader method "load_module(fullname)" which does
load the requested module script files into a null-terminated string. I
know that "load_module" has to return the module PyObject*. But I can't
get the python source in the c-string into a module PyObject*.
[...]
Can anybody help how in gods name one is supposed to create a module
from an in-memory c-string when called from within load_module (or
anywhere)?

Looks like you want to implement a S=
ourceLoader:

https://docs.python.org/=
3.4/library/importlib.html#importlib.abc.SourceLoader

I recommend implementing this in Python code instead of C code, though.
Much easier. Cython can help with the integration between both.

That doesn't work in 2.x, doesn't it?

Is there a reason you have to use Py2?

Anyway, PEP 302 predates Py3 by a couple of years:

http://legacy.python.org/dev/peps/pep-0302/

I'm willing to go to Py3 but only if the s=
olution to the problem is
simpler than getting it fixed in Py2. So some questions first:

- does this importlib stuff you showed there apply to C++ land (I need
to fully drive it from C++ not Python code)?

As I said, implementing this in Python code =
is much simpler than doing it
in C/C++ code. Basically, stop where you got the C string and do the rest=

in Python. All your C code has to do is to take a module lookup request
from your custom Python module Finder and return a byte string with the
code. Then let your Python code wrap that in a Loader and return it to th=
e
import machinery.

I don't get how this is supposed to work. I'm =
running it as fully
embedded Python. There is no main script. The builtin modules are added
as C++ bound classes and a user made main script is loaded but not run
directly (I'm hooking into a create object). For this purpose I load the
script module using C++ code using PyImport_ImportModule(moduleName). At
this time the module loading code has to kick in already (I've added
this one by C++ too before). The problem is now that in this call I end
up in my C++ loader version where there is no Python script involved. I
came to the conclusion that I can solve this only by having the C++ end
properly load the module. I could add Python code with
PyRun_SimpleString but then I'm down to the same problem as before: how
to evaluate code so it is attached to a module or type-class? As I
understand it the problem is the same as before just pushed around a bit.=


No, just run some Python code (using PyRun_Simpl=
eString() if you have to)
and let it do whatever you like. Such as, defining a Finder class and
injecting it into the import hook. Just provide it with the entry point o=
f
your C++ loader as a (CFunction) object when you execute it, and then let=

it call that function at need whenever the Finder gets executed.

Alternatively, compile your Python integration code with Cython and link =
it
into your main program as yet another binary extension module.

I'm not using Cython so that's out of question. Concerning the injection
how would this work? From the PEP I assume it would have to look like thi=
s:

# CODE #
import sys
class VFSModuleLoader:
   def find_module(fullname, path=3DNone):
      return self if VFS.exists( vfsPathFromFullname(fullname) ) else Non=
e
   def load_module(fullname):
      sourceCode =3D VFS.read( vfsPathFromFullname(fullname))
      ???
# CODE #

How does ??? work? If I use "eval" I end up with the code inside
VFSModuleLoader.load_modules as context but it should go into the global
context as an own module.

Yeah, well, the import machinery is rather badly exposed in Py2. This got=

much cleaner in Py3, especially 3.3/3.4.

Here's a hacky way to do it in Py2:

    import imp
    module =3D imp.new_module(module_name)
    module.__importer__ =3D self
    exec source_code in module.__dict__
    return module

Found here:

http://hg.python.org/cpython/file/5=
68041fd8090/Lib/imputil.py#l284

You may have to set a couple of more special attributes on the module
(__name__? __path__? __package__? others?), but the above should at least=
 work.

Stefan
This exec source_code in module.__dict__ , should this not also be doable with PyEval_EvalCode? I've found a snippet like this:

# CODE #
PyCodeObject* code =3D (PyCodeObject*) Py_CompileString(= s, "test", Py_file_input);
PyObject* main_module =3D<= span class=3D"pln"> PyImport_AddModule("__main__");
PyObject* global_dict =3D<= span class=3D"pln"> PyModule_GetDict= (main_module);
PyObject* local_dict =3D PyDict_New();
PyObject* obj =3D PyEval_EvalCode<= span class=3D"pun">(code, global_dict, local_dict);
# CODE #

If I would use a newly created module instead of PyImport_AddModule with the appropriate name would this not evaluate the code (in my case a string not a code object) in the right code space? Or did I misunderstand the snippet there?

--
Yours sincerely
Pl=C3=BCss Roland

Leader and Head Programmer
- Game: Epsylon ( http://www.indiedb.com/games/epsylon = )
- Game Engine: Drag[en]gine ( http://www.indiedb.com/engines/dragengine , http://dragengine.rptd.ch/wiki )
- Normal Map Generator: DENormGen ( http://epsylon.rptd.ch/denormgen.php )
- As well as various Blender export scripts und game tools
--------------050905080907060904060306-- --s8VCnxhERdI8ncB5V5l4a8T59PFdsd0ae Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlN5DN8ACgkQjKGu9ysRWfq+PQCeI6euBxTePoT15Wu1EYJ7kf02 pXoAnjr/lyLoeqzXPPw4nEW8+Q7g10oX =gqKt -----END PGP SIGNATURE----- --s8VCnxhERdI8ncB5V5l4a8T59PFdsd0ae--