Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder4.news.weretis.net!rt.uk.eu.org!newsfeed.xs4all.nl!newsfeed2a.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Date: Sat, 17 May 2014 18:28:18 +0200
From: =?UTF-8?B?Um9sYW5kIFBsw7xzcw==?= <roland@rptd.ch>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.4.0
MIME-Version: 1.0
To: python-list@python.org
Subject: Re: Loading modules from files through C++
References: <5376ACE7.8030706@rptd.ch> <ll7iu1$oge$1@ger.gmane.org> <53775D68.2010009@rptd.ch> <ll7o30$pc3$1@ger.gmane.org> <537768FB.7060303@rptd.ch> <ll7q3m$idv$1@ger.gmane.org> <53778005.8030105@rptd.ch> <ll80dl$bov$1@ger.gmane.org>
In-Reply-To: <ll80dl$bov$1@ger.gmane.org>
Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="hWdCxrmV6WchMcTTFq1F60UIQwgEvjAOs"
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.10092.1400344098.18130.python-list@python.org>
Lines: 153
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:71698

This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--hWdCxrmV6WchMcTTFq1F60UIQwgEvjAOs
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable


On 05/17/2014 05:49 PM, Stefan Behnel wrote:
> Roland Pl=C3=BCss, 17.05.2014 17:28:
>> On 05/17/2014 04:01 PM, Stefan Behnel wrote:
>>> Roland Pl=C3=BCss, 17.05.2014 15:49:
>>>> On 05/17/2014 03:26 PM, Stefan Behnel wrote:
>>>>> Roland Pl=C3=BCss, 17.05.2014 15:00:
>>>>>> On 05/17/2014 01:58 PM, Stefan Behnel wrote:
>>>>>>> Roland Pl=C3=BCss, 17.05.2014 02:27:
>>>>>>>> I'm using Python in an embedded situation. In particular I have =
to load
>>>>>>>> python scripts through a memory interface so regular python modu=
le
>>>>>>>> loading can not be used. I got working so far a module loader ob=
ject
>>>>>>>> I've added using C++ to sys.meta_path . Now I'm totally stuck at=
 the
>>>>>>>> finally loading step.
>>>>>>>>
>>>>>>>> I've got this a C++ loader method "load_module(fullname)" which =
does
>>>>>>>> load the requested module script files into a null-terminated st=
ring. I
>>>>>>>> know that "load_module" has to return the module PyObject*. But =
I can't
>>>>>>>> get the python source in the c-string into a module PyObject*.
>>>>>>>> [...]
>>>>>>>> Can anybody help how in gods name one is supposed to create a mo=
dule
>>>>>>>> from an in-memory c-string when called from within load_module (=
or
>>>>>>>> anywhere)?
>>>>>>> Looks like you want to implement a SourceLoader:
>>>>>>>
>>>>>>> https://docs.python.org/3.4/library/importlib.html#importlib.abc.=
SourceLoader
>>>>>>>
>>>>>>> I recommend implementing this in Python code instead of C code, t=
hough.
>>>>>>> Much easier. Cython can help with the integration between both.
>>>>>> That doesn't work in 2.x, doesn't it?
>>>>> Is there a reason you have to use Py2?
>>>>>
>>>>> Anyway, PEP 302 predates Py3 by a couple of years:
>>>>>
>>>>> http://legacy.python.org/dev/peps/pep-0302/
>>>> I'm willing to go to Py3 but only if the solution to the problem is
>>>> simpler than getting it fixed in Py2. So some questions first:
>>>>
>>>> - does this importlib stuff you showed there apply to C++ land (I ne=
ed
>>>> to fully drive it from C++ not Python code)?
>>> As I said, implementing this in Python code is much simpler than doin=
g it
>>> in C/C++ code. Basically, stop where you got the C string and do the =
rest
>>> in Python. All your C code has to do is to take a module lookup reque=
st
>>> from your custom Python module Finder and return a byte string with t=
he
>>> code. Then let your Python code wrap that in a Loader and return it t=
o the
>>> import machinery.
>> I don't get how this is supposed to work. I'm running it as fully
>> embedded Python. There is no main script. The builtin modules are adde=
d
>> as C++ bound classes and a user made main script is loaded but not run=

>> directly (I'm hooking into a create object). For this purpose I load t=
he
>> script module using C++ code using PyImport_ImportModule(moduleName). =
At
>> this time the module loading code has to kick in already (I've added
>> this one by C++ too before). The problem is now that in this call I en=
d
>> up in my C++ loader version where there is no Python script involved. =
I
>> came to the conclusion that I can solve this only by having the C++ en=
d
>> properly load the module. I could add Python code with
>> PyRun_SimpleString but then I'm down to the same problem as before: ho=
w
>> to evaluate code so it is attached to a module or type-class? As I
>> understand it the problem is the same as before just pushed around a b=
it.
> No, just run some Python code (using PyRun_SimpleString() if you have t=
o)
> and let it do whatever you like. Such as, defining a Finder class and
> injecting it into the import hook. Just provide it with the entry point=
 of
> your C++ loader as a (CFunction) object when you execute it, and then l=
et
> it call that function at need whenever the Finder gets executed.
>
> Alternatively, compile your Python integration code with Cython and lin=
k it
> into your main program as yet another binary extension module.
>
> Stefan
>
>
I'm not using Cython so that's out of question. Concerning the injection
how would this work? From the PEP I assume it would have to look like thi=
s:

# CODE #
import sys
class VFSModuleLoader:
   def find_module(fullname, path=3DNone):
      return self if VFS.exists( vfsPathFromFullname(fullname) ) else Non=
e
   def load_module(fullname):
      sourceCode =3D VFS.read( vfsPathFromFullname(fullname))
      ???
# CODE #

How does ??? work? If I use "eval" I end up with the code inside
VFSModuleLoader.load_modules as context but it should go into the global
context as an own module.

--=20
Yours sincerely
Pl=C3=BCss Roland

Leader and Head Programmer
- Game: Epsylon ( http://www.indiedb.com/games/epsylon )
- Game Engine: Drag[en]gine ( http://www.indiedb.com/engines/dragengine
, http://dragengine.rptd.ch/wiki )
- Normal Map Generator: DENormGen ( http://epsylon.rptd.ch/denormgen.php =
)
- As well as various Blender export scripts und game tools


--hWdCxrmV6WchMcTTFq1F60UIQwgEvjAOs
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlN3jikACgkQjKGu9ysRWfqIbQCfdjdb8FcI0+u3qXjCYCmqaHgD
z0sAniKeruqgfLp9X71BHuhiyIPtzS+a
=k4k0
-----END PGP SIGNATURE-----

--hWdCxrmV6WchMcTTFq1F60UIQwgEvjAOs--