Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #52411 > unrolled thread

Calling Python macro from ctypes

Back to article view | Back to comp.lang.python

Contents

  Calling Python macro from ctypes Steven D'Aprano <steve@pearwood.info> - 2013-08-12 10:01 +0000
    Re: Calling Python macro from ctypes Peter Otten <__peter__@web.de> - 2013-08-12 13:42 +0200
      Re: Calling Python macro from ctypes Steven D'Aprano <steve@pearwood.info> - 2013-08-13 06:25 +0000
        Re: Calling Python macro from ctypes Stefan Behnel <stefan_ml@behnel.de> - 2013-08-13 10:11 +0200
          Re: Calling Python macro from ctypes Steven D'Aprano <steve@pearwood.info> - 2013-08-13 09:13 +0000
        Re: Calling Python macro from ctypes Peter Otten <__peter__@web.de> - 2013-08-13 12:25 +0200
    Re: Calling Python macro from ctypes Dave Angel <davea@davea.name> - 2013-08-12 17:11 +0000

#52411 — Calling Python macro from ctypes

Is it possible to call a Python macro from ctypes? For example, Python 
3.3 introduces some new macros for querying the internal representation 
of strings:

http://www.python.org/dev/peps/pep-0393/#new-api


So I try this in 3.3:

py> import ctypes
py> ctypes.pythonapi.PyUnicode_MAX_CHAR_VALUE
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.3/ctypes/__init__.py", line 366, in 
__getattr__
    func = self.__getitem__(name)
  File "/usr/local/lib/python3.3/ctypes/__init__.py", line 371, in 
__getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: python3.3: undefined symbol: PyUnicode_MAX_CHAR_VALUE





-- 
Steven

[toc] | [next] | [standalone]

#52412

Steven D'Aprano wrote:

> Is it possible to call a Python macro from ctypes? For example, Python
> 3.3 introduces some new macros for querying the internal representation
> of strings:
> 
> http://www.python.org/dev/peps/pep-0393/#new-api
> 
> 
> So I try this in 3.3:
> 
> py> import ctypes
> py> ctypes.pythonapi.PyUnicode_MAX_CHAR_VALUE
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/usr/local/lib/python3.3/ctypes/__init__.py", line 366, in
> __getattr__
>     func = self.__getitem__(name)
>   File "/usr/local/lib/python3.3/ctypes/__init__.py", line 371, in
> __getitem__
>     func = self._FuncPtr((name_or_ordinal, self))
> AttributeError: python3.3: undefined symbol: PyUnicode_MAX_CHAR_VALUE

That's not possible. It may look like a function, but a preprocessor 
replaces the C macro in the C source before compilation. An example of very 
bad usage of macros, just to drive the point home:

$ cat macro.c 
#define IF(expr) if (expr) {
#define ENDIF ;}

main()
{
  IF(1>0)
    printf("It worked\n")
  ENDIF
}

And here's what the compiler sees:

$ gcc -E -P macro.c



main()
{
  if (1>0) {
    printf("It worked\n")
  ;}
}

[toc] | [prev] | [next] | [standalone]

#52440

On Mon, 12 Aug 2013 13:42:14 +0200, Peter Otten wrote:

> Steven D'Aprano wrote:
> 
>> Is it possible to call a Python macro from ctypes? For example, Python
>> 3.3 introduces some new macros for querying the internal representation
>> of strings:
>> 
>> http://www.python.org/dev/peps/pep-0393/#new-api
[...]

> That's not possible. It may look like a function, but a preprocessor
> replaces the C macro in the C source before compilation.

That's what I feared.

In that case, how would I use ctypes to access the underlying fields in 
the new string implementation?



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#52442

Steven D'Aprano, 13.08.2013 08:25:
> On Mon, 12 Aug 2013 13:42:14 +0200, Peter Otten wrote:
>> Steven D'Aprano wrote:
>>
>>> Is it possible to call a Python macro from ctypes? For example, Python
>>> 3.3 introduces some new macros for querying the internal representation
>>> of strings:
>>>
>>> http://www.python.org/dev/peps/pep-0393/#new-api
> [...]
> 
>> That's not possible. It may look like a function, but a preprocessor
>> replaces the C macro in the C source before compilation.
> 
> That's what I feared.
> 
> In that case, how would I use ctypes to access the underlying fields in 
> the new string implementation?

I'd personally use Cython (no surprise here), but in order to use something
like ctypes, which works at the ABI level, not the API level, and doesn't
use a C compiler to get things properly configured for the local platform,
you'd have to manually define the actual PyObject struct in order to access
its fields directly.

The macros are there to give you source code level portability for that,
but if you start defining the struct layout statically, you're pretty much
on your own when it comes to stuff like different CPython versions, debug
builds, etc.

Could you describe your use case a little deeper? Maybe there's a better
way all together to do what you want.

Stefan

[toc] | [prev] | [next] | [standalone]

#52449

On Tue, 13 Aug 2013 10:11:29 +0200, Stefan Behnel wrote:

> Steven D'Aprano, 13.08.2013 08:25:
>> On Mon, 12 Aug 2013 13:42:14 +0200, Peter Otten wrote:
>>> Steven D'Aprano wrote:
>>>
>>>> Is it possible to call a Python macro from ctypes? For example,
>>>> Python 3.3 introduces some new macros for querying the internal
>>>> representation of strings:
>>>>
>>>> http://www.python.org/dev/peps/pep-0393/#new-api
>> [...]
>> 
>>> That's not possible. It may look like a function, but a preprocessor
>>> replaces the C macro in the C source before compilation.
>> 
>> That's what I feared.
>> 
>> In that case, how would I use ctypes to access the underlying fields in
>> the new string implementation?
> 
> I'd personally use Cython (no surprise here), but in order to use
> something like ctypes, which works at the ABI level, not the API level,
> and doesn't use a C compiler to get things properly configured for the
> local platform, you'd have to manually define the actual PyObject struct
> in order to access its fields directly.
> 
> The macros are there to give you source code level portability for that,
> but if you start defining the struct layout statically, you're pretty
> much on your own when it comes to stuff like different CPython versions,
> debug builds, etc.
> 
> Could you describe your use case a little deeper? Maybe there's a better
> way all together to do what you want.

One concrete example of what I want to do is introspect Python 3.3 
strings. E.g. to tell whether they are 1 byte, 2 byte or 4 byte strings. 
Strings have a "kind" field, set to 0-3, that specifies how many bytes 
per char, but it is not exposed to Python.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#52450

Steven D'Aprano wrote:

> On Mon, 12 Aug 2013 13:42:14 +0200, Peter Otten wrote:
> 
>> Steven D'Aprano wrote:
>> 
>>> Is it possible to call a Python macro from ctypes? For example, Python
>>> 3.3 introduces some new macros for querying the internal representation
>>> of strings:
>>> 
>>> http://www.python.org/dev/peps/pep-0393/#new-api
> [...]
> 
>> That's not possible. It may look like a function, but a preprocessor
>> replaces the C macro in the C source before compilation.
> 
> That's what I feared.
> 
> In that case, how would I use ctypes to access the underlying fields in
> the new string implementation?

You'd have to replicate the structure of Python's unicode type with ctypes. 
You could take shortcuts like determining sizeof(PyObject) and then fill in 
bytes for that instead of the actual fields, but it's still tedious.

So while I have no idea why you would want to do this I took your question 
as an occasion to take a glimpse into the C extension tutorial. I then 
adapted the example code to wrap said macro. Here's the result so far 
(bugfixes welcome):

$ cat macrowrapper.c 
#include <Python.h>

static PyObject *
macrowrapper_max_char_value(PyObject *self, PyObject *args)
{
    PyObject *unicode;
    Py_UCS4 maxchar;

    if (!PyArg_ParseTuple(args, "O", &unicode))
        return NULL;
    if (!PyUnicode_Check(unicode)) {
      PyErr_BadArgument();
      return NULL;
    }
    maxchar = PyUnicode_MAX_CHAR_VALUE(unicode);
    return PyLong_FromLong(maxchar);
}

static PyMethodDef MacroWrapperMethods[] = {
    {"max_char_value",  macrowrapper_max_char_value, METH_VARARGS,
     "Wrap PyUnicode_MAX_CHAR_VALUE() macro."},

    {NULL, NULL, 0, NULL}        /* Sentinel */
};

static struct PyModuleDef macrowrapper = {
   PyModuleDef_HEAD_INIT,
   "macrowrapper",   /* name of module */
   NULL, /* module documentation, may be NULL */
   -1,       /* size of per-interpreter state of the module,
                or -1 if the module keeps state in global variables. */
   MacroWrapperMethods
};

PyMODINIT_FUNC
PyInit_macrowrapper(void)
{
    return PyModule_Create(&macrowrapper);
}
$ cat setup.py 
from distutils.core import setup, Extension

module = Extension('macrowrapper',
                    sources = ['macrowrapper.c'])

setup (name = 'MacroWrapper',
       version = '1.0',
       description = 'Wrap the PyUnicode_MAX_CHAR_VALUE macro',
       ext_modules = [module])
$ python3.3 setup.py install --user

[snip]

$ python3.3
Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 30 2012, 14:49:00) 
[GCC 4.6.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from macrowrapper import max_char_value
>>> max_char_value("")
127
>>> max_char_value("Löblich")
255
>>> max_char_value("εὕρηκα")
65535
>>> max_char_value(42)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: bad argument type for built-in operation
>>> max_char_value()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: function takes exactly 1 argument (0 given)

[toc] | [prev] | [next] | [standalone]

#52423

Peter Otten wrote:

> Steven D'Aprano wrote:
>
>> Is it possible to call a Python macro from ctypes? For example, Python
>> 3.3 introduces some new macros for querying the internal representation
>> of strings:
>> 
>> http://www.python.org/dev/peps/pep-0393/#new-api
>> 
>> 
>> So I try this in 3.3:
>> 
>> py> import ctypes
>> py> ctypes.pythonapi.PyUnicode_MAX_CHAR_VALUE
>> Traceback (most recent call last):
>>   File "<stdin>", line 1, in <module>
>>   File "/usr/local/lib/python3.3/ctypes/__init__.py", line 366, in
>> __getattr__
>>     func = self.__getitem__(name)
>>   File "/usr/local/lib/python3.3/ctypes/__init__.py", line 371, in
>> __getitem__
>>     func = self._FuncPtr((name_or_ordinal, self))
>> AttributeError: python3.3: undefined symbol: PyUnicode_MAX_CHAR_VALUE
>
> That's not possible. It may look like a function, but a preprocessor 
> replaces the C macro in the C source before compilation. An example of very 
> bad usage of macros, just to drive the point home:
>
> $ cat macro.c 
> #define IF(expr) if (expr) {
> #define ENDIF ;}
>
> main()
> {
>   IF(1>0)
>     printf("It worked\n")
>   ENDIF
> }
>
> And here's what the compiler sees:
>
> $ gcc -E -P macro.c
>
>
>
> main()
> {
>   if (1>0) {
>     printf("It worked\n")
>   ;}
> }
>

To elaborate a bit more, Python can only see those symbols that are put
into the shared library  They can be functions, and they can be
"values," but they don't include macros, which are processed by the
preprocessor, before the real C compiler even starts.  C Macros are
actually text-substitution rules.  They can look like functions, but
those functions do not end up in the shared library.

In Windows, you can use dumpbin to examine a DLL and see what symbols it
exports.  I don't remember the syntax; it's been years.

I assume there's a similar tool for Linux to examine a shared library
(typically an .so file).  Perhaps "readelf" and/or "nm" is such a tool,
but I don't really know. Although I've been using Python and C++ in
Linux in recent years, I haven't used them together, and neither have
I had to examine a shared library.

The following link looks interesting, but I haven't read it yet.

http://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html

-- 
DaveA

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web