Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #108974 > unrolled thread

Interfacing a dynamic shared library gives me different results in 2.7 versus 3.5

Back to article view | Back to comp.lang.python

Contents

  Interfacing a dynamic shared library gives me different results in 2.7 versus 3.5 Siyi Deng <mr.siyi.deng@gmail.com> - 2016-05-22 20:15 -0700
    Re: Interfacing a dynamic shared library gives me different results in 2.7 versus 3.5 Fabien <fabien.maussion@gmail.com> - 2016-05-23 11:27 +0200
    Re: Interfacing a dynamic shared library gives me different results in 2.7 versus 3.5 Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2016-05-23 20:29 +1000
    Re: Interfacing a dynamic shared library gives me different results in 2.7 versus 3.5 Siyi Deng <mr.siyi.deng@gmail.com> - 2016-05-24 00:15 -0700
      Re: Interfacing a dynamic shared library gives me different results in 2.7 versus 3.5 Chris Angelico <rosuav@gmail.com> - 2016-05-24 17:45 +1000
        Re: Interfacing a dynamic shared library gives me different results in 2.7 versus 3.5 Siyi Deng <mr.siyi.deng@gmail.com> - 2016-05-24 09:18 -0700
          Re: Interfacing a dynamic shared library gives me different results in 2.7 versus 3.5 Chris Angelico <rosuav@gmail.com> - 2016-05-25 02:26 +1000
          Re: Interfacing a dynamic shared library gives me different results in 2.7 versus 3.5 Steven D'Aprano <steve@pearwood.info> - 2016-05-25 02:32 +1000
            Re: Interfacing a dynamic shared library gives me different results in 2.7 versus 3.5 Chris Angelico <rosuav@gmail.com> - 2016-05-25 02:45 +1000
            Re: Interfacing a dynamic shared library gives me different results in 2.7 versus 3.5 Siyi Deng <mr.siyi.deng@gmail.com> - 2016-05-24 10:38 -0700
            Re: Interfacing a dynamic shared library gives me different results in 2.7 versus 3.5 Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2016-05-24 20:14 -0400
    Re: Interfacing a dynamic shared library gives me different results in 2.7 versus 3.5 Siyi Deng <mr.siyi.deng@gmail.com> - 2016-05-24 10:47 -0700

#108974 — Interfacing a dynamic shared library gives me different results in 2.7 versus 3.5

I have a dynamic library doing some numerical computations. 

I used ctypes to interact it by passing numpy arrays back and forth. 

Python 3.5 gives me the correct results. 

Python 2.7 gives me different, erroneous results, but it never crashes. 

How is this possible? There is no string operations involved whatsoever.

[toc] | [next] | [standalone]

#108987

On 05/23/2016 05:15 AM, Siyi Deng wrote:
> I have a dynamic library doing some numerical computations.
> I used ctypes to interact it by passing numpy arrays back and forth.
> Python 3.5 gives me the correct results.
> Python 2.7 gives me different, erroneous results, but it never crashes.
> How is this possible? There is no string operations involved whatsoever.

You might be more successful if you describe your problem in more detail 
(maybe with a minimal working example?) and if you ask the scipy/numpy 
people: https://www.scipy.org/scipylib/mailing-lists.html

Cheers

[toc] | [prev] | [next] | [standalone]

#108989

On Monday 23 May 2016 13:15, Siyi Deng wrote:

> I have a dynamic library doing some numerical computations.
> 
> I used ctypes to interact it by passing numpy arrays back and forth.
> 
> Python 3.5 gives me the correct results.
> 
> Python 2.7 gives me different, erroneous results, but it never crashes.
> 
> How is this possible? There is no string operations involved whatsoever.

This is only a guess, because you haven't shown your code or data, but I guess 
it could be related to the change from integer division to true division in 
Python 3.

In Python 2.7, try this at the interactive interpreter. Do you get the same 
results?

>>> import numpy
>>> a =numpy.array([1, 2, 3])
>>> a/3
array([0, 0, 1])


That's because in Python 2, the / operator performs integer division, like C. 
In Python 3, it performs true division.

Put "from __future__ import division" at the top of your script. In Python 2.7, 
it will give you the same behaviour as Python 3, and in Python 3, it will be 
harmless. Note that there are TWO underscores at the front and end of 
__future__:


>>> from __future__ import division
>>> a/3
array([ 0.33333333,  0.66666667,  1.        ])



Please note that __future__ imports must be the FIRST line of code. They can 
follow comments, but must be before any other code.



-- 
Steve

[toc] | [prev] | [next] | [standalone]

#109046

Thanks for all the replies.  

It turned out that the Apple OS X stock python 2.7 gives the wrong results, but other distributions like 2.7 from miniconda gives the correct results. Facepalm.

[toc] | [prev] | [next] | [standalone]

#109048

On Tue, May 24, 2016 at 5:15 PM, Siyi Deng <mr.siyi.deng@gmail.com> wrote:
> Thanks for all the replies.
>
> It turned out that the Apple OS X stock python 2.7 gives the wrong results, but other distributions like 2.7 from miniconda gives the correct results. Facepalm.

When you use a binary shared library, it has to be compiled against
the correct Python. You're messing around with ctypes, so basically
you've voided your warranty; *everything* you're doing is
platform-specific. Have fun. :)

ChrisA

[toc] | [prev] | [next] | [standalone]

#109064

Hello ChrisA,
I don't quite understand, the binary shared library contains no python interfaces, it should be independent of python. As a matter of fact, I have successfully used it in Conda python 2.7, 3.5,  Julialang as well as c++ executables. I think the fact that only stock python 2.7 failed to run correctly indicates some bug in that python distribution. 


> When you use a binary shared library, it has to be compiled against
> the correct Python. You're messing around with ctypes, so basically
> you've voided your warranty; *everything* you're doing is
> platform-specific. Have fun. :)
> 
> ChrisA

[toc] | [prev] | [next] | [standalone]

#109065

On Wed, May 25, 2016 at 2:18 AM, Siyi Deng <mr.siyi.deng@gmail.com> wrote:
> I don't quite understand, the binary shared library contains no python interfaces, it should be independent of python. As a matter of fact, I have successfully used it in Conda python 2.7, 3.5,  Julialang as well as c++ executables. I think the fact that only stock python 2.7 failed to run correctly indicates some bug in that python distribution.
>

Ah okay. Still, somehow you have to get the data from the numpy array
to the binary library; and that depends on the exact layout of the
Python object (including pointer sizes and stuff). So your ctypes
interface code might need to be adjusted.

ChrisA

[toc] | [prev] | [next] | [standalone]

#109066

On Wed, 25 May 2016 02:18 am, Siyi Deng wrote:

> Hello ChrisA,
> I don't quite understand, the binary shared library contains no python
> interfaces, it should be independent of python. 

In your first post, you said you were using numpy. How is that independent
of Python?

> As a matter of fact, I 
> have successfully used it in Conda python 2.7, 3.5,  Julialang as well as
> c++ executables. I think the fact that only stock python 2.7 failed to run
> correctly indicates some bug in that python distribution.

What do you expect us to say? You still won't show us the code you are
using. Do you expect that we will just take your word for it? If so, then
what? We can't fix this bug (if it is a bug) if we can't identify it.




-- 
Steven

[toc] | [prev] | [next] | [standalone]

#109068

On Wed, May 25, 2016 at 2:32 AM, Steven D'Aprano <steve@pearwood.info> wrote:
> On Wed, 25 May 2016 02:18 am, Siyi Deng wrote:
>
>> Hello ChrisA,
>> I don't quite understand, the binary shared library contains no python
>> interfaces, it should be independent of python.
>
> In your first post, you said you were using numpy. How is that independent
> of Python?

I'm not certain of the in-memory representation of a numpy array, but
it wouldn't surprise me if it's just a series of consecutive
ints/floats - which would be exactly what a C or Fortran library will
be expecting. So my understanding is that there's some basic ctypes
wrapper code to locate the base address of the array, and then it
passes that straight along.

Of course, without seeing the code....

ChrisA

[toc] | [prev] | [next] | [standalone]

#109073

The c function has a signature as follows:

int cfun(int len_data, float* data, int* a, int num_a,
        int flag1, int flag2, int flag3, float* param,
        float* out1, float* out2, float* out3)


and in python:

import numpy as np
import ctypes as ct

    data = np.atleast_2d(np.float32(data))
    a = np.atleast_2d(np.int32(a)) if a else np.zeros((2, 1), dtype=np.int32)
    param = np.atleast_2d(np.float32(param))

    num_a = activity.shape[1]
    len_data = data.shape[1]
    num_inc = len_data//256
    cf = ct.POINTER(ct.c_float)
    
hr = np.zeros((2, num_inc), dtype=np.float32)
    of = np.zeros((num_inc, 207), dtype=np.float32)
    gait = np.zeros((num_inc, 14), dtype=np.float32)
    pt_of = of.ctypes.data_as(cf) if do_of else None
    pt_gait = gait.ctypes.data_as(cf) if do_gait else None
    pt_param = param.ctypes.data_as(cf) if param else None
    dl.run_plt_hrm(len_data, data.ctypes.data_as(cf),
            activity.ctypes.data_as(ct.POINTER(ct.c_int)),
            num_act, do_long_fft+0, do_cls_mitigate+0, do_weighted_average+0,
            pt_param, hr.ctypes.data_as(cf), pt_of, pt_gait)

[toc] | [prev] | [next] | [standalone]

#109094

On Wed, 25 May 2016 02:45:30 +1000, Chris Angelico <rosuav@gmail.com>
declaimed the following:

>
>I'm not certain of the in-memory representation of a numpy array, but
>it wouldn't surprise me if it's just a series of consecutive
>ints/floats - which would be exactly what a C or Fortran library will
>be expecting. So my understanding is that there's some basic ctypes
>wrapper code to locate the base address of the array, and then it
>passes that straight along.
>
	That leads to the question: Row-major or Column-major...

	Could have a very significant affect taking a row-major array and
passing it to code using a column-major convention.
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
    wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [next] | [standalone]

#109076

Here is a summary of what I did with numpy and the dll

I have verified that the values entering the last dll call (dl.cfunction) are identical across platforms.


 

The c function has a signature as follows: 

int cfunction(int len_data, float* data, int* ac, int num_ac, 
        int flag1, int flag2, int flag3, float* param, 
        float* out) 


and in python: 

import numpy as np 
import ctypes as ct 

dl = ct.cdll.LoadLibrary('xxx.dylib')

# data, ac, param are loaded from somewhere else.
data = np.atleast_2d(np.float32(data)) 
ac = np.atleast_2d(np.int32(a)) if a else np.zeros((2, 1), dtype=np.int32) 
param = np.atleast_2d(np.float32(param)) 
flag1 = True
flag2 = True
flag3 = True
num_ac = ac.shape[1] 
len_data = data.shape[1] 
num_inc = len_data//200
out = np.zeros((2, num_inc), dtype=np.float32) 
pt_param = param.ctypes.data_as(cf) if param else None

cf = ct.POINTER(ct.c_float) 
dl.cfunction(len_data, data.ctypes.data_as(cf), 
        ac.ctypes.data_as(ct.POINTER(ct.c_int)), 
        num_ac, flag1+0, flag2+0, flag3+0, 
        pt_param, out.ctypes.data_as(cf)) 
        
        

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web