Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #14404 > unrolled thread

PYTHONPATH with cmake build

Back to article view | Back to linux.debian.maint.python

Contents

  PYTHONPATH with cmake build PICCA Frederic-Emmanuel <frederic-emmanuel.picca@synchrotron-soleil.fr> - 2022-08-30 16:30 +0200
    Re: PYTHONPATH with cmake build Andrey Rahmatullin <wrar@debian.org> - 2022-08-30 16:40 +0200
      Re: PYTHONPATH with cmake build PICCA Frederic-Emmanuel <frederic-emmanuel.picca@synchrotron-soleil.fr> - 2022-08-30 18:00 +0200
        Re: PYTHONPATH with cmake build Andrey Rahmatullin <wrar@debian.org> - 2022-08-30 18:00 +0200
          Re: PYTHONPATH with cmake build PICCA Frederic-Emmanuel <frederic-emmanuel.picca@synchrotron-soleil.fr> - 2022-08-30 18:30 +0200
            Re: PYTHONPATH with cmake build Andrey Rahmatullin <wrar@debian.org> - 2022-08-30 19:00 +0200
              Re: PYTHONPATH with cmake build PICCA Frederic-Emmanuel <frederic-emmanuel.picca@synchrotron-soleil.fr> - 2022-08-30 19:20 +0200
                Re: PYTHONPATH with cmake build Neil Williams <codehelp@debian.org> - 2022-08-30 20:10 +0200
              Re: PYTHONPATH with cmake build PICCA Frederic-Emmanuel <frederic-emmanuel.picca@synchrotron-soleil.fr> - 2022-08-30 19:30 +0200
            Re: PYTHONPATH with cmake build Stefano Rivera <stefanor@debian.org> - 2022-08-31 00:20 +0200
              Re: PYTHONPATH with cmake build PICCA Frederic-Emmanuel <frederic-emmanuel.picca@synchrotron-soleil.fr> - 2022-08-31 11:30 +0200
                Re: PYTHONPATH with cmake build stefanor@debian.org - 2022-08-31 11:40 +0200

#14404 — PYTHONPATH with cmake build

Hello, I am packaging a python extension which use cmake as build system

here the repo

https://salsa.debian.org/science-team/dxtbx

I try to activate the test but this cause this kind of trouble

During handling of the above exception, another exception occurred:
/usr/lib/python3.10/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/serialize/test_xds.py:8: in <module>
    from dxtbx.imageset import ImageSetFactory
dxtbx/imageset.py:5: in <module>
    import dxtbx.format.image  # noqa: F401, import dependency for unpickling
dxtbx/format/image.py:8: in <module>
    from dxtbx_format_image_ext import *  # type: ignore # isort:skip # noqa: F403
E   ModuleNotFoundError: No module named 'dxtbx_format_image_ext'

Indeed the extension are build here.

cd /<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/src/dxtbx && /usr/bin/cmake -E cmake_link_script CMakeFiles/dxtbx_flumpy.dir/link.txt --verbose=1
/usr/bin/c++ -fPIC -g -O2 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -flto -Wl,-z,relro -shared  -o ../../lib/dxtbx_flumpy.cpython-310-x86_64-linux-gnu.so CMakeFiles/dxtbx_flumpy.dir/boost_python/flumpy.cc.o  /usr/lib/x86_64-linux-gnu/libboost_python310.so.1.74.0 
lto-wrapper: warning: using serial compilation of 4 LTRANS jobs
lto-wrapper: note: see the ‘-flto’ option documentation for more information
[ 90%] Linking CXX shared module ../../lib/dxtbx_format_image_ext.so

so I need to add the /<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib directory to the PYTHONPATH.

do you know how to do this ?

thanks for your help

Fred

[toc] | [next] | [standalone]

#14405

[Multipart message — attachments visible in raw view] — view raw

On Tue, Aug 30, 2022 at 03:58:26PM +0200, PICCA Frederic-Emmanuel wrote:
> Hello, I am packaging a python extension which use cmake as build system
> 
> here the repo
> 
> https://salsa.debian.org/science-team/dxtbx
> 
> I try to activate the test but this cause this kind of trouble
> 
> During handling of the above exception, another exception occurred:
> /usr/lib/python3.10/importlib/__init__.py:126: in import_module
>     return _bootstrap._gcd_import(name[level:], package, level)
> tests/serialize/test_xds.py:8: in <module>
>     from dxtbx.imageset import ImageSetFactory
> dxtbx/imageset.py:5: in <module>
>     import dxtbx.format.image  # noqa: F401, import dependency for unpickling
> dxtbx/format/image.py:8: in <module>
>     from dxtbx_format_image_ext import *  # type: ignore # isort:skip # noqa: F403
> E   ModuleNotFoundError: No module named 'dxtbx_format_image_ext'
> 
> Indeed the extension are build here.
> 
> cd /<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/src/dxtbx && /usr/bin/cmake -E cmake_link_script CMakeFiles/dxtbx_flumpy.dir/link.txt --verbose=1
> /usr/bin/c++ -fPIC -g -O2 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -flto -Wl,-z,relro -shared  -o ../../lib/dxtbx_flumpy.cpython-310-x86_64-linux-gnu.so CMakeFiles/dxtbx_flumpy.dir/boost_python/flumpy.cc.o  /usr/lib/x86_64-linux-gnu/libboost_python310.so.1.74.0 
> lto-wrapper: warning: using serial compilation of 4 LTRANS jobs
> lto-wrapper: note: see the ‘-flto’ option documentation for more information
> [ 90%] Linking CXX shared module ../../lib/dxtbx_format_image_ext.so
> 
> so I need to add the /<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/lib directory to the PYTHONPATH.
> 
> do you know how to do this ?
Does the same happen when you run the test in the source tree manually?

-- 
WBR, wRAR

[toc] | [prev] | [next] | [standalone]

#14406

Hello Andrey

> Does the same happen when you run the test in the source tree manually?

I do not know, I am in the process to build the package in sbuild so I just try to fix the build process.
If I have hard code the lib path  the test are running failling for other missing modules, but this is ok.

so my question is more how can I obtain the cmake default build path from the rules file ?

cheers

[toc] | [prev] | [next] | [standalone]

#14407

[Multipart message — attachments visible in raw view] — view raw

On Tue, Aug 30, 2022 at 05:26:01PM +0200, PICCA Frederic-Emmanuel wrote:
> Hello Andrey
> 
> > Does the same happen when you run the test in the source tree manually?
> 
> I do not know
When trying to run tests you should look how does the upstream intend to
run them.


-- 
WBR, wRAR

[toc] | [prev] | [next] | [standalone]

#14408

> When trying to run tests you should look how does the upstream intend to
run them.


Yes they are building inplace the module like this

from https://github.com/dials/dxtbx/blob/main/.azure-pipelines/unix-build.yml


# Build dxtbx
- bash: |
    set -e
    . conda_base/bin/activate
    set -ux
    mkdir build
    cd build
    cmake ../modules/dxtbx
    cmake --build . --target install
    pip install ../modules/dxtbx
  displayName: Build dxtbx
  workingDirectory: $(Pipeline.Workspace)

then

# Finally, run the full regression test suite
- bash: |
    set -e
    . conda_base/bin/activate
    set -ux
    export DIALS_DATA=${PWD}/data
    cd modules/dxtbx
    export PYTHONDEVMODE=1
    pytest -v -ra -n auto --basetemp="$(Pipeline.Workspace)/tests" --durations=10 \
        --cov=dxtbx --cov-report=html --cov-report=xml --cov-branch \
        --timeout=5400 --regression || echo "##vso[task.complete result=Failed;]Some tests failed"
  displayName: Run tests
  workingDirectory: $(Pipeline.Workspace)


-- 
WBR, wRAR

[toc] | [prev] | [next] | [standalone]

#14409

[Multipart message — attachments visible in raw view] — view raw

On Tue, Aug 30, 2022 at 06:02:01PM +0200, PICCA Frederic-Emmanuel wrote:
> > When trying to run tests you should look how does the upstream intend to
> run them.
> 
> 
> Yes they are building inplace the module like this
> 
> from https://github.com/dials/dxtbx/blob/main/.azure-pipelines/unix-build.yml
> 
> 
> # Build dxtbx
> - bash: |
>     set -e
>     . conda_base/bin/activate
>     set -ux
>     mkdir build
>     cd build
>     cmake ../modules/dxtbx
>     cmake --build . --target install
>     pip install ../modules/dxtbx
Oh. In this case setting PYTHONPATH (if it works, and I'm not 100% sure it
will) sounds like a better option.
So, the cmake build path is, as far as I know, defined in
/usr/share/perl5/Debian/Debhelper/Buildsystem.pm to be just
"obj-$DEB_HOST_GNU_TYPE".


-- 
WBR, wRAR

[toc] | [prev] | [next] | [standalone]

#14410

> /usr/share/perl5/Debian/Debhelper/Buildsystem.pm to be just
> "obj-$DEB_HOST_GNU_TYPE".


Thanks for the info, if I an not wrong during the build process we can setup a new builddir.
So is is possible to obtain the real builddir during the build ?

[toc] | [prev] | [next] | [standalone]

#14413

[Multipart message — attachments visible in raw view] — view raw

On Tue, 30 Aug 2022 19:02:24 +0200 (CEST)
PICCA Frederic-Emmanuel <frederic-emmanuel.picca@synchrotron-soleil.fr>
wrote:

> > /usr/share/perl5/Debian/Debhelper/Buildsystem.pm to be just
> > &#034;obj-$DEB_HOST_GNU_TYPE&#034;.  
> 
> 
> Thanks for the info, if I an not wrong during the build process we
> can setup a new builddir. So is is possible to obtain the real
> builddir during the build ?


PYTHONPATH can be a relative path, there is no need to try to identify
the full build path (that can introduce reproducibility issues). No
need to force a change to the build dir either.

So just obj-$DEB_HOST_GNU_TYPE

-- 
Neil Williams
linux@codehelp.co.uk
https://linux.codehelp.co.uk

[toc] | [prev] | [next] | [standalone]

#14411

> Oh. In this case setting PYTHONPATH (if it works, and I'm not 100% sure it
> will) sounds like a better option.
> So, the cmake build path is, as far as I know, defined in
> /usr/share/perl5/Debian/Debhelper/Buildsystem.pm to be just
> obj-$DEB_HOST_GNU_TYPE".

thanks for the info. It seems to me that the builddir can be set directly in the ruels file with --builddir=<something>
So do you know a way to extract the real builddir during the build process ?

Cheers

[toc] | [prev] | [next] | [standalone]

#14414

Hi PICCA (2022.08.30_16:02:01_+0000)
>     cd build
>     cmake ../modules/dxtbx
>     cmake --build . --target install
>     pip install ../modules/dxtbx

This package essentially has 2 build systems, camke for the C++ stuff
and setuptools for the pure-python. I don't see any integration between
the two (which is quite annoying, the upstream really should fix that).

pybuild knows how to drive distutils, and knows how to drive cmake, but
not both together. Ideally, you do want to use pybuild when building
Python extensions, so you can do them for all supported Python versions.

So, can I suggest something like this?

#! /usr/bin/make -f

export PYBUILD_NAME=dxtbx
export PYBUILD_SYSTEM=distutils
export PYBUILD_AFTER_CONFIGURE=cmake -DPython_EXECUTABLE=/usr/bin/{interpreter} -S . -B {build_dir}/lib
export PYBUILD_AFTER_BUILD=make -C {build_dir}/lib
export PYBUILD_BEFORE_TEST=cp {build_dir}/lib/lib/*.so {build_dir}

%:
	dh $@ --with python3 --buildsystem=pybuild

You'll need something for the install too, but I didn't get that far, as
the tests fail (due to missing dependencies in Debian).

SR

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  +1 415 683 3272

[toc] | [prev] | [next] | [standalone]

#14416

Hello Stephano

I end up with this

#! /usr/bin/make -f 

export DH_VERSOBE=1

export PYBUILD_NAME=dxtbx 
export PYBUILD_SYSTEM=distutils
export PYBUILD_AFTER_CONFIGURE=cmake -DPython_EXECUTABLE=/usr/bin/{interpreter} -S . -B {build_dir}/lib
export PYBUILD_AFTER_BUILD=make -C {build_dir}/lib
export PYBUILD_BEFORE_TEST=cp {build_dir}/lib/lib/*.so {build_dir}
export PYBUILD_BEFORE_INSTALL=cp {build_dir}/lib/lib/*.so {build_dir}; rm -rf {build_dir}/lib

%:
        dh $@ --with python3 --buildsystem=pybuild

override_dh_auto_test:                  

it seem to work but I am am not really confident with the 'rm -rf {build_dir}/lib' during the install
If I do not remove these files, they got installed by pybuild.

no more magical obj-$blabla path :)


I now can concentrate on the unit test failures...

Cheers

Fred

[toc] | [prev] | [next] | [standalone]

#14417

Hi PICCA (2022.08.31_09:05:43_+0000)
> export PYBUILD_AFTER_BUILD=make -C {build_dir}/lib

I didn't do anything to support parallel, there. It may be nice to
(makes a huge difference to this package's build time).

> export PYBUILD_BEFORE_TEST=cp {build_dir}/lib/lib/*.so {build_dir}
> export PYBUILD_BEFORE_INSTALL=cp {build_dir}/lib/lib/*.so {build_dir}; rm -rf {build_dir}/lib
> it seem to work but I am am not really confident with the 'rm -rf {build_dir}/lib' during the install

I'd suggest an && instead of ; of course :)

And maybe logically, that should all happen in AFTER_BUILD...

> If I do not remove these files, they got installed by pybuild.

What's actually happening there is that they're getting installed by
setuptools. If we could install them into the right spot with cmake,
maybe we could reduce the copying and deleting. I didn't experiment with
that very much, just got to the point it worked...

SR

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  +1 415 683 3272

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.maint.python

csiph-web