Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #12407 > unrolled thread

Re: Python3 modules not built for all supported Python versions

Back to article view | Back to linux.debian.maint.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Python3 modules not built for all supported Python versions Johannes Schauer <josch@debian.org> - 2020-03-30 15:40 +0200
    Re: Python3 modules not built for all supported Python versions Simon McVittie <smcv@debian.org> - 2020-03-30 16:10 +0200
      Re: Python3 modules not built for all supported Python versions Piotr Ożarowski <piotr@debian.org> - 2020-03-30 21:00 +0200
      Re: Python3 modules not built for all supported Python versions Jochen Sprickerhof <jspricke@debian.org> - 2020-04-06 15:40 +0200

#12407 — Re: Python3 modules not built for all supported Python versions

[Multipart message — attachments visible in raw view] — view raw

Hi,

Quoting Emilio Pozuelo Monfort (2020-03-30 13:24:01)
> We've just finished the transition to python3.8 as the default python3
> interpreter, which was a bit difficult due to some autopkgtest regressions in
> a few rdeps, and to the fact that many modules only build their extensions
> for the default python version, which means they have a strict dependency on
> the python3 version[1] and they need to be rebuilt and migrated in lockstep
> with python3-defaults.
> 
> I have analyzed this based on current sid amd64 contents and have come up with
> the following packages that don't ship extensions for both py3.7 and 3.8 (which
> are the currently supported versions). Note that pure python packages that don't
> build C extensions are not affected.
> 
> It would be great if this situation can be improved in order to help with future
> python transitions. Building for all the supported python versions can be done
> by build-depending on python3-all-dev and compiling your package (or just the
> python bits) with PYTHON pointing to each version. Depending on your package's
> build system, this could be largely automated using some helper, such as
> pybuild. If you don't know how to add support for your package, feel free to ask.

does this mean that build-depending on python3-dev is wrong in general and
should instead be replaced by build-depending on python3-all-dev?

Could wrong build dependencies or binary packages which have a strict
dependency on a python3 version not easily be detected by lintian?

I'm not an expert in Python module packaging -- it just so happens that some
packages I maintain offer Python bindings. Do you have a diff of a source
package that successfully made the transition so that I know what I would have
to change? For example the package src:ros-geometry2 has a super simple
dh-style rules file, basically just doing:

%:
	dh $@ --buildsystem=cmake --with python3

What would I have to change to successfully fix this problem?

Thanks!

cheers, josch

[toc] | [next] | [standalone]

#12408

On Mon, 30 Mar 2020 at 15:30:01 +0200, Johannes Schauer wrote:
> does this mean that build-depending on python3-dev is wrong in general and
> should instead be replaced by build-depending on python3-all-dev?

It is only wrong for packages that build Python 3 extensions (binary
modules) that are intended to be loadable by all supported Python
3 versions (roughly: `find /usr/lib/python3/dist-packages -name '*.so'`).

For packages that embed Python 3, like the versions of vim that
have Python scripting support, or packages that use a Python 3
extension as an internal implementation detail of some tool, like
gobject-introspection, my understanding is that build-depending
on python3-dev continues to be appropriate. These extensions would
ideally be installed in a private directory, like gobject-introspection's
/usr/lib/x86_64-linux-gnu/gobject-introspection/giscanner/_giscanner.cpython-38-x86_64-linux-gnu.so
- but I know some upstreams and some downstream maintainers (arguably
incorrectly) package private extensions as though they were public
extensions, because the mechanics of doing so are much simpler.

> For example the package src:ros-geometry2 has a super simple
> dh-style rules file, basically just doing:
> 
> %:
> 	dh $@ --buildsystem=cmake --with python3
> 
> What would I have to change to successfully fix this problem?

The general answer is that you would have to build it repeatedly in a
loop, with each supported version of Python 3 in turn. I am not aware
of a way to do this in a similarly simple rules file.

    smcv

[toc] | [prev] | [next] | [standalone]

#12409

> I don't know if I'm missing an argument to dh_python3 so that it knows the
> python version, or even if there's a better workaround. But perhaps pybuild

better workaround is to force cmake to install into versioned
dist-packages (that's what I do in distutils) as I didn't find a
reliable way to read Python version/architecture/tag/etc. from .so file

> should be doing this automatically between the dh_auto_install calls so that
> this kind of workarounds aren't necessary.

it's actually a nice idea - pybuild can check if
/usr/lib/python3.X/dist-packages/ is empty and there are .so files in
/usr/lib/python3/dist-packages/ and move them temporarily to
/usr/lib/python3.X/dist-packages/ (it knows which interpreter is used at
this point) and let dh_python3 rename them later

> Piotr, is this a bug in pybuild, or am I doing something wrong?

well, cmake is not really easy to work with, they provide
-DPYTHON_LIBRARY:FILEPATH… but they don't use it or use completely
different name… it basically is different for each library I checked
(and in different official documentation versions I read, sic!)

[toc] | [prev] | [next] | [standalone]

#12431

[Multipart message — attachments visible in raw view] — view raw

Hi Emilio,

* Emilio Pozuelo Monfort <pochu@debian.org> [2020-03-30 17:47]:
>I've heard pybuild now has a cmake backend, so theoretically you could do
>something like
>
>%:
>	dh $@ --buildsystem=pybuild --system=cmake
[..]
>I don't know if I'm missing an argument to dh_python3 so that it knows the
>python version, or even if there's a better workaround. But perhaps pybuild
>should be doing this automatically between the dh_auto_install calls so that
>this kind of workarounds aren't necessary.

Thanks for looking into this and providing a patch!
I polished it a little:

https://salsa.debian.org/science-team/ros-geometry2/-/commit/11739091abe58007485772492fddbdc1a12a59c6

and uploaded a working version to the archive. Will fix all ros-* 
packages now.

Cheers Jochen

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.maint.python

csiph-web