Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #13118 > unrolled thread

Generic Python packages which don’t work on all architectures

Back to article view | Back to linux.debian.maint.python

Contents

  Generic Python packages which don’t work on all  architectures Stephen Kitt <skitt@debian.org> - 2021-01-20 20:50 +0100
    Re: Generic Python packages which don’t work on all architectures Matthias Klose <doko@debian.org> - 2021-01-20 21:10 +0100
    Re: Generic Python packages which don’t work on all architectures Paul Wise <pabs@debian.org> - 2021-01-21 02:50 +0100
      Re: Generic Python packages which don’t work on all architectures Christian Kastner <ckk@debian.org> - 2021-01-21 08:10 +0100
      Re: Generic Python packages which don’t work on all architectures Ole Streicher <olebole@debian.org> - 2021-01-21 14:20 +0100
        Re: Generic Python packages which don’t work on all architectures Paul Wise <pabs@debian.org> - 2021-01-21 14:30 +0100
          Re: Generic Python packages which don’t work on all architectures Ole Streicher <olebole@debian.org> - 2021-01-21 16:50 +0100
            Re: Generic Python packages which don’t work on all architectures Paul Wise <pabs@debian.org> - 2021-01-22 00:40 +0100
              Re: Generic Python packages which don’t work on all architectures Paul Wise <pabs@debian.org> - 2021-01-22 00:50 +0100
              Re: Generic Python packages which don’t work on all architectures Ole Streicher <olebole@debian.org> - 2021-01-22 10:00 +0100
        Re: Generic Python packages which don’t work on  all architectures Ansgar <ansgar@43-1.org> - 2021-01-21 14:40 +0100
          Re: Generic Python packages which don’t work on all architectures Ole Streicher <olebole@debian.org> - 2021-01-21 16:20 +0100

#13118 — Generic Python packages which don’t work on all architectures

[Multipart message — attachments visible in raw view] — view raw

Hi,

I’ve come across a situation which doesn’t seem to be addressed by existing
policies: the python-ptrace source package only ships
architecture-independent content, but it works on a small number of
architectures (currently, 32/64-bit x86, 32-bit ARM, 32/64-bit PPC).

As a result, the packages it builds are installable everywhere, but that’s a
false promise since they don’t work everywhere. Would it make sense to
convert it to an architecture-dependent package, only on those architectures
which are really supported?

(In this specific instance, there’s an optional CPython extension which would
be interesting, so I’d switch to an architecture-dependent package for this
anyway, were I to maintain the package; but I thought this was an interesting
question in general.)

Regards,

Stephen

[toc] | [next] | [standalone]

#13119 — Re: Generic Python packages which don’t work on all architectures

On 1/20/21 8:39 PM, Stephen Kitt wrote:
> Hi,
> 
> I’ve come across a situation which doesn’t seem to be addressed by existing
> policies: the python-ptrace source package only ships
> architecture-independent content, but it works on a small number of
> architectures (currently, 32/64-bit x86, 32-bit ARM, 32/64-bit PPC).
> 
> As a result, the packages it builds are installable everywhere, but that’s a
> false promise since they don’t work everywhere. Would it make sense to
> convert it to an architecture-dependent package, only on those architectures
> which are really supported?

There's no python packaging specific approach.  So yes, either make it
architecture dependent, or maybe add code to the entry points (__init__.py) to
raise an ImportError on unsupported architectures?

Matthias

[toc] | [prev] | [next] | [standalone]

#13120 — Re: Generic Python packages which don’t work on all architectures

On Wed, Jan 20, 2021 at 7:40 PM Stephen Kitt wrote:

> I’ve come across a situation which doesn’t seem to be addressed by existing
> policies: the python-ptrace source package only ships
> architecture-independent content, but it works on a small number of
> architectures (currently, 32/64-bit x86, 32-bit ARM, 32/64-bit PPC).

The iotop package is in a similar situation, it uses Linux kernel APIs
so does not work on Hurd or kFreeBSD.

> As a result, the packages it builds are installable everywhere, but that’s a
> false promise since they don’t work everywhere. Would it make sense to
> convert it to an architecture-dependent package, only on those architectures
> which are really supported?

This is what I eventually chose for iotop. At the time I wanted dpkg
and dak to support something like Architecture: linux-all, which would
build arch: all packages, but only put them in the Packages files for
the Linux architectures.

I am now thinking that a more generic solution than Architecture:
linux-all is needed, in order to cover your case as well. Perhaps
something like Available-Architecures or Runtime-Architectures or
Architecture-all-Architectures: or similar.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

[toc] | [prev] | [next] | [standalone]

#13121 — Re: Generic Python packages which don’t work on all architectures

On 21.01.21 02:44, Paul Wise wrote:
> I am now thinking that a more generic solution than Architecture:
> linux-all is needed, in order to cover your case as well. Perhaps
> something like Available-Architecures or Runtime-Architectures or
> Architecture-all-Architectures: or similar.

To be honest, that sounds like a lot of complicated extra infrastructure
work compared to just making the package architecture-dependent.

Yes, it will save a few builds, but it's not like these particular
builds consume a lot of resources.

Best,
Christian

[toc] | [prev] | [next] | [standalone]

#13123 — Re: Generic Python packages which don’t work on all architectures

Paul Wise <pabs@debian.org> writes:
> This is what I eventually chose for iotop. At the time I wanted dpkg
> and dak to support something like Architecture: linux-all, which would
> build arch: all packages, but only put them in the Packages files for
> the Linux architectures.
>
> I am now thinking that a more generic solution than Architecture:
> linux-all is needed, in order to cover your case as well. Perhaps
> something like Available-Architecures or Runtime-Architectures or
> Architecture-all-Architectures: or similar.

I am still wondering why we don't have just empty some pseudo-packages
that are available only on specific architectures (or groups of them,
like linux, or little endian, or 64 bit or so). If a package works (or
builds) only on a specific arch (or an arch with specific properties),
it can be set dependent (or built-dependent) on that.

This would be a quite flexible and extendible approach to have packages
installable only where they work.

Best regards

Ole

[toc] | [prev] | [next] | [standalone]

#13124 — Re: Generic Python packages which don’t work on all architectures

On Thu, Jan 21, 2021 at 1:12 PM Ole Streicher wrote:

> This would be a quite flexible and extendible approach to have packages
> installable only where they work.

There is precedent for this sort of thing in the isa-support source
package, which fails installation when your CPU doesn't support
particular ISA extensions like SSE or NEON.

It is a reasonable workaround for now, but the packages using the
pseudo-packages would still be available, causing confusion when
people try to install them. It would be better to just not have them
available where they aren't going to work. I also wonder how
multi-arch stuff would interact with the pseudo-packages idea.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

[toc] | [prev] | [next] | [standalone]

#13128 — Re: Generic Python packages which don’t work on all architectures

Paul Wise <pabs@debian.org> writes:
> On Thu, Jan 21, 2021 at 1:12 PM Ole Streicher wrote:
>
>> This would be a quite flexible and extendible approach to have packages
>> installable only where they work.
>
> There is precedent for this sort of thing in the isa-support source
> package, which fails installation when your CPU doesn't support
> particular ISA extensions like SSE or NEON.
>
> It is a reasonable workaround for now, but the packages using the
> pseudo-packages would still be available, causing confusion when
> people try to install them.

But this is also the case for all packages which implicitly depend on
other packages which are not available on some architectures. This is
not very rare:

 * Python arch:all packages may depend on arch:* Python packages which
   are not available everywhere

 * Packages written in an interpreted language are arch:all, but depend
   on the interpreter and can only installed on archs where the
   interpreter exists (like gnudatalanguage or iraf from my portfolio)

Generally, arch:all packages depend on a lot of architecture dependent
packages, and unless one resolves the whole dependency chain, there is
no way to check whether a package is actually installable.

> It would be better to just not have them available where they aren't
> going to work. I also wonder how multi-arch stuff would interact with
> the pseudo-packages idea.

IMO this is a problem that already exists there and is not
solved. Imagine the following constellation:

 * I have a python3-foo package with arch:all.
 * this depends on python3-foo-obj that exists only for i386.

Now I have a x86_64 machine that gets i386 added. On this machine, I
could install python3-foo (since the dependency is resolvable), but
'import foo' with the x86_64 interpreter would fail.

Best regards

Ole

[toc] | [prev] | [next] | [standalone]

#13133 — Re: Generic Python packages which don’t work on all architectures

On Thu, Jan 21, 2021 at 3:42 PM Ole Streicher wrote:

> But this is also the case for all packages which implicitly depend on
> other packages which are not available on some architectures.

This is true, but it doesn't make for a good user experience.

> Generally, arch:all packages depend on a lot of architecture dependent
> packages, and unless one resolves the whole dependency chain, there is
> no way to check whether a package is actually installable.

The buildds have installability checking for build-deps (IIRC using
dose), so some sort of QA service could be doing these sort of checks
more cheaply than installing arch:all packages outside of amd64. Also
unfortunately piuparts.d.o doesn't yet support multiple arches, but
there is probably some coverage of arch:all packages on debci, not
sure if it tests them on multiple arches though.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

[toc] | [prev] | [next] | [standalone]

#13134 — Re: Generic Python packages which don’t work on all architectures

On Thu, Jan 21, 2021 at 11:30 PM Paul Wise wrote:

> there is probably some coverage of arch:all packages on debci, not
> sure if it tests them on multiple arches though.

FTR, #debci says arch:all packages get tested on all debci arches.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

[toc] | [prev] | [next] | [standalone]

#13137 — Re: Generic Python packages which don’t work on all architectures

Paul Wise <pabs@debian.org> writes:
> On Thu, Jan 21, 2021 at 3:42 PM Ole Streicher wrote:
>
>> But this is also the case for all packages which implicitly depend on
>> other packages which are not available on some architectures.
>
> This is true, but it doesn't make for a good user experience.

Yes, but this is a separate problem, and this IMO cannot be reliably
solved by adding a simple arch-specific declaration to the source
package: If we would do (require) f.e. an additional "installs-on" field
in d/control, then each maintainer would be requested to monitor all
dependencies on all archs to find out where the package could be
installable, and to re-upload the package with adjustments whenever one
of the dependencies changes the list of supported architectures.

IMO, a useful solution would be to parse the dependency chain on the
client (aptitude, synaptic or whatever) and to list only those packages
that can be installed.

If we would introduce arch specific marker packages, we wouldn't solve
this ofcourse, but we would solve the problem that you can install a
package that is actually not working on the architecture, which is even
a worse user experience. And we would solve the problem that one can't
specify f.e. all little-endian systems, or all 64-bit systems as build
dependency, except for explicitly listing all of them (which is IMO a
nightmare for ports or for derivatives that have additional
architectures).

It is not perfect, but a pragmatic step forward.

Best

Ole

[toc] | [prev] | [next] | [standalone]

#13125 — Re: Generic Python packages which don’t work on all architectures

On Thu, 2021-01-21 at 13:46 +0100, Ole Streicher wrote:
> I am still wondering why we don't have just empty some pseudo-
packages that are available only on specific architectures
> (or  groups of them, like linux, or little endian, or 64 bit or so).

To solve which problem?

Packages being installable don't mean that they can do anything useful
as they might, for example, require special hardware.  We don't have a
way to express "requires device ${X}" on a package level.

Ansgar

[toc] | [prev] | [next] | [standalone]

#13127 — Re: Generic Python packages which don’t work on all architectures

Ansgar <ansgar@43-1.org> writes:
> On Thu, 2021-01-21 at 13:46 +0100, Ole Streicher wrote:
>> I am still wondering why we don't have just empty some pseudo-
> packages that are available only on specific architectures
>> (or  groups of them, like linux, or little endian, or 64 bit or so).
>
> To solve which problem?

For example, the problem that a certain package is working properly only
on big endian systems. Or on Linux.

Currently, there is (as discussed here) no way to tell that an arch:all
package is not working on a 32-bit system. And for architecture
dependent builds, one needs to specify every single architecture where
it is supposed to build. How do you currently otherwise would specify
"all little endian systems" as build dependency?

> Packages being installable don't mean that they can do anything useful
> as they might, for example, require special hardware.  We don't have a
> way to express "requires device ${X}" on a package level.

Special hardware is another topic, which is surely not solved here (and
not solvable at all in general, since hardware today may be
hot-pluggable, and the hardware configuration at install time does not
need to be the configuration at run time).

However, the proposal to have architecture (group) specific empty
packages would already solve some of the problems if this is limited to
architecture specification.

Best regards

Ole

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.maint.python

csiph-web