Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #12466 > unrolled thread

PEP-517/PEP-518 Support In Debian

Back to article view | Back to linux.debian.maint.python

Contents

  PEP-517/PEP-518 Support In Debian Scott Kitterman <debian@kitterman.com> - 2020-04-13 00:40 +0200
    Re: PEP-517/PEP-518 Support In Debian Paul Wise <pabs@debian.org> - 2020-04-13 05:50 +0200
      Re: PEP-517/PEP-518 Support In Debian Scott Kitterman <debian@kitterman.com> - 2020-04-13 06:00 +0200
    Re: PEP-517/PEP-518 Support In Debian Dmitry Shachnev <mitya57@debian.org> - 2020-04-13 11:20 +0200
      Re: PEP-517/PEP-518 Support In Debian Scott Kitterman <debian@kitterman.com> - 2020-04-13 14:10 +0200
        Re: PEP-517/PEP-518 Support In Debian Nicholas D Steeves <nsteeves@gmail.com> - 2020-06-05 01:50 +0200
          Re: PEP-517/PEP-518 Support In Debian Scott Kitterman <debian@kitterman.com> - 2020-06-05 02:00 +0200
            Re: PEP-517/PEP-518 Support In Debian Nicholas D Steeves <nsteeves@gmail.com> - 2020-06-05 02:50 +0200

#12466 — PEP-517/PEP-518 Support In Debian

[Multipart message — attachments visible in raw view] — view raw

This being roughly the mid-point in the development cycle, I thought it might 
be good to see where we are in terms of future Python packaging developments.

As I understand it, PEP-517 and PEP-518 are 'the future'.

I took a look at the presence of pyproject.toml files in the Debian archive.  
There are currently 99 packages.  Of those, only 28 specify a 'build-backend', 
which is required by 517/518 to be useful for building a package.

25 specify: build-backend = "setuptools.build_meta" (including setuptools)
3 specify: build-backend = "flit_core.buildapi" (including flit)

If build-backend is not specified, the build system has to fall back to 
setup.py.

I've recently package flit (just arrived in Testing) and written a flit plugin 
for pybuild that's pending review for merging[1].  I also packaged pep517 for 
our pip package, so that's available to support future Debian tool development 
in this area.

For the moment, I guess we are in reasonable shape, but it might be useful to 
have a pybuild plugin to use PEP517/setuptools.build_meta in lieu of setup.py 
with setuptools/distutils when available.  In the future, this will be the 
primary API and the sooner we start to use it, the better.

Scott K

[1] https://salsa.debian.org/python-team/tools/dh-python/-/merge_requests/12

[toc] | [next] | [standalone]

#12469

On Sun, Apr 12, 2020 at 10:32 PM Scott Kitterman wrote:

> I took a look at the presence of pyproject.toml files in the Debian archive.
> There are currently 99 packages.  Of those, only 28 specify a 'build-backend',
> which is required by 517/518 to be useful for building a package.

Is there a tool that can report problems in pyproject.toml files?
Either way I think that lintian needs to detect this particular issue.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

[toc] | [prev] | [next] | [standalone]

#12470

On April 13, 2020 3:44:31 AM UTC, Paul Wise <pabs@debian.org> wrote:
>On Sun, Apr 12, 2020 at 10:32 PM Scott Kitterman wrote:
>
>> I took a look at the presence of pyproject.toml files in the Debian
>archive.
>> There are currently 99 packages.  Of those, only 28 specify a
>'build-backend',
>> which is required by 517/518 to be useful for building a package.
>
>Is there a tool that can report problems in pyproject.toml files?
>Either way I think that lintian needs to detect this particular issue.

Not that I know of, but I don't think it's time yet.

They're non-build uses for pyproject.toml, so if build-backend isn't present, it just means that it isn't used for build reasons.

The pep517 package (python3-pep517) can validate metadata generated based on pyproject.toml, but it spins up it's own environment using pip and does a package sdist build to do so.  I don't think that's what you have in mind.

In the pybuild plugin I did for flit, I do check for the correct build-backend when autodetecting if the plugin should be used.  Using python3-toml it's a one liner.  If we really want to head down rlthe path of checking, it ought to be simple enough to write.

Scott K

[toc] | [prev] | [next] | [standalone]

#12471

[Multipart message — attachments visible in raw view] — view raw

Hi Scott!

On Sun, Apr 12, 2020 at 06:31:57PM -0400, Scott Kitterman wrote:
> This being roughly the mid-point in the development cycle, I thought it might 
> be good to see where we are in terms of future Python packaging developments.
>
> As I understand it, PEP-517 and PEP-518 are 'the future'.
>
> I took a look at the presence of pyproject.toml files in the Debian archive.  
> There are currently 99 packages.  Of those, only 28 specify a 'build-backend', 
> which is required by 517/518 to be useful for building a package.
>
> 25 specify: build-backend = "setuptools.build_meta" (including setuptools)
> 3 specify: build-backend = "flit_core.buildapi" (including flit)

pyqt5 and pyqt5webengine specify: build-backend = "sipbuild.api".

> If build-backend is not specified, the build system has to fall back to 
> setup.py.
>
> I've recently package flit (just arrived in Testing) and written a flit plugin 
> for pybuild that's pending review for merging[1].  I also packaged pep517 for 
> our pip package, so that's available to support future Debian tool development 
> in this area.
>
> For the moment, I guess we are in reasonable shape, but it might be useful to 
> have a pybuild plugin to use PEP517/setuptools.build_meta in lieu of setup.py 
> with setuptools/distutils when available.  In the future, this will be the 
> primary API and the sooner we start to use it, the better.

One issue that comes to mind: how will we specify the install location in a
way that will work with any backend? In other words, what is the replacement
for distutils' --install-layout=deb?

--
Dmitry Shachnev

[toc] | [prev] | [next] | [standalone]

#12474

[Multipart message — attachments visible in raw view] — view raw

On Monday, April 13, 2020 5:18:53 AM EDT Dmitry Shachnev wrote:
> Hi Scott!
> 
> On Sun, Apr 12, 2020 at 06:31:57PM -0400, Scott Kitterman wrote:
> > This being roughly the mid-point in the development cycle, I thought it
> > might be good to see where we are in terms of future Python packaging
> > developments.
> > 
> > As I understand it, PEP-517 and PEP-518 are 'the future'.
> > 
> > I took a look at the presence of pyproject.toml files in the Debian
> > archive. There are currently 99 packages.  Of those, only 28 specify a
> > 'build-backend', which is required by 517/518 to be useful for building a
> > package.
> > 
> > 25 specify: build-backend = "setuptools.build_meta" (including setuptools)
> > 3 specify: build-backend = "flit_core.buildapi" (including flit)
> 
> pyqt5 and pyqt5webengine specify: build-backend = "sipbuild.api".

So they do.  They didn't show up in my codesearch.d.n results, that makes me 
wonder what else I missed.  If anyone has an idea of a better way to track 
this, please speak up.

> > If build-backend is not specified, the build system has to fall back to
> > setup.py.
> > 
> > I've recently package flit (just arrived in Testing) and written a flit
> > plugin for pybuild that's pending review for merging[1].  I also packaged
> > pep517 for our pip package, so that's available to support future Debian
> > tool development in this area.

P1otr merged the flit plugin a little while ago, so it'll be in the next dh-
python upload.

> > For the moment, I guess we are in reasonable shape, but it might be useful
> > to have a pybuild plugin to use PEP517/setuptools.build_meta in lieu of
> > setup.py with setuptools/distutils when available.  In the future, this
> > will be the primary API and the sooner we start to use it, the better.
> 
> One issue that comes to mind: how will we specify the install location in a
> way that will work with any backend? In other words, what is the replacement
> for distutils' --install-layout=deb?

I think the answer to that question is going to be build-backend specific.

The good news is that the legacy processing for setup.py with setuptools/
distutils will be around approximately forever, so a complete answer to the 
question isn't urgent.

One related question is if we are willing to bring pip into our package build 
process?  I took a quick look at python3-pep517 to see how hard it would be to 
add support for build-backend = "setuptools.build_meta"  to pybuild.  My quick 
look answer is not too hard if we are willing to bring pip into the process. 
If we aren't willing to involve pip (which does bring a lot of complexity), I 
think there will be substantially more Debian specific code required to support 
it.

I expect build-backend = "sipbuild.api" will need custom support as well, but 
I haven't looked into it at all.

Scott K

[toc] | [prev] | [next] | [standalone]

#12624

[Multipart message — attachments visible in raw view] — view raw

Hi,

Scott Kitterman <debian@kitterman.com> writes:

> On Monday, April 13, 2020 5:18:53 AM EDT Dmitry Shachnev wrote:
>> Hi Scott!
>> 
>> On Sun, Apr 12, 2020 at 06:31:57PM -0400, Scott Kitterman wrote:
>> > This being roughly the mid-point in the development cycle, I thought it
>> > might be good to see where we are in terms of future Python packaging
>> > developments.
>> > 
>> > As I understand it, PEP-517 and PEP-518 are 'the future'.
>> >

While importing the latest release of Fissix (a grammar parser used by
Bowler) I found that it had a pyproject.toml but no setup.py.

>> > I took a look at the presence of pyproject.toml files in the Debian
>> > archive. There are currently 99 packages.  Of those, only 28 specify a
>> > 'build-backend', which is required by 517/518 to be useful for building a
>> > package.
>> > 
>> > 25 specify: build-backend = "setuptools.build_meta" (including setuptools)
>> > 3 specify: build-backend = "flit_core.buildapi" (including flit)
>> 
>> pyqt5 and pyqt5webengine specify: build-backend = "sipbuild.api".
>

Is there existing documentation around if/how build-backend can be used
to work around the absence of setup.py?

> So they do.  They didn't show up in my codesearch.d.n results, that makes me 
> wonder what else I missed.  If anyone has an idea of a better way to track 
> this, please speak up.
>
>> > If build-backend is not specified, the build system has to fall back to
>> > setup.py.
>> > 
>> > I've recently package flit (just arrived in Testing) and written a flit
>> > plugin for pybuild that's pending review for merging[1].  I also packaged
>> > pep517 for our pip package, so that's available to support future Debian
>> > tool development in this area.
>
> P1otr merged the flit plugin a little while ago, so it'll be in the next dh-
> python upload.
>

I think this is probably what is needed to unblock work on fissix (and
thus Bowler), because its Makefile has:

    python -m pip install -r requirements.txt
    python -m pip install -r requirements-dev.txt
    python -m flit install --symlink

[snip]

Any advise for existing workarounds would be appreciated.  So far the
only idea I've had is writing or generating a setup.py, and then adding
it as a quilt patch.

Thanks,
Nicholas

[toc] | [prev] | [next] | [standalone]

#12625

[Multipart message — attachments visible in raw view] — view raw

On Thursday, June 4, 2020 7:39:59 PM EDT Nicholas D Steeves wrote:
> Hi,
> 
> Scott Kitterman <debian@kitterman.com> writes:
> > On Monday, April 13, 2020 5:18:53 AM EDT Dmitry Shachnev wrote:
> >> Hi Scott!
> >> 
> >> On Sun, Apr 12, 2020 at 06:31:57PM -0400, Scott Kitterman wrote:
> >> > This being roughly the mid-point in the development cycle, I thought it
> >> > might be good to see where we are in terms of future Python packaging
> >> > developments.
> >> > 
> >> > As I understand it, PEP-517 and PEP-518 are 'the future'.
> 
> While importing the latest release of Fissix (a grammar parser used by
> Bowler) I found that it had a pyproject.toml but no setup.py.
> 
> >> > I took a look at the presence of pyproject.toml files in the Debian
> >> > archive. There are currently 99 packages.  Of those, only 28 specify a
> >> > 'build-backend', which is required by 517/518 to be useful for building
> >> > a
> >> > package.
> >> > 
> >> > 25 specify: build-backend = "setuptools.build_meta" (including
> >> > setuptools)
> >> > 3 specify: build-backend = "flit_core.buildapi" (including flit)
> >> 
> >> pyqt5 and pyqt5webengine specify: build-backend = "sipbuild.api".
> 
> Is there existing documentation around if/how build-backend can be used
> to work around the absence of setup.py?
> 
> > So they do.  They didn't show up in my codesearch.d.n results, that makes
> > me wonder what else I missed.  If anyone has an idea of a better way to
> > track this, please speak up.
> > 
> >> > If build-backend is not specified, the build system has to fall back to
> >> > setup.py.
> >> > 
> >> > I've recently package flit (just arrived in Testing) and written a flit
> >> > plugin for pybuild that's pending review for merging[1].  I also
> >> > packaged
> >> > pep517 for our pip package, so that's available to support future
> >> > Debian
> >> > tool development in this area.
> > 
> > P1otr merged the flit plugin a little while ago, so it'll be in the next
> > dh- python upload.
> 
> I think this is probably what is needed to unblock work on fissix (and
> thus Bowler), because its Makefile has:
> 
>     python -m pip install -r requirements.txt
>     python -m pip install -r requirements-dev.txt
>     python -m flit install --symlink
> 
> [snip]
> 
> Any advise for existing workarounds would be appreciated.  So far the
> only idea I've had is writing or generating a setup.py, and then adding
> it as a quilt patch.

That is probably the best approach for now.

Ultimately we have two choices:

1.  Add per-backend capability into pybuild as I have done with flit.  
Depending on how many of them there are, this may or may not be a reasonable 
approach.

2.  Use the pep517 module with pip in our package build system.  I think 
Fedora has gone this direction.

I prefer #1 because we want to produce a traditional bdist, not wheels, but 
we'll have to do the work.  I don't think we want an impenetrable morass of 
zip files in /usr/lib/python3/dist-packages.

Scott K 

[toc] | [prev] | [next] | [standalone]

#12628

[Multipart message — attachments visible in raw view] — view raw

Hi Scott,

Thank you for the quick reply!

Scott Kitterman <debian@kitterman.com> writes:

> On Thursday, June 4, 2020 7:39:59 PM EDT Nicholas D Steeves wrote:
>> Scott Kitterman <debian@kitterman.com> writes:
>> > On Monday, April 13, 2020 5:18:53 AM EDT Dmitry Shachnev wrote:
>> >> Hi Scott!
>> >> 
>> >> On Sun, Apr 12, 2020 at 06:31:57PM -0400, Scott Kitterman wrote:
>> >> > This being roughly the mid-point in the development cycle, I thought it
>> >> > might be good to see where we are in terms of future Python packaging
>> >> > developments.
>> >> > 
>> >> > As I understand it, PEP-517 and PEP-518 are 'the future'.
>> 
>> While importing the latest release of Fissix (a grammar parser used by
>> Bowler) I found that it had a pyproject.toml but no setup.py.
>> 
>> >> > I took a look at the presence of pyproject.toml files in the Debian
>> >> > archive. There are currently 99 packages.  Of those, only 28 specify a
>> >> > 'build-backend', which is required by 517/518 to be useful for building
>> >> > a
>> >> > package.
>> >> > 
>> >> > 25 specify: build-backend = "setuptools.build_meta" (including
>> >> > setuptools)
>> >> > 3 specify: build-backend = "flit_core.buildapi" (including flit)
>> >> 
>> >> pyqt5 and pyqt5webengine specify: build-backend = "sipbuild.api".
>> 
>> Is there existing documentation around if/how build-backend can be used
>> to work around the absence of setup.py?
>> 
>> > So they do.  They didn't show up in my codesearch.d.n results, that makes
>> > me wonder what else I missed.  If anyone has an idea of a better way to
>> > track this, please speak up.
>> > 
>> >> > If build-backend is not specified, the build system has to fall back to
>> >> > setup.py.
>> >> > 
>> >> > I've recently package flit (just arrived in Testing) and written a flit
>> >> > plugin for pybuild that's pending review for merging[1].  I also
>> >> > packaged
>> >> > pep517 for our pip package, so that's available to support future
>> >> > Debian
>> >> > tool development in this area.
>> > 
>> > P1otr merged the flit plugin a little while ago, so it'll be in the next
>> > dh- python upload.
>> 
>> I think this is probably what is needed to unblock work on fissix (and
>> thus Bowler), because its Makefile has:
>> 
>>     python -m pip install -r requirements.txt
>>     python -m pip install -r requirements-dev.txt
>>     python -m flit install --symlink
>> 
>> [snip]
>> 
>> Any advise for existing workarounds would be appreciated.  So far the
>> only idea I've had is writing or generating a setup.py, and then adding
>> it as a quilt patch.
>
> That is probably the best approach for now.
>

Thank you for the ACK :-)

> Ultimately we have two choices:
>
> 1.  Add per-backend capability into pybuild as I have done with flit.  
> Depending on how many of them there are, this may or may not be a reasonable 
> approach.
>
> 2.  Use the pep517 module with pip in our package build system.  I think 
> Fedora has gone this direction.
>
> I prefer #1 because we want to produce a traditional bdist, not wheels, but 
> we'll have to do the work.  I don't think we want an impenetrable morass of 
> zip files in /usr/lib/python3/dist-packages.
>

I also prefer #1.  Here's an idea for reducing the work, and more
importantly reducing the Debian-specific investment in time:

Dephell (from the author of Poetry?) claims to support: egginfo, sdist,
wheel, pip, piplock, pipfile, pipfilelock, poetry, poetrylock, already
installed system-wide packages, plus setup.py, flit, conda, and
pyproject.

More interestingly it claims to be able to convert between formats, eg:

  dephell deps convert --from=pyproject.toml --to=setup.py

Within a Debian packaging context this seems like it would be useful,
because packages whose upstreams change formats could continue to use
their existing rules file with the addition of a 'dephell deps convert'
step.

It seems like it might fit well with DPMT Policy, in that we could
recommend conversion to a particular format, for use with a particular
buildsystem, for uniformity of packaging.  Finally, if that conversion
capability works well then an effect is that Debian packaging is
effectively decoupled from upstream buildsystem changes, and this seems
like a good thing.

Downside: Their installation instructions (curl -L dephell.org/install |
python3) make me wonder if there is a fundamental cultural difference
that would make this a no-go.

tldr: Could integrating Dephell into dh_python solve these challenges
once and for all?


Regards,
Nicholas

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.maint.python

csiph-web