Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > linux.debian.maint.python > #8109 > unrolled thread

Amend Debian Python Proposal to Include More Python Metadata?

Started byDonald Stufft <donald@stufft.io>
First post2016-01-21 15:30 +0100
Last post2016-01-26 19:30 +0100
Articles 9 on this page of 29 — 6 participants

Back to article view | Back to linux.debian.maint.python


Contents

  Amend Debian Python Proposal to Include More Python Metadata? Donald Stufft <donald@stufft.io> - 2016-01-21 15:30 +0100
    Re: Amend Debian Python Proposal to Include More Python Metadata? Paul Tagliamonte <paultag@debian.org> - 2016-01-21 15:50 +0100
      Re: Amend Debian Python Proposal to Include More Python Metadata? Donald Stufft <donald@stufft.io> - 2016-01-21 16:10 +0100
        Re: Amend Debian Python Proposal to Include More Python Metadata? Scott Kitterman <debian@kitterman.com> - 2016-01-21 17:00 +0100
    Re: Amend Debian Python Proposal to Include More Python Metadata? Piotr Ożarowski <piotr@debian.org> - 2016-01-22 16:40 +0100
      Re: Amend Debian Python Proposal to Include More Python Metadata? Donald Stufft <donald@stufft.io> - 2016-01-22 17:00 +0100
        Re: Amend Debian Python Proposal to Include More Python Metadata? Piotr Ożarowski <piotr@debian.org> - 2016-01-22 17:20 +0100
          Re: Amend Debian Python Proposal to Include More Python Metadata? Paul Tagliamonte <paultag@debian.org> - 2016-01-22 17:50 +0100
            Re: Amend Debian Python Proposal to Include More Python Metadata? Piotr Ożarowski <piotr@debian.org> - 2016-01-22 18:20 +0100
          Re: Amend Debian Python Proposal to Include More Python Metadata? Donald Stufft <donald@stufft.io> - 2016-01-22 18:00 +0100
          Re: Amend Debian Python Proposal to Include More Python Metadata? Piotr Ożarowski <piotr@debian.org> - 2016-01-22 18:40 +0100
        Re: Amend Debian Python Proposal to Include More Python Metadata? Scott Kitterman <debian@kitterman.com> - 2016-01-22 18:00 +0100
          Re: Amend Debian Python Proposal to Include More Python Metadata? Donald Stufft <donald@stufft.io> - 2016-01-22 18:20 +0100
            Re: Amend Debian Python Proposal to Include More Python Metadata? Scott Kitterman <debian@kitterman.com> - 2016-01-22 18:50 +0100
              Re: Amend Debian Python Proposal to Include More Python Metadata? Fred Drake <fred@fdrake.net> - 2016-01-22 19:40 +0100
                Re: Amend Debian Python Proposal to Include More Python Metadata? Fred Drake <fred@fdrake.net> - 2016-01-22 19:50 +0100
                  Re: Amend Debian Python Proposal to Include More Python Metadata? Scott Kitterman <debian@kitterman.com> - 2016-01-22 20:00 +0100
                Re: Amend Debian Python Proposal to Include More Python Metadata? Scott Kitterman <debian@kitterman.com> - 2016-01-22 19:50 +0100
        Re: Amend Debian Python Proposal to Include More Python Metadata? Scott Kitterman <debian@kitterman.com> - 2016-01-23 09:20 +0100
          Re: Amend Debian Python Proposal to Include More Python Metadata? Scott Kitterman <debian@kitterman.com> - 2016-01-23 09:30 +0100
    Re: Amend Debian Python Proposal to Include More Python Metadata? Barry Warsaw <barry@debian.org> - 2016-01-22 23:40 +0100
      Re: Amend Debian Python Proposal to Include More Python Metadata? Donald Stufft <donald@stufft.io> - 2016-01-23 00:00 +0100
        Re: Amend Debian Python Proposal to Include More Python Metadata? Barry Warsaw <barry@debian.org> - 2016-01-23 00:10 +0100
          Re: Amend Debian Python Proposal to Include More Python Metadata? Donald Stufft <donald@stufft.io> - 2016-01-23 00:20 +0100
        Re: Amend Debian Python Proposal to Include More Python Metadata? Scott Kitterman <debian@kitterman.com> - 2016-01-23 00:10 +0100
          Re: Amend Debian Python Proposal to Include More Python Metadata? Donald Stufft <donald@stufft.io> - 2016-01-23 00:30 +0100
            Re: Amend Debian Python Proposal to Include More Python Metadata? Scott Kitterman <debian@kitterman.com> - 2016-01-23 01:20 +0100
              Re: Amend Debian Python Proposal to Include More Python Metadata? Donald Stufft <donald@stufft.io> - 2016-01-23 02:00 +0100
            pip as distutils/setuptools/setup.py wrapper Piotr Ożarowski <piotr@debian.org> - 2016-01-26 19:30 +0100

Page 2 of 2 — ← Prev page 1 [2]


#8129

FromBarry Warsaw <barry@debian.org>
Date2016-01-22 23:40 +0100
Message-ID<qTSx5-20U-29@gated-at.bofh.it>
In reply to#8109

[Multipart message — attachments visible in raw view] — view raw

Hey Donald, thanks for starting this conversation.  I for one am super
appreciative of all the consideration you give for Debian's little slice of
the world.

There's a lot to unpack in this thread, and I'm a little under the weather[1],
so hopefully this makes sense.

Big +1 for recording the files that get installed via the .egg-info/RECORD
file.  As you know, I've been working on dirtbike, which is a "rewheeling"
tool to turn an installed package into a .whl.  While Debian tightly controls
via policy the set of wheels we'll allow into the archive, dirtbike has code
for parsing the RECORD file.  Unfortunately this is never exercised in
practice because we don't have RECORD files - at least not for the packages we
care about[2].

Big +1 for using setuptools everywhere.  By my count, of the packages that I
happen to randomly have installed on my Ubuntu 16.04 system, I have 67
.egg-info files and 113 .egg-info directories.  I'd rather have .egg-info
directories everywhere.

+1 for a lintian warning if distutils is used.  I guess I'm +0 on forcing that
through pybuild because it'll be unobvious and mysterious, and kind of lets
upstreams off the hook.  I'd mildly prefer to patch packages that use
distutils because that's much more discoverable, but I can appreciate that
that's a lot of work we'd be imposing on maintainers, so I won't argue this
too much (other than to say that if pybuild forces it, let's definitely
document this in its manpage!).

On Jan 22, 2016, at 12:40 PM, Scott Kitterman wrote:

>Currently --record includes the .pyc files which is both unneeded and bad.
>Before this gets added either in setuptools or by us, this needed to be
>fixed.

+1 for (I think) another reason than has already been discussed.  We won't be
generating .egg-info directories on the end-user's machine, but instead the
machine the package is built on.  That could be a maintainer's own system or a
central build machine depending on various conditions.  But because the pycs
are generated on the end-user's machine, we actually don't know what pycs will
be generated when the debs are installed, so the egg-info/RECORD file *can't*
contain them, at least not accurately.

On Jan 22, 2016, at 11:54 AM, Donald Stufft wrote:

>Regardless of what happens in this thread, pip is going to stop mucking
>around in files that are owned by some other tool without some sort of
>opt-in/--force style flag *and* we're going to be restructuring things to try
>and guide people away from using pip outside of a virtual environment or
>through the user of --user as well.

Of course, I'd still like --user to be the default[3].  I think that's still
the eventual goal for pip, but isn't yet implemented because $priorities.

>A more controversial way that comes with possibly some extra benefits (which
>Debian may not care about) is to use ``pip`` itself as the build tool rather
>than directly invoking setup.py. In this pip would be responsible for mucking
>around with the distutils/setuptools mess instead of that needing to be
>handled by Debian specific tooling.

I'd like to better understand how this would work.  IIUC, the Fedora ecosystem
is making or already has made this switch, but I don't know how it works.
Obviously, we don't want to install wheels into /usr/lib/python3/dist-packages.

FTR, I am still working on updating Debian's pip.  I'm slowly shaving the yak,
but there are still a few things to do.  If you want to help, get in touch.

>[3] Import pkg_resources is not the fasest thing in the world, it has to scan
>every directory on sys.path looking for things to activate and it comes with
>a bunch of side effects. This happens implicitly for any project using
>console scripts.

Which frankly sucks.  It's also fragile.  Every once in a while a broken
package gets uploaded that breaks pkg_resources, and it's not easy to debug or
repair.  I really hope Brett can fix this when/if he builds this functionality
into the stdlib.

Cheers,
-Barry

[1] double meaning: I have a cold and we're in the early stages of an historic
snow storm!

[2] dirtbike has two fallbacks, both of which use `dpkg -L` to get the list of
installed files.  The first fallback uses `dpkg -S` to find the binary package
that contains the Python package's .egg-info file/directory (doesn't matter
for this purpose).  pkg_resources doesn't have one of those.  The second
fallback imports the Python package and looks for the binary package
containing the top-level directory.  It's all rather ugly, and I'd like to
just use the .egg-info/RECORD file, but I suspect I'll still need the import
fallback for pkg_resources.

[3] https://github.com/pypa/pip/issues/1668

[toc] | [prev] | [next] | [standalone]


#8131

FromDonald Stufft <donald@stufft.io>
Date2016-01-23 00:00 +0100
Message-ID<qTSQp-28S-7@gated-at.bofh.it>
In reply to#8129

[Multipart message — attachments visible in raw view] — view raw

> On Jan 22, 2016, at 5:30 PM, Barry Warsaw <barry@debian.org> wrote:
> 
> Hey Donald, thanks for starting this conversation.  I for one am super
> appreciative of all the consideration you give for Debian's little slice of
> the world.
> 
> There's a lot to unpack in this thread, and I'm a little under the weather[1],
> so hopefully this makes sense.
> 
> Big +1 for recording the files that get installed via the .egg-info/RECORD
> file.  As you know, I've been working on dirtbike, which is a "rewheeling"
> tool to turn an installed package into a .whl.  While Debian tightly controls
> via policy the set of wheels we'll allow into the archive, dirtbike has code
> for parsing the RECORD file.  Unfortunately this is never exercised in
> practice because we don't have RECORD files - at least not for the packages we
> care about[2].
> 
> Big +1 for using setuptools everywhere.  By my count, of the packages that I
> happen to randomly have installed on my Ubuntu 16.04 system, I have 67
> .egg-info files and 113 .egg-info directories.  I'd rather have .egg-info
> directories everywhere.
> 
> +1 for a lintian warning if distutils is used.  I guess I'm +0 on forcing that
> through pybuild because it'll be unobvious and mysterious, and kind of lets
> upstreams off the hook.  I'd mildly prefer to patch packages that use
> distutils because that's much more discoverable, but I can appreciate that
> that's a lot of work we'd be imposing on maintainers, so I won't argue this
> too much (other than to say that if pybuild forces it, let's definitely
> document this in its manpage!).
> 
> On Jan 22, 2016, at 12:40 PM, Scott Kitterman wrote:
> 
>> Currently --record includes the .pyc files which is both unneeded and bad.
>> Before this gets added either in setuptools or by us, this needed to be
>> fixed.
> 
> +1 for (I think) another reason than has already been discussed.  We won't be
> generating .egg-info directories on the end-user's machine, but instead the
> machine the package is built on.  That could be a maintainer's own system or a
> central build machine depending on various conditions.  But because the pycs
> are generated on the end-user's machine, we actually don't know what pycs will
> be generated when the debs are installed, so the egg-info/RECORD file *can't*
> contain them, at least not accurately.

FWIW It appears that if you do setup.py install —no-compile to tell distutils
not to compile the .pyc during .deb build time, I believe that as is it will
just omit the .pyc files. Not sure if that’s workable for Debian or not.

> 
> On Jan 22, 2016, at 11:54 AM, Donald Stufft wrote:
> 
>> Regardless of what happens in this thread, pip is going to stop mucking
>> around in files that are owned by some other tool without some sort of
>> opt-in/--force style flag *and* we're going to be restructuring things to try
>> and guide people away from using pip outside of a virtual environment or
>> through the user of --user as well.
> 
> Of course, I'd still like --user to be the default[3].  I think that's still
> the eventual goal for pip, but isn't yet implemented because $priorities.

Yes, still a goal (it’s not as simple as —user will be the default, but that’s
the outcome for a non root account in a system Python).

> 
>> A more controversial way that comes with possibly some extra benefits (which
>> Debian may not care about) is to use ``pip`` itself as the build tool rather
>> than directly invoking setup.py. In this pip would be responsible for mucking
>> around with the distutils/setuptools mess instead of that needing to be
>> handled by Debian specific tooling.
> 
> I'd like to better understand how this would work.  IIUC, the Fedora ecosystem
> is making or already has made this switch, but I don't know how it works.
> Obviously, we don't want to install wheels into /usr/lib/python3/dist-packages.

I haven’t actually personally seen if they’ve done it, but the Fedora maintainer
of the python-pip told me they were doing it, and they had a patch to pip to
add a feature they needed for it.

Forget that pip can fetch files from PyPI and install them for a moment and
consider the command ``pip install .``. Fundamentally this is similar to the
command ``make install`` right? It builds the files and then installs them
into the final location. As I understand it, the standard way for a project
like Debian to install a project using a “regular” Makefile install like that
is to do ``make DESTDIR=/tmp/some-tempdir install`` so that it invokes the
build system, sets all the paths in the resulting build system to the final
destination, but then right before it actually copies files to the final paths
it just appends a directory in front of them to let you stage the files in
a different location instead.

We already have an option like this, the —root option which will just append
a different prefix to all of the installation paths. So essentially instead
of invoking ``python setup.py install —root /tmp/something/`` which is what
I think you’re doing now, you’d do ``pip install —root /tmp/something/ —no-deps .``
and it’d just work similarly to what you have now, except pip would be
responsible for interacting with the Python build system.

Slavek had said there was some bug preventing him from using —root at the time
and I’m not entirely sure what that was off the top of my head but the issue
he opened was https://github.com/pypa/pip/issues/1351. I’m not 100% sure what
the right combination of options are, but if it’s something y’all are interested
in I can certainly figure it out (and add anything needed to make it happen).

> 
> FTR, I am still working on updating Debian's pip.  I'm slowly shaving the yak,
> but there are still a few things to do.  If you want to help, get in touch.
> 
>> [3] Import pkg_resources is not the fasest thing in the world, it has to scan
>> every directory on sys.path looking for things to activate and it comes with
>> a bunch of side effects. This happens implicitly for any project using
>> console scripts.
> 
> Which frankly sucks.  It's also fragile.  Every once in a while a broken
> package gets uploaded that breaks pkg_resources, and it's not easy to debug or
> repair.  I really hope Brett can fix this when/if he builds this functionality
> into the stdlib.

For console scripts we don’t actually need anything added, this is the console
scripts that pip generates when it installs from a Wheel (either explicitly
from PyPI or through an implicitly compiled wheel): https://bpaste.net/show/247fe79d91c4

> 
> Cheers,
> -Barry
> 
> [1] double meaning: I have a cold and we're in the early stages of an historic
> snow storm!
> 
> [2] dirtbike has two fallbacks, both of which use `dpkg -L` to get the list of
> installed files.  The first fallback uses `dpkg -S` to find the binary package
> that contains the Python package's .egg-info file/directory (doesn't matter
> for this purpose).  pkg_resources doesn't have one of those.  The second
> fallback imports the Python package and looks for the binary package
> containing the top-level directory.  It's all rather ugly, and I'd like to
> just use the .egg-info/RECORD file, but I suspect I'll still need the import
> fallback for pkg_resources.
> 
> [3] https://github.com/pypa/pip/issues/1668


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[toc] | [prev] | [next] | [standalone]


#8132

FromBarry Warsaw <barry@debian.org>
Date2016-01-23 00:10 +0100
Message-ID<qTT06-2t8-11@gated-at.bofh.it>
In reply to#8131

[Multipart message — attachments visible in raw view] — view raw

On Jan 22, 2016, at 05:50 PM, Donald Stufft wrote:

>Forget that pip can fetch files from PyPI and install them for a moment and
>consider the command ``pip install .``. Fundamentally this is similar to the
>command ``make install`` right?

Please remind me what the long term plan for this is.  Let's say it's 2021 and
you and your amazing team of dozens of distutil-sig developers have been
cranking away the whole time.

What tool will be used to build Python packages?

What tool will be used to upload (built/source?) Python packages to
WarehouseNG?

What tool will be used to download Python packages from WarehouseNG?

What tool will be used to install packages from WarehouseNG into an importable
location?

Cheers,
-Barry

[toc] | [prev] | [next] | [standalone]


#8134

FromDonald Stufft <donald@stufft.io>
Date2016-01-23 00:20 +0100
Message-ID<qTT9M-2wH-15@gated-at.bofh.it>
In reply to#8132

[Multipart message — attachments visible in raw view] — view raw

> On Jan 22, 2016, at 6:02 PM, Barry Warsaw <barry@debian.org> wrote:
> 
> On Jan 22, 2016, at 05:50 PM, Donald Stufft wrote:
> 
>> Forget that pip can fetch files from PyPI and install them for a moment and
>> consider the command ``pip install .``. Fundamentally this is similar to the
>> command ``make install`` right?
> 
> Please remind me what the long term plan for this is.  Let's say it's 2021 and
> you and your amazing team of dozens of distutil-sig developers have been
> cranking away the whole time.
> 
> What tool will be used to build Python packages?

Whatever people want! At least, that’s my hope. Ideally we get away from
the idea that any particular build system is the right build system (because
the needs of a pure python thing like six.py is vastly different from Numpy
which needs Fortran and other crazy things).

My end goal is, as long as your build system can consume a sdist (or maybe
a repository too, still up in the air, but some kind of source input) and
produce a wheel (whether this is actually a zipped up wheel file or a properly
formatted directory is also still up in the air). Then you have a good enough
build system and that is all it needs to do, it doesn’t need to also implement
installing and what not, that’s some other tool’s job.

> 
> What tool will be used to upload (built/source?) Python packages to
> WarehouseNG?

Either twine, or we’ll fold that functionality into pip.

> 
> What tool will be used to download Python packages from WarehouseNG?

Whatever tool you want! We’ll have defined formats so as long as your tool
speaks those formats it can install them. In the case of already “compiled”
(pure Python or not) binary .whl files it can just download them, unzip,
move into the correct location + install deps and such. For source files
(in a not-yet thought out or done sdist 2.0 thing) it can download the
sdist 2.0 file, install the build dependencies, invoke the build tool for
this package (from above), get the wheel it produced, and then install
that as if it had downloaded it from PyPI.

> 
> What tool will be used to install packages from WarehouseNG into an importable
> location?

Same answer here.

> 
> Cheers,
> -Barry


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[toc] | [prev] | [next] | [standalone]


#8133

FromScott Kitterman <debian@kitterman.com>
Date2016-01-23 00:10 +0100
Message-ID<qTT06-2t8-17@gated-at.bofh.it>
In reply to#8131

[Multipart message — attachments visible in raw view] — view raw

On Friday, January 22, 2016 05:50:13 PM Donald Stufft wrote:
...
> We already have an option like this, the —root option which will just append
> a different prefix to all of the installation paths. So essentially instead
> of invoking ``python setup.py install —root /tmp/something/`` which is what
> I think you’re doing now, you’d do ``pip install —root /tmp/something/
> —no-deps .`` and it’d just work similarly to what you have now, except pip
> would be responsible for interacting with the Python build system.
...

Why would pip interacting with the Python build system instead of setuptools 
be better?

Scott K

[toc] | [prev] | [next] | [standalone]


#8135

FromDonald Stufft <donald@stufft.io>
Date2016-01-23 00:30 +0100
Message-ID<qTTjs-2Bp-3@gated-at.bofh.it>
In reply to#8133

[Multipart message — attachments visible in raw view] — view raw

> On Jan 22, 2016, at 6:04 PM, Scott Kitterman <debian@kitterman.com> wrote:
> 
> On Friday, January 22, 2016 05:50:13 PM Donald Stufft wrote:
> ...
>> We already have an option like this, the —root option which will just append
>> a different prefix to all of the installation paths. So essentially instead
>> of invoking ``python setup.py install —root /tmp/something/`` which is what
>> I think you’re doing now, you’d do ``pip install —root /tmp/something/
>> —no-deps .`` and it’d just work similarly to what you have now, except pip
>> would be responsible for interacting with the Python build system.
> ...
> 
> Why would pip interacting with the Python build system instead of setuptools
> be better?
> 
> Scott K

setuptools *is* the build system right now (or distutils if you’re still using
that). So the benefit to Debian right now would be: We maintain the
distutils/setuptools hacks to make them suck less so you don’t have to!

Longer term, we want to enable people to go hog wild and invent their own
build systems instead of trying to do it all one size fits all with everyone
miserable because it doesn’t solve anyones problems nicely. At that point the
benefit to Debian becomes that instead of having to implement the standard
build tool interface that these tools implement (Which will most likely be
geared towards produces wheels, not towards putting files on disk in an
installed location) Debian can just let us maintain that bit of code too.

Of course, if Debian would prefer to interact directly with these tools, it’s
totally fine to do that. The whole point of us doing this work and trying to
define formats and APIs and not bless implementations is so anyone who wants
to can slot in their own tool in the process instead of being forced to use
some blessed tool.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[toc] | [prev] | [next] | [standalone]


#8136

FromScott Kitterman <debian@kitterman.com>
Date2016-01-23 01:20 +0100
Message-ID<qTU5Q-3am-5@gated-at.bofh.it>
In reply to#8135

On January 22, 2016 6:27:08 PM EST, Donald Stufft <donald@stufft.io> wrote:
>
>> On Jan 22, 2016, at 6:04 PM, Scott Kitterman <debian@kitterman.com>
>wrote:
>> 
>> On Friday, January 22, 2016 05:50:13 PM Donald Stufft wrote:
>> ...
>>> We already have an option like this, the —root option which will
>just append
>>> a different prefix to all of the installation paths. So essentially
>instead
>>> of invoking ``python setup.py install —root /tmp/something/`` which
>is what
>>> I think you’re doing now, you’d do ``pip install —root
>/tmp/something/
>>> —no-deps .`` and it’d just work similarly to what you have now,
>except pip
>>> would be responsible for interacting with the Python build system.
>> ...
>> 
>> Why would pip interacting with the Python build system instead of
>setuptools
>> be better?
>> 
>> Scott K
>
>setuptools *is* the build system right now (or distutils if you’re
>still using
>that). So the benefit to Debian right now would be: We maintain the
>distutils/setuptools hacks to make them suck less so you don’t have to!
>
>Longer term, we want to enable people to go hog wild and invent their
>own
>build systems instead of trying to do it all one size fits all with
>everyone
>miserable because it doesn’t solve anyones problems nicely. At that
>point the
>benefit to Debian becomes that instead of having to implement the
>standard
>build tool interface that these tools implement (Which will most likely
>be
>geared towards produces wheels, not towards putting files on disk in an
>installed location) Debian can just let us maintain that bit of code
>too.
>
>Of course, if Debian would prefer to interact directly with these
>tools, it’s
>totally fine to do that. The whole point of us doing this work and
>trying to
>define formats and APIs and not bless implementations is so anyone who
>wants
>to can slot in their own tool in the process instead of being forced to
>use
>some blessed tool.

You probably expected something like this back, but here goes anyway...

The Zen of Python says, among other things, "There should be one-- and preferably only one --obvious way to do it".  Build systems seem to me like a great place to apply that.

Build system proliferation is a nightmare.  In the last several months I've gone through roughly all of the several hundred packages in the Debian archive that build (or were packed like they build) compiled code (e.g. python C extensions, library bindings, etc.).  I was working on two goals:

1) Making sure they built with python3.5

2) Adding support for building with multiple python3 versions

For the former, the issue were sometimes buil system related, but for the latter that was almost always the case.  When packages used setup.py (and it made no difference if it was distutills or setuputils) it was virtually always easy.  For anything else ranged from moderately complex to not worth the trouble.

For the things I need it for in the distro, setup.py (with the changes we've made) works really well.

Why do we need a bazillion options for Python build tools?  To close with another quote from the Zen, "Special cases aren't special enough to break the rules."

Scott K

P.S. I worry about having to go through waf again [1].  That was painful.

[1] https://lists.debian.org/debian-dak/2012/01/msg00009.html

[toc] | [prev] | [next] | [standalone]


#8138

FromDonald Stufft <donald@stufft.io>
Date2016-01-23 02:00 +0100
Message-ID<qTUIy-3qs-17@gated-at.bofh.it>
In reply to#8136

[Multipart message — attachments visible in raw view] — view raw

> On Jan 22, 2016, at 7:18 PM, Scott Kitterman <debian@kitterman.com> wrote:
> 
> The Zen of Python says, among other things, "There should be one-- and preferably only one --obvious way to do it".  Build systems seem to me like a great place to apply that.


We have a sliding scale of complexity in what a project needs from it’s build system that generally breaks down into a few major classes of projects:

1) Projects that are pure python, single source, whose only real “compilation” is shuffling files to the correct location.
2) Projects that are pure python, but require some sort of generation step as part of the build process (2to3, etc).
3) Projects that have some basic C code that needs to be compiled, but which doesn’t link against anything special besides Python itself.
4) Projects that have some basic C code that needs to be compiled, but which links against other libraries like OpenSSL, libpq, etc.
5) Projects that have some basic C code, that needs to be compiled, that links against other libraries, and needs to be able to conditionally link against different libraries based on the capabilities and what is available in the system.
6) Extremely complex projects that need to link against many different libraries, possibly hard to build, possibly not C (e.g., Numpy with it’s blas libraries, Fortran, etc).

The problem with a one size fits all solution is that it’s very difficult to actually cover all of these cases in a way that is not horrible to actually use for each particular case. For an example, there is currently a build system called flit which doesn’t support anything but building straight to wheels (because we don’t have any sort of sdist 2.0 or anything yet). It doesn’t attempt to solve anything but the first class of users up there, and because of that it is able to create a very simple and easy to use packaging experience for authors, you just add a __version__ = “the version” to the top level of your package, and then drop in a flit.ini that looks something like:

    [metadata]
    module=foobar
    author=Sir Robin
    author-email=robin@camelot.uk
    home-page=http://github.com/sirrobin/foobar

    # If you want command line scripts, this is how to declare them.
    # If not, you can leave this section out completely.
    [scripts]
    # foobar:main means the script will do: from foobar import main; main()
    foobar=foobar:main

And that’s it. You’re done. From there flit can do the rest of the work for you because it didn’t need to concern itself with trying to work on anything complex.

In addition to the above types of problems, you also have other things like what the “source of truth” is for your metadata. A common thing that people want is to be able to not have to duplicate their version in multiple locations (sometimes that even extends to the tags in the version control), however it’s not currently possible to do that in a particularly easy way. You have systems like pbr, setuptools_scm, versioner, etc that all do it but which all rely on some level of terribleness to deal with the sort of weird inverted control flow we have.

Once you get to all of the possible options for things people reasonably want to do, you quickly end up in a place where the only reasonable solution is the full complexity of a programming language like what we have in setup.py. However, that has it’s own problems which we’ve discovered over two decades of doing that :) It tends to end up with people doing badly tested adhoc code that they cargo cult from project to project and when there’s a problem there ends up being very little understanding why some code did that since it had been copy/pasted around so much. Pip has to go to a lot of effort to try and work around common mistakes people make in their setup.py files (like, depending on it being invoked with the project root as CURDIR) which most people will just never notice their slightly broken setup.py.

So basically, by allowing multiple build systems we will enable a world where your bog standard pure python project can get an extremely simple tool/ux and projects with crazy hard requirements like Numpy can get something more complex, without the two groups of users fighting each other over simplicity + limited vs complexity + powerful.

In the end, I think it’s likely we’ll end up with 2-3 really popular build tools, one that is for the complex projects, one that is for the simpler projects with some basic C needs, and *maybe* one that is for pure python (but that may be able to be handled by the basic C needs one) though there will be a long tail I’m sure.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[toc] | [prev] | [next] | [standalone]


#8166 — pip as distutils/setuptools/setup.py wrapper

FromPiotr Ożarowski <piotr@debian.org>
Date2016-01-26 19:30 +0100
Subjectpip as distutils/setuptools/setup.py wrapper
Message-ID<qVgxk-83o-13@gated-at.bofh.it>
In reply to#8135
[Donald Stufft, 2016-01-23]
> setuptools *is* the build system right now (or distutils if you’re still using
> that). So the benefit to Debian right now would be: We maintain the
> distutils/setuptools hacks to make them suck less so you don’t have to!

will pip allow me to:
a) clean source dir (remove built files, etc.; setup.py clean ...),
b) configure the program (setup.py config ...),
c) build extensions, etc., without installing them (setup.py build ...)
   (in a dir I want or can easily guess, not one with a ~random name or
   the same for all interpreters/versions with overwritten files),
d) test what's produced in previous step (setup.py test with a ton of hacks),
e) install already built files (setup.py install --root)
?
-- 
Piotr Ożarowski                         Debian GNU/Linux Developer
www.ozarowski.pl          www.griffith.cc           www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

[toc] | [prev] | [standalone]


Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.debian.maint.python


csiph-web