Groups > linux.debian.maint.python > #8109 > unrolled thread

Amend Debian Python Proposal to Include More Python Metadata?

Started by	Donald Stufft <donald@stufft.io>
First post	2016-01-21 15:30 +0100
Last post	2016-01-26 19:30 +0100
Articles	20 on this page of 29 — 6 participants

Back to article view | Back to linux.debian.maint.python

Page 1 of 2 [1] 2 Next page →

#8109 — Amend Debian Python Proposal to Include More Python Metadata?

From	Donald Stufft <donald@stufft.io>
Date	2016-01-21 15:30 +0100
Subject	Amend Debian Python Proposal to Include More Python Metadata?
Message-ID	<qTopk-6qy-11@gated-at.bofh.it>

[Multipart message — attachments visible in raw view] — view raw

Hello!

I'd like to suggest a change to the Debian Policy around Python packages that
will help enable the world of Python packaging to continue to progress forward.

First, a little bit of background:

At the Python level there are three metadata formats for Python packaging:

* The original, setuptools style .egg-info directories.
* The distutils style .egg-info *file* added to distutils at some point.
* The new and improved, wheel based .dist-info directories.

The presence of any of these files will signal to Python tools that a
particular distribution has been installed, however there are two fairly major
and important differences between the distutils style, and the other two.

1. The distutils style has no provisions to record what files on the system
   belong to the installed distribution, making it appear to Python tooling
   that there *are* no files other than the metadata file itself.

2. The distutils style has no provisions to include additional metadata files
   in the metadata, making it impossible to extend the python level metadata
   with additional files.

I have a series of improvements that I'd like to make to the packaging
toolchain that will sort of build on one another, but which is not going to
function correctly with the distutils style metadata and I'm hoping that I can
convince y'all to make it policy to default to generating one of the other two
kinds (with varying methods, more on that later).

Concretely the thing that this is blocking right now, is that with the newly
released pip 8.0 I tried to make it so that pip will refuse to uninstall a
project that is installed with distutils style metadata. This is because we do
not have any way to associate the actual .py (and others) files on disk with
the installed metadata, so all we have ever done is just simply remove the
metadata file, making it appear as if the item is uninstalled but leaving
behind all of the actual files. However I'm going to be reverting this in a
pip 8.0.1 release because it caused a decent amount of breakage amongst pip's
users, almost all of them people who are attempting to upgrade OS provided
packages using pip.

Now, I know that upgrading OS provided packages using pip is less than optimal
and I would greatly prefer that people did not do it (and I'm generally in
agreement) however if we don't enable people to do it, they'll just continue to
use an old version of pip and file bugs. It's a non starter for pip to make it
impossible to do.

In addition to the uninstall bit, it also means that things like pip show -f
return junk information for packages installed in this way.

Beyond just (eventually) enabling pip to disable uninstallations of distutils
based installs this will start to allow some other future changes that I think
will be more interesting to Debian. The uninstallation of distutils based
installs comes hand in hand with pip stomping all over already existing files
willy nilly because the way upgrading a project like that works is pip
uninstalls the metadata file that says X is installed, then it just overwrites
over any of the files that happen to be in it's way when it installs the newer
version. If we can remove the need for pip to gleefully overwrite files to
support these types of installed packages, then we can make it so pip will
hard fail if it attempts to overwrite an already existing file on disk.

An additional benefit here is that by switching to using the directory based
options, we can add additional metadata files to the installed projects, much
like the INSTALLER file from PEP376 (IIRC). This file will likely be the path
to having pip refuse to touch OS owned files all together without some sort of
--force flag to override the safety switch.

As far as compatibility goes, pip has always forced everything to be installed
using setuptools and as far as I am aware, there's no real fallout from doing
so. I think in 2016 it's pretty reasonable to assume that a Python project is
capable of being installed using setuptools instead of distutils.

So without getting into the actual *method* of doing this (of which there are
several different options with different trade offs) does this sound like
something at all that Debian would be interested in?

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[toc] | [next] | [standalone]

#8110

From	Paul Tagliamonte <paultag@debian.org>
Date	2016-01-21 15:50 +0100
Message-ID	<qToIH-6z7-47@gated-at.bofh.it>
In reply to	#8109

[Multipart message — attachments visible in raw view] — view raw

Hey Donald!

As far as using pip to do stuff system-wide, I wrote thoughts on
http://notes.pault.ag/debian-python

As for the rest of it, distutils is actually concretely shitty, and
replacing it with setuptools
sounds sane. Sounds like a solid idea.

Perhaps we can add a Lintian warning for using distutils; it's really nasty
cruft,
and it's going to bite the archive sooner rather than later.

Cheers,
  Paul

On Thu, Jan 21, 2016 at 9:19 AM, Donald Stufft <donald@stufft.io> wrote:

> Hello!
>
> I'd like to suggest a change to the Debian Policy around Python packages
> that
> will help enable the world of Python packaging to continue to progress
> forward.
>
> First, a little bit of background:
>
> At the Python level there are three metadata formats for Python packaging:
>
> * The original, setuptools style .egg-info directories.
> * The distutils style .egg-info *file* added to distutils at some point.
> * The new and improved, wheel based .dist-info directories.
>
> The presence of any of these files will signal to Python tools that a
> particular distribution has been installed, however there are two fairly
> major
> and important differences between the distutils style, and the other two.
>
> 1. The distutils style has no provisions to record what files on the system
>    belong to the installed distribution, making it appear to Python tooling
>    that there *are* no files other than the metadata file itself.
>
> 2. The distutils style has no provisions to include additional metadata
> files
>    in the metadata, making it impossible to extend the python level
> metadata
>    with additional files.
>
> I have a series of improvements that I'd like to make to the packaging
> toolchain that will sort of build on one another, but which is not going to
> function correctly with the distutils style metadata and I'm hoping that I
> can
> convince y'all to make it policy to default to generating one of the other
> two
> kinds (with varying methods, more on that later).
>
> Concretely the thing that this is blocking right now, is that with the
> newly
> released pip 8.0 I tried to make it so that pip will refuse to uninstall a
> project that is installed with distutils style metadata. This is because
> we do
> not have any way to associate the actual .py (and others) files on disk
> with
> the installed metadata, so all we have ever done is just simply remove the
> metadata file, making it appear as if the item is uninstalled but leaving
> behind all of the actual files. However I'm going to be reverting this in a
> pip 8.0.1 release because it caused a decent amount of breakage amongst
> pip's
> users, almost all of them people who are attempting to upgrade OS provided
> packages using pip.
>
> Now, I know that upgrading OS provided packages using pip is less than
> optimal
> and I would greatly prefer that people did not do it (and I'm generally in
> agreement) however if we don't enable people to do it, they'll just
> continue to
> use an old version of pip and file bugs. It's a non starter for pip to
> make it
> impossible to do.
>
> In addition to the uninstall bit, it also means that things like pip show
> -f
> return junk information for packages installed in this way.
>
> Beyond just (eventually) enabling pip to disable uninstallations of
> distutils
> based installs this will start to allow some other future changes that I
> think
> will be more interesting to Debian. The uninstallation of distutils based
> installs comes hand in hand with pip stomping all over already existing
> files
> willy nilly because the way upgrading a project like that works is pip
> uninstalls the metadata file that says X is installed, then it just
> overwrites
> over any of the files that happen to be in it's way when it installs the
> newer
> version. If we can remove the need for pip to gleefully overwrite files to
> support these types of installed packages, then we can make it so pip will
> hard fail if it attempts to overwrite an already existing file on disk.
>
> An additional benefit here is that by switching to using the directory
> based
> options, we can add additional metadata files to the installed projects,
> much
> like the INSTALLER file from PEP376 (IIRC). This file will likely be the
> path
> to having pip refuse to touch OS owned files all together without some
> sort of
> --force flag to override the safety switch.
>
> As far as compatibility goes, pip has always forced everything to be
> installed
> using setuptools and as far as I am aware, there's no real fallout from
> doing
> so. I think in 2016 it's pretty reasonable to assume that a Python project
> is
> capable of being installed using setuptools instead of distutils.
>
> So without getting into the actual *method* of doing this (of which there
> are
> several different options with different trade offs) does this sound like
> something at all that Debian would be interested in?
>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
>
>


-- 
:wq

[toc] | [prev] | [next] | [standalone]

#8111

From	Donald Stufft <donald@stufft.io>
Date	2016-01-21 16:10 +0100
Message-ID	<qTp23-6YC-39@gated-at.bofh.it>
In reply to	#8110

[Multipart message — attachments visible in raw view] — view raw

> On Jan 21, 2016, at 9:32 AM, Paul Tagliamonte <paultag@debian.org> wrote:
> 
> Hey Donald!
> 
> As far as using pip to do stuff system-wide, I wrote thoughts on http://notes.pault.ag/debian-python <http://notes.pault.ag/debian-python>

I just want to be clear, that I totally agree with this. The reason I want to
ensure we can enable this isn't because I think doing so is a great idea and
everyone should be doing it. People *are* doing it and we (pip) have to be
careful about what we make impossible to do because a possible result of going
too break happy is people just never upgrade and things start to stagnate
again.

So for pip at the system level, my goal is try and push people away from it as
much as we can without flat out making it impossible *and* try to make it as
"safe" to do so as possible when people decide to do it anyways.

> 
> As for the rest of it, distutils is actually concretely shitty, and replacing it with setuptools
> sounds sane. Sounds like a solid idea.
> 
> Perhaps we can add a Lintian warning for using distutils; it's really nasty cruft,
> and it's going to bite the archive sooner rather than later.
> 

As far as options for how to implement this goes, I'm not sure what best fits
with Debian so I'll leave that up to y'all, but here are a few options that I
can think of.

A lintian check is probably a pretty good thing, I would guess that it would
specifically check for the existence of a .egg-info or .dist-info directory
(or check to the non-existence of a .egg-info file).

In pip the way we make this work is with a bit of a gross little thunk that
ensures setuptools is imported prior to setup.py being invoked which forces all
setup.py calls to use setuptools instead of distutils even if they import from
distutils.

Of course, many setup.py already unconditionally use setuptools, and many more
conditionally use it (typically falling back to distutils) so for many projects
it may be enough to just add python-setuptools to build-depends. For projects
that unconditionally use distutils, either the pip way or patching setup.py
might be a reasonable way forward.

A more controversial way that comes with possibly some extra benefits (which
Debian may not care about) is to use ``pip`` itself as the build tool rather
than directly invoking setup.py. In this pip would be responsible for mucking
around with the distutils/setuptools mess instead of that needing to be handled
by Debian specific tooling. You could even leverage this to remove some of the
runtime dependencies on python-pkg-resources and speed up some python using
software if you let pip build a wheel as part of the buld process [1][2][3]. I
suspect pip would need to grow some additional options to make it suitable for
you in this role, but I would be more than willing to help if this is desired
(or not, this proposal isn't about trying to ram pip down anyones throat!).

[1] To be clear, this isn't about adding wheels for everything to the archive,
    it would only be using Wheel as an intermediate step in the build process.
[2] Console scripts generated by setuptools entry wrappers when installed by
    setuptools have a dependency on pkg_resources at runtime, however those
    same scripts, when installed by pip from a wheel, do not. This would enable
    removing the runtime dependency on pkg_resources for anything that only has
    it for console scripts.
[3] Import pkg_resources is not the fasest thing in the world, it has to scan
    every directory on sys.path looking for things to activate and it comes
    with a bunch of side effects. This happens implicitly for any project using
    console scripts.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[toc] | [prev] | [next] | [standalone]

#8114

From	Scott Kitterman <debian@kitterman.com>
Date	2016-01-21 17:00 +0100
Message-ID	<qTpOq-7iT-5@gated-at.bofh.it>
In reply to	#8111

[Multipart message — attachments visible in raw view] — view raw

On Thursday, January 21, 2016 10:00:53 AM Donald Stufft wrote:
> > On Jan 21, 2016, at 9:32 AM, Paul Tagliamonte <paultag@debian.org> wrote:
> > 
> > Hey Donald!
> > 
> > As far as using pip to do stuff system-wide, I wrote thoughts on
> > http://notes.pault.ag/debian-python <http://notes.pault.ag/debian-python>
> I just want to be clear, that I totally agree with this. The reason I want
> to ensure we can enable this isn't because I think doing so is a great idea
> and everyone should be doing it. People *are* doing it and we (pip) have to
> be careful about what we make impossible to do because a possible result of
> going too break happy is people just never upgrade and things start to
> stagnate again.
> 
> So for pip at the system level, my goal is try and push people away from it
> as much as we can without flat out making it impossible *and* try to make
> it as "safe" to do so as possible when people decide to do it anyways.

On Debian, if you try to remove an essential package (which would be really 
bad as a rule), the system won't stop you, but you get this:

You are about to do something potentially harmful.
To continue type in the phrase 'Yes, do as I say!'
 ?]

And you have to put in the entire phrase.

If pip did something like that for system level installs, I doubt most of us 
would object to it anymore.

Scott K

[toc] | [prev] | [next] | [standalone]

#8115

From	Piotr Ożarowski <piotr@debian.org>
Date	2016-01-22 16:40 +0100
Message-ID	<qTLYC-5Vz-11@gated-at.bofh.it>
In reply to	#8109

Hi,

[Donald Stufft, 2016-01-21]
> I'd like to suggest a change to the Debian Policy around Python packages that
> will help enable the world of Python packaging to continue to progress forward.
[...]
> I have a series of improvements that I'd like to make to the packaging
> toolchain that will sort of build on one another, but which is not going to
> function correctly with the distutils style metadata and I'm hoping that I can
> convince y'all to make it policy to default to generating one of the other two
> kinds (with varying methods, more on that later).

Isn't that something for distutils-sig or python-dev?
We install whatever setup.py creates, no matter if it's egg-info dir,
file or something else.

[...]
> Now, I know that upgrading OS provided packages using pip is less than optimal

it's more than that, it's something that makes me scream: kill it! kill it with
fire!!! (but since we also allow /bin/rm in Debian, I guess we have to
tolerate pip ;)

> and I would greatly prefer that people did not do it (and I'm generally in
> agreement) however if we don't enable people to do it, they'll just continue to
> use an old version of pip and file bugs. It's a non starter for pip to make it
> impossible to do.

did I mention fire? ;)

[...]
> An additional benefit here is that by switching to using the directory based
> options, we can add additional metadata files to the installed projects, much
> like the INSTALLER file from PEP376 (IIRC). This file will likely be the path
> to having pip refuse to touch OS owned files all together without some sort of
> --force flag to override the safety switch.

why pip cannot do that right now? IMO pip should be allowed to touch 3 types
of paths only: /usr/local/pythonX.Y/, ~/.local/lib/pythonX.Y/ and virtual envs.

[...]
> So without getting into the actual *method* of doing this (of which there are
> several different options with different trade offs) does this sound like
> something at all that Debian would be interested in?

to be honest, I still don't know what you're asking for. What do you
want us to do? Patch 2.7's distutils?
-- 
Piotr Ożarowski                         Debian GNU/Linux Developer
www.ozarowski.pl          www.griffith.cc           www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

[toc] | [prev] | [next] | [standalone]

#8116

From	Donald Stufft <donald@stufft.io>
Date	2016-01-22 17:00 +0100
Message-ID	<qTMhZ-62g-15@gated-at.bofh.it>
In reply to	#8115

[Multipart message — attachments visible in raw view] — view raw

> On Jan 22, 2016, at 10:36 AM, Piotr Ożarowski <piotr@debian.org> wrote:
> 
> to be honest, I still don't know what you're asking for. What do you
> want us to do? Patch 2.7's distutils?

Essentially, ensure that setuptools not distutils is used in a setup.py. There are generally three kinds of setup.py files:

1) Ones that use setuptools unconditionally - These ones you just leave alone, they are already correct and you should already have a build depends on python-setuptools.
2) Ones that conditionally use setuptools - These ones you just need to satisfy whatever condition the setup.py uses to enable setuptools. Typically this is just checking if setuptools is importable but sometimes they use environment variables or similar.
3) Ones that use distutils unconditionally - These ones you switch to making them use setuptools instead of distutils.

Now, that’s the high level overview, there’s an easier, more automatic way that could maybe just be added to pybuild (Not sure exactly how pybuild works) where instead of invoking the setup.py as:

    python setup.py install (or whatever commands/args you’re passing)

You do it as (taken from pip):

    python -c "import setuptools, tokenize;__file__='$PWD/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))” install (or whatever commands/args you’re passing).

The thing is kind of ugly, but that will install things using setuptools (just like pip does) regardless of if it imports setuptools or distutils in it’s setup.py file.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[toc] | [prev] | [next] | [standalone]

#8117

From	Piotr Ożarowski <piotr@debian.org>
Date	2016-01-22 17:20 +0100
Message-ID	<qTMBj-6ri-9@gated-at.bofh.it>
In reply to	#8116

[Donald Stufft, 2016-01-22]
> Now, that’s the high level overview, there’s an easier, more automatic
> way that could maybe just be added to pybuild (Not sure exactly how
> pybuild works) where instead of invoking the setup.py as:
> 
>     python setup.py install (or whatever commands/args you’re passing)
> 
> You do it as (taken from pip):
> 
>     python -c "import setuptools, tokenize;__file__='$PWD/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))” install (or whatever commands/args you’re passing).

let's make a deal. If you will make sure pip doesn't touch system files
(and others will not crucify me for this) - I will make sure pybuild
uses above line (if setuptools is not detected in setup.py but is listed
in Build-Depends).
-- 
Piotr Ożarowski                         Debian GNU/Linux Developer
www.ozarowski.pl          www.griffith.cc           www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

[toc] | [prev] | [next] | [standalone]

#8118

From	Paul Tagliamonte <paultag@debian.org>
Date	2016-01-22 17:50 +0100
Message-ID	<qTN4m-6Dx-5@gated-at.bofh.it>
In reply to	#8117

[Multipart message — attachments visible in raw view] — view raw

On Fri, Jan 22, 2016 at 05:18:36PM +0100, Piotr Ożarowski wrote:
> let's make a deal. If you will make sure pip doesn't touch system files
> (and others will not crucify me for this) - I will make sure pybuild
> uses above line (if setuptools is not detected in setup.py but is listed
> in Build-Depends).

Oh *come on*. That's insane.

I'm sure if you had a real and honest conversation with Donnald, there'd
be middleground. I've never found him to be the sort to bully or ignore
technical arguments.

"Give me a thing I want and I'll fix this broken thing otherwise it'll
 remain broken" is *NOT* how we do things in Debian. Flat out. This
isn't a game of poker, and the archive is not yours to bargain with.

Your argument is basically:

 - Thing X being broken is preventing Y from doing Z
 - We do Z on our platform
 - So, I'll fix X when you make Y stop doing Z

pip is used outside Debian, and another special case is tech debt. pip
touching the system sucks, but that's unrelated to the fact that thing X
sucks.

Why these issues are intertwined is confusing me, perhaps dstufft would
have been better being dishonest and not mentioned pip in his mail.

I've never used my position on the ftpteam to bully someone into doing a
thing, and I seriously hope that was a bad joke about using your
position in DPMT/PAPT to get what you want.

Cheers,
  Paul

[toc] | [prev] | [next] | [standalone]

#8121

From	Piotr Ożarowski <piotr@debian.org>
Date	2016-01-22 18:20 +0100
Message-ID	<qTNxp-77w-21@gated-at.bofh.it>
In reply to	#8118

[Paul Tagliamonte, 2016-01-22]
> I'm sure if you had a real and honest conversation with Donnald, there'd
> be middleground. I've never found him to be the sort to bully or ignore
> technical arguments.

and why we cannot find a middle ground here? Did I say something bad
about Donald? Quite the contrary, I think he's the best guy to talk to
regarding pip/setuptools/distutils.

> "Give me a thing I want and I'll fix this broken thing otherwise it'll
>  remain broken" is *NOT* how we do things in Debian. Flat out. This
> isn't a game of poker, and the archive is not yours to bargain with.

no, it's: "I already use too many hacks, but I will add another one if
you bribe me" ;P

There's nothing "broken" on our side, so I don't have to fix anything
(other than educating our users to not shoot themselves in the foot)

I want to help Daniel (I will even send a pull request if he wants),
all I need is something for me in return (and I'm also OK with adding
"Q: why don't you allow me to touch system files? A: because evil Piotr
made me" to pip's FAQ)

> I've never used my position on the ftpteam to bully someone into doing a
> thing, and I seriously hope that was a bad joke about using your
> position in DPMT/PAPT to get what you want.

it's my time and I will do in my time whatever I want. If I want to add
a hack to pybuild in my free time, I will do it when I want, you will not
bully me into doing something you think is right!
-- 
Piotr Ożarowski                         Debian GNU/Linux Developer
www.ozarowski.pl          www.griffith.cc           www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

[toc] | [prev] | [next] | [standalone]

#8120

From	Donald Stufft <donald@stufft.io>
Date	2016-01-22 18:00 +0100
Message-ID	<qTNe4-6HG-41@gated-at.bofh.it>
In reply to	#8117

[Multipart message — attachments visible in raw view] — view raw

> On Jan 22, 2016, at 11:18 AM, Piotr Ożarowski <piotr@debian.org> wrote:
> 
> let's make a deal. If you will make sure pip doesn't touch system files
> (and others will not crucify me for this) - I will make sure pybuild
> uses above line (if setuptools is not detected in setup.py but is listed
> in Build-Depends).

Regardless of what happens in this thread, pip is going to stop mucking around
in files that are owned by some other tool without some sort of opt-in/--force
style flag *and* we're going to be restructuring things to try and guide people
away from using pip outside of a virtual environment or through the user of
--user as well.

The main reason why that hasn't happened yet isn't because I'm holding those
things hostage, it's because I'm one person and my time is spread amongst
several really important tools and I have to prioritize things in accordance to
how bad it's on fire. In addition, pip is a cross platform program and we have
to ensure that whatever we do works in a cross platform manner. You mentioned
/usr/local/.../site-packages/ before, but plenty of systems don't have a
/usr/local/.../site-packages/ that their Python respects because that is a
Debian specific patch, so if we refused to install to /usr/.../site-packages/
we'd break installing at the system level on all platforms but Debian derived
ones.

What we need is a cross platform way to determine if a particular installed
Python package is "owned" by the system or if it's something we're allowed to
modify. That's going to require some effort to work out the exact best way,
but in my head the way forward on that is to use the metadata directory (the
one I'm asking y'all to start using) and add an additional piece of metadata
inside of it that just says "Hey, This is a system install" that we can inspect
and then take additional logic (like refusing to touch it without a --force)
based on that. The change I'm asking for here helps make that possible (among
other things).

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[toc] | [prev] | [next] | [standalone]

#8123

From	Piotr Ożarowski <piotr@debian.org>
Date	2016-01-22 18:40 +0100
Message-ID	<qTNQJ-7gK-3@gated-at.bofh.it>
In reply to	#8117

> let's make a deal. If you will make sure pip doesn't touch system files
> (and others will not crucify me for this) - I will make sure pybuild
> uses above line (if setuptools is not detected in setup.py but is listed
> in Build-Depends).

FTR: (after talking about it on IRC)

the "deal" was about my free time (for implementing this) and not about
not accepting patches (and Donald already mentioned he wants to
implement the cookie I want so I simply wanted him to give it higher
priority)
-- 
Piotr Ożarowski                         Debian GNU/Linux Developer
www.ozarowski.pl          www.griffith.cc           www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

[toc] | [prev] | [next] | [standalone]

#8119

From	Scott Kitterman <debian@kitterman.com>
Date	2016-01-22 18:00 +0100
Message-ID	<qTNe4-6HG-29@gated-at.bofh.it>
In reply to	#8116

[Multipart message — attachments visible in raw view] — view raw

On Friday, January 22, 2016 10:54:54 AM Donald Stufft wrote:
> > On Jan 22, 2016, at 10:36 AM, Piotr Ożarowski <piotr@debian.org> wrote:
> > 
> > to be honest, I still don't know what you're asking for. What do you
> > want us to do? Patch 2.7's distutils?
> 
> Essentially, ensure that setuptools not distutils is used in a setup.py.
> There are generally three kinds of setup.py files:
> 
> 1) Ones that use setuptools unconditionally - These ones you just leave
> alone, they are already correct and you should already have a build depends
> on python-setuptools. 2) Ones that conditionally use setuptools - These
> ones you just need to satisfy whatever condition the setup.py uses to
> enable setuptools. Typically this is just checking if setuptools is
> importable but sometimes they use environment variables or similar. 3) Ones
> that use distutils unconditionally - These ones you switch to making them
> use setuptools instead of distutils.
> 
> Now, that’s the high level overview, there’s an easier, more automatic way
> that could maybe just be added to pybuild (Not sure exactly how pybuild
> works) where instead of invoking the setup.py as:
> 
>     python setup.py install (or whatever commands/args you’re passing)
> 
> You do it as (taken from pip):
> 
>     python -c "import setuptools,
> tokenize;__file__='$PWD/setup.py';exec(compile(getattr(tokenize, 'open',
> open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))” install
> (or whatever commands/args you’re passing).
> 
> The thing is kind of ugly, but that will install things using setuptools
> (just like pip does) regardless of if it imports setuptools or distutils in
> it’s setup.py file.

I tried this and it works, but what's the big deal?

It provides a PKG-INFO file that has identical content to the old egg-info file, 
an empty dependency links file, and a top_level.txt file with the one line in 
it.

Why is this better (I'm not a huge upstream developer of Python stuff, but I do 
do some and I don't see what this gets me)?

Scott K

[toc] | [prev] | [next] | [standalone]

#8122

From	Donald Stufft <donald@stufft.io>
Date	2016-01-22 18:20 +0100
Message-ID	<qTNxp-77w-25@gated-at.bofh.it>
In reply to	#8119

[Multipart message — attachments visible in raw view] — view raw

> On Jan 22, 2016, at 11:51 AM, Scott Kitterman <debian@kitterman.com> wrote:
> 
> On Friday, January 22, 2016 10:54:54 AM Donald Stufft wrote:
>>> On Jan 22, 2016, at 10:36 AM, Piotr Ożarowski <piotr@debian.org> wrote:
>>> 
>>> to be honest, I still don't know what you're asking for. What do you
>>> want us to do? Patch 2.7's distutils?
>> 
>> Essentially, ensure that setuptools not distutils is used in a setup.py.
>> There are generally three kinds of setup.py files:
>> 
>> 1) Ones that use setuptools unconditionally - These ones you just leave
>> alone, they are already correct and you should already have a build depends
>> on python-setuptools. 2) Ones that conditionally use setuptools - These
>> ones you just need to satisfy whatever condition the setup.py uses to
>> enable setuptools. Typically this is just checking if setuptools is
>> importable but sometimes they use environment variables or similar. 3) Ones
>> that use distutils unconditionally - These ones you switch to making them
>> use setuptools instead of distutils.
>> 
>> Now, that’s the high level overview, there’s an easier, more automatic way
>> that could maybe just be added to pybuild (Not sure exactly how pybuild
>> works) where instead of invoking the setup.py as:
>> 
>>    python setup.py install (or whatever commands/args you’re passing)
>> 
>> You do it as (taken from pip):
>> 
>>    python -c "import setuptools,
>> tokenize;__file__='$PWD/setup.py';exec(compile(getattr(tokenize, 'open',
>> open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))” install
>> (or whatever commands/args you’re passing).
>> 
>> The thing is kind of ugly, but that will install things using setuptools
>> (just like pip does) regardless of if it imports setuptools or distutils in
>> it’s setup.py file.
> 
> I tried this and it works, but what's the big deal?
> 
> It provides a PKG-INFO file that has identical content to the old egg-info file,
> an empty dependency links file, and a top_level.txt file with the one line in
> it.
> 
> Why is this better (I'm not a huge upstream developer of Python stuff, but I do
> do some and I don't see what this gets me)?
> 
> Scott K

This a fair question:

1) distutils is effectively frozen and all improvements to the build system
   are either going to be inside of setuptools, or will be opt in build systems
   that (probably) won't rely on setup.py at all. This includes newer metadata
   formats and the like as well.

2) That top_level.txt is an important file, without it we have no idea that
   $SITE-PACKAGES/requests is in any way associated with the metadata file
   requests-2.9.1.egg-info, that top_level.txt tells us what the top level
   import names are for a particular project.

3) It slipped my mind that you have to pass an additional flag to setuptools
   right now to get the full file list (pip passes that flag unconditionally)
   however I'm going to poke setuptools to see about getting them to add the
   record file unconditionally to the .egg-info directory so it doesn't
   require the --record flag. (Although Debian could add it earlier if they
   wanted, but it's fine to wait for setuptools to change here).

4) Switching it to a directory allows us to add additional files/information to
   the Python level metadata, like the sort of divergent thread about getting
   pip to stop mucking with OS owned files, the most likely path forward is to
   drop an INSTALLER file inside the .egg-info directory that has some special
   value ("system"? "os"? Not sure) which pip will take as a signal that it
   really should not touch.

5) In pip, because of #2 and #3 a regular distutils installed package (which
   gives you an .egg-info file, not directory) we have now way when we're told
   to install a package named "foo" to associate that with any files on disk.
   This means that we're currently lying when we tell a user we've uninstalled
   a distutils installed package, we just remove the .egg-info file but leave
   all the other files behind. We've deprecated the ability to do this, and we
   tried to remove it all together in the recent pip 8.0 release, but we had to
   roll it back because it caused too much breakage, largely due to OS packages
   that were using distutils installed metadata.

   On the surface, it may seem counterproductive for Debian to help pip get
   better metadata about what files belong to a package so that we can
   uninstall it, but we're going to solve the "people modify system files with
   pip when they shouldn't do that" more directly and there exist use cases
   where it's perfectly fine to break your system in this way, because it's an
   emphereal system (think CI servers) where you just want to get the newest
   version of a package (maybe just one package) into the system in order to
   run some tests before you throw the whole server away or something equally
   as reasonable.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[toc] | [prev] | [next] | [standalone]

#8124

From	Scott Kitterman <debian@kitterman.com>
Date	2016-01-22 18:50 +0100
Message-ID	<qTO0q-7kv-11@gated-at.bofh.it>
In reply to	#8122

[Multipart message — attachments visible in raw view] — view raw

On Friday, January 22, 2016 12:11:27 PM Donald Stufft wrote:
...
> 3) It slipped my mind that you have to pass an additional flag to setuptools
> right now to get the full file list (pip passes that flag unconditionally)
> however I'm going to poke setuptools to see about getting them to add the
> record file unconditionally to the .egg-info directory so it doesn't
> require the --record flag. (Although Debian could add it earlier if they
> wanted, but it's fine to wait for setuptools to change here).
...
Just to document this (since I think it's significant and worth memorializing) 
from IRC:

Currently --record includes the .pyc files which is both unneeded and bad.  
Before this gets added either in setuptools or by us, this needed to be fixed.

Scott K

[toc] | [prev] | [next] | [standalone]

#8125

From	Fred Drake <fred@fdrake.net>
Date	2016-01-22 19:40 +0100
Message-ID	<qTOMO-7Ux-19@gated-at.bofh.it>
In reply to	#8124

On Fri, Jan 22, 2016 at 12:40 PM, Scott Kitterman <debian@kitterman.com> wrote:
> Currently --record includes the .pyc files which is both unneeded and bad.
> Before this gets added either in setuptools or by us, this needed to be fixed.

Why is this bad?  Isn't the point that the record file lists the files
installed by the installation process?  If the pyc files are installed
as part of the package, they should be listed.

  -Fred

-- 
Fred L. Drake, Jr.    <fred at fdrake.net>
"A storm broke loose in my mind."  --Albert Einstein

[toc] | [prev] | [next] | [standalone]

#8126

From	Fred Drake <fred@fdrake.net>
Date	2016-01-22 19:50 +0100
Message-ID	<qTOWu-7Yf-13@gated-at.bofh.it>
In reply to	#8125

On Fri, Jan 22, 2016 at 1:41 PM, Scott Kitterman <debian@kitterman.com> wrote:
> For Debian it's bad because we don't ship the .pyc files in the package they
> are managed locally by the installed python system.  They are also unnecessary
> because setuptools/pip/python is smart enough to relate .pyc files to their .py
> files without them being listed.

The .pyc files aren't shipped in the .deb, or they aren't installed
when the .deb is installed?

If they're installed, I think they should be listed.  If not installed
when the package is installed, then they definitely shouldn't be
listed.

> Part of the goal here is to make sure that regardless of the installation
> method, there's a list of files that were installed.  Since Debian packages
> don't install the .pyc files, listing them is wrong.

I agree that listing them if they aren't installed is wrong.

  -Fred

-- 
Fred L. Drake, Jr.    <fred at fdrake.net>
"A storm broke loose in my mind."  --Albert Einstein

[toc] | [prev] | [next] | [standalone]

#8128

From	Scott Kitterman <debian@kitterman.com>
Date	2016-01-22 20:00 +0100
Message-ID	<qTP6a-81w-15@gated-at.bofh.it>
In reply to	#8126

On Friday, January 22, 2016 01:47:28 PM Fred Drake wrote:
> On Fri, Jan 22, 2016 at 1:41 PM, Scott Kitterman <debian@kitterman.com> 
wrote:
> > For Debian it's bad because we don't ship the .pyc files in the package
> > they are managed locally by the installed python system.  They are also
> > unnecessary because setuptools/pip/python is smart enough to relate .pyc
> > files to their .py files without them being listed.
> 
> The .pyc files aren't shipped in the .deb, or they aren't installed
> when the .deb is installed?

They aren't shipped in the .deb, so the list of files shipped in the .deb 
shouldn't include them.

> If they're installed, I think they should be listed.  If not installed
> when the package is installed, then they definitely shouldn't be
> listed.

No  I don't think they should be.  There's no need for that and it may change.  
Then you have a maintenance nightmare.  Currently we have both python3.4 and 
python3.5 as supported versions so as part of installation *.cpython-34.pyc 
and *.cpython-35.pyc are generated on the local file system.  As the python3 
versions change, those will change and Python knows how to deal with that as 
long as the .py files are present.

Scott K

> > Part of the goal here is to make sure that regardless of the installation
> > method, there's a list of files that were installed.  Since Debian
> > packages
> > don't install the .pyc files, listing them is wrong.
> 
> I agree that listing them if they aren't installed is wrong.
> 
> 
>   -Fred

[toc] | [prev] | [next] | [standalone]

#8127

From	Scott Kitterman <debian@kitterman.com>
Date	2016-01-22 19:50 +0100
Message-ID	<qTOWu-7Yf-15@gated-at.bofh.it>
In reply to	#8125

On Friday, January 22, 2016 01:32:45 PM Fred Drake wrote:
> On Fri, Jan 22, 2016 at 12:40 PM, Scott Kitterman <debian@kitterman.com> 
wrote:
> > Currently --record includes the .pyc files which is both unneeded and bad.
> > Before this gets added either in setuptools or by us, this needed to be
> > fixed.
> Why is this bad?  Isn't the point that the record file lists the files
> installed by the installation process?  If the pyc files are installed
> as part of the package, they should be listed.

For Debian it's bad because we don't ship the .pyc files in the package they 
are managed locally by the installed python system.  They are also unnecessary 
because setuptools/pip/python is smart enough to relate .pyc files to their .py 
files without them being listed.

Part of the goal here is to make sure that regardless of the installation 
method, there's a list of files that were installed.  Since Debian packages 
don't install the .pyc files, listing them is wrong.

Scott K

P.S.  I'm subscribed to the list, so there's no need to CC me.

[toc] | [prev] | [next] | [standalone]

#8139

From	Scott Kitterman <debian@kitterman.com>
Date	2016-01-23 09:20 +0100
Message-ID	<qU1Am-8nr-13@gated-at.bofh.it>
In reply to	#8116

[Multipart message — attachments visible in raw view] — view raw

On Friday, January 22, 2016 10:54:54 AM Donald Stufft wrote:
> > On Jan 22, 2016, at 10:36 AM, Piotr Ożarowski <piotr@debian.org> wrote:
> > 
> > to be honest, I still don't know what you're asking for. What do you
> > want us to do? Patch 2.7's distutils?
> 
> Essentially, ensure that setuptools not distutils is used in a setup.py.
> There are generally three kinds of setup.py files:
> 
> 1) Ones that use setuptools unconditionally - These ones you just leave
> alone, they are already correct and you should already have a build depends
> on python-setuptools. 
> 2) Ones that conditionally use setuptools - These ones you just need to
> satisfy whatever condition the setup.py uses to enable setuptools. Typically
> this is just checking if setuptools is importable but sometimes they use
> environment variables or similar. 
> 3) Ones that use distutils unconditionally - These ones you switch to making
> them use setuptools instead of distutils.
> 
> Now, that’s the high level overview, there’s an easier, more automatic way
> that could maybe just be added to pybuild (Not sure exactly how pybuild
> works) where instead of invoking the setup.py as:
> 
>     python setup.py install (or whatever commands/args you’re passing)
> 
> You do it as (taken from pip):
> 
>     python -c "import setuptools,
> tokenize;__file__='$PWD/setup.py';exec(compile(getattr(tokenize, 'open',
> open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))” install
> (or whatever commands/args you’re passing).
> 
> The thing is kind of ugly, but that will install things using setuptools
> (just like pip does) regardless of if it imports setuptools or distutils in
> it’s setup.py file.

I decided to work on switching something I'm upstream for from distutils to 
setuptools today (since hey, it's the future) and discovered there aren't 
three kinds as listed above.  There are four:

4) Packages that use distutils features that aren't available in setuptools 
and break horribly when switched.

As a result, I think I global change is a non-starter.

In my particular case, here's what I have (snipped from setup.py) to install 
man pages and a configuration file for an application written in python3:

    data_files=[(os.path.join('share', 'man', 'man1'),
      ['policyd-spf.1']), (os.path.join('share', 'man', 'man5'),
      ['policyd-spf.conf.5']), (os.path.join('/etc', 'python-policyd-spf'),
      ['policyd-spf.conf']), (os.path.join('share', 'man', 'man5'),
      ['policyd-spf.peruser.5'])],

The man pages end up in build/bdist.linux-x86_64/egg as does the config file if 
I remove the leading "/", but it still won't install it.

codesearch.debian.net claims 720 packages using data_files.  I doubt they are 
all using that pythonically, but they'll need to be checked for breakage.

The standard Google answer to the question of how to install man pages using 
setuptools seems to be use distutils.

No way can we do any automagic switching.

Scott K

[toc] | [prev] | [next] | [standalone]

#8140

From	Scott Kitterman <debian@kitterman.com>
Date	2016-01-23 09:30 +0100
Message-ID	<qU1K2-8qT-11@gated-at.bofh.it>
In reply to	#8139

[Multipart message — attachments visible in raw view] — view raw

On Saturday, January 23, 2016 03:13:48 AM Scott Kitterman wrote:
> On Friday, January 22, 2016 10:54:54 AM Donald Stufft wrote:
> > > On Jan 22, 2016, at 10:36 AM, Piotr Ożarowski <piotr@debian.org> wrote:
> > > 
> > > to be honest, I still don't know what you're asking for. What do you
> > > want us to do? Patch 2.7's distutils?
> > 
> > Essentially, ensure that setuptools not distutils is used in a setup.py.
> > There are generally three kinds of setup.py files:
> > 
> > 1) Ones that use setuptools unconditionally - These ones you just leave
> > alone, they are already correct and you should already have a build
> > depends
> > on python-setuptools.
> > 2) Ones that conditionally use setuptools - These ones you just need to
> > satisfy whatever condition the setup.py uses to enable setuptools.
> > Typically this is just checking if setuptools is importable but sometimes
> > they use environment variables or similar.
> > 3) Ones that use distutils unconditionally - These ones you switch to
> > making them use setuptools instead of distutils.
> > 
> > Now, that’s the high level overview, there’s an easier, more automatic way
> > that could maybe just be added to pybuild (Not sure exactly how pybuild
> > 
> > works) where instead of invoking the setup.py as:
> >     python setup.py install (or whatever commands/args you’re passing)
> > 
> > You do it as (taken from pip):
> >     python -c "import setuptools,
> > 
> > tokenize;__file__='$PWD/setup.py';exec(compile(getattr(tokenize, 'open',
> > open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))” install
> > (or whatever commands/args you’re passing).
> > 
> > The thing is kind of ugly, but that will install things using setuptools
> > (just like pip does) regardless of if it imports setuptools or distutils
> > in
> > it’s setup.py file.
> 
> I decided to work on switching something I'm upstream for from distutils to
> setuptools today (since hey, it's the future) and discovered there aren't
> three kinds as listed above.  There are four:
> 
> 4) Packages that use distutils features that aren't available in setuptools
> and break horribly when switched.
> 
> As a result, I think I global change is a non-starter.
> 
> In my particular case, here's what I have (snipped from setup.py) to install
> man pages and a configuration file for an application written in python3:
> 
>     data_files=[(os.path.join('share', 'man', 'man1'),
>       ['policyd-spf.1']), (os.path.join('share', 'man', 'man5'),
>       ['policyd-spf.conf.5']), (os.path.join('/etc', 'python-policyd-spf'),
>       ['policyd-spf.conf']), (os.path.join('share', 'man', 'man5'),
>       ['policyd-spf.peruser.5'])],
> 
> The man pages end up in build/bdist.linux-x86_64/egg as does the config file
> if I remove the leading "/", but it still won't install it.
> 
> codesearch.debian.net claims 720 packages using data_files.  I doubt they
> are all using that pythonically, but they'll need to be checked for
> breakage.
> 
> The standard Google answer to the question of how to install man pages using
> setuptools seems to be use distutils.
> 
> No way can we do any automagic switching.

BTW, it's probably worth mentioning that this particularly burns because the 
setuptools describes this as a feature[1].

> Note, by the way, that this encapsulation of data files means that you can’t
> actually install data files to some arbitrary location on a user’s machine;
> this is a feature, not a bug. You can always include a script in your
> distribution that extracts and copies your the documentation or data files
> to a user-specified location, at their discretion.

Scott K

[1] http://pythonhosted.org/setuptools/setuptools.html#including-data-files

[toc] | [prev] | [next] | [standalone]

Page 1 of 2 [1] 2 Next page →

csiph-web

Amend Debian Python Proposal to Include More Python Metadata?

Contents

#8109 — Amend Debian Python Proposal to Include More Python Metadata?

#8110

#8111

#8114

#8115

#8116

#8117

#8118

#8121

#8120

#8123

#8119

#8122

#8124

#8125

#8126

#8128

#8127

#8139

#8140