Path: csiph.com!news.mixmin.net!aioe.org!news.servidellagleba.it!bofh.it!news.nic.it!robomod From: Paul Tagliamonte Newsgroups: linux.debian.maint.python Subject: Re: Amend Debian Python Proposal to Include More Python Metadata? Date: Thu, 21 Jan 2016 15:50:03 +0100 Message-ID: References: X-Mailbox-Line: From debian-python-request@lists.debian.org Thu Jan 21 14:48:35 2016 Old-Return-Path: X-Amavis-Spam-Status: No, score=-5.248 tagged_above=-10000 required=5.3 tests=[BAYES_00=-2, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FOURLA=0.1, FREEMAIL_FORGED_FROMDOMAIN=0.25, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.001, HTML_MESSAGE=2, LDO_WHITELIST=-5, MURPHY_SCAM1=0.1, RCVD_IN_DNSWL_LOW=-0.7] autolearn=no autolearn_force=no X-Policyd-Weight: DYN_NJABL=SKIP(0) NOT_IN_SBL_XBL_SPAMHAUS=-1.5 NOT_IN_BL_NJABL=-1.5 CL_IP_EQ_HELO_IP=-2 (check from: .gmail. - helo: .mail-vk0-x22b.google. - helo-domain: .google.) FROM/MX_MATCHES_HELO(DOMAIN)=-2; rate: -7 Dkim-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc:content-type; bh=62dTA/kxXhTdK14WkiFyw1VrP9kb+PLc43+wB59fblQ=; b=WrIy3tQR3xKU8X3kz0G1C5leGMmb9T+nX5AOIpZduvNSdLeFwsKVT/a1FQyCueyKad FKSD0tBnubWvEuPeiAiPAOScqla1Zfw2peI6DGmIyZMeF/s1rMOMd1SSfsVcVexL7Rp3 YuJsgHQlo8gqyLNlcobY9JioxH/1i1wpb/QB1JS/zN5pkyF7tbDJtOvFh7SGKcRK7EWp Z6ZE7fR+uiutDg8S4Y1/NKkD9nvSGldobMepJcGtG9kpEGCd24vNzeDvXiDlrM7dFSN/ e9rH2V5GMH4uhj9yMbycSvMXmHkJFatFBmyu6dUuXHX2JpEdgPGkWLAUNwcBzDfSZDDX bjbQ== X-Google-Dkim-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-type; bh=62dTA/kxXhTdK14WkiFyw1VrP9kb+PLc43+wB59fblQ=; b=cCubp4phQi8+YMMeSyKXQT0Tm8jYIdtr5VYFNYvjSdADhk7aTpuNdIhmztO1Yz7l4E dM9rSGsFJ/vueASiL6E+APZbIxRAUkXXK3RM9okYhUcUwEBzZzDjQV8VPK3UZjmj7R4U pFFJJV2mNHr/hnCxz98eZ7cWJTR5JeCVWhkZfg+rZrP5+Am9Uvh8YFJqEqzZqPjb68gB vAtrZ+lbU3STrZGqtIEnXliXiau/9Fvf5KZSlJJXlXxUe8/E3ZbKdPdrdP2h5zckVS+Z xOSsK82rwFxbyckoSspGve82yXRCvPkV92gYbOa+/4yxkxAhh0v4o4bUKmI0oOEelwJB Axjg== X-Gm-Message-State: ALoCoQm7oE8W4s7N8ytwHtzgSu297sYqGFtCxunwRZNPf03Ahwu0w50J3mkPxLQEUi9c7Q9TusrrGEzK+4dOIT2OgCMNhjPDog== X-Received: by 10.31.52.65 with SMTP id b62mr28777837vka.61.1453386783783; Thu, 21 Jan 2016 06:33:03 -0800 (PST) MIME-Version: 1.0 Sender: robomod@news.nic.it X-Google-Sender-Auth: y7l2NmlCt3nf-x_FzXTngirrrFI Content-Type: multipart/alternative; boundary=001a1143fddef7ffd40529d8faac X-Mailing-List: archive/latest/13433 List-ID: List-URL: List-Archive: https://lists.debian.org/msgid-search/CAO6P2QQ4HQcZ+t75i79zNdVV7CpFcpCi8_auO3bmFC43TvScnQ@mail.gmail.com Approved: robomod@news.nic.it Lines: 291 Organization: linux.* mail to news gateway X-Original-Cc: debian-python X-Original-Date: Thu, 21 Jan 2016 09:32:44 -0500 X-Original-Message-ID: X-Original-References: X-Original-Sender: paultag@gmail.com Xref: csiph.com linux.debian.maint.python:8110 --001a1143fddef7ffd40529d8faac Content-Type: text/plain; charset=UTF-8 Hey Donald! As far as using pip to do stuff system-wide, I wrote thoughts on http://notes.pault.ag/debian-python As for the rest of it, distutils is actually concretely shitty, and replacing it with setuptools sounds sane. Sounds like a solid idea. Perhaps we can add a Lintian warning for using distutils; it's really nasty cruft, and it's going to bite the archive sooner rather than later. Cheers, Paul On Thu, Jan 21, 2016 at 9:19 AM, Donald Stufft wrote: > Hello! > > I'd like to suggest a change to the Debian Policy around Python packages > that > will help enable the world of Python packaging to continue to progress > forward. > > First, a little bit of background: > > At the Python level there are three metadata formats for Python packaging: > > * The original, setuptools style .egg-info directories. > * The distutils style .egg-info *file* added to distutils at some point. > * The new and improved, wheel based .dist-info directories. > > The presence of any of these files will signal to Python tools that a > particular distribution has been installed, however there are two fairly > major > and important differences between the distutils style, and the other two. > > 1. The distutils style has no provisions to record what files on the system > belong to the installed distribution, making it appear to Python tooling > that there *are* no files other than the metadata file itself. > > 2. The distutils style has no provisions to include additional metadata > files > in the metadata, making it impossible to extend the python level > metadata > with additional files. > > I have a series of improvements that I'd like to make to the packaging > toolchain that will sort of build on one another, but which is not going to > function correctly with the distutils style metadata and I'm hoping that I > can > convince y'all to make it policy to default to generating one of the other > two > kinds (with varying methods, more on that later). > > Concretely the thing that this is blocking right now, is that with the > newly > released pip 8.0 I tried to make it so that pip will refuse to uninstall a > project that is installed with distutils style metadata. This is because > we do > not have any way to associate the actual .py (and others) files on disk > with > the installed metadata, so all we have ever done is just simply remove the > metadata file, making it appear as if the item is uninstalled but leaving > behind all of the actual files. However I'm going to be reverting this in a > pip 8.0.1 release because it caused a decent amount of breakage amongst > pip's > users, almost all of them people who are attempting to upgrade OS provided > packages using pip. > > Now, I know that upgrading OS provided packages using pip is less than > optimal > and I would greatly prefer that people did not do it (and I'm generally in > agreement) however if we don't enable people to do it, they'll just > continue to > use an old version of pip and file bugs. It's a non starter for pip to > make it > impossible to do. > > In addition to the uninstall bit, it also means that things like pip show > -f > return junk information for packages installed in this way. > > Beyond just (eventually) enabling pip to disable uninstallations of > distutils > based installs this will start to allow some other future changes that I > think > will be more interesting to Debian. The uninstallation of distutils based > installs comes hand in hand with pip stomping all over already existing > files > willy nilly because the way upgrading a project like that works is pip > uninstalls the metadata file that says X is installed, then it just > overwrites > over any of the files that happen to be in it's way when it installs the > newer > version. If we can remove the need for pip to gleefully overwrite files to > support these types of installed packages, then we can make it so pip will > hard fail if it attempts to overwrite an already existing file on disk. > > An additional benefit here is that by switching to using the directory > based > options, we can add additional metadata files to the installed projects, > much > like the INSTALLER file from PEP376 (IIRC). This file will likely be the > path > to having pip refuse to touch OS owned files all together without some > sort of > --force flag to override the safety switch. > > As far as compatibility goes, pip has always forced everything to be > installed > using setuptools and as far as I am aware, there's no real fallout from > doing > so. I think in 2016 it's pretty reasonable to assume that a Python project > is > capable of being installed using setuptools instead of distutils. > > So without getting into the actual *method* of doing this (of which there > are > several different options with different trade offs) does this sound like > something at all that Debian would be interested in? > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > -- :wq --001a1143fddef7ffd40529d8faac Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hey Donald!

As far as using pip to do stuff system-wide, I wrote t= houghts on=C2=A0http://notes.pault.ag/debian-python

As for the rest of it, distutils i= s actually concretely shitty, and replacing it with setuptools
sounds sane. Sounds like a solid idea.

Perhaps we can add a = Lintian warning for using distutils; it's really nasty cruft,
and it's going to bite the archive sooner rather= than later.

Cheers,
=C2=A0 Paul

On Thu, Jan 21, 2016 at 9:1= 9 AM, Donald Stufft <donald@stufft.io> wrote:
Hello!

I'd like to suggest a change to the Debian Policy around Python package= s that
will help enable the world of Python packaging to continue to progress forw= ard.

First, a little bit of background:

At the Python level there are three metadata formats for Python packaging:<= br>
* The original, setuptools style .egg-info directories.
* The distutils style .egg-info *file* added to distutils at some point. * The new and improved, wheel based .dist-info directories.

The presence of any of these files will signal to Python tools that a
particular distribution has been installed, however there are two fairly ma= jor
and important differences between the distutils style, and the other two.
1. The distutils style has no provisions to record what files on the system=
=C2=A0 =C2=A0belong to the installed distribution, making it appear to Pyth= on tooling
=C2=A0 =C2=A0that there *are* no files other than the metadata file itself.=

2. The distutils style has no provisions to include additional metadata fil= es
=C2=A0 =C2=A0in the metadata, making it impossible to extend the python lev= el metadata
=C2=A0 =C2=A0with additional files.

I have a series of improvements that I'd like to make to the packaging<= br> toolchain that will sort of build on one another, but which is not going to=
function correctly with the distutils style metadata and I'm hoping tha= t I can
convince y'all to make it policy to default to generating one of the ot= her two
kinds (with varying methods, more on that later).

Concretely the thing that this is blocking right now, is that with the newl= y
released pip 8.0 I tried to make it so that pip will refuse to uninstall a<= br> project that is installed with distutils style metadata. This is because we= do
not have any way to associate the actual .py (and others) files on disk wit= h
the installed metadata, so all we have ever done is just simply remove the<= br> metadata file, making it appear as if the item is uninstalled but leaving behind all of the actual files. However I'm going to be reverting this = in a
pip 8.0.1 release because it caused a decent amount of breakage amongst pip= 's
users, almost all of them people who are attempting to upgrade OS provided<= br> packages using pip.

Now, I know that upgrading OS provided packages using pip is less than opti= mal
and I would greatly prefer that people did not do it (and I'm generally= in
agreement) however if we don't enable people to do it, they'll just= continue to
use an old version of pip and file bugs. It's a non starter for pip to = make it
impossible to do.

In addition to the uninstall bit, it also means that things like pip show -= f
return junk information for packages installed in this way.

Beyond just (eventually) enabling pip to disable uninstallations of distuti= ls
based installs this will start to allow some other future changes that I th= ink
will be more interesting to Debian. The uninstallation of distutils based installs comes hand in hand with pip stomping all over already existing fil= es
willy nilly because the way upgrading a project like that works is pip
uninstalls the metadata file that says X is installed, then it just overwri= tes
over any of the files that happen to be in it's way when it installs th= e newer
version. If we can remove the need for pip to gleefully overwrite files to<= br> support these types of installed packages, then we can make it so pip will<= br> hard fail if it attempts to overwrite an already existing file on disk.

An additional benefit here is that by switching to using the directory base= d
options, we can add additional metadata files to the installed projects, mu= ch
like the INSTALLER file from PEP376 (IIRC). This file will likely be the pa= th
to having pip refuse to touch OS owned files all together without some sort= of
--force flag to override the safety switch.

As far as compatibility goes, pip has always forced everything to be instal= led
using setuptools and as far as I am aware, there's no real fallout from= doing
so. I think in 2016 it's pretty reasonable to assume that a Python proj= ect is
capable of being installed using setuptools instead of distutils.

So without getting into the actual *method* of doing this (of which there a= re
several different options with different trade offs) does this sound like something at all that Debian would be interested in?

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCF= A




--
:wq
--001a1143fddef7ffd40529d8faac--