Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #8109

Amend Debian Python Proposal to Include More Python Metadata?

Show key headers only | View raw

[Multipart message — attachments visible in raw view] - view raw

Hello!

I'd like to suggest a change to the Debian Policy around Python packages that
will help enable the world of Python packaging to continue to progress forward.

First, a little bit of background:

At the Python level there are three metadata formats for Python packaging:

* The original, setuptools style .egg-info directories.
* The distutils style .egg-info *file* added to distutils at some point.
* The new and improved, wheel based .dist-info directories.

The presence of any of these files will signal to Python tools that a
particular distribution has been installed, however there are two fairly major
and important differences between the distutils style, and the other two.

1. The distutils style has no provisions to record what files on the system
   belong to the installed distribution, making it appear to Python tooling
   that there *are* no files other than the metadata file itself.

2. The distutils style has no provisions to include additional metadata files
   in the metadata, making it impossible to extend the python level metadata
   with additional files.

I have a series of improvements that I'd like to make to the packaging
toolchain that will sort of build on one another, but which is not going to
function correctly with the distutils style metadata and I'm hoping that I can
convince y'all to make it policy to default to generating one of the other two
kinds (with varying methods, more on that later).

Concretely the thing that this is blocking right now, is that with the newly
released pip 8.0 I tried to make it so that pip will refuse to uninstall a
project that is installed with distutils style metadata. This is because we do
not have any way to associate the actual .py (and others) files on disk with
the installed metadata, so all we have ever done is just simply remove the
metadata file, making it appear as if the item is uninstalled but leaving
behind all of the actual files. However I'm going to be reverting this in a
pip 8.0.1 release because it caused a decent amount of breakage amongst pip's
users, almost all of them people who are attempting to upgrade OS provided
packages using pip.

Now, I know that upgrading OS provided packages using pip is less than optimal
and I would greatly prefer that people did not do it (and I'm generally in
agreement) however if we don't enable people to do it, they'll just continue to
use an old version of pip and file bugs. It's a non starter for pip to make it
impossible to do.

In addition to the uninstall bit, it also means that things like pip show -f
return junk information for packages installed in this way.

Beyond just (eventually) enabling pip to disable uninstallations of distutils
based installs this will start to allow some other future changes that I think
will be more interesting to Debian. The uninstallation of distutils based
installs comes hand in hand with pip stomping all over already existing files
willy nilly because the way upgrading a project like that works is pip
uninstalls the metadata file that says X is installed, then it just overwrites
over any of the files that happen to be in it's way when it installs the newer
version. If we can remove the need for pip to gleefully overwrite files to
support these types of installed packages, then we can make it so pip will
hard fail if it attempts to overwrite an already existing file on disk.

An additional benefit here is that by switching to using the directory based
options, we can add additional metadata files to the installed projects, much
like the INSTALLER file from PEP376 (IIRC). This file will likely be the path
to having pip refuse to touch OS owned files all together without some sort of
--force flag to override the safety switch.

As far as compatibility goes, pip has always forced everything to be installed
using setuptools and as far as I am aware, there's no real fallout from doing
so. I think in 2016 it's pretty reasonable to assume that a Python project is
capable of being installed using setuptools instead of distutils.

So without getting into the actual *method* of doing this (of which there are
several different options with different trade offs) does this sound like
something at all that Debian would be interested in?

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Back to linux.debian.maint.python | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

Amend Debian Python Proposal to Include More Python Metadata? Donald Stufft <donald@stufft.io> - 2016-01-21 15:30 +0100
  Re: Amend Debian Python Proposal to Include More Python Metadata? Paul Tagliamonte <paultag@debian.org> - 2016-01-21 15:50 +0100
    Re: Amend Debian Python Proposal to Include More Python Metadata? Donald Stufft <donald@stufft.io> - 2016-01-21 16:10 +0100
      Re: Amend Debian Python Proposal to Include More Python Metadata? Scott Kitterman <debian@kitterman.com> - 2016-01-21 17:00 +0100
  Re: Amend Debian Python Proposal to Include More Python Metadata? Piotr Ożarowski <piotr@debian.org> - 2016-01-22 16:40 +0100
    Re: Amend Debian Python Proposal to Include More Python Metadata? Donald Stufft <donald@stufft.io> - 2016-01-22 17:00 +0100
      Re: Amend Debian Python Proposal to Include More Python Metadata? Piotr Ożarowski <piotr@debian.org> - 2016-01-22 17:20 +0100
        Re: Amend Debian Python Proposal to Include More Python Metadata? Paul Tagliamonte <paultag@debian.org> - 2016-01-22 17:50 +0100
          Re: Amend Debian Python Proposal to Include More Python Metadata? Piotr Ożarowski <piotr@debian.org> - 2016-01-22 18:20 +0100
        Re: Amend Debian Python Proposal to Include More Python Metadata? Donald Stufft <donald@stufft.io> - 2016-01-22 18:00 +0100
        Re: Amend Debian Python Proposal to Include More Python Metadata? Piotr Ożarowski <piotr@debian.org> - 2016-01-22 18:40 +0100
      Re: Amend Debian Python Proposal to Include More Python Metadata? Scott Kitterman <debian@kitterman.com> - 2016-01-22 18:00 +0100
        Re: Amend Debian Python Proposal to Include More Python Metadata? Donald Stufft <donald@stufft.io> - 2016-01-22 18:20 +0100
          Re: Amend Debian Python Proposal to Include More Python Metadata? Scott Kitterman <debian@kitterman.com> - 2016-01-22 18:50 +0100
            Re: Amend Debian Python Proposal to Include More Python Metadata? Fred Drake <fred@fdrake.net> - 2016-01-22 19:40 +0100
              Re: Amend Debian Python Proposal to Include More Python Metadata? Fred Drake <fred@fdrake.net> - 2016-01-22 19:50 +0100
                Re: Amend Debian Python Proposal to Include More Python Metadata? Scott Kitterman <debian@kitterman.com> - 2016-01-22 20:00 +0100
              Re: Amend Debian Python Proposal to Include More Python Metadata? Scott Kitterman <debian@kitterman.com> - 2016-01-22 19:50 +0100
      Re: Amend Debian Python Proposal to Include More Python Metadata? Scott Kitterman <debian@kitterman.com> - 2016-01-23 09:20 +0100
        Re: Amend Debian Python Proposal to Include More Python Metadata? Scott Kitterman <debian@kitterman.com> - 2016-01-23 09:30 +0100
  Re: Amend Debian Python Proposal to Include More Python Metadata? Barry Warsaw <barry@debian.org> - 2016-01-22 23:40 +0100
    Re: Amend Debian Python Proposal to Include More Python Metadata? Donald Stufft <donald@stufft.io> - 2016-01-23 00:00 +0100
      Re: Amend Debian Python Proposal to Include More Python Metadata? Barry Warsaw <barry@debian.org> - 2016-01-23 00:10 +0100
        Re: Amend Debian Python Proposal to Include More Python Metadata? Donald Stufft <donald@stufft.io> - 2016-01-23 00:20 +0100
      Re: Amend Debian Python Proposal to Include More Python Metadata? Scott Kitterman <debian@kitterman.com> - 2016-01-23 00:10 +0100
        Re: Amend Debian Python Proposal to Include More Python Metadata? Donald Stufft <donald@stufft.io> - 2016-01-23 00:30 +0100
          Re: Amend Debian Python Proposal to Include More Python Metadata? Scott Kitterman <debian@kitterman.com> - 2016-01-23 01:20 +0100
            Re: Amend Debian Python Proposal to Include More Python Metadata? Donald Stufft <donald@stufft.io> - 2016-01-23 02:00 +0100
          pip as distutils/setuptools/setup.py wrapper Piotr Ożarowski <piotr@debian.org> - 2016-01-26 19:30 +0100

csiph-web