Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #8122

Re: Amend Debian Python Proposal to Include More Python Metadata?

Show key headers only | View raw

[Multipart message — attachments visible in raw view] - view raw

> On Jan 22, 2016, at 11:51 AM, Scott Kitterman <debian@kitterman.com> wrote:
> 
> On Friday, January 22, 2016 10:54:54 AM Donald Stufft wrote:
>>> On Jan 22, 2016, at 10:36 AM, Piotr Ożarowski <piotr@debian.org> wrote:
>>> 
>>> to be honest, I still don't know what you're asking for. What do you
>>> want us to do? Patch 2.7's distutils?
>> 
>> Essentially, ensure that setuptools not distutils is used in a setup.py.
>> There are generally three kinds of setup.py files:
>> 
>> 1) Ones that use setuptools unconditionally - These ones you just leave
>> alone, they are already correct and you should already have a build depends
>> on python-setuptools. 2) Ones that conditionally use setuptools - These
>> ones you just need to satisfy whatever condition the setup.py uses to
>> enable setuptools. Typically this is just checking if setuptools is
>> importable but sometimes they use environment variables or similar. 3) Ones
>> that use distutils unconditionally - These ones you switch to making them
>> use setuptools instead of distutils.
>> 
>> Now, that’s the high level overview, there’s an easier, more automatic way
>> that could maybe just be added to pybuild (Not sure exactly how pybuild
>> works) where instead of invoking the setup.py as:
>> 
>>    python setup.py install (or whatever commands/args you’re passing)
>> 
>> You do it as (taken from pip):
>> 
>>    python -c "import setuptools,
>> tokenize;__file__='$PWD/setup.py';exec(compile(getattr(tokenize, 'open',
>> open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))” install
>> (or whatever commands/args you’re passing).
>> 
>> The thing is kind of ugly, but that will install things using setuptools
>> (just like pip does) regardless of if it imports setuptools or distutils in
>> it’s setup.py file.
> 
> I tried this and it works, but what's the big deal?
> 
> It provides a PKG-INFO file that has identical content to the old egg-info file,
> an empty dependency links file, and a top_level.txt file with the one line in
> it.
> 
> Why is this better (I'm not a huge upstream developer of Python stuff, but I do
> do some and I don't see what this gets me)?
> 
> Scott K


This a fair question:

1) distutils is effectively frozen and all improvements to the build system
   are either going to be inside of setuptools, or will be opt in build systems
   that (probably) won't rely on setup.py at all. This includes newer metadata
   formats and the like as well.

2) That top_level.txt is an important file, without it we have no idea that
   $SITE-PACKAGES/requests is in any way associated with the metadata file
   requests-2.9.1.egg-info, that top_level.txt tells us what the top level
   import names are for a particular project.

3) It slipped my mind that you have to pass an additional flag to setuptools
   right now to get the full file list (pip passes that flag unconditionally)
   however I'm going to poke setuptools to see about getting them to add the
   record file unconditionally to the .egg-info directory so it doesn't
   require the --record flag. (Although Debian could add it earlier if they
   wanted, but it's fine to wait for setuptools to change here).

4) Switching it to a directory allows us to add additional files/information to
   the Python level metadata, like the sort of divergent thread about getting
   pip to stop mucking with OS owned files, the most likely path forward is to
   drop an INSTALLER file inside the .egg-info directory that has some special
   value ("system"? "os"? Not sure) which pip will take as a signal that it
   really should not touch.

5) In pip, because of #2 and #3 a regular distutils installed package (which
   gives you an .egg-info file, not directory) we have now way when we're told
   to install a package named "foo" to associate that with any files on disk.
   This means that we're currently lying when we tell a user we've uninstalled
   a distutils installed package, we just remove the .egg-info file but leave
   all the other files behind. We've deprecated the ability to do this, and we
   tried to remove it all together in the recent pip 8.0 release, but we had to
   roll it back because it caused too much breakage, largely due to OS packages
   that were using distutils installed metadata.

   On the surface, it may seem counterproductive for Debian to help pip get
   better metadata about what files belong to a package so that we can
   uninstall it, but we're going to solve the "people modify system files with
   pip when they shouldn't do that" more directly and there exist use cases
   where it's perfectly fine to break your system in this way, because it's an
   emphereal system (think CI servers) where you just want to get the newest
   version of a package (maybe just one package) into the system in order to
   run some tests before you throw the whole server away or something equally
   as reasonable.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Back to linux.debian.maint.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Amend Debian Python Proposal to Include More Python Metadata? Donald Stufft <donald@stufft.io> - 2016-01-21 15:30 +0100
  Re: Amend Debian Python Proposal to Include More Python Metadata? Paul Tagliamonte <paultag@debian.org> - 2016-01-21 15:50 +0100
    Re: Amend Debian Python Proposal to Include More Python Metadata? Donald Stufft <donald@stufft.io> - 2016-01-21 16:10 +0100
      Re: Amend Debian Python Proposal to Include More Python Metadata? Scott Kitterman <debian@kitterman.com> - 2016-01-21 17:00 +0100
  Re: Amend Debian Python Proposal to Include More Python Metadata? Piotr Ożarowski <piotr@debian.org> - 2016-01-22 16:40 +0100
    Re: Amend Debian Python Proposal to Include More Python Metadata? Donald Stufft <donald@stufft.io> - 2016-01-22 17:00 +0100
      Re: Amend Debian Python Proposal to Include More Python Metadata? Piotr Ożarowski <piotr@debian.org> - 2016-01-22 17:20 +0100
        Re: Amend Debian Python Proposal to Include More Python Metadata? Paul Tagliamonte <paultag@debian.org> - 2016-01-22 17:50 +0100
          Re: Amend Debian Python Proposal to Include More Python Metadata? Piotr Ożarowski <piotr@debian.org> - 2016-01-22 18:20 +0100
        Re: Amend Debian Python Proposal to Include More Python Metadata? Donald Stufft <donald@stufft.io> - 2016-01-22 18:00 +0100
        Re: Amend Debian Python Proposal to Include More Python Metadata? Piotr Ożarowski <piotr@debian.org> - 2016-01-22 18:40 +0100
      Re: Amend Debian Python Proposal to Include More Python Metadata? Scott Kitterman <debian@kitterman.com> - 2016-01-22 18:00 +0100
        Re: Amend Debian Python Proposal to Include More Python Metadata? Donald Stufft <donald@stufft.io> - 2016-01-22 18:20 +0100
          Re: Amend Debian Python Proposal to Include More Python Metadata? Scott Kitterman <debian@kitterman.com> - 2016-01-22 18:50 +0100
            Re: Amend Debian Python Proposal to Include More Python Metadata? Fred Drake <fred@fdrake.net> - 2016-01-22 19:40 +0100
              Re: Amend Debian Python Proposal to Include More Python Metadata? Fred Drake <fred@fdrake.net> - 2016-01-22 19:50 +0100
                Re: Amend Debian Python Proposal to Include More Python Metadata? Scott Kitterman <debian@kitterman.com> - 2016-01-22 20:00 +0100
              Re: Amend Debian Python Proposal to Include More Python Metadata? Scott Kitterman <debian@kitterman.com> - 2016-01-22 19:50 +0100
      Re: Amend Debian Python Proposal to Include More Python Metadata? Scott Kitterman <debian@kitterman.com> - 2016-01-23 09:20 +0100
        Re: Amend Debian Python Proposal to Include More Python Metadata? Scott Kitterman <debian@kitterman.com> - 2016-01-23 09:30 +0100
  Re: Amend Debian Python Proposal to Include More Python Metadata? Barry Warsaw <barry@debian.org> - 2016-01-22 23:40 +0100
    Re: Amend Debian Python Proposal to Include More Python Metadata? Donald Stufft <donald@stufft.io> - 2016-01-23 00:00 +0100
      Re: Amend Debian Python Proposal to Include More Python Metadata? Barry Warsaw <barry@debian.org> - 2016-01-23 00:10 +0100
        Re: Amend Debian Python Proposal to Include More Python Metadata? Donald Stufft <donald@stufft.io> - 2016-01-23 00:20 +0100
      Re: Amend Debian Python Proposal to Include More Python Metadata? Scott Kitterman <debian@kitterman.com> - 2016-01-23 00:10 +0100
        Re: Amend Debian Python Proposal to Include More Python Metadata? Donald Stufft <donald@stufft.io> - 2016-01-23 00:30 +0100
          Re: Amend Debian Python Proposal to Include More Python Metadata? Scott Kitterman <debian@kitterman.com> - 2016-01-23 01:20 +0100
            Re: Amend Debian Python Proposal to Include More Python Metadata? Donald Stufft <donald@stufft.io> - 2016-01-23 02:00 +0100
          pip as distutils/setuptools/setup.py wrapper Piotr Ożarowski <piotr@debian.org> - 2016-01-26 19:30 +0100

csiph-web