Path: csiph.com!news.mixmin.net!news.unit0.net!news.panservice.it!bofh.it!news.nic.it!robomod
From: Donald Stufft <donald@stufft.io>
Newsgroups: linux.debian.maint.python
Subject: Re: Amend Debian Python Proposal to Include More Python Metadata?
Date: Fri, 22 Jan 2016 18:20:03 +0100
Message-ID: <qTNxp-77w-25@gated-at.bofh.it>
References: <qTopk-6qy-11@gated-at.bofh.it> <qTLYC-5Vz-11@gated-at.bofh.it> <qTMhZ-62g-15@gated-at.bofh.it> <qTNe4-6HG-29@gated-at.bofh.it>
Old-Return-Path: <donald@stufft.io>
Content-Type: multipart/signed; boundary="Apple-Mail=_963A53DA-5C5B-456F-ADC4-288D7DCADA05"; protocol="application/pgp-signature"; micalg=pgp-sha512
MIME-Version: 1.0 (Mac OS X Mail 9.2 \(3112\))
Approved: robomod@news.nic.it
Lines: 183
Organization: linux.* mail to news gateway
Sender: robomod@news.nic.it
Xref: csiph.com linux.debian.maint.python:8122


--Apple-Mail=_963A53DA-5C5B-456F-ADC4-288D7DCADA05
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8


> On Jan 22, 2016, at 11:51 AM, Scott Kitterman <debian@kitterman.com> =
wrote:
>=20
> On Friday, January 22, 2016 10:54:54 AM Donald Stufft wrote:
>>> On Jan 22, 2016, at 10:36 AM, Piotr O=C5=BCarowski =
<piotr@debian.org> wrote:
>>>=20
>>> to be honest, I still don't know what you're asking for. What do you
>>> want us to do? Patch 2.7's distutils?
>>=20
>> Essentially, ensure that setuptools not distutils is used in a =
setup.py.
>> There are generally three kinds of setup.py files:
>>=20
>> 1) Ones that use setuptools unconditionally - These ones you just =
leave
>> alone, they are already correct and you should already have a build =
depends
>> on python-setuptools. 2) Ones that conditionally use setuptools - =
These
>> ones you just need to satisfy whatever condition the setup.py uses to
>> enable setuptools. Typically this is just checking if setuptools is
>> importable but sometimes they use environment variables or similar. =
3) Ones
>> that use distutils unconditionally - These ones you switch to making =
them
>> use setuptools instead of distutils.
>>=20
>> Now, that=E2=80=99s the high level overview, there=E2=80=99s an =
easier, more automatic way
>> that could maybe just be added to pybuild (Not sure exactly how =
pybuild
>> works) where instead of invoking the setup.py as:
>>=20
>>    python setup.py install (or whatever commands/args you=E2=80=99re =
passing)
>>=20
>> You do it as (taken from pip):
>>=20
>>    python -c "import setuptools,
>> tokenize;__file__=3D'$PWD/setup.py';exec(compile(getattr(tokenize, =
'open',
>> open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))=E2=80=9D=
 install
>> (or whatever commands/args you=E2=80=99re passing).
>>=20
>> The thing is kind of ugly, but that will install things using =
setuptools
>> (just like pip does) regardless of if it imports setuptools or =
distutils in
>> it=E2=80=99s setup.py file.
>=20
> I tried this and it works, but what's the big deal?
>=20
> It provides a PKG-INFO file that has identical content to the old =
egg-info file,
> an empty dependency links file, and a top_level.txt file with the one =
line in
> it.
>=20
> Why is this better (I'm not a huge upstream developer of Python stuff, =
but I do
> do some and I don't see what this gets me)?
>=20
> Scott K


This a fair question:

1) distutils is effectively frozen and all improvements to the build =
system
   are either going to be inside of setuptools, or will be opt in build =
systems
   that (probably) won't rely on setup.py at all. This includes newer =
metadata
   formats and the like as well.

2) That top_level.txt is an important file, without it we have no idea =
that
   $SITE-PACKAGES/requests is in any way associated with the metadata =
file
   requests-2.9.1.egg-info, that top_level.txt tells us what the top =
level
   import names are for a particular project.

3) It slipped my mind that you have to pass an additional flag to =
setuptools
   right now to get the full file list (pip passes that flag =
unconditionally)
   however I'm going to poke setuptools to see about getting them to add =
the
   record file unconditionally to the .egg-info directory so it doesn't
   require the --record flag. (Although Debian could add it earlier if =
they
   wanted, but it's fine to wait for setuptools to change here).

4) Switching it to a directory allows us to add additional =
files/information to
   the Python level metadata, like the sort of divergent thread about =
getting
   pip to stop mucking with OS owned files, the most likely path forward =
is to
   drop an INSTALLER file inside the .egg-info directory that has some =
special
   value ("system"? "os"? Not sure) which pip will take as a signal that =
it
   really should not touch.

5) In pip, because of #2 and #3 a regular distutils installed package =
(which
   gives you an .egg-info file, not directory) we have now way when =
we're told
   to install a package named "foo" to associate that with any files on =
disk.
   This means that we're currently lying when we tell a user we've =
uninstalled
   a distutils installed package, we just remove the .egg-info file but =
leave
   all the other files behind. We've deprecated the ability to do this, =
and we
   tried to remove it all together in the recent pip 8.0 release, but we =
had to
   roll it back because it caused too much breakage, largely due to OS =
packages
   that were using distutils installed metadata.

   On the surface, it may seem counterproductive for Debian to help pip =
get
   better metadata about what files belong to a package so that we can
   uninstall it, but we're going to solve the "people modify system =
files with
   pip when they shouldn't do that" more directly and there exist use =
cases
   where it's perfectly fine to break your system in this way, because =
it's an
   emphereal system (think CI servers) where you just want to get the =
newest
   version of a package (maybe just one package) into the system in =
order to
   run some tests before you throw the whole server away or something =
equally
   as reasonable.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 =
DCFA


--Apple-Mail=_963A53DA-5C5B-456F-ADC4-288D7DCADA05
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename=signature.asc
Content-Type: application/pgp-signature;
	name=signature.asc
Content-Description: Message signed with OpenPGP using GPGMail

-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org

iQIcBAEBCgAGBQJWomLIAAoJEG48vOkzctz6pKcP/3OEi002oHjB0W60hvJatmOz
x71j8GhVPEPmIt3l+SRfVfB8DfcHywOTd16ExhX3faJM7FlHZfFsRAH8yoClVo/i
AuS5gUscqI6zKlSfpPSEHMlSnVSp6QctpQ36osDJYHkCPoFAG6BTViPzDGP4EgwN
QYhZLR4n5Q9NNHOpyHVfxiqqVBpOZyFc0mX/tdZC0n0HXdKTUloYBkx7G8hk9Xtc
4MWzATzTtGu2RwHhjvTidn0QFjFaQOQhziNiH00FA0LT0Gj/av17v6wUTDuqAEjk
1RUUrxXw//kP1tMtMXupTM5lL8CkLIo2EO/+RZShqYU6yz+2CKvLlttPIC1a1HXG
QcS+9onPEV62K1LiyrzWfdRO5wB+hO7x7pZtptbVZj6CPmWhnC1OxQSNZVptChff
RPFt+rN3cTG6hOAPROJJ1NMtJNDPEnh/FCV7H4Z83rBz7e7i/PKnT8A4AFX7o6Oo
b6orZaKBdD+XtXR2WshWP49Xkdgvv+3GEwjtlPWPpPtefn4vjs73BnvIGCkPv3E7
g+RqSWxC/KPgk7UvFLSn2TfFXMRuWIhsHIcGiVQ8hRI84XOf9aA/hMy+K218ShWs
/YzyyfzpVGOCvz7Vh3bYp7OczXAtfsBrRwNZ6vviR5PwrRWAZZ9r0nLnbK0k2G1i
/4A2epA3wVrv8JE11rDB
=gz90
-----END PGP SIGNATURE-----

--Apple-Mail=_963A53DA-5C5B-456F-ADC4-288D7DCADA05--