Path: csiph.com!news.mixmin.net!aioe.org!bofh.it!news.nic.it!robomod
From: Donald Stufft <donald@stufft.io>
Newsgroups: linux.debian.maint.python
Subject: Re: Amend Debian Python Proposal to Include More Python Metadata?
Date: Sat, 23 Jan 2016 00:00:01 +0100
Message-ID: <qTSQp-28S-7@gated-at.bofh.it>
References: <qTopk-6qy-11@gated-at.bofh.it> <qTSx5-20U-29@gated-at.bofh.it>
Old-Return-Path: <donald@stufft.io>
Content-Type: multipart/signed; boundary="Apple-Mail=_16454A55-AD12-41DA-9BEA-39DAE545CA0B"; protocol="application/pgp-signature"; micalg=pgp-sha512
MIME-Version: 1.0 (Mac OS X Mail 9.2 \(3112\))
Approved: robomod@news.nic.it
Lines: 257
Organization: linux.* mail to news gateway
Sender: robomod@news.nic.it
Xref: csiph.com linux.debian.maint.python:8131


--Apple-Mail=_16454A55-AD12-41DA-9BEA-39DAE545CA0B
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8


> On Jan 22, 2016, at 5:30 PM, Barry Warsaw <barry@debian.org> wrote:
>=20
> Hey Donald, thanks for starting this conversation.  I for one am super
> appreciative of all the consideration you give for Debian's little =
slice of
> the world.
>=20
> There's a lot to unpack in this thread, and I'm a little under the =
weather[1],
> so hopefully this makes sense.
>=20
> Big +1 for recording the files that get installed via the =
.egg-info/RECORD
> file.  As you know, I've been working on dirtbike, which is a =
"rewheeling"
> tool to turn an installed package into a .whl.  While Debian tightly =
controls
> via policy the set of wheels we'll allow into the archive, dirtbike =
has code
> for parsing the RECORD file.  Unfortunately this is never exercised in
> practice because we don't have RECORD files - at least not for the =
packages we
> care about[2].
>=20
> Big +1 for using setuptools everywhere.  By my count, of the packages =
that I
> happen to randomly have installed on my Ubuntu 16.04 system, I have 67
> .egg-info files and 113 .egg-info directories.  I'd rather have =
.egg-info
> directories everywhere.
>=20
> +1 for a lintian warning if distutils is used.  I guess I'm +0 on =
forcing that
> through pybuild because it'll be unobvious and mysterious, and kind of =
lets
> upstreams off the hook.  I'd mildly prefer to patch packages that use
> distutils because that's much more discoverable, but I can appreciate =
that
> that's a lot of work we'd be imposing on maintainers, so I won't argue =
this
> too much (other than to say that if pybuild forces it, let's =
definitely
> document this in its manpage!).
>=20
> On Jan 22, 2016, at 12:40 PM, Scott Kitterman wrote:
>=20
>> Currently --record includes the .pyc files which is both unneeded and =
bad.
>> Before this gets added either in setuptools or by us, this needed to =
be
>> fixed.
>=20
> +1 for (I think) another reason than has already been discussed.  We =
won't be
> generating .egg-info directories on the end-user's machine, but =
instead the
> machine the package is built on.  That could be a maintainer's own =
system or a
> central build machine depending on various conditions.  But because =
the pycs
> are generated on the end-user's machine, we actually don't know what =
pycs will
> be generated when the debs are installed, so the egg-info/RECORD file =
*can't*
> contain them, at least not accurately.

FWIW It appears that if you do setup.py install =E2=80=94no-compile to =
tell distutils
not to compile the .pyc during .deb build time, I believe that as is it =
will
just omit the .pyc files. Not sure if that=E2=80=99s workable for Debian =
or not.

>=20
> On Jan 22, 2016, at 11:54 AM, Donald Stufft wrote:
>=20
>> Regardless of what happens in this thread, pip is going to stop =
mucking
>> around in files that are owned by some other tool without some sort =
of
>> opt-in/--force style flag *and* we're going to be restructuring =
things to try
>> and guide people away from using pip outside of a virtual environment =
or
>> through the user of --user as well.
>=20
> Of course, I'd still like --user to be the default[3].  I think that's =
still
> the eventual goal for pip, but isn't yet implemented because =
$priorities.

Yes, still a goal (it=E2=80=99s not as simple as =E2=80=94user will be =
the default, but that=E2=80=99s
the outcome for a non root account in a system Python).

>=20
>> A more controversial way that comes with possibly some extra benefits =
(which
>> Debian may not care about) is to use ``pip`` itself as the build tool =
rather
>> than directly invoking setup.py. In this pip would be responsible for =
mucking
>> around with the distutils/setuptools mess instead of that needing to =
be
>> handled by Debian specific tooling.
>=20
> I'd like to better understand how this would work.  IIUC, the Fedora =
ecosystem
> is making or already has made this switch, but I don't know how it =
works.
> Obviously, we don't want to install wheels into =
/usr/lib/python3/dist-packages.

I haven=E2=80=99t actually personally seen if they=E2=80=99ve done it, =
but the Fedora maintainer
of the python-pip told me they were doing it, and they had a patch to =
pip to
add a feature they needed for it.

Forget that pip can fetch files from PyPI and install them for a moment =
and
consider the command ``pip install .``. Fundamentally this is similar to =
the
command ``make install`` right? It builds the files and then installs =
them
into the final location. As I understand it, the standard way for a =
project
like Debian to install a project using a =E2=80=9Cregular=E2=80=9D =
Makefile install like that
is to do ``make DESTDIR=3D/tmp/some-tempdir install`` so that it invokes =
the
build system, sets all the paths in the resulting build system to the =
final
destination, but then right before it actually copies files to the final =
paths
it just appends a directory in front of them to let you stage the files =
in
a different location instead.

We already have an option like this, the =E2=80=94root option which will =
just append
a different prefix to all of the installation paths. So essentially =
instead
of invoking ``python setup.py install =E2=80=94root /tmp/something/`` =
which is what
I think you=E2=80=99re doing now, you=E2=80=99d do ``pip install =E2=80=94=
root /tmp/something/ =E2=80=94no-deps .``
and it=E2=80=99d just work similarly to what you have now, except pip =
would be
responsible for interacting with the Python build system.

Slavek had said there was some bug preventing him from using =E2=80=94root=
 at the time
and I=E2=80=99m not entirely sure what that was off the top of my head =
but the issue
he opened was https://github.com/pypa/pip/issues/1351. I=E2=80=99m not =
100% sure what
the right combination of options are, but if it=E2=80=99s something =
y=E2=80=99all are interested
in I can certainly figure it out (and add anything needed to make it =
happen).

>=20
> FTR, I am still working on updating Debian's pip.  I'm slowly shaving =
the yak,
> but there are still a few things to do.  If you want to help, get in =
touch.
>=20
>> [3] Import pkg_resources is not the fasest thing in the world, it has =
to scan
>> every directory on sys.path looking for things to activate and it =
comes with
>> a bunch of side effects. This happens implicitly for any project =
using
>> console scripts.
>=20
> Which frankly sucks.  It's also fragile.  Every once in a while a =
broken
> package gets uploaded that breaks pkg_resources, and it's not easy to =
debug or
> repair.  I really hope Brett can fix this when/if he builds this =
functionality
> into the stdlib.

For console scripts we don=E2=80=99t actually need anything added, this =
is the console
scripts that pip generates when it installs from a Wheel (either =
explicitly
from PyPI or through an implicitly compiled wheel): =
https://bpaste.net/show/247fe79d91c4

>=20
> Cheers,
> -Barry
>=20
> [1] double meaning: I have a cold and we're in the early stages of an =
historic
> snow storm!
>=20
> [2] dirtbike has two fallbacks, both of which use `dpkg -L` to get the =
list of
> installed files.  The first fallback uses `dpkg -S` to find the binary =
package
> that contains the Python package's .egg-info file/directory (doesn't =
matter
> for this purpose).  pkg_resources doesn't have one of those.  The =
second
> fallback imports the Python package and looks for the binary package
> containing the top-level directory.  It's all rather ugly, and I'd =
like to
> just use the .egg-info/RECORD file, but I suspect I'll still need the =
import
> fallback for pkg_resources.
>=20
> [3] https://github.com/pypa/pip/issues/1668


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 =
DCFA


--Apple-Mail=_16454A55-AD12-41DA-9BEA-39DAE545CA0B
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename=signature.asc
Content-Type: application/pgp-signature;
	name=signature.asc
Content-Description: Message signed with OpenPGP using GPGMail

-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org

iQIcBAEBCgAGBQJWorIuAAoJEG48vOkzctz6ajMP/RtP0I9K7r9igqcwmYJ2SvzJ
bTwLVByWtmTIVA2/xDz1s+fSpUGdXb5a99/9L0ovCLMIReDTrh93a/X70NOnle7y
SgEcE0HB5lNBzVd5u7xjl1h6o9M8dNcXH107/s0wo0LiNqeaN8F7CBXjXJ92V+4x
+6P4EQlA32b2ZPx4p3E6dnRZviLVfajHTzPpyuxvMbOYCZYh0fLCkGvgcjXPZU6q
lGf4LIoDvs09iSRcokWpl9FS58HkHINrl6b54MFcZ8ATZfNkp3wNp5ayy8DsvvyB
yhmrfRxwXl9TUQjPG6UYSnNVDZFDNTzyXunagz32viXMskX2+DkFwQYbqrUvt5yL
jUNKz+e4+xE4sVmT9d1PLR8LH6sq4aAL1ojrZq6ISfOCkNmtF873DlD5KNVz7NIG
JKnsobodSrXc5EYirXnRzBw2IT33ySgFguAvUNsZRAso41g2ELZZRrjBKf1RbneR
yRPERT5Te5bolzCxlH16meepcKwOliB7pghNeuqf0l6L4zrpGVCgtyfg1tYGS/Op
wsU1CSCCvm269lJ6rHqt4b72WFYyn93sjyefvhFpp8ePJI1pjLDMJgMrRbhvle7+
J+GlVgTofYrXO1r+3bD+xXVylGIUKo0RnVNUByNZT9HSE+GQsqnNWESyCyau8FiF
EigUnW0cLc7n9cpOTUGM
=qBdd
-----END PGP SIGNATURE-----

--Apple-Mail=_16454A55-AD12-41DA-9BEA-39DAE545CA0B--