Path: csiph.com!news.freedyn.net!aioe.org!bofh.it!news.nic.it!robomod From: Donald Stufft Newsgroups: linux.debian.maint.python Subject: Re: Amend Debian Python Proposal to Include More Python Metadata? Date: Thu, 21 Jan 2016 16:10:03 +0100 Message-ID: References: X-Original-To: Paul Tagliamonte X-Mailbox-Line: From debian-python-request@lists.debian.org Thu Jan 21 15:01:23 2016 Old-Return-Path: X-Amavis-Spam-Status: No, score=-10.7 tagged_above=-10000 required=5.3 tests=[BAYES_00=-2, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FOURLA=0.1, HTML_MESSAGE=2, LDO_WHITELIST=-5, MURPHY_DRUGS_REL8=0.02, PGPSIGNATURE=-5, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01] autolearn=ham autolearn_force=no X-Policyd-Weight: using cached result; rate: -7 Dkim-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=stufft.io; h=cc :content-type:date:from:in-reply-to:message-id:mime-version :references:subject:to:x-sasl-enc:x-sasl-enc; s=mesmtp; bh=FPc8g oFL00xxGm9JIATEG6fItaM=; b=GMJNleFxnTay7yoAMyr6xhNVy8Xj1aZFU4K58 tsoL2j66A7ufwEF9rj6th9z45RYFo3739nmzbtG4YNIbJoOTdsWWqGPJvVkqoNHB I7rCudLc5NcnQMiSb8hdF4EWcdGOlZArRlIRwgru0TXL+9LGduzU5i6+p0dOEXw1 STosk0= Dkim-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-sasl-enc :x-sasl-enc; s=smtpout; bh=FPc8goFL00xxGm9JIATEG6fItaM=; b=eEQ+0 Z9GidyzBRd/GnvdBqVfjpgXfPD2orgFkeLRdkvDNICw1ucNS/brjx202bhR8+Mp4 7Dsv+XUqemdRladmNZWF747uB8/r5HC2hYW91lE1gT55AngTUQTAMVFnq/d97wNX eI41VuavB2n2Js2PUzRwL6Puz9AxyFPe0/KKs4= X-Sasl-Enc: PyZk7o32e2txyTAgkAy4g03JB99VBtmVcKMROvMQsbQu 1453388464 MIME-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Content-Type: multipart/signed; boundary="Apple-Mail=_9DF61F7D-36B4-4691-8CD9-B8CF32FA54D6"; protocol="application/pgp-signature"; micalg=pgp-sha512 X-PGP-Agent: GPGMail 2.6b2 X-Mailer: Apple Mail (2.3112) X-Mailing-List: archive/latest/13434 List-ID: List-URL: List-Archive: https://lists.debian.org/msgid-search/940F3FC6-7197-4CE9-B7D0-B40A20D7F409@stufft.io Approved: robomod@news.nic.it Lines: 249 Organization: linux.* mail to news gateway Sender: robomod@news.nic.it X-Original-Cc: debian-python X-Original-Date: Thu, 21 Jan 2016 10:00:53 -0500 X-Original-Message-ID: <940F3FC6-7197-4CE9-B7D0-B40A20D7F409@stufft.io> X-Original-References: Xref: csiph.com linux.debian.maint.python:8111 --Apple-Mail=_9DF61F7D-36B4-4691-8CD9-B8CF32FA54D6 Content-Type: multipart/alternative; boundary="Apple-Mail=_09936913-B007-45B6-9510-D120A50CEF82" --Apple-Mail=_09936913-B007-45B6-9510-D120A50CEF82 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > On Jan 21, 2016, at 9:32 AM, Paul Tagliamonte = wrote: >=20 > Hey Donald! >=20 > As far as using pip to do stuff system-wide, I wrote thoughts on = http://notes.pault.ag/debian-python = I just want to be clear, that I totally agree with this. The reason I = want to ensure we can enable this isn't because I think doing so is a great idea = and everyone should be doing it. People *are* doing it and we (pip) have to = be careful about what we make impossible to do because a possible result of = going too break happy is people just never upgrade and things start to = stagnate again. So for pip at the system level, my goal is try and push people away from = it as much as we can without flat out making it impossible *and* try to make = it as "safe" to do so as possible when people decide to do it anyways. >=20 > As for the rest of it, distutils is actually concretely shitty, and = replacing it with setuptools > sounds sane. Sounds like a solid idea. >=20 > Perhaps we can add a Lintian warning for using distutils; it's really = nasty cruft, > and it's going to bite the archive sooner rather than later. >=20 As far as options for how to implement this goes, I'm not sure what best = fits with Debian so I'll leave that up to y'all, but here are a few options = that I can think of. A lintian check is probably a pretty good thing, I would guess that it = would specifically check for the existence of a .egg-info or .dist-info = directory (or check to the non-existence of a .egg-info file). In pip the way we make this work is with a bit of a gross little thunk = that ensures setuptools is imported prior to setup.py being invoked which = forces all setup.py calls to use setuptools instead of distutils even if they = import from distutils. Of course, many setup.py already unconditionally use setuptools, and = many more conditionally use it (typically falling back to distutils) so for many = projects it may be enough to just add python-setuptools to build-depends. For = projects that unconditionally use distutils, either the pip way or patching = setup.py might be a reasonable way forward. A more controversial way that comes with possibly some extra benefits = (which Debian may not care about) is to use ``pip`` itself as the build tool = rather than directly invoking setup.py. In this pip would be responsible for = mucking around with the distutils/setuptools mess instead of that needing to be = handled by Debian specific tooling. You could even leverage this to remove some = of the runtime dependencies on python-pkg-resources and speed up some python = using software if you let pip build a wheel as part of the buld process = [1][2][3]. I suspect pip would need to grow some additional options to make it = suitable for you in this role, but I would be more than willing to help if this is = desired (or not, this proposal isn't about trying to ram pip down anyones = throat!). [1] To be clear, this isn't about adding wheels for everything to the = archive, it would only be using Wheel as an intermediate step in the build = process. [2] Console scripts generated by setuptools entry wrappers when = installed by setuptools have a dependency on pkg_resources at runtime, however = those same scripts, when installed by pip from a wheel, do not. This would = enable removing the runtime dependency on pkg_resources for anything that = only has it for console scripts. [3] Import pkg_resources is not the fasest thing in the world, it has to = scan every directory on sys.path looking for things to activate and it = comes with a bunch of side effects. This happens implicitly for any = project using console scripts. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 = DCFA --Apple-Mail=_09936913-B007-45B6-9510-D120A50CEF82 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
On Jan 21, 2016, at 9:32 AM, Paul Tagliamonte <paultag@debian.org> = wrote:

Hey Donald!

As = far as using pip to do stuff system-wide, I wrote thoughts on http://notes.pault.ag/debian-python


I = just want to be clear, that I totally agree with this. The reason I want = to
ensure we can enable this isn't because I think doing so is = a great idea and
everyone should be doing it. People *are* = doing it and we (pip) have to be
careful about what we make = impossible to do because a possible result of going
too break = happy is people just never upgrade and things start to = stagnate
again.

So for pip = at the system level, my goal is try and push people away from it = as
much as we can without flat out making it impossible *and* = try to make it as
"safe" to do so as possible when people = decide to do it anyways.


As = for the rest of it, distutils is actually concretely shitty, and = replacing it with setuptools
sounds = sane. Sounds like a solid idea.

Perhaps we can add a Lintian = warning for using distutils; it's really nasty cruft,
and it's going to bite the archive sooner rather = than later.


As far as options for how to implement this = goes, I'm not sure what best fits
with Debian so I'll leave = that up to y'all, but here are a few options that I
can think = of.

A lintian check is probably a = pretty good thing, I would guess that it would
specifically = check for the existence of a .egg-info or .dist-info = directory
(or check to the non-existence of a .egg-info = file).

In pip the way we make this = work is with a bit of a gross little thunk that
ensures = setuptools is imported prior to setup.py being invoked which forces = all
setup.py calls to use setuptools instead of distutils even = if they import from
distutils.

Of course, many setup.py already unconditionally = use setuptools, and many more
conditionally use it (typically = falling back to distutils) so for many projects
it may be = enough to just add python-setuptools to build-depends. For = projects
that unconditionally use distutils, either the pip = way or patching setup.py
might be a reasonable way = forward.

A more controversial way = that comes with possibly some extra benefits (which
Debian may = not care about) is to use ``pip`` itself as the build tool = rather
than directly invoking setup.py. In this pip would be = responsible for mucking
around with the distutils/setuptools = mess instead of that needing to be handled
by Debian specific = tooling. You could even leverage this to remove some of = the
runtime dependencies on python-pkg-resources and speed up = some python using
software if you let pip build a wheel as = part of the buld process [1][2][3]. I
suspect pip would need = to grow some additional options to make it suitable for
you in = this role, but I would be more than willing to help if this is = desired
(or not, this proposal isn't about trying to ram pip = down anyones throat!).


[1] To be clear, this isn't about adding wheels = for everything to the archive,
    it would only be = using Wheel as an intermediate step in the build process.
[2] = Console scripts generated by setuptools entry wrappers when installed = by
    setuptools have a dependency on pkg_resources = at runtime, however those
    same scripts, when = installed by pip from a wheel, do not. This would = enable
    removing the runtime dependency on = pkg_resources for anything that only has
    it for = console scripts.
[3] Import pkg_resources is not the fasest = thing in the world, it has to scan
    every = directory on sys.path looking for things to activate and it = comes
    with a bunch of side effects. This happens = implicitly for any project using
    console = scripts.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // = 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

= --Apple-Mail=_09936913-B007-45B6-9510-D120A50CEF82-- --Apple-Mail=_9DF61F7D-36B4-4691-8CD9-B8CF32FA54D6 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJWoPKwAAoJEG48vOkzctz6yeUP/0mBTCn/sNmQY6FyTaziF0UN 4y67hS9bsD6rQrf5XmgZV4FjDnSwVT9pSNFtCZDSoi/rLaoA8nB90E8UmDgJ1YzM JU16oPmt6pD56ryJ8gHuSDfxfh5Luw5LtQ9Bkz0Cqls9zmIixXr0riH7Or6odmrf c+/oU0X6GIaVLmCBGrRznS4mUwz6+ta6/aGgMvgmGZWLmyI0xBfOv9kug/PzI70h QzjjokloRHqof8AjF0SZAbfZ+NLm/Xjc3Can41KP5inznotUxVUdEa95I8Mielss QeQMDYMKTVJySWt0ftzi238EZBhkVAWmFvaRWMI/w4CYXQVWhh7wXBSLmDyJ7LEG LoqsqFel0GKsAM4e1rIL/78yQmQeLiP7ezERFCpIxAChlNW3TNWME1AtOEYK67Du 4tsj2gXdk0a7t1imXbUDfBJHRAyX7+aDLs665/O8B3pUzx/kw43/8ahOzP5TfUFR P64H68gbU/cN1Cr1RT3jPXvx+9aeawFVvU+i1eVPaSK10CCW8uuyC8Jw5Fzi1+cE JiQO1/gTMitSWl4H7/IQZCgHm/ShJbCGmM4Fac6okAVPwxpY0AayVWVj0GpJpnI0 8MPODiTubC5DtTShtuLYr5EUYPjiFxIlVLskEhuTrvq6c4x+82fWxNSSEGJzcqvR XAygSyIV79AITufqCMmF =UKr9 -----END PGP SIGNATURE----- --Apple-Mail=_9DF61F7D-36B4-4691-8CD9-B8CF32FA54D6--