Path: csiph.com!news.mixmin.net!aioe.org!bofh.it!news.nic.it!robomod From: Barry Warsaw Newsgroups: linux.debian.maint.python Subject: Re: Amend Debian Python Proposal to Include More Python Metadata? Date: Fri, 22 Jan 2016 23:40:03 +0100 Message-ID: References: X-Original-To: debian-python@lists.debian.org X-Mailbox-Line: From debian-python-request@lists.debian.org Fri Jan 22 22:31:11 2016 Old-Return-Path: X-Amavis-Spam-Status: No, score=-11.88 tagged_above=-10000 required=5.3 tests=[BAYES_00=-2, FOURLA=0.1, LDO_WHITELIST=-5, MURPHY_DRUGS_REL8=0.02, PGPSIGNATURE=-5] autolearn=ham autolearn_force=no X-Policyd-Weight: NOT_IN_SBL_XBL_SPAMHAUS=-1.5 NOT_IN_BL_NJABL=-1.5 CL_IP_EQ_HELO_IP=-2 (check from: .debian. - helo: .mail.wooz. - helo-domain: .wooz.) FROM/MX_MATCHES_NOT_HELO(DOMAIN)=0; rate: -5 Organization: The Organization of Unorganized Woozalists X-Mailer: Claws Mail 3.12.0 (GTK+ 2.24.29; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; boundary="Sig_/Zz/jJUEhH_2wOZJ_62eu_9J"; protocol="application/pgp-signature" X-Mailing-List: archive/latest/13452 List-ID: List-URL: List-Archive: https://lists.debian.org/msgid-search/20160122173051.0363c23f@anarchist.wooz.org Approved: robomod@news.nic.it Lines: 142 Sender: robomod@news.nic.it X-Original-Date: Fri, 22 Jan 2016 17:30:51 -0500 X-Original-Message-ID: <20160122173051.0363c23f@anarchist.wooz.org> X-Original-References: Xref: csiph.com linux.debian.maint.python:8129 --Sig_/Zz/jJUEhH_2wOZJ_62eu_9J Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Hey Donald, thanks for starting this conversation. I for one am super appreciative of all the consideration you give for Debian's little slice of the world. There's a lot to unpack in this thread, and I'm a little under the weather[= 1], so hopefully this makes sense. Big +1 for recording the files that get installed via the .egg-info/RECORD file. As you know, I've been working on dirtbike, which is a "rewheeling" tool to turn an installed package into a .whl. While Debian tightly contro= ls via policy the set of wheels we'll allow into the archive, dirtbike has code for parsing the RECORD file. Unfortunately this is never exercised in practice because we don't have RECORD files - at least not for the packages= we care about[2]. Big +1 for using setuptools everywhere. By my count, of the packages that I happen to randomly have installed on my Ubuntu 16.04 system, I have 67 .egg-info files and 113 .egg-info directories. I'd rather have .egg-info directories everywhere. +1 for a lintian warning if distutils is used. I guess I'm +0 on forcing t= hat through pybuild because it'll be unobvious and mysterious, and kind of lets upstreams off the hook. I'd mildly prefer to patch packages that use distutils because that's much more discoverable, but I can appreciate that that's a lot of work we'd be imposing on maintainers, so I won't argue this too much (other than to say that if pybuild forces it, let's definitely document this in its manpage!). On Jan 22, 2016, at 12:40 PM, Scott Kitterman wrote: >Currently --record includes the .pyc files which is both unneeded and bad. >Before this gets added either in setuptools or by us, this needed to be >fixed. +1 for (I think) another reason than has already been discussed. We won't = be generating .egg-info directories on the end-user's machine, but instead the machine the package is built on. That could be a maintainer's own system o= r a central build machine depending on various conditions. But because the pycs are generated on the end-user's machine, we actually don't know what pycs w= ill be generated when the debs are installed, so the egg-info/RECORD file *can'= t* contain them, at least not accurately. On Jan 22, 2016, at 11:54 AM, Donald Stufft wrote: >Regardless of what happens in this thread, pip is going to stop mucking >around in files that are owned by some other tool without some sort of >opt-in/--force style flag *and* we're going to be restructuring things to = try >and guide people away from using pip outside of a virtual environment or >through the user of --user as well. Of course, I'd still like --user to be the default[3]. I think that's still the eventual goal for pip, but isn't yet implemented because $priorities. >A more controversial way that comes with possibly some extra benefits (whi= ch >Debian may not care about) is to use ``pip`` itself as the build tool rath= er >than directly invoking setup.py. In this pip would be responsible for muck= ing >around with the distutils/setuptools mess instead of that needing to be >handled by Debian specific tooling. I'd like to better understand how this would work. IIUC, the Fedora ecosys= tem is making or already has made this switch, but I don't know how it works. Obviously, we don't want to install wheels into /usr/lib/python3/dist-packa= ges. FTR, I am still working on updating Debian's pip. I'm slowly shaving the y= ak, but there are still a few things to do. If you want to help, get in touch. >[3] Import pkg_resources is not the fasest thing in the world, it has to s= can >every directory on sys.path looking for things to activate and it comes wi= th >a bunch of side effects. This happens implicitly for any project using >console scripts. Which frankly sucks. It's also fragile. Every once in a while a broken package gets uploaded that breaks pkg_resources, and it's not easy to debug= or repair. I really hope Brett can fix this when/if he builds this functional= ity into the stdlib. Cheers, -Barry [1] double meaning: I have a cold and we're in the early stages of an histo= ric snow storm! [2] dirtbike has two fallbacks, both of which use `dpkg -L` to get the list= of installed files. The first fallback uses `dpkg -S` to find the binary pack= age that contains the Python package's .egg-info file/directory (doesn't matter for this purpose). pkg_resources doesn't have one of those. The second fallback imports the Python package and looks for the binary package containing the top-level directory. It's all rather ugly, and I'd like to just use the .egg-info/RECORD file, but I suspect I'll still need the import fallback for pkg_resources. [3] https://github.com/pypa/pip/issues/1668 --Sig_/Zz/jJUEhH_2wOZJ_62eu_9J Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJWoq2bAAoJEBJutWOnSwa/WwEQAKV14772QQsOrx1rjkqt9D2N YMIu6XQS3mNE4lJaRfVptlfXKMdKZrdZKtoqvE83BwFay/RhkZsZeJBxniBbWwZD Wiz6Vr5q5Aet4azOnt2dQ0pgxhy0QHQpPqB70CNg+8FiOwMEwLZk6wBQoKELLX+8 oobBEo/OfSeGEgVqZPQBJHbry7AakL/LFBz1HpFHlJ+gpdlIBZaxkEmaReulOE+T fL8VoTkcqAt30zDBVHa79AGXzrNaNrflNGiedUYXP5Y4pwlJU3gPTBKZXpvHIXrr ZCgnTQm27Dpo4DFq6JW3Wbz8SkuDm5WjzPT+HLoOKOeg3Wm6BoRsWJJNUSr5I+Ob eV0s0NRRo8ZG1WAS1LJV7x3o/edxv2fnmElQ4AKDMtaP61un2INGC2B5Td31Chu6 EcAbpBPMAw5S7XSfL5HZRu8+nyP6YKYgnsaBhmypq9G7IVkrH9PotnkT9ULhP1NT 3JC+SJOWp7O6JMTUKYUQdQ7lQtkgwRw42UWzoFkpFkO8d8OoDXXmLYRl4M5VxwHa gnyGOIiSDm9XVAz+kfhs+68sUO1D6uS1Y5accj5bior+fP1qpJg0eOKH4UeqduUv RCaCAFWajAmpNJSIwkuJmeymF3Badyj9pDwaoW1+Jn8dntIba39sOrnjfDAWfiKU GK4FBupE5BdV1oPxHjHF =I91s -----END PGP SIGNATURE----- --Sig_/Zz/jJUEhH_2wOZJ_62eu_9J--