Path: csiph.com!aioe.org!bofh.it!news.nic.it!robomod
From: Donald Stufft <donald@stufft.io>
Newsgroups: linux.debian.maint.python
Subject: Re: Amend Debian Python Proposal to Include More Python Metadata?
Date: Sat, 23 Jan 2016 02:00:02 +0100
Message-ID: <qTUIy-3qs-17@gated-at.bofh.it>
References: <qTopk-6qy-11@gated-at.bofh.it> <qTSx5-20U-29@gated-at.bofh.it> <qTSQp-28S-7@gated-at.bofh.it> <qTT06-2t8-17@gated-at.bofh.it> <qTTjs-2Bp-3@gated-at.bofh.it> <qTU5Q-3am-5@gated-at.bofh.it>
Old-Return-Path: <donald@stufft.io>
MIME-Version: 1.0 (Mac OS X Mail 9.2 \(3112\))
Content-Type: multipart/signed; boundary="Apple-Mail=_78062978-E215-49D5-8D99-07540E19A6AA"; protocol="application/pgp-signature"; micalg=pgp-sha512
Approved: robomod@news.nic.it
Lines: 134
Organization: linux.* mail to news gateway
Sender: robomod@news.nic.it
Xref: csiph.com linux.debian.maint.python:8138


--Apple-Mail=_78062978-E215-49D5-8D99-07540E19A6AA
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8


> On Jan 22, 2016, at 7:18 PM, Scott Kitterman <debian@kitterman.com> =
wrote:
>=20
> The Zen of Python says, among other things, "There should be one-- and =
preferably only one --obvious way to do it".  Build systems seem to me =
like a great place to apply that.


We have a sliding scale of complexity in what a project needs from =
it=E2=80=99s build system that generally breaks down into a few major =
classes of projects:

1) Projects that are pure python, single source, whose only real =
=E2=80=9Ccompilation=E2=80=9D is shuffling files to the correct =
location.
2) Projects that are pure python, but require some sort of generation =
step as part of the build process (2to3, etc).
3) Projects that have some basic C code that needs to be compiled, but =
which doesn=E2=80=99t link against anything special besides Python =
itself.
4) Projects that have some basic C code that needs to be compiled, but =
which links against other libraries like OpenSSL, libpq, etc.
5) Projects that have some basic C code, that needs to be compiled, that =
links against other libraries, and needs to be able to conditionally =
link against different libraries based on the capabilities and what is =
available in the system.
6) Extremely complex projects that need to link against many different =
libraries, possibly hard to build, possibly not C (e.g., Numpy with =
it=E2=80=99s blas libraries, Fortran, etc).

The problem with a one size fits all solution is that it=E2=80=99s very =
difficult to actually cover all of these cases in a way that is not =
horrible to actually use for each particular case. For an example, there =
is currently a build system called flit which doesn=E2=80=99t support =
anything but building straight to wheels (because we don=E2=80=99t have =
any sort of sdist 2.0 or anything yet). It doesn=E2=80=99t attempt to =
solve anything but the first class of users up there, and because of =
that it is able to create a very simple and easy to use packaging =
experience for authors, you just add a __version__ =3D =E2=80=9Cthe =
version=E2=80=9D to the top level of your package, and then drop in a =
flit.ini that looks something like:

    [metadata]
    module=3Dfoobar
    author=3DSir Robin
    author-email=3Drobin@camelot.uk
    home-page=3Dhttp://github.com/sirrobin/foobar

    # If you want command line scripts, this is how to declare them.
    # If not, you can leave this section out completely.
    [scripts]
    # foobar:main means the script will do: from foobar import main; =
main()
    foobar=3Dfoobar:main

And that=E2=80=99s it. You=E2=80=99re done. =46rom there flit can do the =
rest of the work for you because it didn=E2=80=99t need to concern =
itself with trying to work on anything complex.

In addition to the above types of problems, you also have other things =
like what the =E2=80=9Csource of truth=E2=80=9D is for your metadata. A =
common thing that people want is to be able to not have to duplicate =
their version in multiple locations (sometimes that even extends to the =
tags in the version control), however it=E2=80=99s not currently =
possible to do that in a particularly easy way. You have systems like =
pbr, setuptools_scm, versioner, etc that all do it but which all rely on =
some level of terribleness to deal with the sort of weird inverted =
control flow we have.

Once you get to all of the possible options for things people reasonably =
want to do, you quickly end up in a place where the only reasonable =
solution is the full complexity of a programming language like what we =
have in setup.py. However, that has it=E2=80=99s own problems which =
we=E2=80=99ve discovered over two decades of doing that :) It tends to =
end up with people doing badly tested adhoc code that they cargo cult =
from project to project and when there=E2=80=99s a problem there ends up =
being very little understanding why some code did that since it had been =
copy/pasted around so much. Pip has to go to a lot of effort to try and =
work around common mistakes people make in their setup.py files (like, =
depending on it being invoked with the project root as CURDIR) which =
most people will just never notice their slightly broken setup.py.

So basically, by allowing multiple build systems we will enable a world =
where your bog standard pure python project can get an extremely simple =
tool/ux and projects with crazy hard requirements like Numpy can get =
something more complex, without the two groups of users fighting each =
other over simplicity + limited vs complexity + powerful.

In the end, I think it=E2=80=99s likely we=E2=80=99ll end up with 2-3 =
really popular build tools, one that is for the complex projects, one =
that is for the simpler projects with some basic C needs, and *maybe* =
one that is for pure python (but that may be able to be handled by the =
basic C needs one) though there will be a long tail I=E2=80=99m sure.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 =
DCFA


--Apple-Mail=_78062978-E215-49D5-8D99-07540E19A6AA
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename=signature.asc
Content-Type: application/pgp-signature;
	name=signature.asc
Content-Description: Message signed with OpenPGP using GPGMail

-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org

iQIcBAEBCgAGBQJWos7iAAoJEG48vOkzctz6dawP/2vv4xeQXnxSNyV1kP00/4MT
LoK0pRI20PduWz5NLs2y0ztlPdoy1/cm1Cx0tdvuk0UBw06p2uAcGfZtMeKCZbi0
Fo1Q2TwUrUlrOo1F8I/dj+5OBWcOISu43VwTIFzigBrHoz4Xz8R9Hk0JzUSG5+M7
/VaD7ND6T77CS9abhfb08LTnsm/ooFlxHEIRReMWAh8XeVpN6a5gs5zAmP34/cCM
/jZ5Tz+Z4vSzUFt4MJgJRh2sPI5UshXiP+uAQ0goGdAhIMK58qXNEQgt7ElC24rU
BQUXp4s+h8flM+uE+faqpviHMRV52mo9rTaXFh7L6tDp8eGtchbnNkNno+oUMsRQ
Sgx6LzF4vajPuOBdVq6KS4akh/cPcZ+YrZoEoVVbsnCz7vinoEJOiu4eLrQ8zPrv
uMO9+zPUlNymGVUNu4n5DMuYqjXm36vCP1dIxUHd/iKXstt2iEFhXBUomoj/678g
pAkW6FkPrgFZ771kqTY+GVyY1pfsAaI+nQBxN8cm94Z5WqdRbZWxsmNVIwZsbCck
HUZC8ELc851DUQblKEVMUGEmFjn31WwXoPwXyTT7SLB4cHf6z9/FZHTTS4lJ2P5l
SxA0Yfd81M1gqUXKL19ROTFOT6IPxbhcLdYw+NWW9gkybskGHGvRuqRJMREs3uon
SzObUXt/Uo03fnfi3U0f
=00zu
-----END PGP SIGNATURE-----

--Apple-Mail=_78062978-E215-49D5-8D99-07540E19A6AA--