Path: csiph.com!aioe.org!bofh.it!news.nic.it!robomod From: Dmitry Shachnev Newsgroups: linux.debian.maint.python Subject: Re: providing sphinx3-* binaries Date: Thu, 28 Sep 2017 19:00:01 +0200 Message-ID: References: X-Original-To: Antoine =?iso-8859-1?Q?Beaupr=E9?= X-Mailbox-Line: From debian-python-request@lists.debian.org Thu Sep 28 16:59:45 2017 Old-Return-Path: X-Amavis-Spam-Status: No, score=-11.509 tagged_above=-10000 required=5.3 tests=[BAYES_00=-2, DIGITS_LETTERS=1, FOURLA=0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FORGED_FROMDOMAIN=0.199, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.001, LDO_WHITELIST=-5, PGPSIGNATURE=-5, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-2.8, RCVD_IN_SORBS_SPAM=0.5, SARE_LWSHORTT=1.24] autolearn=ham autolearn_force=no X-Policyd-Weight: DYN_NJABL=ERR(0) NOT_IN_SBL_XBL_SPAMHAUS=-1.5 BL_NJABL=ERR(-1.5) CL_IP_EQ_HELO_IP=-2 (check from: .gmail. - helo: .mail-wr0-f175.google. - helo-domain: .google.) FROM/MX_MATCHES_HELO(DOMAIN)=-2; rate: -7 X-Google-Dkim-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=nwR6SNfEf8smzNr0nIqMQ0Z8SkKd/FCNq5MpkQGAVtk=; b=sxNh3rANj6XI4M/Hy/16PdSXPfRDlq5z6I5KhCHT8HJSYFAtYELRkzzOwpXGHfOjIj 9XNHHX4AGla7QOW5sK4PZANOwNF6mZCQf665bkLYDFci6W/+T7gjNH911XkF4WlZSxgT YJCUc1PhDP4IJMEsZv/GLiw1gxqkycXWLtZf/NGADqS2LI+rW1aqQnhC7aI5Z4oyHrJQ 5OW+Y1Yj5iEIeYdz2eeXryhAA+ZRQBlnON20Peggzh4l7lGlLCZDkq/UCsiqjbaADJ+5 3uTxZVVQCmRfn3ufxC6Yiq94HGE3VBtZFUg5zQC2DA67IYfKw+9uehgyn6rT/2ysfsQX TUJQ== X-Gm-Message-State: AHPjjUhyrVT4tzLdATbFUu+P2/nPhuI2vpd6GbgOGhc5sXKzr7B4jw90 yIsXV+R/Pk0U0WSbs+fLb3RsIqiW X-Google-SMTP-Source: AOwi7QD4B90jxksJbmsnCu6hzuKOizHrWG3r8csDHqXVQa/KiUiHAF1fNefiNIPgVvxNAzdyjapmcQ== X-Received: by 10.46.83.81 with SMTP id t17mr2387508ljd.180.1506617956037; Thu, 28 Sep 2017 09:59:16 -0700 (PDT) MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="dm7iopso64xp5qel" Content-Disposition: inline User-Agent: NeoMutt/20170609 (1.8.3) X-Mailing-List: archive/latest/15252 List-ID: List-URL: List-Archive: https://lists.debian.org/msgid-search/20170928165912.rjnitrq44wn25gzx@mitya57.me Approved: robomod@news.nic.it Lines: 245 Organization: linux.* mail to news gateway Sender: robomod@news.nic.it X-Original-Cc: debian-python@lists.debian.org X-Original-Date: Thu, 28 Sep 2017 19:59:12 +0300 X-Original-Message-ID: <20170928165912.rjnitrq44wn25gzx@mitya57.me> X-Original-References: <877ewlqeku.fsf@curie.anarc.at> <20170927220327.45zxzo5yj5sdj7nv@mitya57.me> <87vak3ovo7.fsf@curie.anarc.at> Xref: csiph.com linux.debian.maint.python:9870 --dm7iopso64xp5qel Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Sep 28, 2017 at 08:27:20AM -0400, Antoine Beaupr=C3=A9 wrote: > > And moving Python 3 packages to /usr/bin/sphinx3-build or something like > > that will mean diverging from upstream (see below). > > Note that we already diverge from upstream in numerous places in Python, > first and foremost in the naming of the Python binary itself. How do we diverge there? In my opinion we follow PEP 394 quite closely: https://www.python.org/dev/peps/pep-0394/ > > Good suggestion, I have submitted a pull request upstream: > > https://github.com/sphinx-doc/sphinx/pull/4092 > > Excellent - I meant to do that but ran out of time writing the email in > the first place. :) > > I see, however, it's not getting too much traction.. But they have a > fair point - I didn't realize there was a difference between variables > from the environment and from the commandline in Make. Maybe it's good > enough as it is then. It is not =E2=80=98they=E2=80=99, it is our Simon :) He also replied on thi= s mailing list, just did not CC you. > > And there is no such thing as --with=3Dsphinx. There is --with=3Dsphinx= doc, > > but it operates on already built and installed documentation, so it can= not > > help with building. > > I guess this is what I mean should be automated better. Every time I > used --with=3Dsphinxdoc, I expected it to build the docs and then > remembered, when the produced package lacked any docs whatsoever, that I > needed to build things myself. > > A simple --with=3Dsphinxdoc should be sufficient to build sphinx docs. We > shouldn't expect upstream to do that step in setup.py, as it is rarely > hooked into the main build. Yes, there's the `build_sphinx` command, but > it's not hooked into `setup.py build` except in some rare project. Sphinxdoc is a helper to run after install, not a build system. So it is not a place for such code. > This is why I mention pybuild: maybe *that* is where docs should be > built? I am not sure. In any case, I feel there's a shim missing here, > and there's a good occasion to leverage the automation we have to > generate proper build deps and build-time commands. Maybe pybuild can do this, but it then needs to make some assumptions about the doc source path, build path, wished formats, etc. For finding sou= rce path it can probably use something like this: https://github.com/sphinx-doc/sphinx/blob/stable/sphinx/setup_command.py#L1= 11 But it is up to pybuild maintainer (Piotr) to decide whether he wants this feature or not. > > What is the problem with using /usr/share/sphinx/scripts/python3/sphinx= -build > > explicitly if you want to force Python 3? > > It's hard to find. It's unusal for a user-facing command to be absent > from the path yet to expect users to find it, especially when one > version *is* available in the path. As I said ealier, for most cases just using sphinx-build which is on PATH should be fine. In advanced cases, it should not be hard to run =E2=80=98dp= kg -L python3-sphinx=E2=80=99 to figure out where its scripts are, or use the -m = syntax. > Sure. I just found out about this while writing the email, after almost > a decade of maintaining python packages. That's a clear symptom of bad > discovery, although the upstream Makefile changes may help with that > eventually. Actually quite many packages tend to provide the -m syntax and even prefer it to scripts in /usr/bin/. Examples from packages I maintain: -m markdown, -m keyring, -m nose. > > Also there is no such thing as sphinx3-build upstream (unlike other pac= kages > > that follow this convention). So all upstream projects will try to use > > sphinx-build, not sphinx3-build, even if they are Python 3 only. And if= you > > override this command anyway, you can use two existing ways, I don=E2= =80=99t see a > > need for the third one. > > The reasoning here is that is is more discoverable, namely through > commandline completion, and is consistent with the /usr/bin/python* > binary naming conventions. I have just figured out that there *is* bash completion for python3 -m syntax. It is not difficult for me to provide the sphinx3-* binaries, but that might create false assumptions for developers that it is available on other syste= ms too. It is better to use the -m syntax because it is more universal and wor= ks on any OS. > > I won=E2=80=99t say that this convention is usual. There are many packa= ges named > > python-* or python3-* which provide executable scripts. > > I would argue that is a strange practice: I tend to ignore packages that > start with python* unless I am programming something. As a user, they > are basically useless because I am supposed to code something to use > those libraries. Now, granted, sphinx is a little special because it > *is* a programmer-oriented library, but keep in mind it can be used for > non-programs as well (see the Debian policy for example). > > > Let me summarize what you, in my opinion, should do as a maintainer of > > packages using Sphinx: > > > > 1) If your project is not using autodoc: just build-depend on python3-s= phinx > > and use /usr/bin/sphinx-build. There is no need to care about > > versions. > > 2) If your project uses autodoc, and is compatible with both Python 2 a= nd 3: > > same as in 1), just use /usr/bin/sphinx-build. > > 3) If your project uses autodoc, and is only compatible with Python 3: > > use =E2=80=98python3 -m sphinx=E2=80=99 or =E2=80=98python3 setup.py bu= ild_sphinx=E2=80=99, depending on the > > upstream build system. > > 4) If your project uses autodoc, and is only compatible with Python 2: > > Port it to python 3. Or use =E2=80=98python2 -m sphinx=E2=80=99 or =E2= =80=98python2 setup.py > > build_sphinx=E2=80=99. > > > > If you think this classification is incomplete, then please let me know > > where I am wrong. > > That is a fair description, of course. It would be great to have build > step automated. At the very least, the above should be clearly > documented somewhere, maybe in dh_sphinxdoc's README or > manpage. Documenting that dh_sphinxdoc doesn't build in its manpage > would also go a long way, as that rather implicit right now. (ie. it > doesn't say it builds sphinx docs, but it doesn't say it doesn't either: > we need to assume that the list of things it does is exhaustive and > complete, something I never assume about documentation :) Now it is explicit: https://anonscm.debian.org/cgit/python-modules/packages/sphinx.git/commit/?= id=3D5b2efffcaae8c915 > Okay then the policy seems to point towards shipping sphinx3-build > binary, no? See what I said above (about false assumptions). > > This may happen when setup.py depends on Sphinx and automatically calls= it > > during build. Such packages have to build-depend on both Python 2 and 3 > > versions of Sphinx, even if only one version of documentation actually = gets > > shipped. > > I don't follow. Why do they need to depend on both? We don't need *nor* > want to build the documentation twice - if upstream's setup.py > automatically calls build_sphinx, then this is something we should be > working around (again, pybuild?) if only to save time on the buildds. I do not say that these packages do the right thing. I just tried to find a reason why they may be doing this. It might be better to ask maintainers of these packages for more precise reason. > So maybe there's just nothing else to do here in the short term. I'll > start using "make SPHINXBUILD=3D'python3 -m sphinx'" even though I find > that real clunky. In fact, the more I think about it, the less I see the > point of using the Makefiles at all from debian/rules: i'll probably > just call "python3 -m sphinx" or "setup.py build_sphinx" myself from > there, as needed. Right, there is no reason for Makefiles. If the build process is not integrated in upstream=E2=80=99s main build system, there should be no diff= erence for you which line to write in debian/rules. > The problem, then, of course, is that I lose that precious > automation. Most Python Debian packages really don't need that much > stuff in debian/rules - this is enough for 90% of the cases: > > %: > dh $@ --with=3Dpython2,python3 --buildsystem=3Dpybuild > > Yet, when I want to build docs in there, i need to override the build > targets immediately: > > override_dh_auto_build: > dh_auto_build > # builder=3Dhtml,man only supported in Sphinx 1.6, not yet releas= ed: > # http://www.sphinx-doc.org/en/master/setuptools.html > make -C doc html man SPHINXBUILD=3D"python3 -m sphinx" > # ^untested > > I'd like to be able to just assume sphinxdoc will do the right thing and > build me a set of HTML docs (at least). Ideally, I'd be able to choose > the builders (html, man, epub, etc) through the environment, but then > that'd need to change the build-deps as well, something I understand is > basically impossible. ;) See what I said above about pybuild. > Anyways, I learned a lot through this. Thanks for all the information, > and I hope we can update docs somewhere to reflect the current situation > and move ahead with some better build process for this as well. -- Dmitry Shachnev --dm7iopso64xp5qel Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEbEPcK+5mZmLK5jNU1v5xA2P4XdMFAlnNKlsACgkQ1v5xA2P4 XdN2VQ//TrFkbQ5Iz5CGEzon+JYHj0Tvp4yssE0rPIabnoY3DP6iPPl/rIVANqb6 fN/sH3u9k7dz4c6wYej67JqsygJOrezyXqtAsIxHk2YxcBzjEO7E43/b7WnhKmQP /SFUl+P2FZKwCbXijiOSRhCRXZnaIT5Ln396tPawxrmks8uLyKlNLHQoOp/uw7xq 660oERrwMhc/Zns7uKI55Q0NlnUKqamraIUnZClY5I9fpr1UhveWd80XGKM3lZGG DCQC9QeCZ1q4zBohqNnuPHr8C7wNlATzGiuupkwydVJgeR19TnVMoE7eQXmzDKPj jZz2HAr8rdl+09lH2zoL4JQIjP48yeeuuJaPZ2GdYzBwZw5Wnx2ugZS9RjP2bjzL Iuquqvgoj2lg5vSApgapVuZgxmsBduX2EX/3NmO6bDnnfy23az283UkgpCgauWVn uYWwnVggLMLgN03QkHZOzTQ7K3cSZeUM5iC8A4mGuoh4w+rI5M5j5FHsaws68Uwj TGA5+FazUfpTlQMRdVO4HjNlu398d+K9WW0p5xXa3Com1gfbpDUPMNajpWEhxna2 aBMzSloL/M7gmS89aN/8B8yjoIROxmGC713iQhCkq/60NLJ1br0z5Osj5pfU+A3j vHMH/LiMBDsZMxP77zgY3PQsQqRgd8lQNkQA7tNPPahuMejgM+E= =02RA -----END PGP SIGNATURE----- --dm7iopso64xp5qel--