Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #9860 > unrolled thread

providing sphinx3-* binaries

Back to article view | Back to linux.debian.maint.python

Contents

  providing sphinx3-* binaries Antoine Beaupré <anarcat@debian.org> - 2017-09-27 00:30 +0200
    Re: providing sphinx3-* binaries Dmitry Shachnev <mitya57@debian.org> - 2017-09-28 00:10 +0200
      Re: providing sphinx3-* binaries Simon McVittie <smcv@debian.org> - 2017-09-28 09:40 +0200
      Re: providing sphinx3-* binaries Antoine Beaupré <anarcat@debian.org> - 2017-09-28 14:30 +0200
        Re: providing sphinx3-* binaries Dmitry Shachnev <mitya57@debian.org> - 2017-09-28 19:00 +0200
          Re: providing sphinx3-* binaries Antoine Beaupré <anarcat@debian.org> - 2017-09-28 19:20 +0200
    Re: providing sphinx3-* binaries Thomas Goirand <zigo@debian.org> - 2017-09-29 13:00 +0200
      RE:providing sphinx3-* binaries PICCA Frederic-Emmanuel <frederic-emmanuel.picca@synchrotron-soleil.fr> - 2017-09-29 13:30 +0200
        Re: providing sphinx3-* binaries Ghislain Vaillant <ghisvail@gmail.com> - 2017-09-29 14:00 +0200
          RE:providing sphinx3-* binaries PICCA Frederic-Emmanuel <frederic-emmanuel.picca@synchrotron-soleil.fr> - 2017-09-29 14:20 +0200
        Re: providing sphinx3-* binaries Thomas Goirand <zigo@debian.org> - 2017-10-03 23:50 +0200
          Re: providing sphinx3-* binaries Ghislain Vaillant <ghisvail@gmail.com> - 2017-10-04 04:50 +0200
            Re: providing sphinx3-* binaries Thomas Goirand <zigo@debian.org> - 2017-10-04 11:50 +0200

#9860 — providing sphinx3-* binaries

Hi!

[please keep me in CC, I am not on the list]

We just had a short conversation on the #debian-devel IRC channel
regarding the upcoming Python 2 EOL, in the context of the Sphinx
packages.

As a packager of Python tools (and an upstream), I find the current
situation a little confusing. My Python 2 workflow is okay, but I am
trying, for new projects, to use only Python 3, which gives me a rather
broken workflow. It looks something like this:

 1. compile documentation using `make html`, fail with mysterious
    errors.

 2. realize those are because Sphinx uses Python 2 by default

 3. install python3-sphinx

 4. try to compile again, same errors

 5. try to find the sphinx binary (generally, `dpkg -L python3-sphinx |
    grep bin`, which fails, then `dpkg -L python3-sphinx | grep build`)

 6. hack the Makefile to make the `SPHINXBUILD` variable overridable
    (with `?=`)

 7. fix my docs and debian/rules to call:
    `SPHINXBUILD=/usr/share/sphinx/scripts/python3/sphinx-build`

So that's the first problem I have. The other problem we identified is
that the /usr/bin/sphinx-build symlink is not owned by any package.

$ LANG=C dpkg -S /usr/bin/sphinx-build
dpkg-query: no path found matching pattern /usr/bin/sphinx-build

That's a rather bad practice... We shouldn't install those symlinks in
postinst. They should be alternatives or conflicts or *something*. In
2013, the sphinx maintainer [explained][1] the following:

> True for docutils; however, sphinx doesn't use alternatives, and it
> doesn't do so for good reasons.
> 
> The alternatives mechanisms is only suitable if both commands are
> compatible, i.e. their behavior doesn't vary with Python version. This
> is the case for rst2html and friends, but no so much for
> sphinx-build. The problem is that sphinx-build can import 3rd-party
> Python code, which is not necessarily compatible with both Python 2
> and 3.
> 
> As I understand it, python{2,3}-coverage are NOT compatible, and
> therefore they should NOT use alternatives.

I understand that reasoning, but I don't think I agree with it. Or at
least, I don't think that, in the long term, it should be something that
blocks us.

I am not sure what the solution is, but I would offer the following:

 1. there should be a more easy way to override SPHINXBUILD in
    quickstart-generated Makefiles. IMHO, that means SPHINXBUILD?=
    instead of SPHINXBUILD=

 2. similarly, --with=sphinx should do the right thing depending on the
    detected pybuild Python version, that is, call
    /usr/share/sphinx/scripts/python{,3}/sphinx-build directly depending
    on which version we are building for.

 3. sphinx packages should follow the "python vs python3"
    convention. right now, when you run the "python" binary, you get
    Python 2. if you want Python 3, you run "python3". I don't know if
    there are plans to change this, but that's irrelevant at this stage:
    sphinx should follow convention and call python2 sphinx when you run
    "sphinx-build" and python3 sphinx when you run "sphinx3-build", and
    both should be available in $PATH.

 4. the sphinx package is not following the usual convention of
    providing only libraries in python*. or to be more precise, it's
    rather odd that "sphinx-common" is what installs the python2 or
    python3 binary in $PATH. i would suggest providing new binary
    packages called "sphinx" and "sphinx3" that would provide those
    binaries.

 5. optionally, the sphinx and sphinx3 packages *may* conflict with each
    other to fight over the sphinx-build symlink. then people needing to
    build against *both* packages can use the "python -m sphinx"
    mechanism in their makefiles, since it's what upstream seems to be
    [doing][2]

 6. there are currently packages depending on *both* python-sphinx and
    python3-sphinx. for me, that makes no sense at all: there is a
    single set of documentation files built in a package, and it must be
    on either one of those packages, but not both. if the software
    supports building with python3, then it should depend on
    python3-sphinx, if not, it should depend on python-sphinx. by
    switching to sphinx{,3} binary packages, this would make that
    distinction clearer as well.

My pain points would mostly be fixed with th e simple #1 and #3
fixes. Without breaking anything, we could already provide a
sphinx3-build binary, and we could file a PR upstream to make
SPHINXBUILD more easily overridable. The rest is just a brainstorm of
possible medium-term fixes to try and fix that dangling symlink issue.

I understand this is a complex situation and I would love to have better
answers, but I'm not sure there are any simple answers here. We
shouldn't hesitate to make radical changes, however: we have transition
tools to help us with that stuff, and if we can upgrade libc and gcc
across all of Debian fairly painlessly these days, i don't see why we
couldn't articulate such a transition for ~1000 python packages. :)

Thank you for your time.

A.

 [1]: https://lists.debian.org/debian-python/2013/10/msg00048.html
 [2]: https://github.com/sphinx-doc/sphinx/pull/3523

[toc] | [next] | [standalone]

#9867

[Multipart message — attachments visible in raw view] — view raw

Hi Antoine, and thanks for the detailed mail!

On Tue, Sep 26, 2017 at 06:29:05PM -0400, Antoine Beaupré wrote:
> We just had a short conversation on the #debian-devel IRC channel
> regarding the upcoming Python 2 EOL, in the context of the Sphinx
> packages.
>
> [...]
>
> So that's the first problem I have. The other problem we identified is
> that the /usr/bin/sphinx-build symlink is not owned by any package.
>
> $ LANG=C dpkg -S /usr/bin/sphinx-build
> dpkg-query: no path found matching pattern /usr/bin/sphinx-build
>
> That's a rather bad practice... We shouldn't install those symlinks in
> postinst. They should be alternatives or conflicts or *something*. In
> 2013, the sphinx maintainer [explained][1] the following:
>
> > True for docutils; however, sphinx doesn't use alternatives, and it
> > doesn't do so for good reasons.
> >
> > The alternatives mechanisms is only suitable if both commands are
> > compatible, i.e. their behavior doesn't vary with Python version. This
> > is the case for rst2html and friends, but no so much for
> > sphinx-build. The problem is that sphinx-build can import 3rd-party
> > Python code, which is not necessarily compatible with both Python 2
> > and 3.
>
> I understand that reasoning, but I don't think I agree with it. Or at
> least, I don't think that, in the long term, it should be something that
> blocks us.

I think the reason why we should not use alternatives still applies. The
current approach is conservative and it guarantees that when a package has
python-sphinx in build-dependencies, /usr/bin/sphinx-build will always use
Python 2. With alternatives there is no such guarantee.

And moving Python 3 packages to /usr/bin/sphinx3-build or something like
that will mean diverging from upstream (see below).

> I am not sure what the solution is, but I would offer the following:
>
>  1. there should be a more easy way to override SPHINXBUILD in
>     quickstart-generated Makefiles. IMHO, that means SPHINXBUILD?=
>     instead of SPHINXBUILD=

Good suggestion, I have submitted a pull request upstream:
https://github.com/sphinx-doc/sphinx/pull/4092

>  2. similarly, --with=sphinx should do the right thing depending on the
>     detected pybuild Python version, that is, call
>     /usr/share/sphinx/scripts/python{,3}/sphinx-build directly depending
>     on which version we are building for.

I do not understand this. You mentioned pybuild, so you are talking about
setup.py based build systems, not Makefile based ones, right?

If setup.py calls build_sphinx automatically, then there is nothing else
to do. If it does not, then you should call ‘setup.py build_sphinx’ with
explicit interpreter, i.e. python or python3.

The only thing where pybuild may help is when you want to run Sphinx for
all supported Python versions, then you can use its COMMAND syntax.

And there is no such thing as --with=sphinx. There is --with=sphinxdoc,
but it operates on already built and installed documentation, so it cannot
help with building.

>  3. sphinx packages should follow the "python vs python3"
>     convention. right now, when you run the "python" binary, you get
>     Python 2. if you want Python 3, you run "python3". I don't know if
>     there are plans to change this, but that's irrelevant at this stage:
>     sphinx should follow convention and call python2 sphinx when you run
>     "sphinx-build" and python3 sphinx when you run "sphinx3-build", and
>     both should be available in $PATH.

What is the problem with using /usr/share/sphinx/scripts/python3/sphinx-build
explicitly if you want to force Python 3?

If you like shorter command names, then you can also use ‘python3 -m sphinx’.
Using the -m syntax instead of scripts is even endorsed by Sphinx upstream
these days (see your link [2]).

Also there is no such thing as sphinx3-build upstream (unlike other packages
that follow this convention). So all upstream projects will try to use
sphinx-build, not sphinx3-build, even if they are Python 3 only. And if you
override this command anyway, you can use two existing ways, I don’t see a
need for the third one.

>  4. the sphinx package is not following the usual convention of
>     providing only libraries in python*. or to be more precise, it's
>     rather odd that "sphinx-common" is what installs the python2 or
>     python3 binary in $PATH. i would suggest providing new binary
>     packages called "sphinx" and "sphinx3" that would provide those
>     binaries.

I won’t say that this convention is usual. There are many packages named
python-* or python3-* which provide executable scripts.

Let me summarize what you, in my opinion, should do as a maintainer of
packages using Sphinx:

1) If your project is not using autodoc: just build-depend on python3-sphinx
and use /usr/bin/sphinx-build. There is no need to care about versions.
2) If your project uses autodoc, and is compatible with both Python 2 and 3:
same as in 1), just use /usr/bin/sphinx-build.
3) If your project uses autodoc, and is only compatible with Python 3:
use ‘python3 -m sphinx’ or ‘python3 setup.py build_sphinx’, depending on the
upstream build system.
4) If your project uses autodoc, and is only compatible with Python 2:
Port it to python 3. Or use ‘python2 -m sphinx’ or ‘python2 setup.py
build_sphinx’.

If you think this classification is incomplete, then please let me know
where I am wrong.

>  5. optionally, the sphinx and sphinx3 packages *may* conflict with each
>     other to fight over the sphinx-build symlink. then people needing to
>     build against *both* packages can use the "python -m sphinx"
>     mechanism in their makefiles, since it's what upstream seems to be
>     [doing][2]

Let me quote Policy section 5.4:

“adding Conflicts is normally not the best solution when two packages provide
the same files. Depending on the reason for that conflict, using alternatives
or renaming the files is often a better approach.”

>  6. there are currently packages depending on *both* python-sphinx and
>     python3-sphinx. for me, that makes no sense at all: there is a
>     single set of documentation files built in a package, and it must be
>     on either one of those packages, but not both. if the software
>     supports building with python3, then it should depend on
>     python3-sphinx, if not, it should depend on python-sphinx. by
>     switching to sphinx{,3} binary packages, this would make that
>     distinction clearer as well.

This may happen when setup.py depends on Sphinx and automatically calls it
during build. Such packages have to build-depend on both Python 2 and 3
versions of Sphinx, even if only one version of documentation actually gets
shipped.

> My pain points would mostly be fixed with th e simple #1 and #3
> fixes. Without breaking anything, we could already provide a
> sphinx3-build binary, and we could file a PR upstream to make
> SPHINXBUILD more easily overridable. The rest is just a brainstorm of
> possible medium-term fixes to try and fix that dangling symlink issue.
>
> I understand this is a complex situation and I would love to have better
> answers, but I'm not sure there are any simple answers here. We
> shouldn't hesitate to make radical changes, however: we have transition
> tools to help us with that stuff, and if we can upgrade libc and gcc
> across all of Debian fairly painlessly these days, i don't see why we
> couldn't articulate such a transition for ~1000 python packages. :)

In my opinion there should be a very strong reason to make changes that
break packages. I have to deal with Sphinx upstream breaking compatibility
which results in lots of FTBFS bugs for each major version. And I don’t
want to introduce even more incompatibility myself.

--
Dmitry Shachnev

[toc] | [prev] | [next] | [standalone]

#9868

On Thu, 28 Sep 2017 at 01:03:27 +0300, Dmitry Shachnev wrote:
> On Tue, Sep 26, 2017 at 06:29:05PM -0400, Antoine Beaupré wrote:
> >  1. there should be a more easy way to override SPHINXBUILD in
> >     quickstart-generated Makefiles. IMHO, that means SPHINXBUILD?=
> >     instead of SPHINXBUILD=
> 
> Good suggestion, I have submitted a pull request upstream:
> https://github.com/sphinx-doc/sphinx/pull/4092

I'm not sure this is necessary. If a Makefile has

	SPHINXBUILD = sphinx-build
	# or := or ::=

	some-target:
		$(SPHINXBUILD) some-arguments

then you can override it on the command-line:

	make SPHINXBUILD="python3 -m sphinx"

without needing the ?= syntax. The ?= syntax is only necessary if you want
to pick up a value from the environment:

	export SPHINXBUILD="python3 -m sphinx"; make

or if your Makefile has complicated logic that may or may not have set it
already:

	ifeq(...)
		SPHINXBUILD = $(PYTHON) -m sphinx
	endif

	SPHINXBUILD ?= sphinx-build

The Makefiles generated by GNU Automake, which are very verbose but
generally also very good about doing the right thing in the face of
strange make(1) quirks, use plain "=".

(I've said the same on the upstream PR.)

Regards,
    smcv

[toc] | [prev] | [next] | [standalone]

#9869

On 2017-09-28 01:03:27, Dmitry Shachnev wrote:
> Hi Antoine, and thanks for the detailed mail!
>
> On Tue, Sep 26, 2017 at 06:29:05PM -0400, Antoine Beaupré wrote:
>> We just had a short conversation on the #debian-devel IRC channel
>> regarding the upcoming Python 2 EOL, in the context of the Sphinx
>> packages.
>>
>> [...]
>>
>> So that's the first problem I have. The other problem we identified is
>> that the /usr/bin/sphinx-build symlink is not owned by any package.
>>
>> $ LANG=C dpkg -S /usr/bin/sphinx-build
>> dpkg-query: no path found matching pattern /usr/bin/sphinx-build
>>
>> That's a rather bad practice... We shouldn't install those symlinks in
>> postinst. They should be alternatives or conflicts or *something*. In
>> 2013, the sphinx maintainer [explained][1] the following:
>>
>> > True for docutils; however, sphinx doesn't use alternatives, and it
>> > doesn't do so for good reasons.
>> >
>> > The alternatives mechanisms is only suitable if both commands are
>> > compatible, i.e. their behavior doesn't vary with Python version. This
>> > is the case for rst2html and friends, but no so much for
>> > sphinx-build. The problem is that sphinx-build can import 3rd-party
>> > Python code, which is not necessarily compatible with both Python 2
>> > and 3.
>>
>> I understand that reasoning, but I don't think I agree with it. Or at
>> least, I don't think that, in the long term, it should be something that
>> blocks us.
>
> I think the reason why we should not use alternatives still applies. The
> current approach is conservative and it guarantees that when a package has
> python-sphinx in build-dependencies, /usr/bin/sphinx-build will always use
> Python 2. With alternatives there is no such guarantee.

Understood, fair point.

> And moving Python 3 packages to /usr/bin/sphinx3-build or something like
> that will mean diverging from upstream (see below).

Note that we already diverge from upstream in numerous places in Python,
first and foremost in the naming of the Python binary itself.

>> I am not sure what the solution is, but I would offer the following:
>>
>>  1. there should be a more easy way to override SPHINXBUILD in
>>     quickstart-generated Makefiles. IMHO, that means SPHINXBUILD?=
>>     instead of SPHINXBUILD=
>
> Good suggestion, I have submitted a pull request upstream:
> https://github.com/sphinx-doc/sphinx/pull/4092

Excellent - I meant to do that but ran out of time writing the email in
the first place. :)

I see, however, it's not getting too much traction.. But they have a
fair point - I didn't realize there was a difference between variables
from the environment and from the commandline in Make. Maybe it's good
enough as it is then.

>>  2. similarly, --with=sphinx should do the right thing depending on the
>>     detected pybuild Python version, that is, call
>>     /usr/share/sphinx/scripts/python{,3}/sphinx-build directly depending
>>     on which version we are building for.
>
> I do not understand this. You mentioned pybuild, so you are talking about
> setup.py based build systems, not Makefile based ones, right?

True. Well, technically, I refer to dh_sphinxdoc, but from what I could
tell, that doesn't seem to actually *build* the documentation, but just
clean it up, something I didn't expect.

> If setup.py calls build_sphinx automatically, then there is nothing else
> to do. If it does not, then you should call ‘setup.py build_sphinx’ with
> explicit interpreter, i.e. python or python3.
>
> The only thing where pybuild may help is when you want to run Sphinx for
> all supported Python versions, then you can use its COMMAND syntax.
>
> And there is no such thing as --with=sphinx. There is --with=sphinxdoc,
> but it operates on already built and installed documentation, so it cannot
> help with building.

I guess this is what I mean should be automated better. Every time I
used --with=sphinxdoc, I expected it to build the docs and then
remembered, when the produced package lacked any docs whatsoever, that I
needed to build things myself.

A simple --with=sphinxdoc should be sufficient to build sphinx docs. We
shouldn't expect upstream to do that step in setup.py, as it is rarely
hooked into the main build. Yes, there's the `build_sphinx` command, but
it's not hooked into `setup.py build` except in some rare project.

This is why I mention pybuild: maybe *that* is where docs should be
built? I am not sure. In any case, I feel there's a shim missing here,
and there's a good occasion to leverage the automation we have to
generate proper build deps and build-time commands.

>>  3. sphinx packages should follow the "python vs python3"
>>     convention. right now, when you run the "python" binary, you get
>>     Python 2. if you want Python 3, you run "python3". I don't know if
>>     there are plans to change this, but that's irrelevant at this stage:
>>     sphinx should follow convention and call python2 sphinx when you run
>>     "sphinx-build" and python3 sphinx when you run "sphinx3-build", and
>>     both should be available in $PATH.
>
> What is the problem with using /usr/share/sphinx/scripts/python3/sphinx-build
> explicitly if you want to force Python 3?

It's hard to find. It's unusal for a user-facing command to be absent
from the path yet to expect users to find it, especially when one
version *is* available in the path.

> If you like shorter command names, then you can also use ‘python3 -m sphinx’.
> Using the -m syntax instead of scripts is even endorsed by Sphinx upstream
> these days (see your link [2]).

Sure. I just found out about this while writing the email, after almost
a decade of maintaining python packages. That's a clear symptom of bad
discovery, although the upstream Makefile changes may help with that
eventually.

> Also there is no such thing as sphinx3-build upstream (unlike other packages
> that follow this convention). So all upstream projects will try to use
> sphinx-build, not sphinx3-build, even if they are Python 3 only. And if you
> override this command anyway, you can use two existing ways, I don’t see a
> need for the third one.

The reasoning here is that is is more discoverable, namely through
commandline completion, and is consistent with the /usr/bin/python*
binary naming conventions.

>>  4. the sphinx package is not following the usual convention of
>>     providing only libraries in python*. or to be more precise, it's
>>     rather odd that "sphinx-common" is what installs the python2 or
>>     python3 binary in $PATH. i would suggest providing new binary
>>     packages called "sphinx" and "sphinx3" that would provide those
>>     binaries.
>
> I won’t say that this convention is usual. There are many packages named
> python-* or python3-* which provide executable scripts.

I would argue that is a strange practice: I tend to ignore packages that
start with python* unless I am programming something. As a user, they
are basically useless because I am supposed to code something to use
those libraries. Now, granted, sphinx is a little special because it
*is* a programmer-oriented library, but keep in mind it can be used for
non-programs as well (see the Debian policy for example).

> Let me summarize what you, in my opinion, should do as a maintainer of
> packages using Sphinx:
>
> 1) If your project is not using autodoc: just build-depend on python3-sphinx
> and use /usr/bin/sphinx-build. There is no need to care about
> versions.
> 2) If your project uses autodoc, and is compatible with both Python 2 and 3:
> same as in 1), just use /usr/bin/sphinx-build.
> 3) If your project uses autodoc, and is only compatible with Python 3:
> use ‘python3 -m sphinx’ or ‘python3 setup.py build_sphinx’, depending on the
> upstream build system.
> 4) If your project uses autodoc, and is only compatible with Python 2:
> Port it to python 3. Or use ‘python2 -m sphinx’ or ‘python2 setup.py
> build_sphinx’.
>
> If you think this classification is incomplete, then please let me know
> where I am wrong.

That is a fair description, of course. It would be great to have build
step automated. At the very least, the above should be clearly
documented somewhere, maybe in dh_sphinxdoc's README or
manpage. Documenting that dh_sphinxdoc doesn't build in its manpage
would also go a long way, as that rather implicit right now. (ie. it
doesn't say it builds sphinx docs, but it doesn't say it doesn't either:
we need to assume that the list of things it does is exhaustive and
complete, something I never assume about documentation :)

>>  5. optionally, the sphinx and sphinx3 packages *may* conflict with each
>>     other to fight over the sphinx-build symlink. then people needing to
>>     build against *both* packages can use the "python -m sphinx"
>>     mechanism in their makefiles, since it's what upstream seems to be
>>     [doing][2]
>
> Let me quote Policy section 5.4:
>
> “adding Conflicts is normally not the best solution when two packages provide
> the same files. Depending on the reason for that conflict, using alternatives
> or renaming the files is often a better approach.”

Okay then the policy seems to point towards shipping sphinx3-build
binary, no?

>>  6. there are currently packages depending on *both* python-sphinx and
>>     python3-sphinx. for me, that makes no sense at all: there is a
>>     single set of documentation files built in a package, and it must be
>>     on either one of those packages, but not both. if the software
>>     supports building with python3, then it should depend on
>>     python3-sphinx, if not, it should depend on python-sphinx. by
>>     switching to sphinx{,3} binary packages, this would make that
>>     distinction clearer as well.
>
> This may happen when setup.py depends on Sphinx and automatically calls it
> during build. Such packages have to build-depend on both Python 2 and 3
> versions of Sphinx, even if only one version of documentation actually gets
> shipped.

I don't follow. Why do they need to depend on both? We don't need *nor*
want to build the documentation twice - if upstream's setup.py
automatically calls build_sphinx, then this is something we should be
working around (again, pybuild?) if only to save time on the buildds.

>> My pain points would mostly be fixed with th e simple #1 and #3
>> fixes. Without breaking anything, we could already provide a
>> sphinx3-build binary, and we could file a PR upstream to make
>> SPHINXBUILD more easily overridable. The rest is just a brainstorm of
>> possible medium-term fixes to try and fix that dangling symlink issue.
>>
>> I understand this is a complex situation and I would love to have better
>> answers, but I'm not sure there are any simple answers here. We
>> shouldn't hesitate to make radical changes, however: we have transition
>> tools to help us with that stuff, and if we can upgrade libc and gcc
>> across all of Debian fairly painlessly these days, i don't see why we
>> couldn't articulate such a transition for ~1000 python packages. :)
>
> In my opinion there should be a very strong reason to make changes that
> break packages. I have to deal with Sphinx upstream breaking compatibility
> which results in lots of FTBFS bugs for each major version. And I don’t
> want to introduce even more incompatibility myself.

meh. I saw the current FTBFS transition as a perfect moment to switch
over to python3 myself, but maybe i'm being a little too
enthusiastic. ;)

So maybe there's just nothing else to do here in the short term. I'll
start using "make SPHINXBUILD='python3 -m sphinx'" even though I find
that real clunky. In fact, the more I think about it, the less I see the
point of using the Makefiles at all from debian/rules: i'll probably
just call "python3 -m sphinx" or "setup.py build_sphinx" myself from
there, as needed.

The problem, then, of course, is that I lose that precious
automation. Most Python Debian packages really don't need that much
stuff in debian/rules - this is enough for 90% of the cases:

%:
        dh $@ --with=python2,python3 --buildsystem=pybuild

Yet, when I want to build docs in there, i need to override the build
targets immediately:

override_dh_auto_build:
        dh_auto_build
        # builder=html,man only supported in Sphinx 1.6, not yet released:
        # http://www.sphinx-doc.org/en/master/setuptools.html
        make -C doc html man SPHINXBUILD="python3 -m sphinx"
        # ^untested

I'd like to be able to just assume sphinxdoc will do the right thing and
build me a set of HTML docs (at least). Ideally, I'd be able to choose
the builders (html, man, epub, etc) through the environment, but then
that'd need to change the build-deps as well, something I understand is
basically impossible. ;)

Anyways, I learned a lot through this. Thanks for all the information,
and I hope we can update docs somewhere to reflect the current situation
and move ahead with some better build process for this as well.

A.

-- 
Having failed to discover weapons of mass destruction, Washington
shifted its propaganda to "establishing democracy." That flatly refutes
their earlier claim that the "only question" was whether Saddam would
disarm. But with a sufficiently obedient intellectual class, and loyal
media, the farce can proceed untroubled.
                        - Noam Chomsky, in an interview about Irak

[toc] | [prev] | [next] | [standalone]

#9870

[Multipart message — attachments visible in raw view] — view raw

On Thu, Sep 28, 2017 at 08:27:20AM -0400, Antoine Beaupré wrote:
> > And moving Python 3 packages to /usr/bin/sphinx3-build or something like
> > that will mean diverging from upstream (see below).
>
> Note that we already diverge from upstream in numerous places in Python,
> first and foremost in the naming of the Python binary itself.

How do we diverge there? In my opinion we follow PEP 394 quite closely:
https://www.python.org/dev/peps/pep-0394/

> > Good suggestion, I have submitted a pull request upstream:
> > https://github.com/sphinx-doc/sphinx/pull/4092
>
> Excellent - I meant to do that but ran out of time writing the email in
> the first place. :)
>
> I see, however, it's not getting too much traction.. But they have a
> fair point - I didn't realize there was a difference between variables
> from the environment and from the commandline in Make. Maybe it's good
> enough as it is then.

It is not ‘they’, it is our Simon :) He also replied on this mailing list,
just did not CC you.

> > And there is no such thing as --with=sphinx. There is --with=sphinxdoc,
> > but it operates on already built and installed documentation, so it cannot
> > help with building.
>
> I guess this is what I mean should be automated better. Every time I
> used --with=sphinxdoc, I expected it to build the docs and then
> remembered, when the produced package lacked any docs whatsoever, that I
> needed to build things myself.
>
> A simple --with=sphinxdoc should be sufficient to build sphinx docs. We
> shouldn't expect upstream to do that step in setup.py, as it is rarely
> hooked into the main build. Yes, there's the `build_sphinx` command, but
> it's not hooked into `setup.py build` except in some rare project.

Sphinxdoc is a helper to run after install, not a build system. So it is
not a place for such code.

> This is why I mention pybuild: maybe *that* is where docs should be
> built? I am not sure. In any case, I feel there's a shim missing here,
> and there's a good occasion to leverage the automation we have to
> generate proper build deps and build-time commands.

Maybe pybuild can do this, but it then needs to make some assumptions
about the doc source path, build path, wished formats, etc. For finding source
path it can probably use something like this:

https://github.com/sphinx-doc/sphinx/blob/stable/sphinx/setup_command.py#L111

But it is up to pybuild maintainer (Piotr) to decide whether he wants this
feature or not.

> > What is the problem with using /usr/share/sphinx/scripts/python3/sphinx-build
> > explicitly if you want to force Python 3?
>
> It's hard to find. It's unusal for a user-facing command to be absent
> from the path yet to expect users to find it, especially when one
> version *is* available in the path.

As I said ealier, for most cases just using sphinx-build which is on PATH
should be fine. In advanced cases, it should not be hard to run ‘dpkg -L
python3-sphinx’ to figure out where its scripts are, or use the -m syntax.

> Sure. I just found out about this while writing the email, after almost
> a decade of maintaining python packages. That's a clear symptom of bad
> discovery, although the upstream Makefile changes may help with that
> eventually.

Actually quite many packages tend to provide the -m syntax and even prefer
it to scripts in /usr/bin/. Examples from packages I maintain: -m markdown,
-m keyring, -m nose.

> > Also there is no such thing as sphinx3-build upstream (unlike other packages
> > that follow this convention). So all upstream projects will try to use
> > sphinx-build, not sphinx3-build, even if they are Python 3 only. And if you
> > override this command anyway, you can use two existing ways, I don’t see a
> > need for the third one.
>
> The reasoning here is that is is more discoverable, namely through
> commandline completion, and is consistent with the /usr/bin/python*
> binary naming conventions.

I have just figured out that there *is* bash completion for python3 -m
syntax.

It is not difficult for me to provide the sphinx3-* binaries, but that might
create false assumptions for developers that it is available on other systems
too. It is better to use the -m syntax because it is more universal and works
on any OS.

> > I won’t say that this convention is usual. There are many packages named
> > python-* or python3-* which provide executable scripts.
>
> I would argue that is a strange practice: I tend to ignore packages that
> start with python* unless I am programming something. As a user, they
> are basically useless because I am supposed to code something to use
> those libraries. Now, granted, sphinx is a little special because it
> *is* a programmer-oriented library, but keep in mind it can be used for
> non-programs as well (see the Debian policy for example).
>
> > Let me summarize what you, in my opinion, should do as a maintainer of
> > packages using Sphinx:
> >
> > 1) If your project is not using autodoc: just build-depend on python3-sphinx
> > and use /usr/bin/sphinx-build. There is no need to care about
> > versions.
> > 2) If your project uses autodoc, and is compatible with both Python 2 and 3:
> > same as in 1), just use /usr/bin/sphinx-build.
> > 3) If your project uses autodoc, and is only compatible with Python 3:
> > use ‘python3 -m sphinx’ or ‘python3 setup.py build_sphinx’, depending on the
> > upstream build system.
> > 4) If your project uses autodoc, and is only compatible with Python 2:
> > Port it to python 3. Or use ‘python2 -m sphinx’ or ‘python2 setup.py
> > build_sphinx’.
> >
> > If you think this classification is incomplete, then please let me know
> > where I am wrong.
>
> That is a fair description, of course. It would be great to have build
> step automated. At the very least, the above should be clearly
> documented somewhere, maybe in dh_sphinxdoc's README or
> manpage. Documenting that dh_sphinxdoc doesn't build in its manpage
> would also go a long way, as that rather implicit right now. (ie. it
> doesn't say it builds sphinx docs, but it doesn't say it doesn't either:
> we need to assume that the list of things it does is exhaustive and
> complete, something I never assume about documentation :)

Now it is explicit:
https://anonscm.debian.org/cgit/python-modules/packages/sphinx.git/commit/?id=5b2efffcaae8c915

> Okay then the policy seems to point towards shipping sphinx3-build
> binary, no?

See what I said above (about false assumptions).

> > This may happen when setup.py depends on Sphinx and automatically calls it
> > during build. Such packages have to build-depend on both Python 2 and 3
> > versions of Sphinx, even if only one version of documentation actually gets
> > shipped.
>
> I don't follow. Why do they need to depend on both? We don't need *nor*
> want to build the documentation twice - if upstream's setup.py
> automatically calls build_sphinx, then this is something we should be
> working around (again, pybuild?) if only to save time on the buildds.

I do not say that these packages do the right thing. I just tried to find
a reason why they may be doing this. It might be better to ask maintainers
of these packages for more precise reason.

> So maybe there's just nothing else to do here in the short term. I'll
> start using "make SPHINXBUILD='python3 -m sphinx'" even though I find
> that real clunky. In fact, the more I think about it, the less I see the
> point of using the Makefiles at all from debian/rules: i'll probably
> just call "python3 -m sphinx" or "setup.py build_sphinx" myself from
> there, as needed.

Right, there is no reason for Makefiles. If the build process is not
integrated in upstream’s main build system, there should be no difference
for you which line to write in debian/rules.

> The problem, then, of course, is that I lose that precious
> automation. Most Python Debian packages really don't need that much
> stuff in debian/rules - this is enough for 90% of the cases:
>
> %:
>         dh $@ --with=python2,python3 --buildsystem=pybuild
>
> Yet, when I want to build docs in there, i need to override the build
> targets immediately:
>
> override_dh_auto_build:
>         dh_auto_build
>         # builder=html,man only supported in Sphinx 1.6, not yet released:
>         # http://www.sphinx-doc.org/en/master/setuptools.html
>         make -C doc html man SPHINXBUILD="python3 -m sphinx"
>         # ^untested
>
> I'd like to be able to just assume sphinxdoc will do the right thing and
> build me a set of HTML docs (at least). Ideally, I'd be able to choose
> the builders (html, man, epub, etc) through the environment, but then
> that'd need to change the build-deps as well, something I understand is
> basically impossible. ;)

See what I said above about pybuild.

> Anyways, I learned a lot through this. Thanks for all the information,
> and I hope we can update docs somewhere to reflect the current situation
> and move ahead with some better build process for this as well.

--
Dmitry Shachnev

[toc] | [prev] | [next] | [standalone]

#9871

On 2017-09-28 19:59:12, Dmitry Shachnev wrote:
> On Thu, Sep 28, 2017 at 08:27:20AM -0400, Antoine Beaupré wrote:
>> > And moving Python 3 packages to /usr/bin/sphinx3-build or something like
>> > that will mean diverging from upstream (see below).
>>
>> Note that we already diverge from upstream in numerous places in Python,
>> first and foremost in the naming of the Python binary itself.
>
> How do we diverge there? In my opinion we follow PEP 394 quite closely:
> https://www.python.org/dev/peps/pep-0394/

Ah. Oops. :) I guess I was thinking of virtualenvs...

>> > Good suggestion, I have submitted a pull request upstream:
>> > https://github.com/sphinx-doc/sphinx/pull/4092
>>
>> Excellent - I meant to do that but ran out of time writing the email in
>> the first place. :)
>>
>> I see, however, it's not getting too much traction.. But they have a
>> fair point - I didn't realize there was a difference between variables
>> from the environment and from the commandline in Make. Maybe it's good
>> enough as it is then.
>
> It is not ‘they’, it is our Simon :) He also replied on this mailing list,
> just did not CC you.

Oops again. :) I replied there as well.

[...]

>> This is why I mention pybuild: maybe *that* is where docs should be
>> built? I am not sure. In any case, I feel there's a shim missing here,
>> and there's a good occasion to leverage the automation we have to
>> generate proper build deps and build-time commands.
>
> Maybe pybuild can do this, but it then needs to make some assumptions
> about the doc source path, build path, wished formats, etc. For finding source
> path it can probably use something like this:
>
> https://github.com/sphinx-doc/sphinx/blob/stable/sphinx/setup_command.py#L111
>
> But it is up to pybuild maintainer (Piotr) to decide whether he wants this
> feature or not.

Well, it would sure be nice to have some place for this, of course.

[...]

>> > Also there is no such thing as sphinx3-build upstream (unlike other packages
>> > that follow this convention). So all upstream projects will try to use
>> > sphinx-build, not sphinx3-build, even if they are Python 3 only. And if you
>> > override this command anyway, you can use two existing ways, I don’t see a
>> > need for the third one.
>>
>> The reasoning here is that is is more discoverable, namely through
>> commandline completion, and is consistent with the /usr/bin/python*
>> binary naming conventions.
>
> I have just figured out that there *is* bash completion for python3 -m
> syntax.

What i meant is that "sphinx<TAB>" doesn't show there is a python2 vs
python3 version.

[...]

> Now it is explicit:
> https://anonscm.debian.org/cgit/python-modules/packages/sphinx.git/commit/?id=5b2efffcaae8c915

Excellent, thanks again.

A.

-- 
Pour marcher au pas d'une musique militaire, il n'y a pas besoin de
cerveau, une moelle épinière suffit.
                        - Albert Einstein

[toc] | [prev] | [next] | [standalone]

#9873

Hi Antoine,

Thanks for your post.

On 09/27/2017 12:29 AM, Antoine Beaupré wrote:
>  6. there are currently packages depending on *both* python-sphinx and
>     python3-sphinx. for me, that makes no sense at all: there is a
>     single set of documentation files built in a package, and it must be
>     on either one of those packages, but not both. if the software
>     supports building with python3, then it should depend on
>     python3-sphinx, if not, it should depend on python-sphinx. by
>     switching to sphinx{,3} binary packages, this would make that
>     distinction clearer as well.

It does make sense if the package is a sphinx extension, and it has
tests that you may want to run in both Python 2 and 3 cases. I do have a
few of these cases (even though I'm planning on switching to Python3
support only in the short term).

On 09/28/2017 02:27 PM, Antoine Beaupré wrote:
> A simple --with=sphinxdoc should be sufficient to build sphinx docs.
> We shouldn't expect upstream to do that step in setup.py, as it is
> rarely hooked into the main build.

I don't agree with this. I generally prefer to write things this way:

override_dh_sphinxdoc:
ifeq (,$(findstring nodocs, $(DEB_BUILD_OPTIONS)))
        sphinx-build -b html doc/source \
  $(CURDIR)/debian/python-foo-doc/usr/share/doc/python-foo-doc/html
        dh_sphinxdoc
endif

(sorry for the indentation above, lines are too short in emails)

This way, there's no need to pickup/copy the docs in doc/build, it ends
up directly in the package, ready for dh_sphinxdoc. If I need to add
some variables (for example to fix reproducibility), I can easily do it.
I will be switching to the "python3 -m sphinx" style soon, though that's
unrelated.

Over the years, I found using this method a way better than using
"python{3,} setup.py build_sphinx". I'd suggest you try my way too.

Cheers,

Thomas Goirand (zigo)

[toc] | [prev] | [next] | [standalone]

#9874

Hello guyes.

> override_dh_sphinxdoc:
> ifeq (,$(findstring nodocs, $(DEB_BUILD_OPTIONS)))
                                ^^^^
nodocs or nodoc

I alsa do something like this when there is extensions.

override_dh_sphinxdoc:
ifeq (,$(findstring nodoc, $(DEB_BUILD_OPTIONS)))
	PYBUILD_SYSTEM=custom \
	PYBUILD_BUILD_ARGS="cd docs && PYTHONPATH={build_dir} http_proxy='127.0.0.1:9' {interpreter} -m sphinx -N -bhtml source build/html" dh_auto_build  # HTML generator
	dh_installdocs "docs/build/html" -p python-gpyfft-doc
	dh_sphinxdoc -O--buildsystem=pybuild
endif

cheers

Frederic

[toc] | [prev] | [next] | [standalone]

#9875

On 29/09/17 12:08, PICCA Frederic-Emmanuel wrote:
>> override_dh_sphinxdoc:
>> ifeq (,$(findstring nodocs, $(DEB_BUILD_OPTIONS)))
>                                  ^^^^
> nodocs or nodoc
> 
> I alsa do something like this when there is extensions.
> 
> override_dh_sphinxdoc:
> ifeq (,$(findstring nodoc, $(DEB_BUILD_OPTIONS)))
> 	PYBUILD_SYSTEM=custom \
> 	PYBUILD_BUILD_ARGS="cd docs && PYTHONPATH={build_dir} http_proxy='127.0.0.1:9' {interpreter} -m sphinx -N -bhtml source build/html" dh_auto_build  # HTML generator
> 	dh_installdocs "docs/build/html" -p python-gpyfft-doc
> 	dh_sphinxdoc -O--buildsystem=pybuild
> endif

Or just use the sphinx-generated Makefile if there is one:

override_dh_auto_clean:
	dh_auto_clean
ifeq (,$(filter nodoc, $(DEB_BUILD_OPTIONS)))
	$(MAKE) -C docs clean
endif

override_dh_auto_build: export http_proxy=127.0.0.1:9
override_dh_auto_build: export https_proxy=127.0.0.1:9
override_dh_auto_build:
	dh_auto_build
ifeq (,$(filter nodoc, $(DEB_BUILD_OPTIONS)))
	PYTHONPATH=$(CURDIR) $(MAKE) -C docs html
endif

And you're done. The intent is also clearer.

Ghis

[toc] | [prev] | [next] | [standalone]

#9876

Or just use the sphinx-generated Makefile if there is one:

Except that  when there is an autodoc in the documentation, I like to build the doc with all {interpreters}.
It is a sort of unit test

Cheers

Fred

[toc] | [prev] | [next] | [standalone]

#9910

On 09/29/2017 01:08 PM, PICCA Frederic-Emmanuel wrote:
> Hello guyes.
> 
>> override_dh_sphinxdoc:
>> ifeq (,$(findstring nodocs, $(DEB_BUILD_OPTIONS)))
>                                 ^^^^
> nodocs or nodoc
> 
> I alsa do something like this when there is extensions.
> 
> override_dh_sphinxdoc:
> ifeq (,$(findstring nodoc, $(DEB_BUILD_OPTIONS)))
> 	PYBUILD_SYSTEM=custom \
> 	PYBUILD_BUILD_ARGS="cd docs && PYTHONPATH={build_dir} http_proxy='127.0.0.1:9' {interpreter} -m sphinx -N -bhtml source build/html" dh_auto_build  # HTML generator
> 	dh_installdocs "docs/build/html" -p python-gpyfft-doc
> 	dh_sphinxdoc -O--buildsystem=pybuild
> endif

In fact, I was thinking that probably, it'd be nicer to even do:

ifeq (,$(findstring nodoc, $(DEB_BUILD_PROFILES)))

and have Build-Profiles: <!nodoc> for the python-foo-doc package. This
could even become a standard in the DPMT if everyone agrees.

Thoughts anyone?

Cheers,

Thomas Goirand (zigo)

[toc] | [prev] | [next] | [standalone]

#9911

On 03/10/17 22:46, Thomas Goirand wrote:
> On 09/29/2017 01:08 PM, PICCA Frederic-Emmanuel wrote:
>> Hello guyes.
>>
>>> override_dh_sphinxdoc:
>>> ifeq (,$(findstring nodocs, $(DEB_BUILD_OPTIONS)))
>>                                  ^^^^
>> nodocs or nodoc
>>
>> I alsa do something like this when there is extensions.
>>
>> override_dh_sphinxdoc:
>> ifeq (,$(findstring nodoc, $(DEB_BUILD_OPTIONS)))
>> 	PYBUILD_SYSTEM=custom \
>> 	PYBUILD_BUILD_ARGS="cd docs && PYTHONPATH={build_dir} http_proxy='127.0.0.1:9' {interpreter} -m sphinx -N -bhtml source build/html" dh_auto_build  # HTML generator
>> 	dh_installdocs "docs/build/html" -p python-gpyfft-doc
>> 	dh_sphinxdoc -O--buildsystem=pybuild
>> endif
> 
> In fact, I was thinking that probably, it'd be nicer to even do:
> 
> ifeq (,$(findstring nodoc, $(DEB_BUILD_PROFILES)))
> 
> and have Build-Profiles: <!nodoc> for the python-foo-doc package. This
> could even become a standard in the DPMT if everyone agrees.
> 
> Thoughts anyone?

s/DEB_BUILD_PROFILES/DEB_BUILD_OPTIONS

Quoting the relevant part of the documentation for the nodoc build 
profile [1]:

"Builds that set this profile must also add nodoc to DEB_BUILD_OPTIONS"

[1] https://wiki.debian.org/BuildProfileSpec

Cheers,
Ghis

[toc] | [prev] | [next] | [standalone]

#9912

On 10/04/2017 04:41 AM, Ghislain Vaillant wrote:
> 
> 
> On 03/10/17 22:46, Thomas Goirand wrote:
>> On 09/29/2017 01:08 PM, PICCA Frederic-Emmanuel wrote:
>>> Hello guyes.
>>>
>>>> override_dh_sphinxdoc:
>>>> ifeq (,$(findstring nodocs, $(DEB_BUILD_OPTIONS)))
>>>                                  ^^^^
>>> nodocs or nodoc
>>>
>>> I alsa do something like this when there is extensions.
>>>
>>> override_dh_sphinxdoc:
>>> ifeq (,$(findstring nodoc, $(DEB_BUILD_OPTIONS)))
>>>     PYBUILD_SYSTEM=custom \
>>>     PYBUILD_BUILD_ARGS="cd docs && PYTHONPATH={build_dir}
>>> http_proxy='127.0.0.1:9' {interpreter} -m sphinx -N -bhtml source
>>> build/html" dh_auto_build  # HTML generator
>>>     dh_installdocs "docs/build/html" -p python-gpyfft-doc
>>>     dh_sphinxdoc -O--buildsystem=pybuild
>>> endif
>>
>> In fact, I was thinking that probably, it'd be nicer to even do:
>>
>> ifeq (,$(findstring nodoc, $(DEB_BUILD_PROFILES)))
>>
>> and have Build-Profiles: <!nodoc> for the python-foo-doc package. This
>> could even become a standard in the DPMT if everyone agrees.
>>
>> Thoughts anyone?
> 
> s/DEB_BUILD_PROFILES/DEB_BUILD_OPTIONS
> 
> Quoting the relevant part of the documentation for the nodoc build
> profile [1]:
> 
> "Builds that set this profile must also add nodoc to DEB_BUILD_OPTIONS"
> 
> [1] https://wiki.debian.org/BuildProfileSpec
> 
> Cheers,
> Ghis

Yeah, I saw it, though I was wondering if it was a good idea or not to
attempt to change this. I don't really see the point of having to set
both options, when only a single one should be enough. Also, if you only
set DEB_BUILD_OPTIONS=nodoc, then you end up with an empty -doc package,
which isn't very useful. So, I don't really see the use case of having 2
options here.

Your thoughts?

Cheers,

Thomas Goirand (zigo)

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.maint.python

csiph-web