Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #9660 > unrolled thread

building manpages via setup.py

Back to article view | Back to linux.debian.maint.python

Contents

  building manpages via setup.py PICCA Frederic-Emmanuel <frederic-emmanuel.picca@synchrotron-soleil.fr> - 2017-08-01 16:50 +0200
    Re: building manpages via setup.py Ghislain Vaillant <ghisvail@gmail.com> - 2017-08-01 17:00 +0200
      RE:building manpages via setup.py PICCA Frederic-Emmanuel <frederic-emmanuel.picca@synchrotron-soleil.fr> - 2017-08-01 17:50 +0200
    Re: building manpages via setup.py Piotr Ożarowski <piotr@debian.org> - 2017-08-01 17:40 +0200
      RE:building manpages via setup.py PICCA Frederic-Emmanuel <frederic-emmanuel.picca@synchrotron-soleil.fr> - 2017-08-02 09:20 +0200
        Re: building manpages via setup.py Piotr Ożarowski <piotr@debian.org> - 2017-08-02 09:30 +0200
          Re: building manpages via setup.py Piotr Ożarowski <piotr@debian.org> - 2017-08-02 09:50 +0200
            Re: building manpages via setup.py Ghislain Vaillant <ghisvail@gmail.com> - 2017-08-02 09:50 +0200
              Re: building manpages via setup.py Ghislain Vaillant <ghisvail@gmail.com> - 2017-08-02 10:20 +0200
                RE:building manpages via setup.py PICCA Frederic-Emmanuel <frederic-emmanuel.picca@synchrotron-soleil.fr> - 2017-08-02 10:40 +0200
                  Re: building manpages via setup.py Ghislain Vaillant <ghisvail@gmail.com> - 2017-08-02 10:40 +0200
                    RE:building manpages via setup.py PICCA Frederic-Emmanuel <frederic-emmanuel.picca@synchrotron-soleil.fr> - 2017-08-02 11:20 +0200
                      Re: building manpages via setup.py Ghislain Vaillant <ghisvail@gmail.com> - 2017-08-02 11:30 +0200
                        RE:building manpages via setup.py PICCA Frederic-Emmanuel <frederic-emmanuel.picca@synchrotron-soleil.fr> - 2017-08-02 12:10 +0200
                          Re: building manpages via setup.py Ghislain Vaillant <ghisvail@gmail.com> - 2017-08-02 12:50 +0200
                            RE:building manpages via setup.py PICCA Frederic-Emmanuel <frederic-emmanuel.picca@synchrotron-soleil.fr> - 2017-08-02 14:30 +0200
              RE:building manpages via setup.py PICCA Frederic-Emmanuel <frederic-emmanuel.picca@synchrotron-soleil.fr> - 2017-08-02 10:30 +0200
          RE:building manpages via setup.py PICCA Frederic-Emmanuel <frederic-emmanuel.picca@synchrotron-soleil.fr> - 2017-08-02 09:50 +0200
          RE:building manpages via setup.py PICCA Frederic-Emmanuel <frederic-emmanuel.picca@synchrotron-soleil.fr> - 2017-08-03 12:20 +0200
            RE:building manpages via setup.py PICCA Frederic-Emmanuel <frederic-emmanuel.picca@synchrotron-soleil.fr> - 2017-08-03 15:50 +0200
              RE:building manpages via setup.py PICCA Frederic-Emmanuel <frederic-emmanuel.picca@synchrotron-soleil.fr> - 2017-08-03 16:20 +0200

Page 1 of 2  [1] 2  Next page →

#9660 — building manpages via setup.py

Hello,

I am working on the pyfai package.
This pacakge contain one module with extensions (the important point)

The new upstream version 0.14.0 provide a build_man target via the setup.py

So in ordert to generate the doc I need to do

python setup.py build_man

Now if I look at this target, I can find this code

-----

class BuildMan(Command):
    """Command to build man pages"""
    user_options = []

    def initialize_options(self):
        pass

    def finalize_options(self):
        pass

    def entry_points_iterator(self):
        """Iterate other entry points available on the project."""
        entry_points = self.distribution.entry_points
        console_scripts = entry_points.get('console_scripts', [])
        gui_scripts = entry_points.get('gui_scripts', [])
        scripts = []
        scripts.extend(console_scripts)
        scripts.extend(gui_scripts)
        for script in scripts:
            elements = script.split("=")
            target_name = elements[0].strip()
            elements = elements[1].split(":")
            module_name = elements[0].strip()
            function_name = elements[1].strip()
            yield target_name, module_name, function_name

    def run(self):
        build = self.get_finalized_command('build')
        path = sys.path
        path.insert(0, os.path.abspath(build.build_lib))

        env = dict((str(k), str(v)) for k, v in os.environ.items())
        env["PYTHONPATH"] = os.pathsep.join(path)

        import subprocess

        status = subprocess.call(["mkdir", "-p", "build/man"])
        if status != 0:
            raise RuntimeError("Fail to create build/man directory")

        import tempfile
        import stat
        script_name = None

        entry_points = self.entry_points_iterator()
        for target_name, module_name, function_name in entry_points:
            logger.info("Build man for entry-point target '%s'" % target_name)
            # help2man expect a single executable file to extract the help
            # we create it, execute it, and delete it at the end

            py3 = sys.version_info >= (3, 0)
            try:
                # create a launcher using the right python interpreter
                script_fid, script_name = tempfile.mkstemp(prefix="%s_" % target_name, text=True)
                script = os.fdopen(script_fid, 'wt')
                script.write("#!%s\n" % sys.executable)
                script.write("import %s as app\n" % module_name)
                script.write("app.%s()\n" % function_name)
                script.close()
                # make it executable
                mode = os.stat(script_name).st_mode
                os.chmod(script_name, mode + stat.S_IEXEC)

                # execute help2man
                man_file = "build/man/%s.1" % target_name
                command_line = ["help2man", script_name, "-o", man_file]
                if not py3:
                    # Before Python 3.4, ArgParser --version was using
                    # stderr to print the version
                    command_line.append("--no-discard-stderr")

                p = subprocess.Popen(command_line, env=env)
                status = p.wait()
                if status != 0:
                    raise RuntimeError("Fail to generate '%s' man documentation" % target_name)
            finally:
                # clean up the script
                if script_name is not None:
                    os.remove(script_name)
-----


As you can see this create a launch script for each entry point found in the setup.py and run help2man on it.

For now I would like to use this setup.py without modification.

So what should I do to run

python setup.py build_man with the options provided by pybuild during the normal build in order to let the  generated script find the pyFAI modules and its extensions ?


Second questions what is the right way to generate the man pages for a python application ?

thanks for your help

Frederic

[toc] | [next] | [standalone]

#9661

On 01/08/17 15:15, PICCA Frederic-Emmanuel wrote:
> Hello,
> 
> I am working on the pyfai package.
> This pacakge contain one module with extensions (the important point)
> 
> The new upstream version 0.14.0 provide a build_man target via the setup.py
> 
> So in ordert to generate the doc I need to do
> 
> python setup.py build_man
> 
> Now if I look at this target, I can find this code
> 
> -----
> 
> class BuildMan(Command):
>      """Command to build man pages"""
>      user_options = []
> 
>      def initialize_options(self):
>          pass
> 
>      def finalize_options(self):
>          pass
> 
>      def entry_points_iterator(self):
>          """Iterate other entry points available on the project."""
>          entry_points = self.distribution.entry_points
>          console_scripts = entry_points.get('console_scripts', [])
>          gui_scripts = entry_points.get('gui_scripts', [])
>          scripts = []
>          scripts.extend(console_scripts)
>          scripts.extend(gui_scripts)
>          for script in scripts:
>              elements = script.split("=")
>              target_name = elements[0].strip()
>              elements = elements[1].split(":")
>              module_name = elements[0].strip()
>              function_name = elements[1].strip()
>              yield target_name, module_name, function_name
> 
>      def run(self):
>          build = self.get_finalized_command('build')
>          path = sys.path
>          path.insert(0, os.path.abspath(build.build_lib))
> 
>          env = dict((str(k), str(v)) for k, v in os.environ.items())
>          env["PYTHONPATH"] = os.pathsep.join(path)
> 
>          import subprocess
> 
>          status = subprocess.call(["mkdir", "-p", "build/man"])
>          if status != 0:
>              raise RuntimeError("Fail to create build/man directory")
> 
>          import tempfile
>          import stat
>          script_name = None
> 
>          entry_points = self.entry_points_iterator()
>          for target_name, module_name, function_name in entry_points:
>              logger.info("Build man for entry-point target '%s'" % target_name)
>              # help2man expect a single executable file to extract the help
>              # we create it, execute it, and delete it at the end
> 
>              py3 = sys.version_info >= (3, 0)
>              try:
>                  # create a launcher using the right python interpreter
>                  script_fid, script_name = tempfile.mkstemp(prefix="%s_" % target_name, text=True)
>                  script = os.fdopen(script_fid, 'wt')
>                  script.write("#!%s\n" % sys.executable)
>                  script.write("import %s as app\n" % module_name)
>                  script.write("app.%s()\n" % function_name)
>                  script.close()
>                  # make it executable
>                  mode = os.stat(script_name).st_mode
>                  os.chmod(script_name, mode + stat.S_IEXEC)
> 
>                  # execute help2man
>                  man_file = "build/man/%s.1" % target_name
>                  command_line = ["help2man", script_name, "-o", man_file]
>                  if not py3:
>                      # Before Python 3.4, ArgParser --version was using
>                      # stderr to print the version
>                      command_line.append("--no-discard-stderr")
> 
>                  p = subprocess.Popen(command_line, env=env)
>                  status = p.wait()
>                  if status != 0:
>                      raise RuntimeError("Fail to generate '%s' man documentation" % target_name)
>              finally:
>                  # clean up the script
>                  if script_name is not None:
>                      os.remove(script_name)
> -----
> 
> 
> As you can see this create a launch script for each entry point found in the setup.py and run help2man on it.
> 
> For now I would like to use this setup.py without modification.
> 
> So what should I do to run
> 
> python setup.py build_man with the options provided by pybuild during the normal build in order to let the  generated script find the pyFAI modules and its extensions ?

Simplest I can think of would be to build the extensions inplace 
followed by the call to build_man. Something like:

override_dh_auto_build:
	dh_auto_build
	python3 setup.py build_ext --inplace
	python3 setup.py build_man

I left the http_proxy exports and nodoc guards out for clarity.

> Second questions what is the right way to generate the man pages for a python application ?

Usually via Sphinx if the upstream documentation uses it. Regardless of 
the stack, `help2man` is often considered the poor man choice for 
generating manpages.

Let me know if you'd like me to have a look :-)

Ghis

[toc] | [prev] | [next] | [standalone]

#9663

> Simplest I can think of would be to build the extensions inplace
> followed by the call to build_man. Something like:

> override_dh_auto_build:
>         dh_auto_build
>         python3 setup.py build_ext --inplace
>         python3 setup.py build_man

I do not want to build once more the ext (it takes a lot's of time on my computer and it is a wast of ressources).
I just want to run {interpreter} setup.py build_man <options>

where

<options> are provided by pybuild

> I left the http_proxy exports and nodoc guards out for clarity.

thanks ;)

>  Second questions what is the right way to generate the man pages for a python application ?

> Usually via Sphinx if the upstream documentation uses it. Regardless of
> the stack, `help2man` is often considered the poor man choice for
> generating manpages.

Thanks, I know this but I need more time to convince the upstream to find a better solution :)

> Let me know if you'd like me to have a look :-)

thanks a lot

Fred

[toc] | [prev] | [next] | [standalone]

#9662

[PICCA Frederic-Emmanuel, 2017-08-01]
> This pacakge contain one module with extensions (the important point)
> 
> The new upstream version 0.14.0 provide a build_man target via the setup.py
> 
> So in ordert to generate the doc I need to do
> 
> python setup.py build_man
> 
[...]
> So what should I do to run
> 
> python setup.py build_man with the options provided by pybuild during the normal build in order to let the  generated script find the pyFAI modules and its extensions ?


if you want it to be run only once (for default Python 3 interpreter):

| override_dh_auto_build:
|	dh_auto_build
|	pybuild --system=custom --build --build-args '{interpreter} setup.py build_man' --interpreter=python3


if you want to test all requested Python interpreters:

| override_dh_auto_build:
|	dh_auto_build -- --after-build '{interpreter} setup.py build_man'


there's no need to prepend --after-build argument with
"env PYTHONPATH={build_dir}; " - pybuild sets that to build dir automatically

-- 
GPG: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

[toc] | [prev] | [next] | [standalone]

#9664

Hello piotr

> if you want to test all requested Python interpreters:

I prefer this solution in order to check that the built extensions are working for all the python interpreters.
The doc use autodoc so it is nice to have this functionnality

| override_dh_auto_build:
|       dh_auto_build -- --after-build '{interpreter} setup.py build_man'


It is possible to replace this by 

export PYBUILD_AFTER_BUILD='{interpreter} setup.py build_man'

> there's no need to prepend --after-build argument with
> "env PYTHONPATH={build_dir}; " - pybuild sets that to build dir automatically

Do you mean that I should remove the PYTHONPATH in this ?

override_dh_sphinxdoc
ifeq (,$(findstring nodocs, $(DEB_BUILD_OPTIONS)))
	PYBUILD_SYSTEM=custom \
	PYBUILD_BUILD_ARGS="cd doc && PYTHONPATH={build_dir} http_proxy='127.0.0.1:9' {interpreter} -m sphinx -N -bhtml source build/html" dh_auto_build  # HTML generator
	dh_installdocs "doc/build/html" -p python-pyfai-doc
	dh_sphinxdoc -O--buildsystem=pybuild
endif



--
GPG: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

[toc] | [prev] | [next] | [standalone]

#9665

[PICCA Frederic-Emmanuel, 2017-08-02]
> > if you want to test all requested Python interpreters:
> 
> I prefer this solution in order to check that the built extensions are working for all the python interpreters.
> The doc use autodoc so it is nice to have this functionnality
> 
> | override_dh_auto_build:
> |       dh_auto_build -- --after-build '{interpreter} setup.py build_man'
> 
> 
> It is possible to replace this by 
> 
> export PYBUILD_AFTER_BUILD='{interpreter} setup.py build_man'

export PYBUILD_AFTER_BUILD={interpreter} setup.py build_man

(i.e. without quotes if you want to use Make's export)

> > there's no need to prepend --after-build argument with
> > "env PYTHONPATH={build_dir}; " - pybuild sets that to build dir automatically
> 
> Do you mean that I should remove the PYTHONPATH in this ?

you can drop it, PYTHONPATH and http_proxy should be set by pybuild

> override_dh_sphinxdoc
> ifeq (,$(findstring nodocs, $(DEB_BUILD_OPTIONS)))
> 	PYBUILD_SYSTEM=custom \
> 	PYBUILD_BUILD_ARGS="cd doc && PYTHONPATH={build_dir} http_proxy='127.0.0.1:9' {interpreter} -m sphinx -N -bhtml source build/html" dh_auto_build  # HTML generator
> 	dh_installdocs "doc/build/html" -p python-pyfai-doc
> 	dh_sphinxdoc -O--buildsystem=pybuild
> endif
-- 
GPG: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

[toc] | [prev] | [next] | [standalone]

#9667

[PICCA Frederic-Emmanuel, 2017-08-02]
> > you can drop it, PYTHONPATH and http_proxy should be set by pybuild
> 
> Is it true  for jessie
> 
> I need to support jessie and stretch
> 
> And even debian7...

I didn't test it even for unstable, but IIRC pybuild exports those in
all steps since a long time ago. You'll know after first builds...


PS please don't CC me, I'm subscribing the list
-- 
GPG: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

[toc] | [prev] | [next] | [standalone]

#9669

On 02/08/17 08:44, Piotr Ożarowski wrote:
> [PICCA Frederic-Emmanuel, 2017-08-02]
>>> you can drop it, PYTHONPATH and http_proxy should be set by pybuild
>>
>> Is it true  for jessie
>>
>> I need to support jessie and stretch
>>
>> And even debian7...
> 
> I didn't test it even for unstable, but IIRC pybuild exports those in
> all steps since a long time ago. You'll know after first builds...

Perhaps the LibraryStyleGuide should be updated to reflect on this 
change? I believe we are still advising explicit http_proxy / 
https_proxy exports prior to running sphinx-build.

Ghis

[toc] | [prev] | [next] | [standalone]

#9670

On 02/08/17 09:03, PICCA Frederic-Emmanuel wrote:
>> Perhaps the LibraryStyleGuide should be updated to reflect on this
>> change? I believe we are still advising explicit http_proxy /
>> https_proxy exports prior to running sphinx-build.
> 
> And running sphinx-build does not work expecially if there is extensions in the documentation.
> sphinx-build should be run via pybuild in order to know about the build_dir.
> 
> right ?

At the end of the day, it is just a matter of providing an appropriate 
PYTHONPATH, regardless of whether pybuild is used or not.

The default is $(CURDIR) for the vast majority of packages. But, it may 
also be something else if extension packages are involved, or if the 
package directory is under a different folder than the root directory, 
such as src/.

Ghis

[toc] | [prev] | [next] | [standalone]

#9672

> At the end of the day, it is just a matter of providing an appropriate
> PYTHONPATH, regardless of whether pybuild is used or not.

Yes but to avoid the multiplications of way to provide this PYTHONPATH.

Is it possible to have the recommended way which works for modules and extensions.

once agreed, we should put this in the wiki

Cheers

Fred

[toc] | [prev] | [next] | [standalone]

#9673

On 02/08/17 09:19, PICCA Frederic-Emmanuel wrote:
>> At the end of the day, it is just a matter of providing an appropriate
>> PYTHONPATH, regardless of whether pybuild is used or not.
> 
> Yes but to avoid the multiplications of way to provide this PYTHONPATH.

For the vast majority of packages, the current method listed in 
LibraryStyleGuide applies, i.e.:

PYTHONPATH=. sphinx-build -N -b html <docs_dir> <build_dir>

One can also use the sphinx-generated Makefile if available:

PYTHONPATH=$(CURDIR) $(MAKE) -C <docs_dir> html

Both are simple one-liners and do not rely on pybuild.

> Is it possible to have the recommended way which works for modules and extensions.

If it does not cost much to build the extension inplace, then the 
simplest option is to prefix one of these calls above with:

python3 setup.py build_ext --inplace

If the cost is prohibitive, which arguably applies to a very limited set 
of packages (yours included) then you would use pybuild for that, as 
Piotr kindly suggested.

> once agreed, we should put this in the wiki

Considering the rarity of this use case though, I wonder whether it is 
worth adding a separate section to the style guide.

Ghis

[toc] | [prev] | [next] | [standalone]

#9674

> PYTHONPATH=. sphinx-build -N -b html <docs_dir> <build_dir>

> One can also use the sphinx-generated Makefile if available:

> PYTHONPATH=$(CURDIR) $(MAKE) -C <docs_dir> html

> Both are simple one-liners and do not rely on pybuild.

Yes it works but this is fragile since the organisation of the module can change in the sources.

at least the .pybuild directory is under the responsability of pybuild and we should use pybuild instead of relying
on the maintainer snipset. (typo error, change during the time.)

It would be nice if the doc generation in python shold be standardize.

> If it does not cost much to build the extension inplace, then the
> simplest option is to prefix one of these calls above with:

> python3 setup.py build_ext --inplace

when you have multiple verison of the interpreter you prefer to avoid --inplace.

> If the cost is prohibitive, which arguably applies to a very limited set
> of packages (yours included) then you would use pybuild for that, as
> Piotr kindly suggested.

Yes it depends olsa of the arch. Some are really slow....

> Considering the rarity of this use case though, I wonder whether it is
> worth adding a separate section to the style guide.

Found a total of 643 reverse build-depend(s) for python-all-dev.

not that small

cheers

Fred

[toc] | [prev] | [next] | [standalone]

#9675

On 02/08/17 09:55, PICCA Frederic-Emmanuel wrote:
>> PYTHONPATH=. sphinx-build -N -b html <docs_dir> <build_dir>
> 
>> One can also use the sphinx-generated Makefile if available:
> 
>> PYTHONPATH=$(CURDIR) $(MAKE) -C <docs_dir> html
> 
>> Both are simple one-liners and do not rely on pybuild.
> 
> Yes it works but this is fragile since the organisation of the module can change in the sources.

First, that's very speculative. Second, that's upstream's problem.

I have not seen many upstream projects playing with the layout of their 
modules from one version to the next. If that's the case, then there are 
worst things to be worried about (API breakage for instance) than the docs.

> at least the .pybuild directory is under the responsability of pybuild and we should use pybuild instead of relying
> on the maintainer snipset. (typo error, change during the time.)

The upstream Makefile and conf.py are likely generated by Sphinx itself 
via sphinx-quickstart. Did your upstream tinker with them that much that 
they cannot be trusted?

> It would be nice if the doc generation in python shold be standardize.

Some upstream do use a build_sphinx command, but it is far from common 
and it does not solve the extension module problem.

>> If it does not cost much to build the extension inplace, then the
>> simplest option is to prefix one of these calls above with:
> 
>> python3 setup.py build_ext --inplace
> 
> when you have multiple verison of the interpreter you prefer to avoid --inplace.

Why? All you need is *one* occurrence of the extension modules somewhere 
in the PYTHONPATH in order to generate the docs. Chances are that's how 
upstream generates them.

I fail to picture how this is an issue in practice considering your 
build override will run the isolated pybuilds first and the sphinx call 
last.

>> If the cost is prohibitive, which arguably applies to a very limited set
>> of packages (yours included) then you would use pybuild for that, as
>> Piotr kindly suggested.
> 
> Yes it depends olsa of the arch. Some are really slow....

That's unfortunate, indeed.

>> Considering the rarity of this use case though, I wonder whether it is
>> worth adding a separate section to the style guide.
> 
> Found a total of 643 reverse build-depend(s) for python-all-dev.
> 
> not that small

How is the ratio over all the Python packages? I suspect very small.

[toc] | [prev] | [next] | [standalone]

#9676

> First, that's very speculative. Second, that's upstream's problem.

> The upstream Makefile and conf.py are likely generated by Sphinx itself
> via sphinx-quickstart. Did your upstream tinker with them that much that
> they cannot be trusted?

No this is just that does not fit well with extension.

> Some upstream do use a build_sphinx command, but it is far from common
> and it does not solve the extension module problem.

Yes this is what they are doing. extension are part of the python ec-system but it seems thaht this is not well suported out of the box by sphinx.

# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
# import os
# import sys
# sys.path.insert(0, os.path.abspath('.'))

You need to customize the sys.path in order to find the extensions.

This path can change depending on the setup.py build options so this is not reliable.

> Why? All you need is *one* occurrence of the extension modules somewhere
> in the PYTHONPATH in order to generate the docs. Chances are that's how
> upstream generates them.

Because by experience I find issue in the build system and the python code when building the doc for multiple
interpreter (python2/python3 differences...)

> > Found a total of 643 reverse build-depend(s) for python-all-dev.
> >
> > not that small

> How is the ratio over all the Python packages? I suspect very small.

Found a total of 1968 reverse build-depend(s) for python-all.

not that small 32 % ;)


Cheers

Fred

[toc] | [prev] | [next] | [standalone]

#9677

On 02/08/17 10:45, PICCA Frederic-Emmanuel wrote:
>> First, that's very speculative. Second, that's upstream's problem.
> # If extensions (or modules to document with autodoc) are in another directory,
> # add these directories to sys.path here. If the directory is relative to the
> # documentation root, use os.path.abspath to make it absolute, like shown here.
> #
> # import os
> # import sys
> # sys.path.insert(0, os.path.abspath('.'))
> 
> You need to customize the sys.path in order to find the extensions.

The snippet you quoted is not specific to extension modules but to the 
use of the autodoc feature, which requires the modules to be in the 
PYTHONPATH. The `sys.path.insert` hack is just here so that you don't 
have to specify PYTHONPATH yourself when running the upstream Makefile.

> This path can change depending on the setup.py build options so this is not reliable.

I don't understand how setup.py / build options are affecting the Sphinx 
documentation. You are supposed to either call the generator manually 
via sphinx-build (which the style guide recommends), or using the 
upstream Makefile (which upstream often does).

As far as reliability is concerned, my reference here is upstream. If 
upstream can produce the docs, then I should be able to do it too. And 
they don't have pybuild for that.

>> Why? All you need is *one* occurrence of the extension modules somewhere
>> in the PYTHONPATH in order to generate the docs. Chances are that's how
>> upstream generates them.
> 
> Because by experience I find issue in the build system and the python code when building the doc for multiple
> interpreter (python2/python3 differences...)

Why would you build the docs for each supported Python version, 
considering you'll end up shipping only one instance of the generated 
HTML? I am probably missing something here.

You might as well generate the docs just once with the default Python 2 
or Python 3 interpreter, the same way you would do without extension 
modules, no?

Pardon my insistence, but I really fail to understand what issues you 
are referring to.

>>> Found a total of 643 reverse build-depend(s) for python-all-dev.
>>>
>>> not that small
> 
>> How is the ratio over all the Python packages? I suspect very small.
> 
> Found a total of 1968 reverse build-depend(s) for python-all.
> 
> not that small 32 % ;)

More than I expected, indeed.

And amongst these 643 packages, how many have such large build times 
that the overhead of an additional inplace call would be considered 
prohibitive, I wonder?

Besides, the docs will typically be processed by arch-all builders 
(provided you use -indep targets), so arch-any builds won't even feel 
that overhead, right?

I could be wrong though.

[toc] | [prev] | [next] | [standalone]

#9678

> The snippet you quoted is not specific to extension modules but to the
> use of the autodoc feature, which requires the modules to be in the
> PYTHONPATH. The `sys.path.insert` hack is just here so that you don't
> have to specify PYTHONPATH yourself when running the upstream Makefile.

I just says that this snipset is in the conf.py and I yes we need this for autodoc.

> I don't understand how setup.py / build options are affecting the Sphinx
> documentation. You are supposed to either call the generator manually
> via sphinx-build (which the style guide recommends), or using the
> upstream Makefile (which upstream often does).

autodoc introspect the module so you need a working modules from python, whcih is avhieved via the right PYTHONPATH.
So, during the build phase you can gives  options on the python setup.py command line.
This has an effect on the location where the module  and the extensions are built.

build seems to be the default location BUT when we use pybuild it is under .pybuild

So the default PYTHONPATH does not work.

there is no way for sphinx to know where the module and its extensions are excepty by providing the right PYTHONPATH.

Except if sphinx would provide a build_sphinx command whcih take care of this problem and take into account the parameters of the setup.py.

> As far as reliability is concerned, my reference here is upstream. If
> upstream can produce the docs, then I should be able to do it too. And
> they don't have pybuild for that.

Yes but sometime there si some hack done by the upstream in conf.py or setup.py
in order to look for the module at the standards path (under build/) which doesn not work with pybuild.

> Why would you build the docs for each supported Python version,
> considering you'll end up shipping only one instance of the generated
> HTML? I am probably missing something here.

I use this as a sort of unit test. And it is interesting because I catche a bunch of python2/python3 issues like this.
Then I can find issues between sphinx running on python2 or python3.

This is at least a good way to check thaht all the modules can be laoded.

autodoc is interesting for this.

> You might as well generate the docs just once with the default Python 2
> or Python 3 interpreter, the same way you would do without extension
> modules, no?

> Pardon my insistence, but I really fail to understand what issues you
> are referring to.

incompatibilities vetween python2 and python 3 code.
peoples trie to  write code python2 and python3 compatible but this is not thaht easy...

> More than I expected, indeed.

> And amongst these 643 packages, how many have such large build times
> that the overhead of an additional inplace call would be considered
> prohibitive, I wonder?

I do not know, I just think that we should be able to automatically generate packages for python modules from pip in order to
reduce the effort of pacakging, So there should be only one way to build the documentation , manpages.
Escpecillay wheren there is extensions.


> Besides, the docs will typically be processed by arch-all builders
> (provided you use -indep targets), so arch-any builds won't even feel
> that overhead, right?

Yes except that this is ot because --inplace work that the result of python setup.py build are identical.

it is possible to  forget a module referenced by another module in the setup.py.
It workd with --inplace but it does not work for real after build ans installation.

Cheers

Fred


I could be wrong though.

[toc] | [prev] | [next] | [standalone]

#9671

> Perhaps the LibraryStyleGuide should be updated to reflect on this
> change? I believe we are still advising explicit http_proxy /
> https_proxy exports prior to running sphinx-build.

And running sphinx-build does not work expecially if there is extensions in the documentation.
sphinx-build should be run via pybuild in order to know about the build_dir.

right ?

Fred

[toc] | [prev] | [next] | [standalone]

#9668

> export PYBUILD_AFTER_BUILD={interpreter} setup.py build_man

Thanks, The only drawback I see with this solution is that I want to run

dh_auto_build 2 times,
 - 1 for the arch part (pyhon modules, extensions and manpages)
 - 2 for the indep part (doc build)

nevertheless thans a lot it is flexxible enought for my case.

> you can drop it, PYTHONPATH and http_proxy should be set by pybuild

Is it true  for jessie

I need to support jessie and stretch

And even debian7...


Fred

[toc] | [prev] | [next] | [standalone]

#9681

Hello Piotr,

I am struggling, with the build system.

I will speak about this solution

> | override_dh_auto_build:
> |       dh_auto_build -- --after-build '{interpreter} setup.py build_man'

the code instrumented of the BuildMan is this one

class BuildMan(Command):
    """Command to build man pages"""
    user_options = []

    def initialize_options(self):
        pass

    def finalize_options(self):
        pass

    def run(self):
        build = self.get_finalized_command('build')
        path = sys.path
        path.insert(0, os.path.abspath(build.build_lib))

        print("PYTHONPATH", os.environ.get("PYTHONPATH", None))
        print("sys.path", sys.path, os.path.exists(build.build_lib))
        env = dict((str(k), str(v)) for k, v in os.environ.items())
        env["PYTHONPATH"] = os.pathsep.join(path)
        print("env[PYTHONPATH]", env["PYTHONPATH"])

        import subprocess

        status = subprocess.call(["mkdir", "-p", "build/man"])
        if status != 0:
            raise RuntimeError("Fail to create build/man directory")

        try:
            import tempfile
            import stat
            script_name = None

            # help2man expect a single executable file to extract the help
            # we create it, execute it, and delete it at the end

            # create a launcher using the right python interpreter
            script_fid, script_name = tempfile.mkstemp(prefix="%s_" % PROJECT, text=True)
            script = os.fdopen(script_fid, 'wt')
            script.write("#!%s\n" % sys.executable)
            script.write("import runpy\n")
            script.write("runpy.run_module('%s', run_name='__main__')\n" % PROJECT)
            script.close()

            # make it executable
            mode = os.stat(script_name).st_mode
            os.chmod(script_name, mode + stat.S_IEXEC)

            # execute help2man
            p = subprocess.Popen(["help2man", script_name, "-o", "build/man/silx.1"], env=env)
            status = p.wait()
            if status != 0:
                raise RuntimeError("Fail to generate man documentation")
        finally:
            # clean up the script
            if script_name is not None:
                os.remove(script_name)


When I run the build it generate a pydistconfig file wit this content

D: pybuild plugin_distutils:55: pydistutils config file:
[clean]
all=1
[build]
build-lib=/home/picca/Debian/silx/silx/.pybuild/pythonX.Y_2.7/build
[install]
force=1
install-layout=deb
install-scripts=/usr/bin
install-lib=/usr/lib/python2.7/dist-packages
[easy_install]
allow_hosts=None


But when it comes to the --after step, I get this output

('PYTHONPATH', '/home/picca/Debian/silx/silx/.pybuild/pythonX.Y_2.7/build')
('sys.path', ['/home/picca/Debian/silx/silx/build/lib.linux-i386-2.7', '/home/picca/Debian/silx/silx', '/home/picca/Debian/silx/silx/.pybuild/pythonX.Y_2.7/build', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-i386-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PILcompat', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/pymodules/python2.7'], False)
('env[PYTHONPATH]', '/home/picca/Debian/silx/silx/build/lib.linux-i386-2.7:/home/picca/Debian/silx/silx:/home/picca/Debian/silx/silx/.pybuild/pythonX.Y_2.7/build:/usr/lib/python2.7:/usr/lib/python2.7/plat-i386-linux-gnu:/usr/lib/python2.7/lib-tk:/usr/lib/python2.7/lib-old:/usr/lib/python2.7/lib-dynload:/usr/local/lib/python2.7/dist-packages:/usr/lib/python2.7/dist-packages:/usr/lib/python2.7/dist-packages/PILcompat:/usr/lib/python2.7/dist-packages/gtk-2.0:/usr/lib/pymodules/python2.7')

So as you can see the path returned by build.build_lib
seems to be

/home/picca/Debian/silx/silx/build/lib.linux-i386-2.7'

and not

/home/picca/Debian/silx/silx/.pybuild/pythonX.Y_2.7/build (provided via pydist)

So my question is what is wrong.

self.get_finalized_command('build')

doesn not take into account the pydistconfig file ?


Thanks for your help


Frederic

[toc] | [prev] | [next] | [standalone]

#9683

So I am sltill investigating

with some debug output, I can see this the first build part gives

dh_auto_build -- --after-build '{interpreter} setup.py build_man'
	pybuild --build -i python{version} -p 2.7 --after-build "{interpreter} setup.py build_man"
D: pybuild pybuild:474: version: 2.20170125
D: pybuild pybuild:475: ['/usr/bin/pybuild', '--build', '-i', 'python{version}', '-p', '2.7', '--after-build', '{interpreter} setup.py build_man']
D: pybuild pybuild:36: cfg: Namespace(after_build='{interpreter} setup.py build_man', after_clean=None, after_configure=None, after_install=None, after_test=None, before_build=None, before_clean=None, before_configure=None, before_install=None, before_test=None, build_args=None, build_only=True, clean_args=None, clean_only=False, configure_args=None, configure_only=False, custom_tests=False, destdir='debian/tmp', detect_only=False, dir='/home/picca/Debian/silx/silx', disable=None, ext_destdir=None, ext_pattern='\\.so(\\.[^/]*)?$', install_args=None, install_dir=None, install_only=False, interpreter=['python{version}'], list_systems=False, name='silx', quiet=False, really_quiet=False, system=None, test_args=None, test_nose=False, test_only=False, test_pytest=False, test_tox=False, verbose=True, versions=['2.7'])
D: pybuild pybuild:103: detected build system: distutils (certainty: 61%)
I: pybuild base:184: /usr/bin/python setup.py build 
D: pybuild tools:217: invoking: /usr/bin/python setup.py build 
INFO:silx.setup:Use setuptools
running build
running build_py
creating /home/picca/Debian/silx/silx/.pybuild/pythonX.Y_2.7/build/silx    <----- right path

the after part execute the build_man part BUT

I: pybuild pybuild:242: python2.7 setup.py build_man
D: pybuild tools:217: invoking: python2.7 setup.py build_man
INFO:silx.setup:Use setuptools
running build_man
running build_py
creating build/lib.linux-i386-2.7
creating build/lib.linux-i386-2.7/silx     <- ko


So the after part does not seems to take into account the .pydistutils.cfg file ???

Is it normal ?

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.debian.maint.python

csiph-web