Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #8427 > unrolled thread

Properly splitting Python "-doc" packages

Back to article view | Back to linux.debian.maint.python

Contents

  Properly splitting Python "-doc" packages Tiago Ilieve <tiago.myhro@gmail.com> - 2016-04-14 15:20 +0200
    Re: Properly splitting Python "-doc" packages Piotr Ożarowski <piotr@debian.org> - 2016-04-14 16:00 +0200
      Re: Properly splitting Python "-doc" packages Tiago Ilieve <tiago.myhro@gmail.com> - 2016-04-14 17:10 +0200
        Re: Properly splitting Python "-doc" packages Piotr Ożarowski <piotr@debian.org> - 2016-04-14 17:40 +0200
          Re: Properly splitting Python "-doc" packages Tiago Ilieve <tiago.myhro@gmail.com> - 2016-04-14 22:20 +0200
            Re: Properly splitting Python "-doc" packages Piotr Ożarowski <piotr@debian.org> - 2016-04-14 23:30 +0200
              Re: Properly splitting Python "-doc" packages Tiago Ilieve <tiago.myhro@gmail.com> - 2016-04-15 08:40 +0200
            Re: Properly splitting Python "-doc" packages Ben Finney <ben+debian@benfinney.id.au> - 2016-04-15 06:00 +0200

#8427 — Properly splitting Python "-doc" packages

Hi,

I was working on the "boostrap-vz" package and noticed something
really annoying when creating a "boostrap-vz-doc"[1] binary package
with its Sphinx documentation: the actual Python files that composes
the application weren't being packaged on the main "boostrap-vz" one.
I had to add "usr/lib/python*/dist-packages/bootstrapvz/" to its
"debian/boostrap-vz.install"[2] to fix this, but I feel this is wrong.
Looking at other Python packages that have separate "-doc" packages,
like "python-django", they don't seem to need this.

Also, its "*.egg-info/" folder also went missing, as we can see with "debdiff":

Files in first .deb but not in second
-------------------------------------
-rw-r--r--  root/root
/usr/lib/python2.7/dist-packages/bootstrap_vz-0.9.5.egg-info/PKG-INFO
-rw-r--r--  root/root
/usr/lib/python2.7/dist-packages/bootstrap_vz-0.9.5.egg-info/dependency_links.txt
-rw-r--r--  root/root
/usr/lib/python2.7/dist-packages/bootstrap_vz-0.9.5.egg-info/entry_points.txt
-rw-r--r--  root/root
/usr/lib/python2.7/dist-packages/bootstrap_vz-0.9.5.egg-info/requires.txt
-rw-r--r--  root/root
/usr/lib/python2.7/dist-packages/bootstrap_vz-0.9.5.egg-info/top_level.txt

This broke its entry point scripts (at least I finally find out the
cause of the "pkg_resources.DistributionNotFound" error), so I had to
either add custom ones[3] or add
"usr/lib/python2.7/dist-packages/bootstrap_vz-*.egg-info/" to its
"debian/boostrap-vz.install" file as well.

Any ideas on what I missed here?

Regards,
Tiago.

[1]: https://anonscm.debian.org/git/cloud/bootstrap-vz.git/commit/?id=899e841f89d17418de77e5d7f56ff48627415e79
[2]: https://anonscm.debian.org/git/cloud/bootstrap-vz.git/commit/?id=7d0cf538f8e806f83529b3b7cad9af3ee93cca42
[3]: https://anonscm.debian.org/git/cloud/bootstrap-vz.git/commit/?id=d8ec5b17af1f96d9b1221963abf5abc3ef087900

-- 
Tiago "Myhro" Ilieve
Blog: https://blog.myhro.info/
GitHub: https://github.com/myhro
LinkedIn: https://br.linkedin.com/in/myhro
Montes Claros - MG, Brasil

[toc] | [next] | [standalone]

#8428

[Tiago Ilieve, 2016-04-14]
> I was working on the "boostrap-vz" package and noticed something
> really annoying when creating a "boostrap-vz-doc"[1] binary package
> with its Sphinx documentation: the actual Python files that composes
> the application weren't being packaged on the main "boostrap-vz" one.
> I had to add "usr/lib/python*/dist-packages/bootstrapvz/" to its
> "debian/boostrap-vz.install"[2] to fix this, but I feel this is wrong.
> Looking at other Python packages that have separate "-doc" packages,
> like "python-django", they don't seem to need this.

that's because python-django is using --buildsystem=pybuild in dh; if you add

  export PYBUILD_NAME=bootstrapvz

it will install .py / egg-info files into python-bootstrapvz binary package
(please add python-bootstrapvz binary package, BTW)

> Also, its "*.egg-info/" folder also went missing, as we can see with "debdiff":

"usr/lib/python*/dist-packages/bootstrapvz/" in bootstrap-vz.install
doesn't match /usr/lib/python2.7/dist-packages/bootstrap_vz-0.9.5.egg-info,
add this line if you want egg-info to be included:

  /usr/lib/python*/dist-packages/bootstrap_vz-*.egg-info/
-- 
Piotr Ożarowski                         Debian GNU/Linux Developer
www.ozarowski.pl          www.griffith.cc           www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

[toc] | [prev] | [next] | [standalone]

#8429

Hi Piotr,

On 14 April 2016 at 10:52, Piotr Ożarowski <piotr@debian.org> wrote:
> that's because python-django is using --buildsystem=pybuild in dh; if you add
>
>   export PYBUILD_NAME=bootstrapvz
>
> it will install .py / egg-info files into python-bootstrapvz binary package
> (please add python-bootstrapvz binary package, BTW)

I had made a test with Pybuild, but ended up facing the same problem.
Now I see that the problem is that the variable "PYBUILD_NAME" should
have a value of "bootstrap-vz" (notice the hyphen), not "boostrapvz".
Probably because this has to be consistent with whatever is the
package name defined in "setup.py", right?

Anyway, if the package is named "python-boostrapvz" it ended up
properly packaging the files, but as I've mentioned here a couple
times before[1][2], I don't want to call any Python application
package as "python-*". bootstrap-vz is a CLI application and should
end up in a package named "bootstrap-vz". The problem is that if I do
this, I'll and up with an empty binary package package again. Is there
a way to tell Pybuild that the Python files should end up in the
package "bootstrap-vz" and not "python-bootstrap-vz"?

> "usr/lib/python*/dist-packages/bootstrapvz/" in bootstrap-vz.install
> doesn't match /usr/lib/python2.7/dist-packages/bootstrap_vz-0.9.5.egg-info,
> add this line if you want egg-info to be included:
>
>   /usr/lib/python*/dist-packages/bootstrap_vz-*.egg-info/

If I can fix the package name mentioned above, Pybuild will add the
"*.egg-info/" folder as well.

Regards,
Tiago.

[1]: https://lists.debian.org/debian-python/2016/04/msg00017.html
[2]: https://lists.debian.org/debian-python/2016/04/msg00027.html

-- 
Tiago "Myhro" Ilieve
Blog: https://blog.myhro.info/
GitHub: https://github.com/myhro
LinkedIn: https://br.linkedin.com/in/myhro
Montes Claros - MG, Brasil

[toc] | [prev] | [next] | [standalone]

#8430

[Tiago Ilieve, 2016-04-14]
> On 14 April 2016 at 10:52, Piotr Ożarowski <piotr@debian.org> wrote:
> > that's because python-django is using --buildsystem=pybuild in dh; if you add
> >
> >   export PYBUILD_NAME=bootstrapvz
> >
> > it will install .py / egg-info files into python-bootstrapvz binary package
> > (please add python-bootstrapvz binary package, BTW)
> 
> I had made a test with Pybuild, but ended up facing the same problem.
> Now I see that the problem is that the variable "PYBUILD_NAME" should
> have a value of "bootstrap-vz" (notice the hyphen), not "boostrapvz".
> Probably because this has to be consistent with whatever is the
> package name defined in "setup.py", right?

PYBUILD_NAME is used to guess the Debian binary package name (and that's
THE only thing it is used for), whatever is in setup.py doesn't matter.

If you use PYBUILD_NAME=foo, it will install into:

  debian/python-foo/    # python2.X and python2.X-dbg
  debian/python3-foo/   # python3.X and python3.X-dbg
  debian/pypy-foo/      # pypy

that's all, there's no more magic with PYBUILD_NAME

> Anyway, if the package is named "python-boostrapvz" it ended up

that's the correct binary package name for public Python 2.X "boostrapvz" module,
"boostrap_vz" is egg/dist name which we ignore. For module name
"boostrap_vz" correct binary package names would be python-boostrap-vz

> properly packaging the files, but as I've mentioned here a couple
> times before[1][2], I don't want to call any Python application
> package as "python-*". bootstrap-vz is a CLI application and should
> end up in a package named "bootstrap-vz". The problem is that if I do

then PLEASE PLEASE PLEASE do not install as public modules - install into
/usr/share/boostrap-vz/ f.e. with this pybuild args:

  export PYBUILD_INSTALL_ARGS=--install-lib=/usr/share/boostrap-vz/ --install-scripts=/usr/share/boostrap-vz/

and

  /usr/share/boostrap-vz/boostrap-vz /usr/bin/boostrap-vz
  /usr/share/boostrap-vz/boostrap-remote /usr/bin/boostrap-remote
  /usr/share/boostrap-vz/boostrap-server /usr/bin/boostrap-server

in debian/boostrap-vz.links

> this, I'll and up with an empty binary package package again. Is there
> a way to tell Pybuild that the Python files should end up in the
> package "bootstrap-vz" and not "python-bootstrap-vz"?

see PYBUILD_INSTALL_ARGS above (and use PYBUILD_DESTDIR=debian/bootstrap-vz/
instead of PYBUILD_NAME)
-- 
Piotr Ożarowski                         Debian GNU/Linux Developer
www.ozarowski.pl          www.griffith.cc           www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

[toc] | [prev] | [next] | [standalone]

#8431

Piotr,

On 14 April 2016 at 12:33, Piotr Ożarowski <piotr@debian.org> wrote:
> PYBUILD_NAME is used to guess the Debian binary package name (and that's
> THE only thing it is used for), whatever is in setup.py doesn't matter.
>
> If you use PYBUILD_NAME=foo, it will install into:
>
>   debian/python-foo/    # python2.X and python2.X-dbg
>   debian/python3-foo/   # python3.X and python3.X-dbg
>   debian/pypy-foo/      # pypy
>
> that's all, there's no more magic with PYBUILD_NAME

Got it. Thanks for the explanation.

> then PLEASE PLEASE PLEASE do not install as public modules - install into
> /usr/share/boostrap-vz/ f.e. with this pybuild args:
>
>   export PYBUILD_INSTALL_ARGS=--install-lib=/usr/share/boostrap-vz/ --install-scripts=/usr/share/boostrap-vz/
>
> and
>
>   /usr/share/boostrap-vz/boostrap-vz /usr/bin/boostrap-vz
>   /usr/share/boostrap-vz/boostrap-remote /usr/bin/boostrap-remote
>   /usr/share/boostrap-vz/boostrap-server /usr/bin/boostrap-server
>
> in debian/boostrap-vz.links
>
> see PYBUILD_INSTALL_ARGS above (and use PYBUILD_DESTDIR=debian/bootstrap-vz/
> instead of PYBUILD_NAME)

Ok. As this seems to be considered very wrong, I've separated the
package[1], between "bootstrap-vz" and "python-bootstrap-vz". The
first one contains binaries/man pages/etc. and the later contains the
library with everything packaged by Pybuild.

I've tried to use "/usr/share/boostrap-vz" for it, but as it run as
root it ends up writing "*.pyc" files in there, resulting in a
non-clean package removal. I guess the split between application and
library package is a better form of organization.

Thanks for your help and your awesome work in Pybuild!

Regards,
Tiago.

[1]: https://anonscm.debian.org/git/cloud/bootstrap-vz.git/commit/?id=1c075a7c300fa541abe2ccdcf8c5ab35128f7a76
-- 
Tiago "Myhro" Ilieve
Blog: https://blog.myhro.info/
GitHub: https://github.com/myhro
LinkedIn: https://br.linkedin.com/in/myhro
Montes Claros - MG, Brasil

[toc] | [prev] | [next] | [standalone]

#8432

[Multipart message — attachments visible in raw view] — view raw

[Tiago Ilieve, 2016-04-14]
> > then PLEASE PLEASE PLEASE do not install as public modules - install into
> > /usr/share/boostrap-vz/ f.e. with this pybuild args:
> >
> >   export PYBUILD_INSTALL_ARGS=--install-lib=/usr/share/boostrap-vz/ --install-scripts=/usr/share/boostrap-vz/
> >
> > and
> >
> >   /usr/share/boostrap-vz/boostrap-vz /usr/bin/boostrap-vz
> >   /usr/share/boostrap-vz/boostrap-remote /usr/bin/boostrap-remote
> >   /usr/share/boostrap-vz/boostrap-server /usr/bin/boostrap-server
> >
> > in debian/boostrap-vz.links
> >
> > see PYBUILD_INSTALL_ARGS above (and use PYBUILD_DESTDIR=debian/bootstrap-vz/
> > instead of PYBUILD_NAME)
> 
> Ok. As this seems to be considered very wrong, I've separated the
> package[1], between "bootstrap-vz" and "python-bootstrap-vz". The

if you decide to go this way, please use python-bootstrapvz, not
python-bootstrap-vz (module name is bootstrapvz, not bootstrap-vz)

> first one contains binaries/man pages/etc. and the later contains the
> library with everything packaged by Pybuild.
> 
> I've tried to use "/usr/share/boostrap-vz" for it, but as it run as
> root it ends up writing "*.pyc" files in there, resulting in a

I copy-pasted your typo in package name so dh_python2 didn't find the
right directory and didn't do its job.

See attached patch (now it uses private dir)
-- 
Piotr Ożarowski                         Debian GNU/Linux Developer
www.ozarowski.pl          www.griffith.cc           www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

[toc] | [prev] | [next] | [standalone]

#8434

Piotr,

On 14 April 2016 at 18:28, Piotr Ożarowski <piotr@debian.org> wrote:
> if you decide to go this way, please use python-bootstrapvz, not
> python-bootstrap-vz (module name is bootstrapvz, not bootstrap-vz)
>
> I copy-pasted your typo in package name so dh_python2 didn't find the
> right directory and didn't do its job.
>
> See attached patch (now it uses private dir)

I've reverted[1] the package split and make a few changes based on
your patch. Nice to see that it wasn't working because of a typo and
not anything more serious.

I also like the idea of not hardcoding a Python version (even a Python
2 one) in some "install" file path, as now could be changed from
"usr/lib/python2.7/dist-packages/" to "usr/share/".

Thanks again for the help and the patch.

-

Ben,

On 15 April 2016 at 00:57, Ben Finney <ben+debian@benfinney.id.au> wrote:
> Is there really a need for this separation? If the Python modules are
> installed to an application-private directory, then by definition they
> will not be publicly importable. So the Python libraries don't make much
> sense as a separately installable package.

Mostly because of the problem that I faced earlier, where the package
hadn't worked as I expected because of the "*.pyc" file. Turns out
that this was caused by a typo spotted by Piotr. The package is now
being properly in a private application directory.

Regards,
Tiago.

[1]: https://anonscm.debian.org/git/cloud/bootstrap-vz.git/commit/?id=c25f5cc8dd70456fb25e666687f68419c553059e

-- 
Tiago "Myhro" Ilieve
Blog: https://blog.myhro.info/
GitHub: https://github.com/myhro
LinkedIn: https://br.linkedin.com/in/myhro
Montes Claros - MG, Brasil

[toc] | [prev] | [next] | [standalone]

#8433

Tiago Ilieve <tiago.myhro@gmail.com> writes:

> Ok. As this seems to be considered very wrong, I've separated the
> package[1], between "bootstrap-vz" and "python-bootstrap-vz". The
> first one contains binaries/man pages/etc. and the later contains the
> library with everything packaged by Pybuild.

Is there really a need for this separation? If the Python modules are
installed to an application-private directory, then by definition they
will not be publicly importable. So the Python libraries don't make much
sense as a separately installable package.

Why not simply have the application package also contain the private
Python libraries it needs?

For an example of how I've done this, see my ongoing refactor of ‘dput’
<URL:https://notabug.org/bignose/dput/src/packaging-upgrade/debian>. The
pacakging uses Pybuild to manage the Python libraries and installs them
to an application-private location.

There is no separate Python library package, because ‘dput’ is the only
binary package that would use them and it just installs them itself.

-- 
 \        “A right is not what someone gives you; it's what no one can |
  `\                                     take from you.” —Ramsey Clark |
_o__)                                                                  |
Ben Finney

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.maint.python

csiph-web