Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #8177 > unrolled thread

New, updated pip coming

Back to article view | Back to linux.debian.maint.python

Contents

  New, updated pip coming Barry Warsaw <barry@debian.org> - 2016-01-29 20:30 +0100
    Re: New, updated pip coming Robert Collins <robertc@robertcollins.net> - 2016-01-29 23:40 +0100
      Re: New, updated pip coming Donald Stufft <donald@stufft.io> - 2016-01-29 23:50 +0100
      Re: New, updated pip coming Barry Warsaw <barry@debian.org> - 2016-01-30 00:30 +0100
        Re: New, updated pip coming Julien Cristau <julien.cristau@logilab.fr> - 2016-02-01 14:40 +0100
    Re: New, updated pip coming Piotr Ożarowski <piotr@debian.org> - 2016-02-01 15:10 +0100
      Re: New, updated pip coming Barry Warsaw <barry@debian.org> - 2016-02-01 16:00 +0100

#8177 — New, updated pip coming

[Multipart message — attachments visible in raw view] — view raw

TL;DR By the end of today, I expect to upload pip 8.0.2 to unstable, finally
bringing us up to the latest version.

It's been a long slog, with many people helping out.  I hope that all the hard
work done to get us here means it will be much, much easier to track new pip,
virtualenv, and pyvenv updates going forward.  And now, details!

pip requires the use of PEP 427 wheels, which are essentially zip files of
importable, pure-Python, bilingual code, with a little bit of metadata.  Put a
wheel on sys.path and Python can import from it.  pip and friends use wheels
because it ensures that they can't break themselves when their dependent
packages are updated.  E.g. if pip depends on requests, and you `pip install
--upgrade requests` it's possible the new version would be incompatible with
pip.  Then you'd have a broken pip on your hands.  Remember that you can use
pip to install pre-release packages too.

So by putting the wheels on sys.path *before* anything else, even if you pip
install an incompatible requests, it won't break pip itself.

pip "vendorizes" (i.e. bundles) the wheels of its dependencies with known good
versions, but this breaks Debian policy, so we have to unbundle these.
Several upstreams use this technique, with requests and urllib3 as probably
the most popular examples.  In all these cases, we have to remove the bundled
wheels and hope that the versions of their dependencies in Debian remain
compatible.  It's a chore, but that's our job as distro developers, and it's
the best option given the conflicting requirements.

Fortunately, we can create our own wheels and use them with pip and friends.
Until now, we had to build the wheels at the time that the dependent packages
where built, and we had to have a corresponding foo-whl binary package in the
archives, because we could only use setup.py's bdist_wheel command (which
comes with python{,3}-wheel) to create these.  With 15 run time dependencies
as of pip 7.1.2, this has become untenable, and was a huge reason why Debian's
pip has lagged so far behind upstream.  (That, and a few ITPs for new
dependencies. ;)

Along comes dirtbike.  https://github.com/paulproteus/dirtbike

While it has been much discussed and attempted over the years, it would be
really nice to have a tool that turned installed Python packages into wheels.
I think we discussed such a tool for Debian at last year's (two years ago?)
Debian BOF at Pycon.  Anyway, Asheesh Laroia finally took hold of the, ahem,
wheel, and wrote the initial implementation, which he called dirtbike.  Since
then I've been enhancing it to work with several oddball cases in the Debian
archive, and now it's good enough to turn pip's dependencies into wheels.

Yesterday I uploaded the last of the new pip dependencies (modulo any
surprises in 8.0.2 ;), and dirtbike so now all the pieces are in place.  What
pip now does during its own build process, is turn its installed dependencies
into .whl files and puts them in the policy defined location.  This allows us
to eliminate all those scattered -whl binary packages, and just
Build-Depend/Built-Using in pip to install the standard Python packages,
rewheel them, and put them in a python-pip-whl package.  Going forward, this
will make life so much easier, because only python-pip needs to know anything
about wheels, even if new dependencies are added.

I have everything working locally.  I need to update the master branch for
pip, which involves many curse words hurled at git-dpm (and that's a post for
another day!).  I also have to update my local branch to 8.0.2, but I'm
assuming there won't be any new surprises there.

Eventually, I'll be using the same techique for virtualenv and pyvenv, but not
right away.

Huge thanks also go to Donald Stufft, who always is so helpful with various
technical details, and works really hard to understand our needs and balance
them with upstream's.

Finally, we need to update Python policy to describe the new wheel policy.
Fortunately, this is much better because it removes constraints on other
packages, and reduces the number of -whl binary packages in the distro.  Yes,
I will update those packages that already ship -whls, or file bugs, but only
once I'm sure everything is working correctly.

Below is the policy diff, which I'll commit to the python-defaults bzr.

As always, once everything's uploaded, please do let me know via this list or
BTS, if you notice any problems with pip.

Cheers,
-Barry

=== modified file 'debian/python-policy.sgml'
--- debian/python-policy.sgml	2016-01-24 06:01:19 +0000
+++ debian/python-policy.sgml	2016-01-29 16:49:59 +0000
@@ -458,13 +458,12 @@
         <p>
           <url id="http://legacy.python.org/dev/peps/pep-0427/"
                name="PEP 427">
-          defines a built-package format called "wheels", which is a zip
-          format archive containing Python code and a "dist-info" metadata
+          defines a built-package format called "wheels", a zip
+          archive containing Python code and a "dist-info" metadata
           directory, in a single file named with the .whl suffix.  As zip
           files, wheels containing pure-Python can be put on sys.path and
           modules in the wheel can be imported directly by Python's "import"
-          statement. (Importing extension modules from wheels is not yet
-          supported as of Python 3.4.)
+          statement.
         </p><p>
           Except as described below, packages must not build or provide
           wheels.  They are redundant to the established way of providing
@@ -475,18 +474,20 @@
         </p><p>
           A very limited set of wheel packages are available in the archive,
           but these support the narrow purpose of enabling
-          the <prgn>pip</prgn> tool, in a Debian policy compliant way.  The
-          set of packages providing wheels for this purpose are (by source
-          package name): chardet, distlib, html5lib, python-colorama,
-          python-pip, python-setuptools, python-urllib3, requests, and six.
+          the <prgn>pip</prgn>, <prgn>virtualenv</prgn>,
+          and <prgn>pyvenv</prgn> tools in a Debian policy compliant way.
+          These packages build their own dependent wheels through the use of
+          the <prgn>dirtbike</prgn> "rewheeling" tool, which takes installed
+          Debian packages and turns them back into wheels.  Only universal
+          wheels (i.e. pure-Python, Python 2 and 3 compatible packages) are
+          supported.  Since only the programs that require wheels need build
+          them, only they may provide <var>-whl</var> packages,
+          e.g. <package>python-pip-whl</package>.
         </p><p>
-          Wheel packages supporting <prgn>pyvenv</prgn> and <prgn>pip</prgn>
-          are named with the <var>python-</var> prefix, and the <var>-whl</var>
-          suffix, e.g. <package>python-chardet-whl</package>.  When these
-          binary packages are installed, their .whl files must be placed in
-          the /usr/share/python-wheels directory.  Such wheels must be built
-          with the <tt>--universal</tt> flag so as to generate wheels
-          compatible with both Python 2 and Python 3.
+          When these binary packages are installed, .whl files must be placed
+          in the /usr/share/python-wheels directory.  The location inside a
+          virtual environment will be rooted in the virtual environment,
+          instead of in /usr.
         </p>
       </sect>
       <sect id="package_names">

[toc] | [next] | [standalone]

#8180

I am confused. Here's my understanding of things...

 - Pip doesn't need wheels at all - its vendoring technique doesn't use wheels.
 - Virtualenv needs to install pip, wheel, setuptools when it makes a
new environment, and it does that by some oogly code
 - venv might do the same thing, but I haven't check its actual implementation
 - the pip-whl package was used to give virtualenv a thing to use to
install pip when making a virtualenv

If that aligns with your understanding, I'll put it down to the prose
you wrote rather than us actually having a different worldview; OTOH
if it doesn't, I'd like to find out where I'm wrong, so as to improve
my understanding.

With my understanding in mind - Can I just check something?

 If a new requests package is built, uploaded to the archive and
apt-get installed on my machine, and I then do:
virtualenv test
. test/bin/activate
pip install foobar
 ^--- what version of requests will be used by this in-virtualenv
invocation of pip?

>From your description I expect it to be the version of requests that
was dpkg installed at the time the pip deb was built?

-Rob


On 30 January 2016 at 08:28, Barry Warsaw <barry@debian.org> wrote:
> TL;DR By the end of today, I expect to upload pip 8.0.2 to unstable, finally
> bringing us up to the latest version.
>
> It's been a long slog, with many people helping out.  I hope that all the hard
> work done to get us here means it will be much, much easier to track new pip,
> virtualenv, and pyvenv updates going forward.  And now, details!
>
> pip requires the use of PEP 427 wheels, which are essentially zip files of
> importable, pure-Python, bilingual code, with a little bit of metadata.  Put a
> wheel on sys.path and Python can import from it.  pip and friends use wheels
> because it ensures that they can't break themselves when their dependent
> packages are updated.  E.g. if pip depends on requests, and you `pip install
> --upgrade requests` it's possible the new version would be incompatible with
> pip.  Then you'd have a broken pip on your hands.  Remember that you can use
> pip to install pre-release packages too.
>
> So by putting the wheels on sys.path *before* anything else, even if you pip
> install an incompatible requests, it won't break pip itself.
>
> pip "vendorizes" (i.e. bundles) the wheels of its dependencies with known good
> versions, but this breaks Debian policy, so we have to unbundle these.
> Several upstreams use this technique, with requests and urllib3 as probably
> the most popular examples.  In all these cases, we have to remove the bundled
> wheels and hope that the versions of their dependencies in Debian remain
> compatible.  It's a chore, but that's our job as distro developers, and it's
> the best option given the conflicting requirements.
>
> Fortunately, we can create our own wheels and use them with pip and friends.
> Until now, we had to build the wheels at the time that the dependent packages
> where built, and we had to have a corresponding foo-whl binary package in the
> archives, because we could only use setup.py's bdist_wheel command (which
> comes with python{,3}-wheel) to create these.  With 15 run time dependencies
> as of pip 7.1.2, this has become untenable, and was a huge reason why Debian's
> pip has lagged so far behind upstream.  (That, and a few ITPs for new
> dependencies. ;)
>
> Along comes dirtbike.  https://github.com/paulproteus/dirtbike
>
> While it has been much discussed and attempted over the years, it would be
> really nice to have a tool that turned installed Python packages into wheels.
> I think we discussed such a tool for Debian at last year's (two years ago?)
> Debian BOF at Pycon.  Anyway, Asheesh Laroia finally took hold of the, ahem,
> wheel, and wrote the initial implementation, which he called dirtbike.  Since
> then I've been enhancing it to work with several oddball cases in the Debian
> archive, and now it's good enough to turn pip's dependencies into wheels.
>
> Yesterday I uploaded the last of the new pip dependencies (modulo any
> surprises in 8.0.2 ;), and dirtbike so now all the pieces are in place.  What
> pip now does during its own build process, is turn its installed dependencies
> into .whl files and puts them in the policy defined location.  This allows us
> to eliminate all those scattered -whl binary packages, and just
> Build-Depend/Built-Using in pip to install the standard Python packages,
> rewheel them, and put them in a python-pip-whl package.  Going forward, this
> will make life so much easier, because only python-pip needs to know anything
> about wheels, even if new dependencies are added.
>
> I have everything working locally.  I need to update the master branch for
> pip, which involves many curse words hurled at git-dpm (and that's a post for
> another day!).  I also have to update my local branch to 8.0.2, but I'm
> assuming there won't be any new surprises there.
>
> Eventually, I'll be using the same techique for virtualenv and pyvenv, but not
> right away.
>
> Huge thanks also go to Donald Stufft, who always is so helpful with various
> technical details, and works really hard to understand our needs and balance
> them with upstream's.
>
> Finally, we need to update Python policy to describe the new wheel policy.
> Fortunately, this is much better because it removes constraints on other
> packages, and reduces the number of -whl binary packages in the distro.  Yes,
> I will update those packages that already ship -whls, or file bugs, but only
> once I'm sure everything is working correctly.
>
> Below is the policy diff, which I'll commit to the python-defaults bzr.
>
> As always, once everything's uploaded, please do let me know via this list or
> BTS, if you notice any problems with pip.
>
> Cheers,
> -Barry
>
> === modified file 'debian/python-policy.sgml'
> --- debian/python-policy.sgml   2016-01-24 06:01:19 +0000
> +++ debian/python-policy.sgml   2016-01-29 16:49:59 +0000
> @@ -458,13 +458,12 @@
>          <p>
>            <url id="http://legacy.python.org/dev/peps/pep-0427/"
>                 name="PEP 427">
> -          defines a built-package format called "wheels", which is a zip
> -          format archive containing Python code and a "dist-info" metadata
> +          defines a built-package format called "wheels", a zip
> +          archive containing Python code and a "dist-info" metadata
>            directory, in a single file named with the .whl suffix.  As zip
>            files, wheels containing pure-Python can be put on sys.path and
>            modules in the wheel can be imported directly by Python's "import"
> -          statement. (Importing extension modules from wheels is not yet
> -          supported as of Python 3.4.)
> +          statement.
>          </p><p>
>            Except as described below, packages must not build or provide
>            wheels.  They are redundant to the established way of providing
> @@ -475,18 +474,20 @@
>          </p><p>
>            A very limited set of wheel packages are available in the archive,
>            but these support the narrow purpose of enabling
> -          the <prgn>pip</prgn> tool, in a Debian policy compliant way.  The
> -          set of packages providing wheels for this purpose are (by source
> -          package name): chardet, distlib, html5lib, python-colorama,
> -          python-pip, python-setuptools, python-urllib3, requests, and six.
> +          the <prgn>pip</prgn>, <prgn>virtualenv</prgn>,
> +          and <prgn>pyvenv</prgn> tools in a Debian policy compliant way.
> +          These packages build their own dependent wheels through the use of
> +          the <prgn>dirtbike</prgn> "rewheeling" tool, which takes installed
> +          Debian packages and turns them back into wheels.  Only universal
> +          wheels (i.e. pure-Python, Python 2 and 3 compatible packages) are
> +          supported.  Since only the programs that require wheels need build
> +          them, only they may provide <var>-whl</var> packages,
> +          e.g. <package>python-pip-whl</package>.
>          </p><p>
> -          Wheel packages supporting <prgn>pyvenv</prgn> and <prgn>pip</prgn>
> -          are named with the <var>python-</var> prefix, and the <var>-whl</var>
> -          suffix, e.g. <package>python-chardet-whl</package>.  When these
> -          binary packages are installed, their .whl files must be placed in
> -          the /usr/share/python-wheels directory.  Such wheels must be built
> -          with the <tt>--universal</tt> flag so as to generate wheels
> -          compatible with both Python 2 and Python 3.
> +          When these binary packages are installed, .whl files must be placed
> +          in the /usr/share/python-wheels directory.  The location inside a
> +          virtual environment will be rooted in the virtual environment,
> +          instead of in /usr.
>          </p>
>        </sect>
>        <sect id="package_names">
>



-- 
Robert Collins <rbtcollins@hpe.com>
Distinguished Technologist
HP Converged Cloud

[toc] | [prev] | [next] | [standalone]

#8181

[Multipart message — attachments visible in raw view] — view raw

> On Jan 29, 2016, at 5:33 PM, Robert Collins <robertc@robertcollins.net> wrote:
> 
> I am confused. Here's my understanding of things...
> 
> - Pip doesn't need wheels at all - its vendoring technique doesn't use wheels.
> - Virtualenv needs to install pip, wheel, setuptools when it makes a
> new environment, and it does that by some oogly code
> - venv might do the same thing, but I haven't check its actual implementation
> - the pip-whl package was used to give virtualenv a thing to use to
> install pip when making a virtualenv
> 
> If that aligns with your understanding, I'll put it down to the prose
> you wrote rather than us actually having a different worldview; OTOH
> if it doesn't, I'd like to find out where I'm wrong, so as to improve
> my understanding.
> 
> With my understanding in mind - Can I just check something?
> 
> If a new requests package is built, uploaded to the archive and
> apt-get installed on my machine, and I then do:
> virtualenv test
> . test/bin/activate
> pip install foobar
> ^--- what version of requests will be used by this in-virtualenv
> invocation of pip?

Debian doesn’t want to utilize the bundled libraries in pip for various
reasons (some I agree with, some I don't, but suffice it to say it's a
requirement) but there is recognition that pip is a bit special here and can't
just be debundled in the standard way.

In conjunction with Barry, pip has some basic support for debundling by
deleting the contents of pip/_vendor/ except for pip/_vendor/__init__.py,
creating wheel files for all of pip's dependencies, and then making a small
patch to pip/_vendor/__init__.py to turn on the debundling feature and tell it
where those wheel files live. Pip will then add all of those wheel files to
the front of sys.path.

This gives us a few things:

* Pip will *always* use the versions of those libraries from the wheels,
  regardless of what is installed. This makes it so you can't break yourself
  with ``pip install requests`` in any environment (virtual or otherwise).

* Debian gets to have all of the code in pip come from the correct source
  packages.

These wheel files that represent pip's dependencies are generated when
python-pip is being built from the versions of the packages in Debian's
repositories and the python-pip package has a Built-Using header to indicate
which dependencies have been rebundled inside of pip through this process. This
has essentially recreated static linking but for Python. It is my understanding
that all of the wheels for pip's dependencies will exist inside of the
python-pip-whl .deb.

On top of that, virtualenv and venv will both need wheels that it needs to
install for pip, setuptools, and in the case of virtualenv, wheel. I'm not sure
what the plan is for those.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[toc] | [prev] | [next] | [standalone]

#8183

[Multipart message — attachments visible in raw view] — view raw

On Jan 30, 2016, at 11:33 AM, Robert Collins wrote:

> If a new requests package is built, uploaded to the archive and
>apt-get installed on my machine, and I then do:
>virtualenv test
>. test/bin/activate
>pip install foobar
> ^--- what version of requests will be used by this in-virtualenv
>invocation of pip?
>
>>From your description I expect it to be the version of requests that was
>dpkg installed at the time the pip deb was built?

Correct.

On Jan 29, 2016, at 05:45 PM, Donald Stufft wrote:

>These wheel files that represent pip's dependencies are generated when
>python-pip is being built from the versions of the packages in Debian's
>repositories and the python-pip package has a Built-Using header to indicate
>which dependencies have been rebundled inside of pip through this
>process. This has essentially recreated static linking but for Python. It is
>my understanding that all of the wheels for pip's dependencies will exist
>inside of the python-pip-whl .deb.

Yep.

I don't actually know whether Built-Using *does* anything useful[*], but IWBNI
(maybe) some kind of notification or auto-rebuild happened.  I don't think it
does, but Donald's right.  It's the same problem that Go or statically linked
C/C++ has.

>On top of that, virtualenv and venv will both need wheels that it needs to
>install for pip, setuptools, and in the case of virtualenv, wheel. I'm not
>sure what the plan is for those.

I plan to use the same technique for virtualenv/pyvenv dependencies, although
they will probably Depend on python-pip-whl to just get those for free.  TBH,
I haven't spent a ton of time on those higher level tools, just because of how
much work it was to sort out pip.  But I'll look at those soon.

Cheers,
-Barry

[*] There are some plans to honor this field in Ubuntu, where Build-Depends
may not need to go through the same main-inclusion-report (MIR) process, but
Built-Using packages would.  It's all currently speculative, so I don't have
any more details right now.

[toc] | [prev] | [next] | [standalone]

#8185

On Fri, Jan 29, 2016 at 18:28:53 -0500, Barry Warsaw wrote:

> I don't actually know whether Built-Using *does* anything useful[*], but IWBNI
> (maybe) some kind of notification or auto-rebuild happened.  I don't think it
> does, but Donald's right.  It's the same problem that Go or statically linked
> C/C++ has.
> 
ATM there's no automatic rebuild for out of date built-using.  Its main
use is to force the archive software to keep the source packages listed
as Built-Using in the archive, to comply with GPL requirements.

Cheers,
Julien
-- 
Julien Cristau          <julien.cristau@logilab.fr>
Logilab		        http://www.logilab.fr/
Informatique scientifique & gestion de connaissances

[toc] | [prev] | [next] | [standalone]

#8186

Hi Barry,

Did you consider creating whl files at install time¹ rather than pip's
build time? This way whl can be regenerated whenever one of needed
packages is updated and there's no need to rebuild pip package.

[¹] using dpkg (file) triggers (/usr/share/doc/dpkg-dev/triggers.txt.gz)
-- 
Piotr Ożarowski                         Debian GNU/Linux Developer
www.ozarowski.pl          www.griffith.cc           www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

[toc] | [prev] | [next] | [standalone]

#8187

On Feb 01, 2016, at 03:01 PM, Piotr Ożarowski wrote:

>Did you consider creating whl files at install time¹ rather than pip's
>build time? This way whl can be regenerated whenever one of needed
>packages is updated and there's no need to rebuild pip package.
>
>[¹] using dpkg (file) triggers (/usr/share/doc/dpkg-dev/triggers.txt.gz)

I did think about it, and maybe it's where we want to end up, but I didn't go
this way right now because of the added complexity.  Well, maybe "different"
complexity, since none of this stuff is simple. ;)

What we have now is better than what we had before, but once I get virtualenv
fixed and uploaded, I think I'll come back around to looking at what you
suggest.  (Certainly, patches welcome in the meantime.)  Fortunately, I
believe we have all the tools in place (e.g. dirtbike) to generate the wheels
at install time.  It does mean some extra run-time dependencies, but that's
probably fine.

Cheers,
-Barry

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.maint.python

csiph-web