Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #8180

Re: New, updated pip coming

Show key headers only | View raw

I am confused. Here's my understanding of things...

 - Pip doesn't need wheels at all - its vendoring technique doesn't use wheels.
 - Virtualenv needs to install pip, wheel, setuptools when it makes a
new environment, and it does that by some oogly code
 - venv might do the same thing, but I haven't check its actual implementation
 - the pip-whl package was used to give virtualenv a thing to use to
install pip when making a virtualenv

If that aligns with your understanding, I'll put it down to the prose
you wrote rather than us actually having a different worldview; OTOH
if it doesn't, I'd like to find out where I'm wrong, so as to improve
my understanding.

With my understanding in mind - Can I just check something?

 If a new requests package is built, uploaded to the archive and
apt-get installed on my machine, and I then do:
virtualenv test
. test/bin/activate
pip install foobar
 ^--- what version of requests will be used by this in-virtualenv
invocation of pip?

>From your description I expect it to be the version of requests that
was dpkg installed at the time the pip deb was built?

-Rob


On 30 January 2016 at 08:28, Barry Warsaw <barry@debian.org> wrote:
> TL;DR By the end of today, I expect to upload pip 8.0.2 to unstable, finally
> bringing us up to the latest version.
>
> It's been a long slog, with many people helping out.  I hope that all the hard
> work done to get us here means it will be much, much easier to track new pip,
> virtualenv, and pyvenv updates going forward.  And now, details!
>
> pip requires the use of PEP 427 wheels, which are essentially zip files of
> importable, pure-Python, bilingual code, with a little bit of metadata.  Put a
> wheel on sys.path and Python can import from it.  pip and friends use wheels
> because it ensures that they can't break themselves when their dependent
> packages are updated.  E.g. if pip depends on requests, and you `pip install
> --upgrade requests` it's possible the new version would be incompatible with
> pip.  Then you'd have a broken pip on your hands.  Remember that you can use
> pip to install pre-release packages too.
>
> So by putting the wheels on sys.path *before* anything else, even if you pip
> install an incompatible requests, it won't break pip itself.
>
> pip "vendorizes" (i.e. bundles) the wheels of its dependencies with known good
> versions, but this breaks Debian policy, so we have to unbundle these.
> Several upstreams use this technique, with requests and urllib3 as probably
> the most popular examples.  In all these cases, we have to remove the bundled
> wheels and hope that the versions of their dependencies in Debian remain
> compatible.  It's a chore, but that's our job as distro developers, and it's
> the best option given the conflicting requirements.
>
> Fortunately, we can create our own wheels and use them with pip and friends.
> Until now, we had to build the wheels at the time that the dependent packages
> where built, and we had to have a corresponding foo-whl binary package in the
> archives, because we could only use setup.py's bdist_wheel command (which
> comes with python{,3}-wheel) to create these.  With 15 run time dependencies
> as of pip 7.1.2, this has become untenable, and was a huge reason why Debian's
> pip has lagged so far behind upstream.  (That, and a few ITPs for new
> dependencies. ;)
>
> Along comes dirtbike.  https://github.com/paulproteus/dirtbike
>
> While it has been much discussed and attempted over the years, it would be
> really nice to have a tool that turned installed Python packages into wheels.
> I think we discussed such a tool for Debian at last year's (two years ago?)
> Debian BOF at Pycon.  Anyway, Asheesh Laroia finally took hold of the, ahem,
> wheel, and wrote the initial implementation, which he called dirtbike.  Since
> then I've been enhancing it to work with several oddball cases in the Debian
> archive, and now it's good enough to turn pip's dependencies into wheels.
>
> Yesterday I uploaded the last of the new pip dependencies (modulo any
> surprises in 8.0.2 ;), and dirtbike so now all the pieces are in place.  What
> pip now does during its own build process, is turn its installed dependencies
> into .whl files and puts them in the policy defined location.  This allows us
> to eliminate all those scattered -whl binary packages, and just
> Build-Depend/Built-Using in pip to install the standard Python packages,
> rewheel them, and put them in a python-pip-whl package.  Going forward, this
> will make life so much easier, because only python-pip needs to know anything
> about wheels, even if new dependencies are added.
>
> I have everything working locally.  I need to update the master branch for
> pip, which involves many curse words hurled at git-dpm (and that's a post for
> another day!).  I also have to update my local branch to 8.0.2, but I'm
> assuming there won't be any new surprises there.
>
> Eventually, I'll be using the same techique for virtualenv and pyvenv, but not
> right away.
>
> Huge thanks also go to Donald Stufft, who always is so helpful with various
> technical details, and works really hard to understand our needs and balance
> them with upstream's.
>
> Finally, we need to update Python policy to describe the new wheel policy.
> Fortunately, this is much better because it removes constraints on other
> packages, and reduces the number of -whl binary packages in the distro.  Yes,
> I will update those packages that already ship -whls, or file bugs, but only
> once I'm sure everything is working correctly.
>
> Below is the policy diff, which I'll commit to the python-defaults bzr.
>
> As always, once everything's uploaded, please do let me know via this list or
> BTS, if you notice any problems with pip.
>
> Cheers,
> -Barry
>
> === modified file 'debian/python-policy.sgml'
> --- debian/python-policy.sgml   2016-01-24 06:01:19 +0000
> +++ debian/python-policy.sgml   2016-01-29 16:49:59 +0000
> @@ -458,13 +458,12 @@
>          <p>
>            <url id="http://legacy.python.org/dev/peps/pep-0427/"
>                 name="PEP 427">
> -          defines a built-package format called "wheels", which is a zip
> -          format archive containing Python code and a "dist-info" metadata
> +          defines a built-package format called "wheels", a zip
> +          archive containing Python code and a "dist-info" metadata
>            directory, in a single file named with the .whl suffix.  As zip
>            files, wheels containing pure-Python can be put on sys.path and
>            modules in the wheel can be imported directly by Python's "import"
> -          statement. (Importing extension modules from wheels is not yet
> -          supported as of Python 3.4.)
> +          statement.
>          </p><p>
>            Except as described below, packages must not build or provide
>            wheels.  They are redundant to the established way of providing
> @@ -475,18 +474,20 @@
>          </p><p>
>            A very limited set of wheel packages are available in the archive,
>            but these support the narrow purpose of enabling
> -          the <prgn>pip</prgn> tool, in a Debian policy compliant way.  The
> -          set of packages providing wheels for this purpose are (by source
> -          package name): chardet, distlib, html5lib, python-colorama,
> -          python-pip, python-setuptools, python-urllib3, requests, and six.
> +          the <prgn>pip</prgn>, <prgn>virtualenv</prgn>,
> +          and <prgn>pyvenv</prgn> tools in a Debian policy compliant way.
> +          These packages build their own dependent wheels through the use of
> +          the <prgn>dirtbike</prgn> "rewheeling" tool, which takes installed
> +          Debian packages and turns them back into wheels.  Only universal
> +          wheels (i.e. pure-Python, Python 2 and 3 compatible packages) are
> +          supported.  Since only the programs that require wheels need build
> +          them, only they may provide <var>-whl</var> packages,
> +          e.g. <package>python-pip-whl</package>.
>          </p><p>
> -          Wheel packages supporting <prgn>pyvenv</prgn> and <prgn>pip</prgn>
> -          are named with the <var>python-</var> prefix, and the <var>-whl</var>
> -          suffix, e.g. <package>python-chardet-whl</package>.  When these
> -          binary packages are installed, their .whl files must be placed in
> -          the /usr/share/python-wheels directory.  Such wheels must be built
> -          with the <tt>--universal</tt> flag so as to generate wheels
> -          compatible with both Python 2 and Python 3.
> +          When these binary packages are installed, .whl files must be placed
> +          in the /usr/share/python-wheels directory.  The location inside a
> +          virtual environment will be rooted in the virtual environment,
> +          instead of in /usr.
>          </p>
>        </sect>
>        <sect id="package_names">
>



-- 
Robert Collins <rbtcollins@hpe.com>
Distinguished Technologist
HP Converged Cloud

Back to linux.debian.maint.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

New, updated pip coming Barry Warsaw <barry@debian.org> - 2016-01-29 20:30 +0100
  Re: New, updated pip coming Robert Collins <robertc@robertcollins.net> - 2016-01-29 23:40 +0100
    Re: New, updated pip coming Donald Stufft <donald@stufft.io> - 2016-01-29 23:50 +0100
    Re: New, updated pip coming Barry Warsaw <barry@debian.org> - 2016-01-30 00:30 +0100
      Re: New, updated pip coming Julien Cristau <julien.cristau@logilab.fr> - 2016-02-01 14:40 +0100
  Re: New, updated pip coming Piotr Ożarowski <piotr@debian.org> - 2016-02-01 15:10 +0100
    Re: New, updated pip coming Barry Warsaw <barry@debian.org> - 2016-02-01 16:00 +0100

csiph-web