Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #108661 > unrolled thread

How to create development Python environment on Linux.

Back to article view | Back to comp.lang.python

Contents

  How to create development Python environment on Linux. redirect.null@gmail.com - 2016-05-16 04:22 -0700
    Re: How to create development Python environment on Linux. redirect.null@gmail.com - 2016-05-16 04:29 -0700
    Re: How to create development Python environment on Linux. Zachary Ware <zachary.ware+pylist@gmail.com> - 2016-05-16 11:25 -0500
      Re: How to create development Python environment on Linux. Lawrence D’Oliveiro <lawrencedo99@gmail.com> - 2016-06-01 17:17 -0700
        Re: How to create development Python environment on Linux. Marc Brooks <marcwbrooks@gmail.com> - 2016-06-01 20:30 -0400
    Re: How to create development Python environment on Linux. redirect.null@gmail.com - 2016-05-16 14:28 -0700
      Re: How to create development Python environment on Linux. Michael Selik <michael.selik@gmail.com> - 2016-05-16 21:56 +0000
      Re: How to create development Python environment on Linux. Zachary Ware <zachary.ware+pylist@gmail.com> - 2016-05-17 10:32 -0500

#108661 — How to create development Python environment on Linux.

I have a Linux system (Mint 17.3 based in Ubuntu 14.04) on which I wish to do some Python development. The system has Python 2.7.6 installed already (there is a Python 3 installation too but I won't be needing to use that to start with). I need to install various Python modules for the work I'm going to do and I'm trying to work out how best to do this in a way that does not risk corrupting the system's Python installation.

I've learned that there are (at least) two mechanisms for installing Python packages in a way that does not impact the system installation, virtualenv and user scheme installations.

I have several questions/issues.


1. I am unsure which of these technologies is most suitable for my needs. Virtualenv sounds like what I need, but two things give me pause for thought. First I came across this github issue 

https://github.com/pypa/virtualenv/pull/697

...which highlights what seem like severe issues in the vitualenv implementation and proposes to rewrite the whole library. This raises two questions.

   i) Is this rewrite complete?
   ii) Is 'legacy' virtualenv or virtualenv-rewrite now the recommended library? 

The second thing that gives me pause for thought is the remark in this article

http://python-packaging-user-guide.readthedocs.io/en/latest/install_requirements_linux/#installing-pip-setuptools-wheel-with-linux-package-managers

...where it says

'Recent Debian/Ubuntu versions have modified pip to use the “User Scheme” by default, which is a significant behavior change that can be surprising to some users.'

These concerns caused me to try and research more in the area of user installs and package installation, which leads me to my next two issues.


2. User install site.py behaviour confusing.

The first thing I wanted to work out was whether 'user install' was indeed enabled by default on my system as described in the article above. Here is the documentation for site.py

https://docs.python.org/2/library/site.html

This is the documenation for 2.7.11 while I only have 2.7.6. I don't see a link to documentation for 2.7.6 and I dont know whether this version disparity is significant or not.

However, the documentation starts by saying that this module is loaded by default. That being so I would expect sys.prefix to be /usr/local.

If I fire up Python import sys and dump sys.prefix it outputs /usr.
If I dump site.USER_BASE I get 'name 'site' is not defined'.
If I import site and dump site.USER_BASE I get '/home/<user_name>/.local'.

This confuses the hell out of me. Two documentary sources suggest this behaviour should be enabled by default, yet it doesn't seem to be, I don't know why, and I don't know how to enable it, or even whether I should.

Can anyone either explain why I'm seeing what I'm seeing here, or point the in the direction to some documentation that can.


3. Installers

My system Python installation doesn't appear to have either easy_install nor pip nor virtualenv installed. This leaves me with what seems like a circular problem.

I'm pretty sure I need pip, but there are (at least) two ways to get it. 
   i) apt-get pip
   ii) or download and run get-pip.py.

I've come across various credible warnings of the dangers of using apt-get and pip to manage the same packages, but once pip is installed I was under the impression you need to use pip to update itself before it will install any packages as pip must be up to date in order to access PyPI. 

This page

http://python-packaging-user-guide.readthedocs.io/en/latest/install_requirements_linux/#installing-pip-setuptools-wheel-with-linux-package-managers

...suggests I should use apt-get to install pip, but that gives rise to the following sequence of operations

apt-get pip
pip install -U pip

..and isn't that exactly what I should be trying to avoid, installing a package using apt-get and then managing that package using pip?

I'd have though that using get-pip.py would be safer, but even then that might only be with the provision that I execute get-pip.py in a Python environment separate from the system core Python installation, otherwise I'm in the same boat of using get-pip.py to upgrade my apt-get managed system Python installation. What to do?


4. Python version.

In addition to all this I'm wondering whether I wouldn't be a good idea to have the Python installation I use for development up to 2.7.11 before I start as my system installation of 2.7.6 is 3 years out of date.

----


In short I'm hopelessly confused and could really use some guidance.
P.S. What happened to 'batteries included'?

Thanks.
Neut.

[toc] | [next] | [standalone]

#108662

Rereading this through I notice I'm a bit vague on one point.

In Point 2. User install site.py behaviour confusing, one of the things that confuses me is that the documentation for site.py here

https://docs.python.org/2/library/site.html

...states (in bold) that 'This module is automatically imported during initialization', so I'm confused as to why I have to manaully import it before I can access it? Is the documentation wrong, or did the behaviour change between versions 2.7.6 and 2.7.11 or does the documentation mean something other that what it seems to mean?

[toc] | [prev] | [next] | [standalone]

#108669

On Mon, May 16, 2016 at 6:22 AM,  <redirect.null@gmail.com> wrote:
> I have a Linux system (Mint 17.3 based in Ubuntu 14.04) on which I wish to do some Python development. The system has Python 2.7.6 installed already (there is a Python 3 installation too but I won't be needing to use that to start with).

Not what you asked for, but I would encourage you to look into whether
it's possible for you to use Python 3 instead of Python 2 for what
you're doing.  If it's possible, starting with Python 3 will save you
several headaches in the future.

> [...]
> 1. I am unsure which of these technologies is most suitable for my needs. Virtualenv sounds like what I need, but two things give me pause for thought. First I came across this github issue
>
> https://github.com/pypa/virtualenv/pull/697
>
> ...which highlights what seem like severe issues in the vitualenv implementation and proposes to rewrite the whole library. This raises two questions.
>
>    i) Is this rewrite complete?
>    ii) Is 'legacy' virtualenv or virtualenv-rewrite now the recommended library?

virtualenv is a bit of a hack, by necessity.  Python 2 doesn't truly
support virtualenv, so virtualenv has to make do with what it has
available.  That said, it has worked well for my needs and for many
others.

(The real solution is Python 3.3+'s venv module :))

> The second thing that gives me pause for thought is the remark in this article
>
> http://python-packaging-user-guide.readthedocs.io/en/latest/install_requirements_linux/#installing-pip-setuptools-wheel-with-linux-package-managers
>
> ...where it says
>
> 'Recent Debian/Ubuntu versions have modified pip to use the “User Scheme” by default, which is a significant behavior change that can be surprising to some users.'
>
> These concerns caused me to try and research more in the area of user installs and package installation, which leads me to my next two issues.

The real issue here is mixing the use of the OS package manager with
the use of pip.  If you only install python packages using the OS
package manager (e.g., `apt-get install python-requests`) or only with
pip (e.g., `pip install requests`), you'll probably be fine.  If you
try to use both, though, you may wind up in a weird situation.

> 2. User install site.py behaviour confusing.
>
> The first thing I wanted to work out was whether 'user install' was indeed enabled by default on my system as described in the article above. Here is the documentation for site.py
>
> https://docs.python.org/2/library/site.html
>
> This is the documenation for 2.7.11 while I only have 2.7.6. I don't see a link to documentation for 2.7.6 and I dont know whether this version disparity is significant or not.

It shouldn't be, but here's the 2.7.6 documentation anyway:
https://docs.python.org/release/2.7.6/library/site.html

> However, the documentation starts by saying that this module is loaded by default. That being so I would expect sys.prefix to be /usr/local.

Why do you expect that?  sys.prefix is baked in at compile time of the
python interpreter, and refers to where Python is installed on the
system.  `/usr/local` is the default prefix for custom builds, but
most Linux distributions install Python under `/usr`.  That the site
module is loaded by default just means that particular locations
outside of the standard library path are automatically added to
sys.path (the list of directories where Python will look when you try
to import something).  Compare the output of `python -c "import
pprint,sys;pprint.pprint(sys.path)"` with and without the -S and -s
options.

> If I fire up Python import sys and dump sys.prefix it outputs /usr.
> If I dump site.USER_BASE I get 'name 'site' is not defined'.

Even though site is imported by default by the interpreter, it doesn't
bind the name 'site' anywhere in the default namespace.  If you want
to use the contents of the site module, you still need to import site
yourself (this second import simply looks up 'site' in the sys.modules
dict and returns the module it finds there).

> If I import site and dump site.USER_BASE I get '/home/<user_name>/.local'.
>
> This confuses the hell out of me. Two documentary sources suggest this behaviour should be enabled by default, yet it doesn't seem to be, I don't know why, and I don't know how to enable it, or even whether I should.
>
> Can anyone either explain why I'm seeing what I'm seeing here, or point the in the direction to some documentation that can.

Hopefully I've cleared things up a bit above.  Please ask again if
it's still murky!

> 3. Installers
>
> My system Python installation doesn't appear to have either easy_install nor pip nor virtualenv installed. This leaves me with what seems like a circular problem.
>
> I'm pretty sure I need pip, but there are (at least) two ways to get it.
>    i) apt-get pip
>    ii) or download and run get-pip.py.
>
> I've come across various credible warnings of the dangers of using apt-get and pip to manage the same packages, but once pip is installed I was under the impression you need to use pip to update itself before it will install any packages as pip must be up to date in order to access PyPI.
>
> This page
>
> http://python-packaging-user-guide.readthedocs.io/en/latest/install_requirements_linux/#installing-pip-setuptools-wheel-with-linux-package-managers
>
> ...suggests I should use apt-get to install pip, but that gives rise to the following sequence of operations
>
> apt-get pip
> pip install -U pip
>
> ..and isn't that exactly what I should be trying to avoid, installing a package using apt-get and then managing that package using pip?
>
> I'd have though that using get-pip.py would be safer, but even then that might only be with the provision that I execute get-pip.py in a Python environment separate from the system core Python installation, otherwise I'm in the same boat of using get-pip.py to upgrade my apt-get managed system Python installation. What to do?

This is where virtualenv helps: create your venv, then use the venv's
pip to update pip and then install the packages you want.

> 4. Python version.
>
> In addition to all this I'm wondering whether I wouldn't be a good idea to have the Python installation I use for development up to 2.7.11 before I start as my system installation of 2.7.6 is 3 years out of date.

This depends somewhat on where your code will be running in
production: if you develop with 2.7.11 but deploy to 2.7.6, you may
run into bugs that have been fixed in the interim, or you may try to
use some of the security-sensitive features that have been added in
more recent 2.7 releases (particularly in the ssl module).  Ideally,
you should probably be using the latest version for both development
and deployment.  The simplest way to do that is likely to compile and
install your own Python.  Here's a quick-start guide for that:

 - `sudo apt-get build-dep python` to install dependencies (like a C
compiler and a few external library headers, like openssl-dev)

 - Either download and unzip the source from
https://python.org/downloads, or clone the repository from
hg.python.org/cpython or github.com/python/cpython.  If you go the
repository route, you'll want to update to the correct revision for
the release you want; for the hg repo it'll be tagged "vM.m.p" where M
is the major version (2), m is the minor version (7), and p is the
patch version (11) (the github repo has no tags yet, it's currently an
unofficial mirror).

 - From the root of the source tree, run `./configure && make
profile-opt && make test && sudo make install`.  This will build and
install a production-ready Python under `/usr/local`.  Note that
depending on how your PATH is set up, `python` may refer to either the
system Python at `/usr/bin/python` or your own Python at
`/usr/local/bin/python`.  You can avoid a bit of ambiguity by doing
`make altinstall` instead of `make install`, which will not create
`/usr/local/bin/python`; you'd have to invoke your Python using
`python2.7` (again subject to your PATH setup), but `python` would
always be the system Python.

> In short I'm hopelessly confused and could really use some guidance.
> P.S. What happened to 'batteries included'?

The standard library is a large complement of high-capacity batteries;
PyPI is the Three Gorges Dam.

-- 
Zach

[toc] | [prev] | [next] | [standalone]

#109323

On Tuesday, May 17, 2016 at 4:26:23 AM UTC+12, Zachary Ware wrote:
> Not what you asked for, but I would encourage you to look into whether
> it's possible for you to use Python 3 instead of Python 2 for what
> you're doing.  If it's possible, starting with Python 3 will save you
> several headaches in the future.

Let me add my vote for this.

> sys.prefix is baked in at compile time of the python interpreter ...

    ldo@theon:~> ~/virtualenv/jupyter/bin/python -c "import sys; print(sys.prefix)"
    /home/ldo/virtualenv/jupyter

[toc] | [prev] | [next] | [standalone]

#109346

I am pretty sure (but not 100%) that the pip that virtualenv installs when
it first creates the virtualenv is the version of pip installed on the
system.  Here's the process I used to bootstrap a new Python 2.7 dev
environment.

1. Download and install the latest version of pip as sudo so it's system
wide.
2. Install virtualenv and virtualenvwrapper (a collection of utilities
scripts/aliases for virtualenv).
3. Update my .bash_profile to source the virtualenvwrapper script.

Then for any new virtualenvs I just type 'mkvirtualenv <foo>'

I can update the version of pip in the virtualenv or run pip install for
any of my required libraries at that point.  One wrinkle that can come up
is if you want to use virtualenvwrapper and you are not using bash.  Fish
(another moderately popular shell) has an addon that mimics the macros that
virtualenvwrapper provides.

Mar

On Wed, Jun 1, 2016 at 8:17 PM, Lawrence D’Oliveiro <lawrencedo99@gmail.com>
wrote:

> On Tuesday, May 17, 2016 at 4:26:23 AM UTC+12, Zachary Ware wrote:
> > Not what you asked for, but I would encourage you to look into whether
> > it's possible for you to use Python 3 instead of Python 2 for what
> > you're doing.  If it's possible, starting with Python 3 will save you
> > several headaches in the future.
>
> Let me add my vote for this.
>
> > sys.prefix is baked in at compile time of the python interpreter ...
>
>     ldo@theon:~> ~/virtualenv/jupyter/bin/python -c "import sys;
> print(sys.prefix)"
>     /home/ldo/virtualenv/jupyter
> --
> https://mail.python.org/mailman/listinfo/python-list
>

[toc] | [prev] | [next] | [standalone]

#108699

Thanks Zach, that's a big help. The only reason I want to get a Python 2.7 environment working first is because I'll be working on third party code and that's the platform it uses. For any new projects I would use Python 3.

After considering your guidance I think what I will do is install virtualenv using apt-get and then use that to create a dev environment. Is it ok to run get-pip.py in a virtual environment?

I won't worry about using the latest version of 2.7 for now, since it's only one or two third party open source projects I'll use 2.7 for and they don't need a more recent version.

There are a couple of other things I'm not quite clear on, such as where it would be best to create my new virtual environment (I'm tempted to put it in /usr/local if that means it can be used by all user accounts on my machine), and how I can can control which Python environment is used by the various system and user programs that depend on them, but I expect I can find that information on the web, though I'll make another post here if I do get stuck.

[toc] | [prev] | [next] | [standalone]

#108701

On Mon, May 16, 2016 at 5:31 PM <redirect.null@gmail.com> wrote:

> After considering your guidance I think what I will do is install
> virtualenv using apt-get and then use that to create a dev environment. Is
> it ok to run get-pip.py in a virtual environment?
>

Recent versions of the virtualenv application create virtual environments
that already have pip. It's the default setting last time I checked.

[toc] | [prev] | [next] | [standalone]

#108746

On Mon, May 16, 2016 at 4:28 PM,  <redirect.null@gmail.com> wrote:
> Thanks Zach, that's a big help. The only reason I want to get a Python 2.7 environment working first is because I'll be working on third party code and that's the platform it uses. For any new projects I would use Python 3.

Fair enough :)

> After considering your guidance I think what I will do is install virtualenv using apt-get and then use that to create a dev environment. Is it ok to run get-pip.py in a virtual environment?

Sounds like a plan.  get-pip.py should work fine in a venv, but first
check to see if pip is already there, as Michael mentioned.

> I won't worry about using the latest version of 2.7 for now, since it's only one or two third party open source projects I'll use 2.7 for and they don't need a more recent version.
>
> There are a couple of other things I'm not quite clear on, such as where it would be best to create my new virtual environment (I'm tempted to put it in /usr/local if that means it can be used by all user accounts on my machine), and how I can can control which Python environment is used by the various system and user programs that depend on them, but I expect I can find that information on the web, though I'll make another post here if I do get stuck.

The thing about virtual environments is that you can create as many as
you want wherever you want, and they're all independent of each other.
All you need to do is have a way to consistently recreate the same
environment, which is usually easiest to do with a 'requirements.txt'
file with all of your dependencies' versions pinned.  Then recreating
the venv is as simple as `virtualenv --python=some_python
/path/to/venv && /path/to/venv/bin/pip install -r
/path/to/requirements.txt`.  Then you can do your development wherever
you want and install into somewhere in `/usr/local` when you deploy.
For example, one project I work on creates a venv in
`/usr/local/lib/project_name/`, then creates links to the entry point
scripts (that actually live in `/usr/local/lib/project_name/bin/`) in
`/usr/local/bin/`.

As for controlling which environment is used: all system scripts/apps
should specify some version-specific flavor of /usr/bin/python in
their shebang--if they don't it's an OS bug and should be filed as
such.  For your stuff, the path of least headache is probably to
create a venv for each one, and specify the path to the venv's python
in shebang lines.

Hope this helps,
-- 
Zach

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web