Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #74643 > unrolled thread

Multiple python versions, one dev environment???

Back to article view | Back to comp.lang.python

Contents

  Multiple python versions, one dev environment??? Joep van Delft <joepvandelft@xs4all.nl> - 2014-07-17 17:32 +0200
    Re: Multiple python versions, one dev environment??? Javier <nospam@nospam.com> - 2014-07-17 16:05 +0000
      Re: Multiple python versions, one dev environment??? Joep van Delft <joepvandelft@xs4all.nl> - 2014-07-17 21:52 +0200
        Re: Multiple python versions, one dev environment??? Javier <nospam@nospam.com> - 2014-07-17 21:52 +0000
    Re: Multiple python versions, one dev environment??? Akira <4kir4.1i@gmail.com> - 2014-07-17 16:29 +0000
    Re: Multiple python versions, one dev environment??? Ned Batchelder <ned@nedbatchelder.com> - 2014-07-17 15:41 -0400
      Re: Multiple python versions, one dev environment??? Joep van Delft <joepvandelft@xs4all.nl> - 2014-07-17 21:54 +0200
      Re: Multiple python versions, one dev environment??? Roy Smith <roy@panix.com> - 2014-07-17 19:44 -0400
        Re: Multiple python versions, one dev environment??? alex23 <wuwei23@gmail.com> - 2014-07-18 11:15 +1000
          Re: Multiple python versions, one dev environment??? Roy Smith <roy@panix.com> - 2014-07-17 21:29 -0400
            Re: Multiple python versions, one dev environment??? Chris Angelico <rosuav@gmail.com> - 2014-07-18 12:42 +1000

#74643 — Multiple python versions, one dev environment???

Hello! 

The condensed version of the question would probably be: How does one
deal with multiple interpreters and one package where you want to try
some changes? 

The background: 
I made a trivial change to some package (docutils) to scratch a
personal itch, and I want to offer this back to the community
(whether it will be accepted or not). Because of this, I ran into
some issues. 

Some facts:
1. Python3 is my main interpreter. 
2. The tests of docutils only run under python2.
3. I desire not to touch my distribution's version of
   site-packagesX-Y.
4. I prefer to have one and only one development directory of
   my target package. 

My confusions: 
1. Is it possible to have one source control managed directory of
   some package, which is used by two versions of python? 
2. I assume that the *.pyc-files generated while using some python
   source are version dependent. What is the recommended way to have
   'em both installed? 
3. The way I have stumped a wall over here, is the puzzle of how to
   make python2 have a different $PYTHONPATH as python3. I hope to
   hear how this strategy is silly :) 
4. I have contemplated the way of linking the source files from my
   development directory into user specified site-packages
   directories. Problem 3. still is valid. 
5. Should venv and friends/foes com into play? If so: How? 

Appreciate any light shed on these issues. 

Thanks! 


	Joep

[toc] | [next] | [standalone]

#74646

Are you using arch linux.  


I deal with multiple interpreters putting fake executables in
/usr/local/bin for everything: (python, sphinx, virtualenv, pydoc,
idle, python-config...) selecting 2 or 3.  You can do the same for
selecting 2.3, 2.5, 2.7.  What the scripts do is to detect whether it
is a system script whose prefix starts with /usr/bin, or whether it is
a user script.  Being in /usr/local/bin they will override executables
in /usr/bin. Remember to chmod a+x the files in /usr/local/bin

http://sindhus.bitbucket.org/manage-python-2-3.html
http://stackoverflow.com/questions/15400985/how-to-completely-
replace-python-3-with-python-2-in-arch-linux
https://wiki.archlinux.org/index.php/Python#Dealing_with_
version_problem_in_build_scripts

I use these scripts, but there is more than one way to do it

========/usr/local/bin/python===========================================
#!/bin/bash
script=`readlink -f -- "$1"`
case "$script" in
/usr/bin*)
    exec python3 "$@"
    ;;
esac
exec python2 "$@"

========/usr/local/bin/virtualenv===========================================
#!/bin/bash
script=`readlink -f -- "$1"`
case "$script" in
/usr/bin*)
    exec virtualenv3 "$@"
    ;;
esac

exec virtualenv2 "$@"








Joep van Delft <joepvandelft@xs4all.nl> wrote:
> Hello! 
> 
> The condensed version of the question would probably be: How does one
> deal with multiple interpreters and one package where you want to try
> some changes? 
> 
> The background: 
> I made a trivial change to some package (docutils) to scratch a
> personal itch, and I want to offer this back to the community
> (whether it will be accepted or not). Because of this, I ran into
> some issues. 
> 
> Some facts:
> 1. Python3 is my main interpreter. 
> 2. The tests of docutils only run under python2.
> 3. I desire not to touch my distribution's version of
>   site-packagesX-Y.
> 4. I prefer to have one and only one development directory of
>   my target package. 
> 
> My confusions: 
> 1. Is it possible to have one source control managed directory of
>   some package, which is used by two versions of python? 
> 2. I assume that the *.pyc-files generated while using some python
>   source are version dependent. What is the recommended way to have
>   'em both installed? 
> 3. The way I have stumped a wall over here, is the puzzle of how to
>   make python2 have a different $PYTHONPATH as python3. I hope to
>   hear how this strategy is silly :) 
> 4. I have contemplated the way of linking the source files from my
>   development directory into user specified site-packages
>   directories. Problem 3. still is valid. 
> 5. Should venv and friends/foes com into play? If so: How? 
> 
> Appreciate any light shed on these issues. 
> 
> Thanks! 
> 
> 
>        Joep
> 
> 
> 

[toc] | [prev] | [next] | [standalone]

#74684

Hello Javier! 

Thanks, those links are helping a bit. And: yes, I am using Archlinux.

But still those links assume that there are totally separate
site-packages* directories installed for both. I am not sure how I
would surpass this distinction between py-X.P and py-Y.Q. 

Should I really create a massive amount of symlinks like this?: 

| #!/usr/bin/zsh
| for d in ~/src/mypackage/**/*(/); do 
|    # matches all directories
|    mkdir -p "~/src/py-2.7/mypackage/${d#*src/mypackage}"
|    mkdir -p "~/src/py-3.4/mypackage/${d#*src/mypackage}"
| done
| for f in ~/src/mypackage/**/^*.pyc(.); do 
|    # matches all files except for *.pyc
|    ln -s "$f" "~/src/py-2.7/mypackage${f#*src/mypackage}"
|    ln -s "$f" "~/src/py-3.4/mypackage${f#*src/mypackage}"
| done

...and then set $PYTHONPATH according to the target version in a
#!/usr/local/bin-script? 

I can work with this (have not tried though), but there must be a
more elegant solution than symlinking my way forward... 

Cheers!


    Joep



On Thu, 17 Jul 2014 16:05:27 +0000 (UTC)
Javier <nospam@nospam.com> wrote:

> Are you using arch linux.  
> 
> 
> I deal with multiple interpreters putting fake executables in
> /usr/local/bin for everything: (python, sphinx, virtualenv, pydoc,
> idle, python-config...) selecting 2 or 3.  You can do the same for
> selecting 2.3, 2.5, 2.7.  What the scripts do is to detect whether
> it is a system script whose prefix starts with /usr/bin, or whether
> it is a user script.  Being in /usr/local/bin they will override
> executables in /usr/bin. Remember to chmod a+x the files
> in /usr/local/bin
> 
> http://sindhus.bitbucket.org/manage-python-2-3.html
> http://stackoverflow.com/questions/15400985/how-to-completely-replace-python-3-with-python-2-in-arch-linux
> https://wiki.archlinux.org/index.php/Python#Dealing_with_version_problem_in_build_scripts
> 
> I use these scripts, but there is more than one way to do it
> 
> ========/usr/local/bin/python===========================================
> #!/bin/bash
> script=`readlink -f -- "$1"`
> case "$script" in
> /usr/bin*)
>     exec python3 "$@"
>     ;;
> esac
> exec python2 "$@"
> 
> ========/usr/local/bin/virtualenv===========================================
> #!/bin/bash
> script=`readlink -f -- "$1"`
> case "$script" in
> /usr/bin*)
>     exec virtualenv3 "$@"
>     ;;
> esac
> 
> exec virtualenv2 "$@"
> 
> 
> 
> 
> 
> 
> 
> 
> Joep van Delft <joepvandelft@xs4all.nl> wrote:
> > Hello! 
> > 
> > The condensed version of the question would probably be: How does
> > one deal with multiple interpreters and one package where you
> > want to try some changes? 
> > 
> > The background: 
> > I made a trivial change to some package (docutils) to scratch a
> > personal itch, and I want to offer this back to the community
> > (whether it will be accepted or not). Because of this, I ran into
> > some issues. 
> > 
> > Some facts:
> > 1. Python3 is my main interpreter. 
> > 2. The tests of docutils only run under python2.
> > 3. I desire not to touch my distribution's version of
> >   site-packagesX-Y.
> > 4. I prefer to have one and only one development directory of
> >   my target package. 
> > 
> > My confusions: 
> > 1. Is it possible to have one source control managed directory of
> >   some package, which is used by two versions of python? 
> > 2. I assume that the *.pyc-files generated while using some python
> >   source are version dependent. What is the recommended way to
> > have 'em both installed? 
> > 3. The way I have stumped a wall over here, is the puzzle of how
> > to make python2 have a different $PYTHONPATH as python3. I hope to
> >   hear how this strategy is silly :) 
> > 4. I have contemplated the way of linking the source files from my
> >   development directory into user specified site-packages
> >   directories. Problem 3. still is valid. 
> > 5. Should venv and friends/foes com into play? If so: How? 
> > 
> > Appreciate any light shed on these issues. 
> > 
> > Thanks! 
> > 
> > 
> >        Joep
> > 
> > 
> > 

[toc] | [prev] | [next] | [standalone]

#74694

> I can work with this (have not tried though), but there must be a
> more elegant solution than symlinking my way forward... 


I don't really understand what you are trying to do, but I would advise
to use environment variables to control the behaviour of the fake scripts
in /usr/local/bin

In bash you can do 

export PYVERSION=2.5

And from that time onwards everything defaults to python2.5.

note the ${PYVERSION} that I have included now in the sample scripts
below to select the python version.


I don't think you need to, but if you want to change the environment
variable inside python you can set environment variables with something like
os.environ.?????  Look at the docs.


Caveat1: I have not tested this, but it should work ok.  The setup I use is
much simpler: just default to python2.7

Caveat2: Arch linux packagers dont use a consistent naming of things:
There exists /usr/bin/virtualenv3, but there does not exist
/usr/bin/sphinx-build3 (it is /usr/bin/sphinx-build)
Somebody should send a bug to the package maintainer.

PS: Once you setup a workaround to bypass all the python=python3 nonsense,
Arch Linux is a nice linux distro, the best out there.  I will stick
to it.

HTH

========/usr/local/bin/python===========================================
#!/bin/bash
script=`readlink -f -- "$1"`
case "$script" in
/usr/bin*)
    exec python3 "$@"
    ;;
esac
exec python${PYVERSION} "$@"

========/usr/local/bin/virtualenv===========================================
#!/bin/bash
script=`readlink -f -- "$1"`
case "$script" in
/usr/bin*)
    exec virtualenv3 "$@"
    ;;
esac

exec virtualenv${PYVERSION} "$@"



Joep van Delft <joepvandelft@xs4all.nl> wrote:
> Hello Javier! 
> 
> Thanks, those links are helping a bit. And: yes, I am using Archlinux.
> 
> But still those links assume that there are totally separate
> site-packages* directories installed for both. I am not sure how I
> would surpass this distinction between py-X.P and py-Y.Q. 
> 
> Should I really create a massive amount of symlinks like this?: 
> 
> | #!/usr/bin/zsh
> | for d in ~/src/mypackage/**/*(/); do 
> |    # matches all directories
> |    mkdir -p "~/src/py-2.7/mypackage/${d#*src/mypackage}"
> |    mkdir -p "~/src/py-3.4/mypackage/${d#*src/mypackage}"
> | done
> | for f in ~/src/mypackage/**/^*.pyc(.); do 
> |    # matches all files except for *.pyc
> |    ln -s "$f" "~/src/py-2.7/mypackage${f#*src/mypackage}"
> |    ln -s "$f" "~/src/py-3.4/mypackage${f#*src/mypackage}"
> | done
> 
> ...and then set $PYTHONPATH according to the target version in a
> #!/usr/local/bin-script? 
> 
> I can work with this (have not tried though), but there must be a
> more elegant solution than symlinking my way forward... 
> 
> Cheers!
> 
> 
>    Joep
> 

[toc] | [prev] | [next] | [standalone]

#74648

Joep van Delft <joepvandelft@xs4all.nl> wrote:
> Hello! 
> 
> The condensed version of the question would probably be: How does one
> deal with multiple interpreters and one package where you want to try
> some changes? 

You could use tox to test a package using different Python versions.


--
Akira

[toc] | [prev] | [next] | [standalone]

#74683

On 7/17/14 11:32 AM, Joep van Delft wrote:
> Hello!
>
> The condensed version of the question would probably be: How does one
> deal with multiple interpreters and one package where you want to try
> some changes?
>
> The background:
> I made a trivial change to some package (docutils) to scratch a
> personal itch, and I want to offer this back to the community
> (whether it will be accepted or not). Because of this, I ran into
> some issues.
>
> Some facts:
> 1. Python3 is my main interpreter.
> 2. The tests of docutils only run under python2.
> 3. I desire not to touch my distribution's version of
>     site-packagesX-Y.
> 4. I prefer to have one and only one development directory of
>     my target package.
>
> My confusions:
> 1. Is it possible to have one source control managed directory of
>     some package, which is used by two versions of python?
> 2. I assume that the *.pyc-files generated while using some python
>     source are version dependent. What is the recommended way to have
>     'em both installed?
> 3. The way I have stumped a wall over here, is the puzzle of how to
>     make python2 have a different $PYTHONPATH as python3. I hope to
>     hear how this strategy is silly :)
> 4. I have contemplated the way of linking the source files from my
>     development directory into user specified site-packages
>     directories. Problem 3. still is valid.
> 5. Should venv and friends/foes com into play? If so: How?
>
> Appreciate any light shed on these issues.

Virtualenv is definitely the right way to isolate different Python 
environments from each other.  Each one has its own PYTHONPATH, so each 
project of yours can have different packages installed.

For testing one project on multiple versions of Python, use tox.  Its 
entire reason for being is to test Python code against multiple 
environments, generally for different Python versions, but possibly for 
other reasons, like different versions of dependencies.

Tox will manage the virtualenvs for you, it makes multi-version testing 
very simple.

>
> Thanks!
>
>
> 	Joep
>
>
>


-- 
Ned Batchelder, http://nedbatchelder.com

[toc] | [prev] | [next] | [standalone]

#74685

On Thu, 17 Jul 2014 15:41:44 -0400
Ned Batchelder <ned@nedbatchelder.com> wrote:
>
> For testing one project on multiple versions of Python, use tox.
> Its entire reason for being is to test Python code against multiple 
> environments, generally for different Python versions, but possibly
> for other reasons, like different versions of dependencies.
> 
> Tox will manage the virtualenvs for you, it makes multi-version
> testing very simple.
> 

Excellent, Ned and Akira, I will look into it! 

Cheers, 

    Joep

[toc] | [prev] | [next] | [standalone]

#74701

In article <mailman.11957.1405626121.18130.python-list@python.org>,
 Ned Batchelder <ned@nedbatchelder.com> wrote:

> On 7/17/14 11:32 AM, Joep van Delft wrote:
> > Hello!
> >
> > The condensed version of the question would probably be: How does one
> > deal with multiple interpreters and one package where you want to try
> > some changes?

> Virtualenv is definitely the right way to isolate different Python 
> environments from each other.  Each one has its own PYTHONPATH, so each 
> project of yours can have different packages installed.

Absolutely.  Don't even consider any other alternative.  Just do it.

Our deployment process builds a new virtualenv for every release.  Then 
you don't even have to think about what got added and what got deleted.  
We have a requirements file which we feed to pip and it creates exactly 
what we need for that release.  We use wheel to speed things up, that's 
just an optimization.

[toc] | [prev] | [next] | [standalone]

#74706

On 18/07/2014 9:44 AM, Roy Smith wrote:
> In article <mailman.11957.1405626121.18130.python-list@python.org>,
>   Ned Batchelder <ned@nedbatchelder.com> wrote:
>> Virtualenv is definitely the right way to isolate different Python
>> environments from each other.
>
> Absolutely.  Don't even consider any other alternative.  Just do it.

Not even buildout? :)

[toc] | [prev] | [next] | [standalone]

#74707

In article <lq9sf2$p9r$1@dont-email.me>, alex23 <wuwei23@gmail.com> 
wrote:

> On 18/07/2014 9:44 AM, Roy Smith wrote:
> > In article <mailman.11957.1405626121.18130.python-list@python.org>,
> >   Ned Batchelder <ned@nedbatchelder.com> wrote:
> >> Virtualenv is definitely the right way to isolate different Python
> >> environments from each other.
> >
> > Absolutely.  Don't even consider any other alternative.  Just do it.
> 
> Not even buildout? :)

OK, I will admit I'm not familiar with buildout.

Perhaps I should have said, "Don't even consider trying to manage all 
this stuff manually.

[toc] | [prev] | [next] | [standalone]

#74714

On Fri, Jul 18, 2014 at 11:29 AM, Roy Smith <roy@panix.com> wrote:
> In article <lq9sf2$p9r$1@dont-email.me>, alex23 <wuwei23@gmail.com>
> wrote:
>
>> On 18/07/2014 9:44 AM, Roy Smith wrote:
>> > In article <mailman.11957.1405626121.18130.python-list@python.org>,
>> >   Ned Batchelder <ned@nedbatchelder.com> wrote:
>> >> Virtualenv is definitely the right way to isolate different Python
>> >> environments from each other.
>> >
>> > Absolutely.  Don't even consider any other alternative.  Just do it.
>>
>> Not even buildout? :)
>
> OK, I will admit I'm not familiar with buildout.
>
> Perhaps I should have said, "Don't even consider trying to manage all
> this stuff manually.

I'd put this on par with source control. You'll see arguments about
whether git or hg is the better option, but frankly, either of them
(or bzr, or any of a bunch of others) is just so far ahead of *not*
using source control that it's less important to distinguish than to
say "Don't even consider trying to do this manually".

ChrisA

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web