Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #19717 > unrolled thread

changing sys.path

Back to article view | Back to comp.lang.python

Contents

  changing sys.path Andrea Crotti <andrea.crotti.0@gmail.com> - 2012-02-01 16:15 +0000
    Re: changing sys.path jmfauth <wxjmfauth@gmail.com> - 2012-02-01 09:06 -0800
    Re: changing sys.path Rick Johnson <rantingrickjohnson@gmail.com> - 2012-02-01 09:17 -0800
      Re: changing sys.path Andrea Crotti <andrea.crotti.0@gmail.com> - 2012-02-10 13:08 +0000
      Re: changing sys.path Dave Angel <d@davea.name> - 2012-02-10 09:24 -0500
      Re: changing sys.path Andrea Crotti <andrea.crotti.0@gmail.com> - 2012-02-10 14:51 +0000
      Re: changing sys.path Dave Angel <d@davea.name> - 2012-02-10 10:06 -0500
      Re: changing sys.path Andrea Crotti <andrea.crotti.0@gmail.com> - 2012-02-10 15:14 +0000
      Re: changing sys.path Peter Otten <__peter__@web.de> - 2012-02-10 16:27 +0100
      Re: changing sys.path Andrea Crotti <andrea.crotti.0@gmail.com> - 2012-02-10 15:38 +0000
      Re: changing sys.path Peter Otten <__peter__@web.de> - 2012-02-10 16:40 +0100
      Re: changing sys.path Peter Otten <__peter__@web.de> - 2012-02-10 17:00 +0100
      Re: changing sys.path Andrea Crotti <andrea.crotti.0@gmail.com> - 2012-02-10 20:58 +0000
    Re: changing sys.path Tim Roberts <timr@probo.com> - 2012-02-02 21:10 -0800
    Re: changing sys.path John Nagle <nagle@animats.com> - 2012-02-08 13:43 -0800

#19717 — changing sys.path

So suppose I want to modify the sys.path on the fly before running some code
which imports from one of the modules added.

at run time I do
sys.path.extend(paths_to_add)

but it still doesn't work and I get an import error.

If I take these paths and add them to site-packages/my_paths.pth
everything works, but at run-time the paths which I actually see before
importing are exactly the same.

So there is something I guess that depends on the order, but what can I
reset/reload to make these paths available (I thought I didn't need 
anything in theory)?

[toc] | [next] | [standalone]

#19722

On 1 fév, 17:15, Andrea Crotti <andrea.crott...@gmail.com> wrote:
> So suppose I want to modify the sys.path on the fly before running some code
> which imports from one of the modules added.
>
> at run time I do
> sys.path.extend(paths_to_add)
>
> but it still doesn't work and I get an import error.
>
> If I take these paths and add them to site-packages/my_paths.pth
> everything works, but at run-time the paths which I actually see before
> importing are exactly the same.
>
> So there is something I guess that depends on the order, but what can I
> reset/reload to make these paths available (I thought I didn't need
> anything in theory)?


>>> import mod
Traceback (most recent call last):
  File "<eta last command>", line 1, in <module>
ImportError: No module named mod
>>> sys.path.append(r'd:\\jm\\junk')
>>> import mod
>>> mod
<module 'mod' from 'd:\\jm\\junk\mod.py'>
>>> mod.hello()
fct hello in mod.py


sys.path? Probably, the most genious Python idea.

jmf

[toc] | [prev] | [next] | [standalone]

#19731

On Feb 1, 10:15 am, Andrea Crotti <andrea.crott...@gmail.com> wrote:
> So suppose I want to modify the sys.path on the fly before running some code
> which imports from one of the modules added.
>
> at run time I do
> sys.path.extend(paths_to_add)
>
> but it still doesn't work and I get an import error.
>
> If I take these paths and add them to site-packages/my_paths.pth
> everything works, but at run-time the paths which I actually see before
> importing are exactly the same.

1. Is paths_to_add a nested list?
2. Have you tried inspecting the contents of sys.path AFTER calling
extend method?

Consider:
py> sys.path.__len__()
14
py> sys.path.extend([[1,2,3]])
py> sys.path.__len__()
15

[toc] | [prev] | [next] | [standalone]

#20162

I think I finally located the issue with the sys.path extension.

The problem is that I have many namespace directories, for example

lib:
  - sub1
  - sub2

lib:
   - sub3
   - sub4

But to have everything working I had lib.sub3 in easy-install.pth.
Now if I try to add something else to the path it doesn't take care of 
the namespace
declaration
(every __init__.py in the packages contains: 
__import__('pkg_resources').declare_namespace(__name__))
and just doesn't find the other submodules..

If I try to add manually lib.sub1, lib.sub2 changing the sys.path the 
imports will only work for the first one.

Strangely if I just create a dev_main.pth in site-packages containing 
the same paths, everything works perfectly.

Any suggestions now that the problem is more clear?
Thanks,
Andrea

[toc] | [prev] | [next] | [standalone]

#20163

On 02/10/2012 08:08 AM, Andrea Crotti wrote:
> I think I finally located the issue with the sys.path extension.
>
> The problem is that I have many namespace directories, for example
>
> lib:
>  - sub1
>  - sub2
>
> lib:
>   - sub3
>   - sub4
>
> But to have everything working I had lib.sub3 in easy-install.pth.
> Now if I try to add something else to the path it doesn't take care of 
> the namespace
> declaration
> (every __init__.py in the packages contains: 
> __import__('pkg_resources').declare_namespace(__name__))
> and just doesn't find the other submodules..
>
> If I try to add manually lib.sub1, lib.sub2 changing the sys.path the 
> imports will only work for the first one.
>
> Strangely if I just create a dev_main.pth in site-packages containing 
> the same paths, everything works perfectly.
>
> Any suggestions now that the problem is more clear?
> Thanks,
> Andrea

The only code I saw in this thread was:

sys.path.extend(paths_to_add)

Can you add a print of paths_to_add, and of sys.path after you execute it?

If there's only one path, are you putting it in a list anyway?  If not 
then it won't do what you expect.



-- 

DaveA

[toc] | [prev] | [next] | [standalone]

#20165

Ok now it's getting really confusing, I tried a small example to see 
what is the real behaviour,
so I created some package namespaces (where the __init__.py declare the 
namespace package).

   /home/andrea/test_ns:
   total used in directory 12 available 5655372
   drwxr-xr-x 3 andrea andrea 4096 Feb 10 14:46 a.b
   drwxr-xr-x 3 andrea andrea 4096 Feb 10 14:46 a.c
   -rw-r--r-- 1 andrea andrea  125 Feb 10 14:46 test.py

   /home/andrea/test_ns/a.b:
   total 8
   drwxr-xr-x 3 andrea andrea 4096 Feb 10 14:47 a
   -rw-r--r-- 1 andrea andrea   56 Feb 10 14:35 __init__.py

   /home/andrea/test_ns/a.b/a:
   total 8
   drwxr-xr-x 2 andrea andrea 4096 Feb 10 14:47 b
   -rw-r--r-- 1 andrea andrea   56 Feb 10 14:35 __init__.py

   /home/andrea/test_ns/a.b/a/b:
   total 12
   -rw-r--r-- 1 andrea andrea  25 Feb 10 14:36 api.py
   -rw-r--r-- 1 andrea andrea 153 Feb 10 14:37 api.pyc
   -rw-r--r-- 1 andrea andrea  56 Feb 10 14:35 __init__.py

   /home/andrea/test_ns/a.c:
   total 8
   drwxr-xr-x 3 andrea andrea 4096 Feb 10 14:47 a
   -rw-r--r-- 1 andrea andrea   56 Feb 10 14:35 __init__.py

   /home/andrea/test_ns/a.c/a:
   total 8
   drwxr-xr-x 2 andrea andrea 4096 Feb 10 14:47 c
   -rw-r--r-- 1 andrea andrea   56 Feb 10 14:35 __init__.py

   /home/andrea/test_ns/a.c/a/c:
   total 12
   -rw-r--r-- 1 andrea andrea  20 Feb 10 14:36 api.py
   -rw-r--r-- 1 andrea andrea 148 Feb 10 14:38 api.pyc
   -rw-r--r-- 1 andrea andrea  56 Feb 10 14:35 __init__.py


So this test.py works perfectly:
import sys
sys.path.insert(0, 'a.c')
sys.path.insert(0, 'a.b')

from a.b import api as api_ab

from a.c import api as api_ac

While just mixing the order:
import sys
sys.path.insert(0, 'a.b')

from a.b import api as api_ab

sys.path.insert(0, 'a.c')
from a.c import api as api_ac

Doesn't work anymore

[andrea@precision test_ns]$ python2 test.py
Traceback (most recent call last):
   File "test.py", line 7, in <module>
     from a.c import api as api_ac
ImportError: No module named c



Am I missing something/doing something stupid?

[toc] | [prev] | [next] | [standalone]

#20166

On 02/10/2012 09:51 AM, Andrea Crotti wrote:
> Ok now it's getting really confusing, I tried a small example to see
> what is the real behaviour,
> so I created some package namespaces (where the __init__.py declare the
> namespace package).
>
> /home/andrea/test_ns:
> total used in directory 12 available 5655372
> drwxr-xr-x 3 andrea andrea 4096 Feb 10 14:46 a.b
> drwxr-xr-x 3 andrea andrea 4096 Feb 10 14:46 a.c
> -rw-r--r-- 1 andrea andrea 125 Feb 10 14:46 test.py
>
> /home/andrea/test_ns/a.b:
> total 8
> drwxr-xr-x 3 andrea andrea 4096 Feb 10 14:47 a
> -rw-r--r-- 1 andrea andrea 56 Feb 10 14:35 __init__.py
>
> /home/andrea/test_ns/a.b/a:
> total 8
> drwxr-xr-x 2 andrea andrea 4096 Feb 10 14:47 b
> -rw-r--r-- 1 andrea andrea 56 Feb 10 14:35 __init__.py
>
> /home/andrea/test_ns/a.b/a/b:
> total 12
> -rw-r--r-- 1 andrea andrea 25 Feb 10 14:36 api.py
> -rw-r--r-- 1 andrea andrea 153 Feb 10 14:37 api.pyc
> -rw-r--r-- 1 andrea andrea 56 Feb 10 14:35 __init__.py
>
> /home/andrea/test_ns/a.c:
> total 8
> drwxr-xr-x 3 andrea andrea 4096 Feb 10 14:47 a
> -rw-r--r-- 1 andrea andrea 56 Feb 10 14:35 __init__.py
>
> /home/andrea/test_ns/a.c/a:
> total 8
> drwxr-xr-x 2 andrea andrea 4096 Feb 10 14:47 c
> -rw-r--r-- 1 andrea andrea 56 Feb 10 14:35 __init__.py
>
> /home/andrea/test_ns/a.c/a/c:
> total 12
> -rw-r--r-- 1 andrea andrea 20 Feb 10 14:36 api.py
> -rw-r--r-- 1 andrea andrea 148 Feb 10 14:38 api.pyc
> -rw-r--r-- 1 andrea andrea 56 Feb 10 14:35 __init__.py
>
>
> So this test.py works perfectly:
> import sys
> sys.path.insert(0, 'a.c')
> sys.path.insert(0, 'a.b')
>
> from a.b import api as api_ab
>
> from a.c import api as api_ac
>
> While just mixing the order:
> import sys
> sys.path.insert(0, 'a.b')
>
> from a.b import api as api_ab
>
> sys.path.insert(0, 'a.c')
> from a.c import api as api_ac
>
> Doesn't work anymore
>
> [andrea@precision test_ns]$ python2 test.py
> Traceback (most recent call last):
> File "test.py", line 7, in <module>
> from a.c import api as api_ac
> ImportError: No module named c
>
>
>
> Am I missing something/doing something stupid?

Yes, you've got periods in your directory names.  A period means 
something special within python, and specifically within the import.

When you say  from a.c import api

You're telling it:    from package a  get module c, and from there 
impoort the symbol api

But package a has no module c, so it complains.


In an earlier message you asserted you were using all absolute paths in 
your additions to sys.path.  Here you're inserting relative ones.  How come?

-- 

DaveA

[toc] | [prev] | [next] | [standalone]

#20167

On 02/10/2012 03:06 PM, Dave Angel wrote:
>
> Yes, you've got periods in your directory names.  A period means 
> something special within python, and specifically within the import.
>
> When you say  from a.c import api
>
> You're telling it:    from package a  get module c, and from there 
> impoort the symbol api
>
> But package a has no module c, so it complains.
>
>
> In an earlier message you asserted you were using all absolute paths 
> in your additions to sys.path.  Here you're inserting relative ones.  
> How come?
>

Well yes I have periods, but that's also the real-world situation.
We have many directories that are contributing to the same namespace in 
the same superdirectory
which should not interfere, so it was decided to give this naming.
It would be quite hard to change I guess and I have to prove that this 
is problem.

I renamed everything and this
import sys
from os import path
sys.path.insert(0, path.abspath('ab'))

from a.b import api as api_ab

sys.path.insert(0, path.abspath('ac'))
from a.c import api as api_ac

still fails, so the period in the name is not the problem.

Also absolute or relative paths in this small example doesn't make any 
difference.

Adding all the paths in one go works perfectly fine anyway, so I 
probably have to make sure
I add them *all* before anything is imported.

If there are better solutions I would like to hear them :)

[toc] | [prev] | [next] | [standalone]

#20168

Andrea Crotti wrote:

> Ok now it's getting really confusing, I tried a small example to see
> what is the real behaviour,
> so I created some package namespaces (where the __init__.py declare the
> namespace package).
> 
>    /home/andrea/test_ns:
>    total used in directory 12 available 5655372
>    drwxr-xr-x 3 andrea andrea 4096 Feb 10 14:46 a.b
>    drwxr-xr-x 3 andrea andrea 4096 Feb 10 14:46 a.c
>    -rw-r--r-- 1 andrea andrea  125 Feb 10 14:46 test.py
> 
>    /home/andrea/test_ns/a.b:
>    total 8
>    drwxr-xr-x 3 andrea andrea 4096 Feb 10 14:47 a
>    -rw-r--r-- 1 andrea andrea   56 Feb 10 14:35 __init__.py
> 
>    /home/andrea/test_ns/a.b/a:
>    total 8
>    drwxr-xr-x 2 andrea andrea 4096 Feb 10 14:47 b
>    -rw-r--r-- 1 andrea andrea   56 Feb 10 14:35 __init__.py
> 
>    /home/andrea/test_ns/a.b/a/b:
>    total 12
>    -rw-r--r-- 1 andrea andrea  25 Feb 10 14:36 api.py
>    -rw-r--r-- 1 andrea andrea 153 Feb 10 14:37 api.pyc
>    -rw-r--r-- 1 andrea andrea  56 Feb 10 14:35 __init__.py
> 
>    /home/andrea/test_ns/a.c:
>    total 8
>    drwxr-xr-x 3 andrea andrea 4096 Feb 10 14:47 a
>    -rw-r--r-- 1 andrea andrea   56 Feb 10 14:35 __init__.py
> 
>    /home/andrea/test_ns/a.c/a:
>    total 8
>    drwxr-xr-x 2 andrea andrea 4096 Feb 10 14:47 c
>    -rw-r--r-- 1 andrea andrea   56 Feb 10 14:35 __init__.py
> 
>    /home/andrea/test_ns/a.c/a/c:
>    total 12
>    -rw-r--r-- 1 andrea andrea  20 Feb 10 14:36 api.py
>    -rw-r--r-- 1 andrea andrea 148 Feb 10 14:38 api.pyc
>    -rw-r--r-- 1 andrea andrea  56 Feb 10 14:35 __init__.py
> 
> 
> So this test.py works perfectly:
> import sys
> sys.path.insert(0, 'a.c')
> sys.path.insert(0, 'a.b')
> 
> from a.b import api as api_ab
> 
> from a.c import api as api_ac
> 
> While just mixing the order:
> import sys
> sys.path.insert(0, 'a.b')
> 
> from a.b import api as api_ab
> 
> sys.path.insert(0, 'a.c')
> from a.c import api as api_ac
> 
> Doesn't work anymore
> 
> [andrea@precision test_ns]$ python2 test.py
> Traceback (most recent call last):
>    File "test.py", line 7, in <module>
>      from a.c import api as api_ac
> ImportError: No module named c
> 
> 
> 
> Am I missing something/doing something stupid?

The package a will be either a.c/a/ or a.b/a/ depending on whether a.c/ or 
a.b/ appears first in sys.path.

If it's  a.c/a, that does not contain a c submodule or subpackage.

[toc] | [prev] | [next] | [standalone]

#20169

On 02/10/2012 03:27 PM, Peter Otten wrote:
> The package a will be either a.c/a/ or a.b/a/ depending on whether 
> a.c/ or a.b/ appears first in sys.path. If it's a.c/a, that does not 
> contain a c submodule or subpackage. 


I would agree if I didn't have this declaration
__import__('pkg_resources').declare_namespace(__name__)
in each subdirectory.

And how do you explain the fact that changing the order everything works?
Namespace packages are supposed to work exactly like this, if it doesn't 
resolve the
"c" instead of raising an Exception it goes forward in the sys.path and 
try again, which
is what actually happens when I do this

sys.path.append(path.abspath('ab'))
sys.path.append(path.abspath('ac'))

from a.b import api as api_ab
from a.c import api as api_ac

Maybe this:
Definition: pkgutil.extend_path(path, name)
Docstring:
Extend a package's path.

Intended use is to place the following code in a package's __init__.py:

     from pkgutil import extend_path
     __path__ = extend_path(__path__, __name__)


might come handy, from what I'm gathering is the only way to have a more 
dynamic path
manipulation with namespace packages..

[toc] | [prev] | [next] | [standalone]

#20170

Peter Otten wrote:

> If it's  a.c/a, that does not contain a c submodule or subpackage.

Sorry, I meant a.b/a 

[toc] | [prev] | [next] | [standalone]

#20171

Andrea Crotti wrote:

> On 02/10/2012 03:27 PM, Peter Otten wrote:
>> The package a will be either a.c/a/ or a.b/a/ depending on whether
>> a.c/ or a.b/ appears first in sys.path. If it's a.c/a, that does not
>> contain a c submodule or subpackage.
> 
> 
> I would agree if I didn't have this declaration
> __import__('pkg_resources').declare_namespace(__name__)
> in each subdirectory.

Sorry, you didn't mention that in the post I responded to and I didn't 
follow the thread closely.

I found a description for declare_namespace() at
http://peak.telecommunity.com/DevCenter/PkgResources

but the text explaining the function is completely unintelligible to me, so 
I cannot contribute anything helpful here :(

[toc] | [prev] | [next] | [standalone]

#20186

On 02/10/2012 04:00 PM, Peter Otten wrote:
> Sorry, you didn't mention that in the post I responded to and I didn't
> follow the thread closely.
>
> I found a description for declare_namespace() at
> http://peak.telecommunity.com/DevCenter/PkgResources
>
> but the text explaining the function is completely unintelligible to me, so
> I cannot contribute anything helpful here :(
>

Well in the end I submitted a bug report
http://bugs.python.org/issue13991
I'm not sure it's really a bug and maybe I'm just doing something wrong,
but to me the behavior is at least unexpected..

[toc] | [prev] | [next] | [standalone]

#19824

Andrea Crotti <andrea.crotti.0@gmail.com> wrote:
>
>So suppose I want to modify the sys.path on the fly before running some code
>which imports from one of the modules added.
>
>at run time I do
>sys.path.extend(paths_to_add)
>
>but it still doesn't work and I get an import error.

Are you actually adding multiple paths?  One possible cause for error would
be this:
    sys.path.extend( '/usr/local/lib' )

That succeeds, but it doesn't do what you meant.  It adds "/" as a path,
then "u", then "s", then "r", and so on.
-- 
Tim Roberts, timr@probo.com
Providenza & Boekelheide, Inc.

[toc] | [prev] | [next] | [standalone]

#20034

On 2/1/2012 8:15 AM, Andrea Crotti wrote:
> So suppose I want to modify the sys.path on the fly before running some
> code
> which imports from one of the modules added.
>
> at run time I do
> sys.path.extend(paths_to_add)
>
> but it still doesn't work and I get an import error.

    Do

	import sys

first.

					John Nagle

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web