Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #12273 > unrolled thread

Understanding .pth in site-packages

Back to article view | Back to comp.lang.python

Contents

  Understanding .pth in site-packages Josh English <Joshua.R.English@gmail.com> - 2011-08-27 09:56 -0700
    Re: Understanding .pth in site-packages Philip Semanchuk <philip@semanchuk.com> - 2011-08-27 13:18 -0400
      Re: Understanding .pth in site-packages Josh English <Joshua.R.English@gmail.com> - 2011-08-27 10:57 -0700
        Re: Understanding .pth in site-packages "OKB (not okblacke)" <brenNOSPAMbarn@NObrenSPAMbarn.net> - 2011-08-27 19:45 +0000
          Re: Understanding .pth in site-packages Josh English <Joshua.R.English@gmail.com> - 2011-08-27 15:39 -0700
            Re: Understanding .pth in site-packages "OKB (not okblacke)" <brenNOSPAMbarn@NObrenSPAMbarn.net> - 2011-08-28 05:20 +0000
      Re: Understanding .pth in site-packages Josh English <Joshua.R.English@gmail.com> - 2011-08-27 10:57 -0700
        Re: Understanding .pth in site-packages Philip Semanchuk <philip@semanchuk.com> - 2011-08-27 14:07 -0400
        Re: Understanding .pth in site-packages Terry Reedy <tjreedy@udel.edu> - 2011-08-27 16:14 -0400
        Re: Understanding .pth in site-packages Philip Semanchuk <philip@semanchuk.com> - 2011-08-27 16:28 -0400
          Re: Understanding .pth in site-packages Josh English <Joshua.R.English@gmail.com> - 2011-08-27 15:49 -0700
          Re: Understanding .pth in site-packages Josh English <Joshua.R.English@gmail.com> - 2011-08-27 15:49 -0700
            Re: Understanding .pth in site-packages Philip Semanchuk <philip@semanchuk.com> - 2011-08-27 23:01 -0400
        Re: Understanding .pth in site-packages Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2011-08-27 13:53 -0700
          Re: Understanding .pth in site-packages Josh English <Joshua.R.English@gmail.com> - 2011-08-27 15:41 -0700
          Re: Understanding .pth in site-packages Josh English <Joshua.R.English@gmail.com> - 2011-08-27 15:41 -0700
            Re: Understanding .pth in site-packages Stephen Hansen <me+list/python@ixokai.io> - 2011-08-27 16:44 -0700
    Re: Understanding .pth in site-packages Peter Otten <__peter__@web.de> - 2011-08-27 19:29 +0200
    Re: Understanding .pth in site-packages Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2011-08-27 13:53 -0700

#12273 — Understanding .pth in site-packages

(This may be a shortened double post)

I have a development version of a library in c:\dev\XmlDB\xmldb

After testing the setup script I also have c:\python27\lib\site-packages\xmldb

Now I'm continuing to develop it and simultaneously building an application with it.

I thought I could plug into my site-packages directory a file called xmldb.pth with:

c:\dev\XmlDB\xmldb

which should redirect import statements to the development version of the library.

This doesn't seem to work.

Is there a better way to redirect import statements without messing with the system path or the PYTHONPATH variable?

Josh

[toc] | [next] | [standalone]

#12276

On Aug 27, 2011, at 12:56 PM, Josh English wrote:

> (This may be a shortened double post)
> 
> I have a development version of a library in c:\dev\XmlDB\xmldb
> 
> After testing the setup script I also have c:\python27\lib\site-packages\xmldb
> 
> Now I'm continuing to develop it and simultaneously building an application with it.
> 
> I thought I could plug into my site-packages directory a file called xmldb.pth with:
> 
> c:\dev\XmlDB\xmldb
> 
> which should redirect import statements to the development version of the library.
> 
> This doesn't seem to work.


xmldb.pth should contain the directory that contains xmldb:
c:\dev\XmlDB

Examining sys.path at runtime probably would have helped you to debug the effect of your .pth file.

On another note, I don't know if the behavior of 'import xmldb' is defined when xmldb is present both as a directory in site-pacakges and also as a .pth file. You're essentially giving Python two choices from where to import xmldb, and I don't know which Python will choose. It may be arbitrary. I've looked for some sort of statement on this topic in the documentation, but haven't come across it yet. 


> Is there a better way to redirect import statements without messing with the system path or the PYTHONPATH variable?

Personally I have never used PYTHONPATH.


Hope this helps
Philip

[toc] | [prev] | [next] | [standalone]

#12284

Philip,

Yes, the proper path should be c:\dev\XmlDB, which has the setup.py, xmldb subfolder, the docs subfolder, and example subfolder, and the other text files proscribed by the package development folder.

I could only get it to work, though, by renaming the xmldb folder in the site-packages directory, and deleting the egg file created in the site-packages directory. 

Why the egg file, which doesn't list any paths, would interfere I do not know.

But with those changes, the xmldb.pth file is being read.

So I think the preferred search order is:

1. a folder in the site-packages directory
2. an Egg file (still unsure why)
3. A .pth file

It's a strange juju that I haven't figured out yet. 

Thanks for the hint.

Josh

[toc] | [prev] | [next] | [standalone]

#12290

Josh English wrote:

> Philip,
> 
> Yes, the proper path should be c:\dev\XmlDB, which has the
> setup.py, xmldb subfolder, the docs subfolder, and example
> subfolder, and the other text files proscribed by the package
> development folder. 
> 
> I could only get it to work, though, by renaming the xmldb folder
> in the site-packages directory, and deleting the egg file created
> in the site-packages directory. 
> 
> Why the egg file, which doesn't list any paths, would interfere I
> do not know. 
> 
> But with those changes, the xmldb.pth file is being read.
> 
> So I think the preferred search order is:
> 
> 1. a folder in the site-packages directory
> 2. an Egg file (still unsure why)
> 3. A .pth file

    	You say that the egg file was created by the setup script for the 
library.  Are you sure that this script did not also create or modify a 
.pth file of its own, adding the egg to the path?

    	.pth files do not "redirect" imports from site-packages; they add 
EXTRA directories to sys.path.  Also note that this means the .pth file 
itself is not part of the search path; it's not like you shadow a 
package xyz by creating a .pth file xyz.pth "instead".  A single .pth 
file can list multiple directories, and it's those directories that are 
added to the path.

    	I'm not sure how your package is set up, but easy_install, for 
instance, creates an easy_install.pth file in site-packages.  This file 
contains references to egg files (or, at least in my case, .egg 
directories created by unpacking the eggs) for each package installed 
with easy_install.  As far as I'm aware, Python doesn't have special 
rules for putting egg files in the search path, so my guess is that 
it's something like that: the "setup script" is creating a .pth file (or 
modifying an existing .pth file) to add the egg to the path.

    	Read http://docs.python.org/library/site.html for the description 
of how .PTH files work.  I don't think there is a general way to 
globally "shadow" a package that exists in site-packages.  However, 
according to the docs the .pth files are added in alphabetical order, so 
if it is indeed easy_install.pth that is adding your egg, you could hack 
around it by making a file with an alphabetically earlier name (e.g., 
a_xmldb.pth).

-- 
--OKB (not okblacke)
Brendan Barnwell
"Do not follow where the path may lead.  Go, instead, where there is
no path, and leave a trail."
	--author unknown

[toc] | [prev] | [next] | [standalone]

#12311

OKB,

The setup.py script created the egg, but not the .pth file. I created that myself.

Thank you for clarifying about how .pth works. I know "redirect imports" was the wrong phrase, but it worked in my head at the time. It appears, at least on my system, that Python will find site-packages/foo before it finds and reads site-packages/foo.pth.

At least this solution gives me a way to develop my libraries outside of site-packages.

Josh

[toc] | [prev] | [next] | [standalone]

#12332

Josh English wrote:

> OKB,
> 
> The setup.py script created the egg, but not the .pth file. I
> created that myself. 
> 
> Thank you for clarifying about how .pth works. I know "redirect
> imports" was the wrong phrase, but it worked in my head at the
> time. It appears, at least on my system, that Python will find
> site-packages/foo before it finds and reads site-packages/foo.pth. 
> 
> At least this solution gives me a way to develop my libraries
> outside of site-packages. 

    	Well, I'm still not totally sure what your setup is, but assuming 
site-packages/foo is a directory containing an __init__.py (that is, it 
is a package), then yes, it will be found before an alternative package 
in a directory named with a .pth file.  Note that I don't say it will be 
found before the .pth file, because, again, the finding of the package 
(when you do "import foo") happens much later than the processing of the 
.pth file.  So it doesn't find site-packages/foo before it reads 
foo.pth; it just finds site-packages/foo before it finds the other foo 
that foo.pth was trying to point to.

    	Let's say your .pth file specifies the directory /elsewhere.  The 
.pth file is processed by site.py when the interpreter starts up, and at 
that time /elsewhere  will be appended to sys.path.  Later, when you do 
the import, it searches sys.path in order.  site-packages itself will be 
earlier in sys.path than /elsewhere, so a package site-packages/foo will 
be found before /elsewhere/foo.  The key here is that the .pth file is 
processed at interpreter-start time, but the search for foo doesn't take 
place until you actually execute "import foo".

    	If you want to make your /elsewhere "jump the line" and go to the 
front, look at easy_install.pth, which seems to have some magic code at 
the end that moves its eggs ahead of site-packages in sys.path.  I'm not 
sure how this works, though, and it seems like a risky proposition.


-- 
--OKB (not okblacke)
Brendan Barnwell
"Do not follow where the path may lead.  Go, instead, where there is
no path, and leave a trail."
	--author unknown

[toc] | [prev] | [next] | [standalone]

#12285

Philip,

Yes, the proper path should be c:\dev\XmlDB, which has the setup.py, xmldb subfolder, the docs subfolder, and example subfolder, and the other text files proscribed by the package development folder.

I could only get it to work, though, by renaming the xmldb folder in the site-packages directory, and deleting the egg file created in the site-packages directory. 

Why the egg file, which doesn't list any paths, would interfere I do not know.

But with those changes, the xmldb.pth file is being read.

So I think the preferred search order is:

1. a folder in the site-packages directory
2. an Egg file (still unsure why)
3. A .pth file

It's a strange juju that I haven't figured out yet. 

Thanks for the hint.

Josh

[toc] | [prev] | [next] | [standalone]

#12287

On Aug 27, 2011, at 1:57 PM, Josh English wrote:

> Philip,
> 
> Yes, the proper path should be c:\dev\XmlDB, which has the setup.py, xmldb subfolder, the docs subfolder, and example subfolder, and the other text files proscribed by the package development folder.
> 
> I could only get it to work, though, by renaming the xmldb folder in the site-packages directory, and deleting the egg file created in the site-packages directory. 
> 
> Why the egg file, which doesn't list any paths, would interfere I do not know.
> 
> But with those changes, the xmldb.pth file is being read.
> 
> So I think the preferred search order is:
> 
> 1. a folder in the site-packages directory
> 2. an Egg file (still unsure why)
> 3. A .pth file


That might be implementation-dependent or it might even come down to something as simple as the in which order the operating system returns files/directories when asked for a listing. In other words, unless you can find something in the documentation (or Python's import implementation) that confirms your preferred search order observation, I would not count on it working the same way with all systems, all Pythons, or even all directory names.




Good luck
Philip

[toc] | [prev] | [next] | [standalone]

#12295

On 8/27/2011 2:07 PM, Philip Semanchuk wrote:
>
> On Aug 27, 2011, at 1:57 PM, Josh English wrote:
>
>> Philip,
>>
>> Yes, the proper path should be c:\dev\XmlDB, which has the
>> setup.py, xmldb subfolder, the docs subfolder, and example
>> subfolder, and the other text files proscribed by the package
>> development folder.
>>
>> I could only get it to work, though, by renaming the xmldb folder
>> in the site-packages directory, and deleting the egg file created
>> in the site-packages directory.
>>
>> Why the egg file, which doesn't list any paths, would interfere I
>> do not know.
>>
>> But with those changes, the xmldb.pth file is being read.
>>
>> So I think the preferred search order is:
>>
>> 1. a folder in the site-packages directory 2. an Egg file (still
>> unsure why) 3. A .pth file
>
>
> That might be implementation-dependent or it might even come down to
> something as simple as the in which order the operating system
> returns files/directories when asked for a listing.

Doc says first match, and I presume that includes first match within a 
directory.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#12297

On Aug 27, 2011, at 4:14 PM, Terry Reedy wrote:

> On 8/27/2011 2:07 PM, Philip Semanchuk wrote:
>> 
>> On Aug 27, 2011, at 1:57 PM, Josh English wrote:
>> 
>>> Philip,
>>> 
>>> Yes, the proper path should be c:\dev\XmlDB, which has the
>>> setup.py, xmldb subfolder, the docs subfolder, and example
>>> subfolder, and the other text files proscribed by the package
>>> development folder.
>>> 
>>> I could only get it to work, though, by renaming the xmldb folder
>>> in the site-packages directory, and deleting the egg file created
>>> in the site-packages directory.
>>> 
>>> Why the egg file, which doesn't list any paths, would interfere I
>>> do not know.
>>> 
>>> But with those changes, the xmldb.pth file is being read.
>>> 
>>> So I think the preferred search order is:
>>> 
>>> 1. a folder in the site-packages directory 2. an Egg file (still
>>> unsure why) 3. A .pth file
>> 
>> 
>> That might be implementation-dependent or it might even come down to
>> something as simple as the in which order the operating system
>> returns files/directories when asked for a listing.
> 
> Doc says first match, and I presume that includes first match within a directory.

First match using which ordering? Do the docs clarify that?


Thanks
Philip

[toc] | [prev] | [next] | [standalone]

#12312

When I run: os.listdir('c:\Python27\lib\site-packages') I get the contents in order, so the folders come before .pth files (as nothing comes before something.) I would guess Python is using os.listdir. Why wouldn't it?

[toc] | [prev] | [next] | [standalone]

#12313

When I run: os.listdir('c:\Python27\lib\site-packages') I get the contents in order, so the folders come before .pth files (as nothing comes before something.) I would guess Python is using os.listdir. Why wouldn't it?

[toc] | [prev] | [next] | [standalone]

#12327

On Aug 27, 2011, at 6:49 PM, Josh English wrote:

> When I run: os.listdir('c:\Python27\lib\site-packages') I get the contents in order, so the folders come before .pth files (as nothing comes before something.)

That's one definition of "in order". =)


> I would guess Python is using os.listdir. Why wouldn't it?

If you mean that Python uses os.listdir() during import resolution, then yes I agree that's probable. And os.listdir() doesn't guarantee any consistent order. In fact, the documentation explicitly states that the list is returned in arbitrary order. Like a lot of things in Python, os.listdir() probably relies on the underlying C library which varies from system to system. (Case in point -- on my Mac, os.listdir() returns things in the same order as the 'ls' command, which is case-sensitive alphabetical, files & directories mixed -- different from Windows.)

So if import relies on os.listdir(), then you're relying on arbitrary resolution when you have a .pth file that shadows a site-packages directory. Those rules will probably work consistently on your particular system, you're developing a habit around what is essentially an implementation quirk.  

Cheers
Philip

[toc] | [prev] | [next] | [standalone]

#12299

On Sat, 27 Aug 2011 10:57:01 -0700 (PDT), Josh English
<Joshua.R.English@gmail.com> declaimed the following in
gmane.comp.python.general:


> So I think the preferred search order is:
> 
> 1. a folder in the site-packages directory
> 2. an Egg file (still unsure why)
> 3. A .pth file
> 
> It's a strange juju that I haven't figured out yet. 
>
	.egg files, as I recall, are basically just ZIP compressed files --
and since Python can (again, as I recall) import from a ZIP compressed
file, .egg files get added to sys.path

	(just open an interactive Python session and do

import sys
sys.path

and review the results)
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
        wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [next] | [standalone]

#12309

I have .egg files in my system path. The Egg file created by my setup script doesn't include anything but the introductory text. If I open other eggs I see the zipped data, but not for my own files.

Is having a zipped egg file any faster than a regular package? or does it just prevent people from seeing the code?

Josh

[toc] | [prev] | [next] | [standalone]

#12310

I have .egg files in my system path. The Egg file created by my setup script doesn't include anything but the introductory text. If I open other eggs I see the zipped data, but not for my own files.

Is having a zipped egg file any faster than a regular package? or does it just prevent people from seeing the code?

Josh

[toc] | [prev] | [next] | [standalone]

#12319

[Multipart message — attachments visible in raw view] — view raw

On 8/27/11 3:41 PM, Josh English wrote:
> I have .egg files in my system path. The Egg file created by my setup script doesn't include anything but the introductory text. If I open other eggs I see the zipped data, but not for my own files.

Sounds like your setup.py isn't actually including your source.
> 
> Is having a zipped egg file any faster than a regular package? or does it just prevent people from seeing the code?

IIUC, its nominally very slightly faster to use an egg, because it can
skip a lot of filesystem calls. But I've only heard that and can't
completely confirm it (internal testing at my day job did not
conclusively support this, but our environments are uniquely weird).

But that speed boost (if even true) isn't really the point of
eggs-as-files -- eggs are just easy to deal with as files is all. They
don't prevent people from seeing the code*, they're just regular zip
files and can be unzipped fine.

I almost always install unzip my eggs on a developer machine, because I
inevitably want to go poke inside and see what's actually going on.

-- 

   Stephen Hansen
   ... Also: Ixokai
   ... Mail: me+list/python (AT) ixokai (DOT) io
   ... Blog: http://meh.ixokai.io/

* Although you can make an egg and then go and remove all the .PY files
from it, and leave just the compiled .PYC files, and Python will load it
fine. At the day job, that's what we do. But, you have to be aware that
this ties the egg to a specific version of Python, and its not difficult
for someone industrious to disassemble and/or decompile the PYC back to
effectively equivalent PY files to edit away if they want.

[toc] | [prev] | [next] | [standalone]

#12280

Josh English wrote:

> I have a development version of a library in c:\dev\XmlDB\xmldb
> 
> After testing the setup script I also have
> c:\python27\lib\site-packages\xmldb
> 
> Now I'm continuing to develop it and simultaneously building an
> application with it.
> 
> I thought I could plug into my site-packages directory a file called
> xmldb.pth with:
> 
> c:\dev\XmlDB\xmldb
> 
> which should redirect import statements to the development version of the
> library.
> 
> This doesn't seem to work.

You have to put the directory containing the package into the pth-file. 
That's probably

c:\dev\XmlDB

in your case. Also, Python will stop at the first matching module or 
package; if you keep c:\python27\lib\site-packages\xmldb that will shadow 
c:\dev\XmlDB\xmldb.

%APPDATA%/Python/Python26/site-packages

may be a good place for the pth-file (I'm not on Windows and too lazy to 
figure out where %APPDATA% actually is. The PEP 
http://www.python.org/dev/peps/pep-0370/ may help)

[toc] | [prev] | [next] | [standalone]

#12301

On Sat, 27 Aug 2011 19:29:07 +0200, Peter Otten <__peter__@web.de>
declaimed the following in gmane.comp.python.general:


> 
> %APPDATA%/Python/Python26/site-packages
> 
> may be a good place for the pth-file (I'm not on Windows and too lazy to 
> figure out where %APPDATA% actually is. The PEP 
> http://www.python.org/dev/peps/pep-0370/ may help)

	%APPDATA% is a user specific directory, on WinXP:
C:\Documents and Settings\<username>\Application Data

	If PEP-370 has been implemented, I'm running too old of a Python for
it <G>; my site.py doesn't have the control logic for it.
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
        wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web