Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #19412 > unrolled thread

Where to put data

Back to article view | Back to comp.lang.python

Contents

  Where to put data bvdp <bob@mellowood.ca> - 2012-01-25 09:26 -0800
    Re: Where to put data Rick Johnson <rantingrickjohnson@gmail.com> - 2012-01-25 10:33 -0800
      Re: Where to put data bvdp <bob@mellowood.ca> - 2012-01-25 14:27 -0800
    Re: Where to put data Evan Driscoll <edriscoll@wisc.edu> - 2012-01-25 12:48 -0600
      Re: Where to put data bvdp <bob@mellowood.ca> - 2012-01-25 14:29 -0800
        Re: Where to put data Michael Torrie <torriem@gmail.com> - 2012-01-25 20:30 -0700
          Re: Where to put data bvdp <bob@mellowood.ca> - 2012-01-26 08:30 -0800
            Re: Where to put data Michael Torrie <torriem@gmail.com> - 2012-01-26 20:20 -0700
              Re: Where to put data bvdp <bob@mellowood.ca> - 2012-01-27 19:11 -0800
              Re: Where to put data bvdp <bob@mellowood.ca> - 2012-01-27 19:11 -0800
          Re: Where to put data bvdp <bob@mellowood.ca> - 2012-01-26 08:30 -0800
      Re: Where to put data bvdp <bob@mellowood.ca> - 2012-01-25 14:29 -0800
    Re: Where to put data "Martin P. Hellwig" <martin.hellwig@gmail.com> - 2012-01-26 02:16 +0000
    Re: Where to put data John Nagle <nagle@animats.com> - 2012-01-27 14:15 -0800
      Re: Where to put data bvdp <bob@mellowood.ca> - 2012-01-27 19:13 -0800

#19412 — Where to put data

I'm having a disagreement with a buddy on the packaging of a program we're doing in Python. It's got a number of modules and large number of library files. The library stuff is data, not code. 

I'd like to put the modules in /usr/lib/pythonX.Y/mymodules or wherever setup.py decides. And the data in /usr/share/lib/myprogram.

My buddy says, that it'll be hard to be consistant in the /usr/share/.. when we consider platforms other than linux. So, he wants:

   /usr/lib/pythonX.Y/myprogram
       mymodules ...
       mydata  ....

I've got 2 issues with this:

   1. I don't know if putting data in the python tree is "legit".
   2. I'd have to do a lot of rewritting. My modules currently use:

           include mymodules.foobar
                x=mymodules.foobar.func()

   and I would need to change that to:

            include myprogram.mymodules.foobar

                x=myprogram.mymodules.foobar.func()


unless there is python way to drop the "myprogram" bit?

[toc] | [next] | [standalone]

#19417

On Jan 25, 11:26 am, bvdp <b...@mellowood.ca> wrote:

> I've got 2 issues with this:
>
>    1. I don't know if putting data in the python tree is "legit".
>    2. I'd have to do a lot of rewritting. My modules currently use:

I would not put anything in the toplevel Python folder. You need to
place everything under site-packages --> "Python27\Lib\site-packages
\PackageName\blah". Of course client created files should be saved to
a more accessible place.

> [...]
> unless there is python way to drop the "myprogram" bit?

Considering mymodules is a valid python package, you can do:
py> from mymodules import foobar

[toc] | [prev] | [next] | [standalone]

#19441

> I would not put anything in the toplevel Python folder. You need to
> place everything under site-packages --> "Python27\Lib\site-packages
> \PackageName\blah". Of course client created files should be saved to
> a more accessible place.

Oh. Just looking at my setup (Ubunutu 11.10) and I see that /usr/lib/python2.7 doesn't have a site-packages directory. However, /usr/local/lib/python2.7 has both dist-packages and site-packages. 

So, my stuff should probably go into /usr/local/lib/python2.7/site-packages?

Interesting (?) that these are empty dirs right now?

Also, if I look at my sys.path value I see that /usr/local/lib/python2.7/dist-packages is in the path; but site-packages is not.

> Considering mymodules is a valid python package, you can do:
> py> from mymodules import foobar

Yes. Understand that part. And then I can just call 'foobar()'. What I was wondering is if there was a way to set something in __init__.py to shorten the calls. So, if I have:

     /usr/local/lib/python2.7/dist-packages/myprogram
           mymods
               __init__.py
                mod1.py
                mod2.py
           mylibs
           __init__.py


Is there some magic I can put into myprogram/__init__.py which forces modules to be imported from mymods instead of myprogram/mymods?

Thanks.

[toc] | [prev] | [next] | [standalone]

#19419

I would just like to make a strong plea that you make it possible to 
install in places other than /usr. Bascially, 'python setup.py install 
--prefix /some/alternative/place' should work.

Evan


On 01/25/2012 11:26 AM, bvdp wrote:
> I'm having a disagreement with a buddy on the packaging of a program we're doing in Python. It's got a number of modules and large number of library files. The library stuff is data, not code.
>
> I'd like to put the modules in /usr/lib/pythonX.Y/mymodules or wherever setup.py decides. And the data in /usr/share/lib/myprogram.
>
> My buddy says, that it'll be hard to be consistant in the /usr/share/.. when we consider platforms other than linux. So, he wants:
>
>     /usr/lib/pythonX.Y/myprogram
>         mymodules ...
>         mydata  ....
>
> I've got 2 issues with this:
>
>     1. I don't know if putting data in the python tree is "legit".
>     2. I'd have to do a lot of rewritting. My modules currently use:
>
>             include mymodules.foobar
>                  x=mymodules.foobar.func()
>
>     and I would need to change that to:
>
>              include myprogram.mymodules.foobar
>
>                  x=myprogram.mymodules.foobar.func()
>
>
> unless there is python way to drop the "myprogram" bit?

[toc] | [prev] | [next] | [standalone]

#19442

Right now my program does a search for modules in "all the normal places", which seems to work for windows, mac and linux. Once the modules are found I just insert that location into sys.path[0].

Which permits the modules to reside anywhere on the HDD. However, I have feeling that this isn't quite pythonic.

[toc] | [prev] | [next] | [standalone]

#19469

On 01/25/2012 03:29 PM, bvdp wrote:
> Right now my program does a search for modules in "all the normal
> places", which seems to work for windows, mac and linux. Once the
> modules are found I just insert that location into sys.path[0].
> 
> Which permits the modules to reside anywhere on the HDD. However, I
> have feeling that this isn't quite pythonic

Unless you are writing a python library that will be used by others, I
don't think that where you put your files has anything to do with being
"pythonic" or not.  Just do what works for your OS.

On Linux, many packages written in many languages have to deal with the
problem of where to put their files.  For example, firefox, even when
installed as a package on Fedora, puts just about everything in
/usr/lib/firefox-#.#, and then symlinks the start binary back into
/usr/bin.  Other packages, such as Calibre, put things in
/usr/lib/calibre (most python modules go here), and some things like
extension scripts in /usr/share/calibre.  Other packages usr
/usr/libexec/packagename/.  In the case of both firefox and calibre, if
you install from tarball it makes a "firefox" or "calibre" folder in a
place of your choosing and dumps its stuff inside, sometimes with a
mini-unix directory structure (bin, share, lib etc).

On Mac of course, you can put everything in your application bundle.
That's how most standalone apps written in python work.

On Windows you could stick all your python modules in your application's
directory in Program Files.

[toc] | [prev] | [next] | [standalone]

#19493

On Wednesday, January 25, 2012 8:30:54 PM UTC-7, Michael Torrie wrote:

> Unless you are writing a python library that will be used by others, I
> don't think that where you put your files has anything to do with being
> "pythonic" or not.  Just do what works for your OS.

Yes. I agree and it's nice to have a confirmation. So far I've been putting all my program into /usr/local/share/MYPROGRAM and then simply inserting an entry into sys.path. 

Then, for other systems, I check a few common locations until I find the installation. 

I'm getting mangled by the debian maintainers and friends who seem to believe that python modules need to go into /usr/lib/python...

[toc] | [prev] | [next] | [standalone]

#19518

On 01/26/2012 09:30 AM, bvdp wrote:
> Yes. I agree and it's nice to have a confirmation. So far I've been
> putting all my program into /usr/local/share/MYPROGRAM and then
> simply inserting an entry into sys.path.
> 
> Then, for other systems, I check a few common locations until I find
> the installation.
> 
> I'm getting mangled by the debian maintainers and friends who seem to
> believe that python modules need to go into /usr/lib/python...

I guess the maintainers aren't distinguishing between python apps and
their submodules and general python modules (libraries), which is pretty
silly.  Even as a mere user I would not like my /usr/lib/python
directory cluttered with python code that is not useful generally but is
only for specific apps.  Namespace collisions are inevitable with other
python apps (not libraries) if folks insist on doing this.

Calibre appears to be in the Ubuntu standard repositories.  I just
checked and in calibre proper (not talking about dependent libraries and
things that would be useful outside of calibre), there are no python
files installed in /usr/lib/python/.  Calibre modules that belong to
calibre proper are in /usr/lib/calibre.  Recipes (really just python
scripts) are in /usr/share/calibre.  Maybe Ubuntu is doing things
differently than Debian, but I'm hard pressed to see the logic in
forcing everything ever written in python, such as submodules, installed
to /usr/lib/python.  Baffles the mind.

[toc] | [prev] | [next] | [standalone]

#19530

On Thursday, January 26, 2012 8:20:24 PM UTC-7, Michael Torrie wrote:
>
> > I'm getting mangled by the debian maintainers and friends who seem to
> > believe that python modules need to go into /usr/lib/python...
> 
> I guess the maintainers aren't distinguishing between python apps and
> their submodules and general python modules (libraries), which is pretty
> silly.  Even as a mere user I would not like my /usr/lib/python
> directory cluttered with python code that is not useful generally but is
> only for specific apps.  Namespace collisions are inevitable with other
> python apps (not libraries) if folks insist on doing this.

Well, I might be wrong in my assumptions. Never got invited to join the inner circle so I'm looking at all this from the outside.

> 
> Calibre appears to be in the Ubuntu standard repositories.  I just

Yeah, I looked at that as well after your earlier post.

> checked and in calibre proper (not talking about dependent libraries and
> things that would be useful outside of calibre), there are no python
> files installed in /usr/lib/python/.  Calibre modules that belong to

But, when you dl from the calibre site the default location is /opt.

> calibre proper are in /usr/lib/calibre.  Recipes (really just python
> scripts) are in /usr/share/calibre.  Maybe Ubuntu is doing things
> differently than Debian, but I'm hard pressed to see the logic in
> forcing everything ever written in python, such as submodules, installed
> to /usr/lib/python.  Baffles the mind.

I completely agree.

Mind you, one consolation in putting things in, for example, /usr/lib/pythonX.Y are:

   - you can let setup find that that magic location
   - you don't need to worry about your app finding the lib (python modules).

But, I've pretty much decided that the easy way (and dare I say the correct way?) is to let my packager decide where to install the modules. My program really doesn't care (nor do I). And, then I'll end up with my program's stuff something like:

      myprogram
         mymodules ...
         mylib ...
         program-bin

And then have a link in the users path which is a link or a one line call to program-bin. With modules in a directory at the same level as program-bin I don't have to do any module searches, etc. Seems to be a simple and sort-of-elegant solution.

I've tried this with one-line callers and links and it seems to work in all cases. Any gotchas? 

[toc] | [prev] | [next] | [standalone]

#19531

On Thursday, January 26, 2012 8:20:24 PM UTC-7, Michael Torrie wrote:
>
> > I'm getting mangled by the debian maintainers and friends who seem to
> > believe that python modules need to go into /usr/lib/python...
> 
> I guess the maintainers aren't distinguishing between python apps and
> their submodules and general python modules (libraries), which is pretty
> silly.  Even as a mere user I would not like my /usr/lib/python
> directory cluttered with python code that is not useful generally but is
> only for specific apps.  Namespace collisions are inevitable with other
> python apps (not libraries) if folks insist on doing this.

Well, I might be wrong in my assumptions. Never got invited to join the inner circle so I'm looking at all this from the outside.

> 
> Calibre appears to be in the Ubuntu standard repositories.  I just

Yeah, I looked at that as well after your earlier post.

> checked and in calibre proper (not talking about dependent libraries and
> things that would be useful outside of calibre), there are no python
> files installed in /usr/lib/python/.  Calibre modules that belong to

But, when you dl from the calibre site the default location is /opt.

> calibre proper are in /usr/lib/calibre.  Recipes (really just python
> scripts) are in /usr/share/calibre.  Maybe Ubuntu is doing things
> differently than Debian, but I'm hard pressed to see the logic in
> forcing everything ever written in python, such as submodules, installed
> to /usr/lib/python.  Baffles the mind.

I completely agree.

Mind you, one consolation in putting things in, for example, /usr/lib/pythonX.Y are:

   - you can let setup find that that magic location
   - you don't need to worry about your app finding the lib (python modules).

But, I've pretty much decided that the easy way (and dare I say the correct way?) is to let my packager decide where to install the modules. My program really doesn't care (nor do I). And, then I'll end up with my program's stuff something like:

      myprogram
         mymodules ...
         mylib ...
         program-bin

And then have a link in the users path which is a link or a one line call to program-bin. With modules in a directory at the same level as program-bin I don't have to do any module searches, etc. Seems to be a simple and sort-of-elegant solution.

I've tried this with one-line callers and links and it seems to work in all cases. Any gotchas? 

[toc] | [prev] | [next] | [standalone]

#19494

On Wednesday, January 25, 2012 8:30:54 PM UTC-7, Michael Torrie wrote:

> Unless you are writing a python library that will be used by others, I
> don't think that where you put your files has anything to do with being
> "pythonic" or not.  Just do what works for your OS.

Yes. I agree and it's nice to have a confirmation. So far I've been putting all my program into /usr/local/share/MYPROGRAM and then simply inserting an entry into sys.path. 

Then, for other systems, I check a few common locations until I find the installation. 

I'm getting mangled by the debian maintainers and friends who seem to believe that python modules need to go into /usr/lib/python...

[toc] | [prev] | [next] | [standalone]

#19443

Right now my program does a search for modules in "all the normal places", which seems to work for windows, mac and linux. Once the modules are found I just insert that location into sys.path[0].

Which permits the modules to reside anywhere on the HDD. However, I have feeling that this isn't quite pythonic.

[toc] | [prev] | [next] | [standalone]

#19498

On 25/01/2012 17:26, bvdp wrote:

<cut explanation of bikeshed argument, where do I put the darn things>

Well once you think about distributing, here is the guide line I use:

- If it is meant as a library that can be 'imported' in python:
 > site-packages is the place to be, some linux distros are rather 
creative with them so be careful.

- If it is a 'stand-alone' application that just happens to use the 
python interpreter as a dependency:
 > /usr/local/bin for the executable script and /usr/local/lib for the 
program module(s) itself, of course this is platform dependent, consult 
posix first and then distribution/os specific preferences.

- If the additional (binary) data is static in nature:
 > /usr/local/share is the right place (or whichever preference the 
distribution/os has)

- If the additional (binary) data is dynamic in nature (like databases) 
and the program is run as a daemon:
 > /usr/local/,  /var/ or /opt/

- If the additional (binary) data is dynamic and it is run per user:
 > $HOME/.[application name]/ (the famous dot files in your home folder).

All that is unix like of course, Windows tend to put it all in the 
application folder in the Program Files folder, and user specific data 
in the profiles Application Data.

Of course opinions vary so I can only say this is what I usually follow, 
with the wisdom bestowed upon me by unix admins that where much more 
experience then I ever will be and always had a fit when I didn't put it 
in the right directory.

YMMV
-- 
mph

[toc] | [prev] | [next] | [standalone]

#19527

On 1/25/2012 9:26 AM, bvdp wrote:
> I'm having a disagreement with a buddy on the packaging of a program
> we're doing in Python. It's got a number of modules and large number
> of library files. The library stuff is data, not code.

     How much data?  Megabytes? Gigabytes?

     I have some modules which contain nothing but big
constants, written by a program in Python format.

				John Nagle

[toc] | [prev] | [next] | [standalone]

#19532

On Friday, January 27, 2012 3:15:44 PM UTC-7, John Nagle wrote:
> On 1/25/2012 9:26 AM, bvdp wrote:
> > I'm having a disagreement with a buddy on the packaging of a program
> > we're doing in Python. It's got a number of modules and large number
> > of library files. The library stuff is data, not code.
> 
>      How much data?  Megabytes? Gigabytes?
> 
>      I have some modules which contain nothing but big
> constants, written by a program in Python format.
> 
> 				John Nagle

A couple of hundred files totaling about 2 meg. Not a lot. Gosh, I remember when this was several full floppies :)

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web