Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #80449 > unrolled thread

Python modules

Back to article view | Back to comp.lang.python

Contents

  Python modules Steve Hayes <hayesstw@telkomsa.net> - 2014-11-10 06:55 +0200
    Re: Python modules Deepfriedice <Deepfriedice@openmailbox.org> - 2014-11-10 14:55 +1000
    Re: Python modules Ben Finney <ben+python@benfinney.id.au> - 2014-11-10 16:12 +1100
      Re: Python modules Steve Hayes <hayesstw@telkomsa.net> - 2014-11-10 08:58 +0200
        Re: Python modules Ben Finney <ben+python@benfinney.id.au> - 2014-11-10 18:48 +1100
    Re: Python modules Roy Smith <roy@panix.com> - 2014-11-10 08:36 -0500
      Re: Python modules Chris Angelico <rosuav@gmail.com> - 2014-11-11 00:49 +1100
    Re: Python modules Will Acheson <willyach07@gmail.com> - 2014-11-11 12:06 -0800
      Re: Python modules Ben Finney <ben+python@benfinney.id.au> - 2014-11-12 09:18 +1100

#80449 — Python modules

I have a book on Python that advocates dividing programs into modules, and
importing them when needed. 

I have a question about this.

I can understand doing that in a compiled language, where different modules
can be imported from all sorts of places when the program is compiled. 

But I understand that Python is an interpreted language, and If I wrote a
program in Python like that, and wanted to run it on another computer, how
would it find all the modules to import at run-time, unless I copied the whole
directory structure over to the other computer?




-- 
Steve Hayes from Tshwane, South Africa
Web:  http://www.khanya.org.za/stevesig.htm
Blog: http://khanya.wordpress.com
E-mail - see web page, or parse: shayes at dunelm full stop org full stop uk

[toc] | [next] | [standalone]

#80450

On 10/11/14 14:55, Steve Hayes wrote:
> I have a book on Python that advocates dividing programs into modules, and
> importing them when needed.
>
> I have a question about this.
>
> I can understand doing that in a compiled language, where different modules
> can be imported from all sorts of places when the program is compiled.
>
> But I understand that Python is an interpreted language, and If I wrote a
> program in Python like that, and wanted to run it on another computer, how
> would it find all the modules to import at run-time, unless I copied the whole
> directory structure over to the other computer?

You copy over the directory structure, or wrap it in a compressed archive.

[toc] | [prev] | [next] | [standalone]

#80451

Steve Hayes <hayesstw@telkomsa.net> writes:

> I have a book on Python that advocates dividing programs into modules,
> and importing them when needed.

Which book is this? (This is not essential to your question, but it
might help to gauge your broader learning environment.)

> I can understand doing that in a compiled language, where different
> modules can be imported from all sorts of places when the program is
> compiled.

Python is a compiled language; when you run a Python program a necessary
step is to compile the Python source to a bytecode for actual execution.

Usually, the compilation step is done dynamically; but (barring
contrived examples) it is always done prior to running the program.

> But I understand that Python is an interpreted language

The two are not mutually exclusive. The Python interpreter works with
compiled Python code, it does not execute the source directly.

> If I wrote a program in Python like that, and wanted to run it on
> another computer, how would it find all the modules to import at
> run-time, unless I copied the whole directory structure over to the
> other computer?

That's the idea, yes. You need to distinguish between:

* The standard library: installed along with the Python interpreter when
  you install that, and available on the default module search path.

* Third-party modules: installed using your package manager (ideally by
  the operating system package manager), again to a location already
  part of the default module search path on your system.

* Modules specific to the application you're writing: Keep these in a
  known hierarchy, and use the distribution tools to package them to
  keep them together when distributing to oher machines.

Use abolute import for standard library and third-party modules. Use
relative import for application-private modules.

This ensures your aplication's private modules don't conflict with
current or future names of modules from the standard library or third
parties.

You are working through the Python tutorial step by step, right?
<URL:https://docs.python.org/3/tutorial/> This topic is covered when you
learn about modules <URL:https://docs.python.org/3/tutorial/modules.html>.

-- 
 \       “I am amazed, O Wall, that you have not collapsed and fallen, |
  `\            since you must bear the tedious stupidities of so many |
_o__)                  scrawlers.” —anonymous graffiti, Pompeii, 79 CE |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#80452

On Mon, 10 Nov 2014 16:12:07 +1100, Ben Finney <ben+python@benfinney.id.au>
wrote:

>Steve Hayes <hayesstw@telkomsa.net> writes:
>
>> I have a book on Python that advocates dividing programs into modules,
>> and importing them when needed.
>
>Which book is this? (This is not essential to your question, but it
>might help to gauge your broader learning environment.)

Cunningham, Katie. 2014. Teach yourself Python in 24 hours.
               Indianapolis: Sams.
               ISBN: 978-0-672-33687-4
                   For Python 2.7.5

>> I can understand doing that in a compiled language, where different
>> modules can be imported from all sorts of places when the program is
>> compiled.
>
>Python is a compiled language; when you run a Python program a necessary
>step is to compile the Python source to a bytecode for actual execution.
>
>Usually, the compilation step is done dynamically; but (barring
>contrived examples) it is always done prior to running the program.
>
>> But I understand that Python is an interpreted language
>
>The two are not mutually exclusive. The Python interpreter works with
>compiled Python code, it does not execute the source directly.
>
>> If I wrote a program in Python like that, and wanted to run it on
>> another computer, how would it find all the modules to import at
>> run-time, unless I copied the whole directory structure over to the
>> other computer?
>
>That's the idea, yes. You need to distinguish between:
>
>* The standard library: installed along with the Python interpreter when
>  you install that, and available on the default module search path.
>
>* Third-party modules: installed using your package manager (ideally by
>  the operating system package manager), again to a location already
>  part of the default module search path on your system.
>
>* Modules specific to the application you're writing: Keep these in a
>  known hierarchy, and use the distribution tools to package them to
>  keep them together when distributing to oher machines.
>
>Use abolute import for standard library and third-party modules. Use
>relative import for application-private modules.
>
>This ensures your aplication's private modules don't conflict with
>current or future names of modules from the standard library or third
>parties.

So if I want to run it on another computer, where do I look for the compiled
executable program to copy?


-- 
Steve Hayes from Tshwane, South Africa
Web:  http://www.khanya.org.za/stevesig.htm
Blog: http://khanya.wordpress.com
E-mail - see web page, or parse: shayes at dunelm full stop org full stop uk

[toc] | [prev] | [next] | [standalone]

#80453

Steve Hayes <hayesstw@telkomsa.net> writes:

> So if I want to run it on another computer, where do I look for the
> compiled executable program to copy?

You generally don't do that (the compiled files tend to be specific to
various aspects of the target platform). This is a way that i's
important to remember that most Python interpreters require compiled
Python code.

Instead, to have a Python program run on a different computer, the files
to copy are the *source* files — preferably, in the hierarchy those
files expect.

You can do this with ‘rsync’ <URL:https://rsync.samba.org/> for simple
cases.

For more complex cases, look into Python's distribution and packaging
tools <URL:https://python-packaging-user-guide.readthedocs.org/>.

For even more complex cases (e.g. where the Python code is only part of
a more diverse code base), look into deployment systems and operating
system packages.

-- 
 \         “When we pray to God we must be seeking nothing — nothing.” |
  `\                                                —Francis of Assisi |
_o__)                                                                  |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#80462

In article <i0h06a9pj8h3olo5rnrgc64i7ckpvtvti6@4ax.com>,
 Steve Hayes <hayesstw@telkomsa.net> wrote:

> I have a book on Python that advocates dividing programs into modules, and
> importing them when needed.

Yes, this is a good idea.  Breaking your program down into modules, each 
of which does a small set of closely related things, makes it easier to 
manage.  You can test each module in isolation.  When you look at your 
version control log, you can see just the changes which apply to just 
that module.  When somebody new joins the team, they can learn about 
each module one at a time.

None of this is specific to Python.  Good software engineering practice 
is to break large applications into manageable pieces.  The vocabulary 
may change from language to language (module, package, class, library, 
dll, etc), but the basic concept is the same.

> But I understand that Python is an interpreted language, and If I wrote a
> program in Python like that, and wanted to run it on another computer, how
> would it find all the modules to import at run-time, unless I copied the whole
> directory structure over to the other computer?

Yes, exactly.  When you deploy your application someplace, you need to 
include all the things it depends on.  In the simple case of a few 
python files (say, a main program and a few modules that you're 
written), the easiest thing to do might be to just clone your source 
repository on the other machine and run it directly from that.  Another 
possibility would be to package up all the files in some sort of archive 
(tar, zip, whatever) and unpack that wherever you need it.

You will also have to set up a correct environment.  This usually means 
having an appropriate version of Python already installed, plus whatever 
third-party modules you use.

[toc] | [prev] | [next] | [standalone]

#80463

On Tue, Nov 11, 2014 at 12:36 AM, Roy Smith <roy@panix.com> wrote:
> Yes, exactly.  When you deploy your application someplace, you need to
> include all the things it depends on.  In the simple case of a few
> python files (say, a main program and a few modules that you're
> written), the easiest thing to do might be to just clone your source
> repository on the other machine and run it directly from that.

Even in less simple cases, that's often a good way to run things. As
long as your source repo has no large binary files in it, it'll be
reasonably small; for 400KB of source code and ~1600 commits spanning
~3 years of history, around about 2MB. When your deployment is a
source clone, it's really easy to pull changes and see what's new; and
if you ever find a bug on a deployment machine (maybe a different OS
from your usual dev system), you can make a patch right there and send
it along. There's no huge "okay, let's make a new release now"
overhead - you just keep committing (maybe pushing) changes, same as
you do any other time, and perhaps tag some commit with a version
number. Very very easy. I recommend it.

ChrisA

[toc] | [prev] | [next] | [standalone]

#80557

On Sunday, November 9, 2014 11:51:41 PM UTC-5, Steve Hayes wrote:
> I have a book on Python that advocates dividing programs into modules, and
> importing them when needed. 
> 
> I have a question about this.
> 
> I can understand doing that in a compiled language, where different modules
> can be imported from all sorts of places when the program is compiled. 
> 
> But I understand that Python is an interpreted language, and If I wrote a
> program in Python like that, and wanted to run it on another computer, how
> would it find all the modules to import at run-time, unless I copied the whole
> directory structure over to the other computer?
> 
> 
> 
> 
> -- 
> Steve Hayes from Tshwane, South Africa
> Web:  http://www.khanya.org.za/stevesig.htm
> Blog: http://khanya.wordpress.com
> E-mail - see web page, or parse: shayes at dunelm full stop org full stop uk

I have had a lot of trouble with executing relative imports with some of my projects in python.  

Are there any best practices or methods besides '../../' type hard-coding?

In one project, a spider using scrapy, I used this method to import a module from 'the other side' of a file directory. 

----
spider_lib_path = os.path.realpath(os.path.dirname(__file__) + '/../../../../../library/python')
object_builder_printer = imp.load_source('object_builder_printer', spider_lib_path + '/object_builder_printer.py')
object_builder = object_builder_printer.object_builder_printer(settings)
----

the file structure was:

spiders/
-current/
--app/
---spiders/
----scrapy/
-----Charlotte/
------1.0/
-------pipelines.py (the file that needs to import object_builder_printer.py)

--library/
---python/
----object_builder_printer.py


I think the issue had something to do with there being duplicitous file names, as in, there are multiple directories named 'spiders'.

[toc] | [prev] | [next] | [standalone]

#80571

Will Acheson <willyach07@gmail.com> writes:


> I have had a lot of trouble with executing relative imports with some
> of my projects in python.
>
> Are there any best practices or methods besides '../../' type
> hard-coding?

The important point to learn with Python's import system, as contrasted
with various other languages, is that it is *intentionally* abstracted
from the filesystem hierarchy.

That allows all kinds of helpful infrastructure, but it also comes with
the cost of learning how to let Python know where modules are to be
imported from; it's not a simple case of stating filesystem locations.

The point which tripped me up for a long time, and which I have to keep
reminding myself of:


Python looks for modules in its import search path. Corollary: structure
your project into a hierarchy of packages, including the top level, and
specify all relative imports starting at that top-level package.


Often the cause of my intra-project import woes comes from having
several *discrete* packages, but trying to do relative imports between
them::

    hurgen/            # Note: ‘hurgen’ is not a Python package.
        foo/
            __init__.py
            spam.py
            beans.py
        bar/
            __init__.py
            eggs.py
            ham.py

This is a poor structure, because Python needs to be told about each of
‘foo’ and ‘bar’ separately if relative imports are to work. Worse, a
module invoked as the top-level module (e.g. ‘eggs.py’) can't do imports
relative to itself, because it isn't even aware it's in a package!

Better is to structure the project explicitly under a top-level named
Python package::

    hurgen/
        __init__.py
        foo/
            __init__.py
            spam.py
            beans.py
        bar/
            __init__.py
            eggs.py
            ham.py

and import all the project's modules relative to the ‘hurgen’ package.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web