Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #8319 > unrolled thread

Re: performance critical Python features

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: performance critical Python features Chris Angelico <rosuav@gmail.com> - 2011-06-24 04:00 +1000
    Re: performance critical Python features Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-06-24 00:07 +0000
      Re: performance critical Python features Chris Angelico <rosuav@gmail.com> - 2011-06-24 12:08 +1000

#8319 — Re: performance critical Python features

On Fri, Jun 24, 2011 at 2:58 AM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
> So, which are the other pieces of Python that really need the heavy
> optimization and which are those that don't?  Thanks.
>

Things that are executed once (imports, class/func definitions) and
things that primarily wait for user input don't need to be optimized.
Things that get executed millions of times a second MAY need to be
optimized.

ChrisA
(The keyword MAY is to be interpreted as per RFC 2119.)

[toc] | [next] | [standalone]

#8338

On Fri, 24 Jun 2011 04:00:17 +1000, Chris Angelico wrote:

> On Fri, Jun 24, 2011 at 2:58 AM, Eric Snow <ericsnowcurrently@gmail.com>
> wrote:
>> So, which are the other pieces of Python that really need the heavy
>> optimization and which are those that don't?  Thanks.
>>
>>
> Things that are executed once (imports, class/func definitions) and

You can't assume that either of those things are executed once. Consider 
this toy example:

def outer(a, b):
    def inner(x):
        return (x*a - b)*(x*b - a) - 1
    return inner(b**2 - a**2)

results = [outer(a, b) for (a, b) in coordinate_pairs()]

The function definition for inner gets executed repeatedly, inside a 
tight loop.

Fortunately Python does optimize this case. The heavy lifting (parsing 
the source of inner, compiling a code object) is done once, when outer is 
defined, and the only work done at runtime is assembling the pieces into 
a function object, which is fast.

Similarly, imports are so expensive that it makes sense to optimize them. 
A single line like "import module" requires the following work:

- expensive searches of the file system, looking for a module.py file 
  or a module/__init__.py package, possibly over a slow network or 
  inside zip files;
- once found, parse the file;
- compile it;
- execute it, which could be arbitrarily expensive;
- and which may require any number of new imports.

Again, imports are already optimized in Python: firstly, once a module 
has been imported the first time, the module object is cached in 
sys.modules so that subsequent imports of that same module are much 
faster: it becomes little more than a name lookup in a dict. Only if that 
fails does Python fall back on the expensive import from disk.

Secondly, Python tries to cache the compiled code in a .pyc or .pyo file, 
so that parsing and compiling can be skipped next time you import from 
disk (unless the source code changes, naturally).

And even so, importing is still slow. That's the primary reason why 
Python is not suitable for applications where you need to execute lots of 
tiny scripts really fast: each invocation of the interpreter requires a 
whole lot of imports, which are slow the first time.

(Still, Python's overhead at startup time is nowhere near as expensive as 
that of Java... but Java is faster once started up.)


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#8341

On Fri, Jun 24, 2011 at 10:07 AM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> On Fri, 24 Jun 2011 04:00:17 +1000, Chris Angelico wrote:
>
>> On Fri, Jun 24, 2011 at 2:58 AM, Eric Snow <ericsnowcurrently@gmail.com>
>> wrote:
>>> So, which are the other pieces of Python that really need the heavy
>>> optimization and which are those that don't?  Thanks.
>>>
>>>
>> Things that are executed once (imports, class/func definitions) and
>
> You can't assume that either of those things are executed once. Consider
> this toy example:

Sure. I was talking in generalities; of course you can do expensive
operations frequently. If you wanted to, you could do this:

radius=5
circum=0
for i in range(10,1000):
    c=radius*calculate_pi_to_n_decimals(i)
    if c>circum: circum=c

Calculates the highest possible circumference of a circle of that
radius. Does this mean we now have to optimize the pi calculation
algorithm so it can be used in a tight loop? Well, apart from the fact
that this code is moronic, no. All you need to do is cache. (Although
I guess in a way that's an optimization of the algorithm. It's the
same optimization as is done for imports.)

But generally speaking, functions are called more often than they're
defined, especially when we're talking about tight loops. And while
your example could be written without the repeated definition:

def outer(a, b):
   x=b**2 - a**2
   return (x*a - b)*(x*b - a) - 1

results = [outer(a, b) for (a, b) in coordinate_pairs()]

(at least, I think this is the same functionality), if inner() were
recursive, that would be different. But recursive inner functions
aren't nearly as common as write-once-call-many functions.

ChrisA

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web