Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #16094 > unrolled thread

Writing code to be optimizable

Back to article view | Back to comp.lang.python

Contents

  Writing code to be optimizable snorble <snorble@hotmail.com> - 2011-11-22 21:19 -0800
    Re: Writing code to be optimizable Chris Angelico <rosuav@gmail.com> - 2011-11-23 17:45 +1100
    Re: Writing code to be optimizable Stefan Behnel <stefan_ml@behnel.de> - 2011-11-23 08:37 +0100
    Re: Writing code to be optimizable Roy Smith <roy@panix.com> - 2011-11-23 08:31 -0500

#16094 — Writing code to be optimizable

Sometimes I want to prototype a program in Python, with the idea of
optimizing it later by rewriting parts of it in C or Cython. But I
usually find that in order to rewrite the slow parts, I end up writing
those parts very much like C or C++ anyway, and I end up wondering
what is the point of using Python in such a project.

I liked the idea of Cython, mainly the idea of being able to write in
pure Python, then create a file that modifies the pure Python file
(with type declarations and such), but I ran into the same problems.
In order to be able to modify the code, I can't make a lot of use of
the really helpful things in Python like dicts and such. I have to
dumb down Python to where I'm basically writing C code in Python, so
again I ask myself what is the point?

Is it reasonable to prototype an application in Python that will
require performance? Are there any recommendations on how to write
code in such a way that it can later be optimized or replaced with a
module written in C or Cython? Or any good examples of this being done
that I could learn from?

[toc] | [next] | [standalone]

#16096

On Wed, Nov 23, 2011 at 4:19 PM, snorble <snorble@hotmail.com> wrote:
> Sometimes I want to prototype a program in Python, with the idea of
> optimizing it later by rewriting parts of it in C or Cython. But I
> usually find that in order to rewrite the slow parts, I end up writing
> those parts very much like C or C++ anyway, and I end up wondering
> what is the point of using Python in such a project.

There's a few ways to prototype. If you use Python as the language of
your specs documents (or pseudo-Python, perhaps) - consider the entire
Python implementation as ephemeral, and use it to verify that your
specifications are correct, but not for any sort of performance - then
it doesn't matter that you've rewritten it completely, nor that you've
written C code in Python. You would have been writing C code in
English otherwise, anyway.

If you're not sure which parts you're prototyping (ie will rewrite in
C/C++) and which parts you're writing properly (ie will keep the
Python version as the production code), I'd still recommend using all
of Python's best features... but keep your function docstrings
comprehensive. When you rewrite it in C, you're able to take advantage
of what you learned first time around, but otherwise it's just
strictly reimplementing the docstring in a different environment.

Chris Angelico

[toc] | [prev] | [next] | [standalone]

#16098

snorble, 23.11.2011 06:19:
> Sometimes I want to prototype a program in Python, with the idea of
> optimizing it later by rewriting parts of it in C or Cython. But I
> usually find that in order to rewrite the slow parts, I end up writing
> those parts very much like C or C++ anyway, and I end up wondering
> what is the point of using Python in such a project.
>
> I liked the idea of Cython, mainly the idea of being able to write in
> pure Python, then create a file that modifies the pure Python file
> (with type declarations and such), but I ran into the same problems.
> In order to be able to modify the code, I can't make a lot of use of
> the really helpful things in Python like dicts and such. I have to
> dumb down Python to where I'm basically writing C code in Python, so
> again I ask myself what is the point?
>
> Is it reasonable to prototype an application in Python that will
> require performance? Are there any recommendations on how to write
> code in such a way that it can later be optimized or replaced with a
> module written in C or Cython?

I think the usual approach is to write it in Python and make sure it works 
by writing "enough" tests, then benchmark it, then decide if a non-Python 
level optimisation is required.

If it's required, Cython compile the entire module and add static types to 
your hot spots, either in Python notation ("pure mode") or in an external 
.pxd file. If that doesn't yield enough performance, copy code into a 
separate Cython module and optimise it there using C paradigms instead of 
Python paradigms, i.e. apply algorithmic optimisations by dropping data 
structures into C. Then use an import to use the code from your so-called 
accelerator module in your original module. Try to provide both a pure 
Python implementation and a Cython implementation, as that will allow you 
to run your code in other Python implementations directly and to compare 
your two implementations for equivalence and performance.

The advantage of stepping down into C-ish Cython code incrementally is that 
you will have a working implementation at each step and can decide to stop 
optimising because it is "good enough". If you started 'optimising' your 
code while still writing it, it will take longer to write it and you can 
never be sure that the complexity you add to the long-term maintenance by 
writing C-ish code is truly worth the performance gain (or if there even is 
any gain).

Stefan

[toc] | [prev] | [next] | [standalone]

#16107

In article 
<63e78437-c76b-4a9e-9a62-bfea8d078208@v5g2000yqn.googlegroups.com>,
 snorble <snorble@hotmail.com> wrote:

> Is it reasonable to prototype an application in Python that will
> require performance?

Yes.  Several observations:

1) The classic 80/20 rule.  80% of the time is spent running 20% of the 
code.  Sometimes quoted as 90/10.  Why waste time optimizing the 80%?

2) Surprisingly to many people, it's difficult to predict in advance 
which 20% will be the slow parts.  Don't sweat it until the whole thing 
is up and running and you can begin to gather profiling data.

3) Often enough, when you're done writing your program, you'll discover 
that it's fast enough to get value from.  Be happy and move onto the 
next task on your list.

> Are there any recommendations on how to write code in such a way that 
> it can later be optimized or replaced with a module written in C or 
> Cython?

Sure.  Make your code modular and loosely coupled.  Each module or class 
should do one thing, and have narrow, well-defined interfaces.  Limit 
the number of other modules or classes it interacts with directly.  If 
you've done that, it becomes easier to a) identify the slow parts and b) 
plug something better in its place.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web