Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #16158 > unrolled thread

suitability of python

Back to article view | Back to comp.lang.python

Contents

  suitability of python Rudra Banerjee <bnrj.rudra@gmail.com> - 2011-11-24 18:01 +0530
    Re: suitability of python Laurent Claessens <moky.math@gmail.com> - 2011-11-24 13:46 +0100
    Re: suitability of python Dave Angel <d@davea.name> - 2011-11-24 08:08 -0500
    Re: suitability of python Terry Reedy <tjreedy@udel.edu> - 2011-11-24 19:51 -0500
      Re: suitability of python 88888 Dihedral <dihedral88888@googlemail.com> - 2011-11-24 19:27 -0800
      Re: suitability of python 88888 Dihedral <dihedral88888@googlemail.com> - 2011-11-24 19:27 -0800
      Re: suitability of python alex23 <wuwei23@gmail.com> - 2011-11-24 19:37 -0800
    Re: suitability of python Alan Meyer <ameyer2@yahoo.com> - 2011-11-24 23:06 -0500
    Re: suitability of python Grant Edwards <invalid@invalid.invalid> - 2011-11-27 17:19 +0000
    Re: suitability of python Stefan Behnel <stefan_ml@behnel.de> - 2011-11-27 17:54 +0000

#16158 — suitability of python

Dear friends,
I am a newbie in python and basically i use python for postprocessing
like plotting, data manipulation etc.
Based on ease of programming on python I am wondering if I can consider
it for the main development as well. My jobs (written on fortran) runs
for weeks and quite CPU intensive. How python works on these type of
heavy computation? 
Any comment or reference is welcome.

[toc] | [next] | [standalone]

#16159

Le 24/11/2011 13:31, Rudra Banerjee a écrit :
> Dear friends,
> I am a newbie in python and basically i use python for postprocessing
> like plotting, data manipulation etc.
> Based on ease of programming on python I am wondering if I can consider
> it for the main development as well. My jobs (written on fortran) runs
> for weeks and quite CPU intensive. How python works on these type of
> heavy computation?
> Any comment or reference is welcome.


If you need mathematical power (especially symbolic computations), you 
should also consider Sage[1] which is kind of a module of math over 
python. In some situations, Sage is the "correct" successor of Fortran 
instead of plain python.

Well, it does not answers the question, but ...

Laurent

[1] http://sagemath.org

[toc] | [prev] | [next] | [standalone]

#16160

On 11/24/2011 07:31 AM, Rudra Banerjee wrote:
> Dear friends,
> I am a newbie in python and basically i use python for postprocessing
> like plotting, data manipulation etc.
> Based on ease of programming on python I am wondering if I can consider
> it for the main development as well. My jobs (written on fortran) runs
> for weeks and quite CPU intensive. How python works on these type of
> heavy computation?
> Any comment or reference is welcome.
>
If I take your description at face value, then I'd say that stock 
CPython would be slower than Fortran.  If the CPU-intensive parts had to 
be rewritten in CPython, they'd be slower than the Fortran they replace, 
by somewhere between 10:1 and 500:1.  Further, if you've already got 
those Fortran algorithms written and debugged, why rewrite them?  And 
finally, even for new code, you might be getting ideas for your 
algorithms from journals and other resources, where the examples may 
well be done in Fortran, so productivity might be best in Fortran as well.

HOWEVER, you don't have to use stock CPython, alone.  It could be that 
some of your Fortran algorithms are written in shared libraries, and 
that you could get your CPython code to call them to do the "heavy 
lifting."  Or it could be that numpy, sage, or other 3rd party libraries 
might be usable for your particular problems, and that speed is then 
comparable to Fortran.  Or it could be that one of the alternative 
Python implementations might be fast enough.

Or it could even be that you're mistaken that the present code is even 
CPU intensive.

Or it could be that by the time you recode the problem in Python, you 
discover a more efficient algorithm, and that way gain back all the 
speed you theoretically lost.

There are tools to measure things, though I'm not the one to recommend 
specifics.  And those probably depend on your platform as well.

The last Fortran that I wrote was over 40 years ago.  I'm afraid when I 
need speed, I usually use C++.  But if I were starting a personal 
math-intensive project now, I'd try to prototype it in Python, and only 
move portions of it to Fortran or other compiled language.  Only the 
portions that measurably took too long.  And those portions might be 
rewritten in Cython, C++, or Fortran, depending on what kind of work 
they actually did.

Another alternative that might make sense is to use Python as a "macro 
language" to Fortran, where you call out to Python to automate some 
tasks within the main program.  I have no experience with doing that, 
but I assume it'd be something like how MSWord can call out to VBA 
routines.  And it'd make the most sense when the main app is already 
written, and the macro stuff is an afterthought.

I think the real point is that it doesn't have to be "all or nothing."  
I suspect that the pieces you're already doing in Python are calling out 
to pre-existing libraries as well.  So your plotting code does some 
massaging, and then calls into some plotting library, or even execs a 
plotting executable.

-- 

DaveA

[toc] | [prev] | [next] | [standalone]

#16185

On 11/24/2011 7:31 AM, Rudra Banerjee wrote:
> Dear friends,
> I am a newbie in python and basically i use python for postprocessing
> like plotting, data manipulation etc.
> Based on ease of programming on python I am wondering if I can consider
> it for the main development as well. My jobs (written on fortran) runs
> for weeks and quite CPU intensive. How python works on these type of
> heavy computation?

The first killer app for Python was running Fortran code from within 
Python. People use Python for both pre- and post-processing. For small 
jobs, this enabled running Fortran interactively.

This lead to Numerical Python, now Numpy, SciPy, and later Sage and 
other scientific and Python packages. I believe SciPy has an f2py 
(fortran to py) module to help with running Fortran under Python (but it 
has been years since I read the details).

Detailed questions might get better answers on, for instance, a scipy list.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#16189

On Friday, November 25, 2011 8:51:10 AM UTC+8, Terry Reedy wrote:
> On 11/24/2011 7:31 AM, Rudra Banerjee wrote:
> > Dear friends,
> > I am a newbie in python and basically i use python for postprocessing
> > like plotting, data manipulation etc.
> > Based on ease of programming on python I am wondering if I can consider
> > it for the main development as well. My jobs (written on fortran) runs
> > for weeks and quite CPU intensive. How python works on these type of
> > heavy computation?
> 
> The first killer app for Python was running Fortran code from within 
> Python. People use Python for both pre- and post-processing. For small 
> jobs, this enabled running Fortran interactively.
> 
> This lead to Numerical Python, now Numpy, SciPy, and later Sage and 
> other scientific and Python packages. I believe SciPy has an f2py 
> (fortran to py) module to help with running Fortran under Python (but it 
> has been years since I read the details).
> 
> Detailed questions might get better answers on, for instance, a scipy list.
> 
> -- 
> Terry Jan Reedy

If pyhthon just handles the user interface and glue logics of well written python modules that are most written c, the speed of running python pyc  is OK. 
  
Of course the object reference updating required  in OOP is completely supported by python. 

[toc] | [prev] | [next] | [standalone]

#16190

On Friday, November 25, 2011 8:51:10 AM UTC+8, Terry Reedy wrote:
> On 11/24/2011 7:31 AM, Rudra Banerjee wrote:
> > Dear friends,
> > I am a newbie in python and basically i use python for postprocessing
> > like plotting, data manipulation etc.
> > Based on ease of programming on python I am wondering if I can consider
> > it for the main development as well. My jobs (written on fortran) runs
> > for weeks and quite CPU intensive. How python works on these type of
> > heavy computation?
> 
> The first killer app for Python was running Fortran code from within 
> Python. People use Python for both pre- and post-processing. For small 
> jobs, this enabled running Fortran interactively.
> 
> This lead to Numerical Python, now Numpy, SciPy, and later Sage and 
> other scientific and Python packages. I believe SciPy has an f2py 
> (fortran to py) module to help with running Fortran under Python (but it 
> has been years since I read the details).
> 
> Detailed questions might get better answers on, for instance, a scipy list.
> 
> -- 
> Terry Jan Reedy

If pyhthon just handles the user interface and glue logics of well written python modules that are most written c, the speed of running python pyc  is OK. 
  
Of course the object reference updating required  in OOP is completely supported by python. 

[toc] | [prev] | [next] | [standalone]

#16191

Terry Reedy <tjre...@udel.edu> wrote:
> This lead to Numerical Python, now Numpy, SciPy, and later Sage and
> other scientific and Python packages. I believe SciPy has an f2py
> (fortran to py) module to help with running Fortran under Python (but it
> has been years since I read the details).

Andrew Dalke recently did some work on f2pypy, as a step toward
running Fortran under PyPy:

http://www.dalkescientific.com/writings/diary/archive/2011/11/09/f2pypy.html

If PyPy's Numpy support was more advanced, I'd probably recommend the
OP start there.

[toc] | [prev] | [next] | [standalone]

#16195

On 11/24/2011 07:31 AM, Rudra Banerjee wrote:
> Dear friends,
> I am a newbie in python and basically i use python for postprocessing
> like plotting, data manipulation etc.
> Based on ease of programming on python I am wondering if I can consider
> it for the main development as well. My jobs (written on fortran) runs
> for weeks and quite CPU intensive. How python works on these type of
> heavy computation?
> Any comment or reference is welcome.
>

I would expect that a language that compiles intensive math programming 
to machine language will be much more than an order of magnitude faster 
than a program that does the same thing by interpreting byte code.

If you study all of the Python math libraries I'm guessing you'll find 
modules that do a lot, conceivably all, of what you want in compiled 
machine language, but when held together with Python it may or may not 
be as efficient as fortran.  I'm guessing there's not much out there 
that is as efficient as fortran for purely numerical work.

I think your division of labor using fortran for the CPU intensive math 
parts and python for post-processing is a pretty good one.  It takes 
advantage of the strength of each language.  In addition, it completely 
separates the two parts so that they aren't really dependent on each 
other.  You can change the fortran any way you want without breaking the 
python code as long as you output the same format, and of course you can 
change the python any way you want. Programs in each language don't even 
have to know that any other language is involved.

My only suggestion is to see if you can get a profiler to see what's 
happening inside that weeks long running fortran program.  You might 
find some surprises.  I once wrote a 5,000 line program that was slower 
than I had hoped.  I ran it through a profiler and it showed me that I 
was spending more than 50 percent of my time on one single line of my 
code that called a simple library routine ("strcpy").  I wrote the 
simple library routine inline instead adding just a few lines of code. 
It cut the total execution time of the whole program in half.

     Alan

[toc] | [prev] | [next] | [standalone]

#16293

On 2011-11-24, Rudra Banerjee <bnrj.rudra@gmail.com> wrote:

> I am a newbie in python and basically i use python for postprocessing
> like plotting, data manipulation etc.
> Based on ease of programming on python I am wondering if I can consider
> it for the main development as well. My jobs (written on fortran) runs
> for weeks and quite CPU intensive. How python works on these type of
> heavy computation?

You'll have to tell us what "these type of heavy computation" are
before we can answer.  There are a _lot_ of heavy-duty computational
libraries (many of them written in FORTAN) that have been interfaced
to Python (BLAS and so on).  If the heavy lifting can be done by those
libraries, Python might be very suitable.

You might want to check out scipy, Scientific Python, and the
Enthought python distro.

http://www.scipy.org/
http://dirac.cnrs-orleans.fr/plone/software/scientificpython/overview/
http://www.enthought.com/products/epd.php

[toc] | [prev] | [next] | [standalone]

#16294

Rudra Banerjee, 24.11.2011 12:31:
> I am a newbie in python and basically i use python for postprocessing
> like plotting, data manipulation etc.
> Based on ease of programming on python I am wondering if I can consider
> it for the main development as well. My jobs (written on fortran) runs
> for weeks and quite CPU intensive. How python works on these type of
> heavy computation?

You already got a lot of answers that pointed you to the scientific 
computing tools that are available for Python. The reason why they exist is 
because (and nowadays also "why") Python is so extremely popular in that 
field: it's an easy to learn and use language and the standard 
implementation (often referred to as CPython) makes it really easy to 
interface with external code (C/C++/Fortran/etc.) in a very efficient way.

In addition to looking at NumPy/SciPy and/or Sage (depending on the kind of 
computations you are involved with), you should also look at fwrap and 
Cython. They will allow you to easily wrap your existing Fortran code for 
Python, and to quickly write very fast glue code for the two language 
environments. Thus, you can keep your existing code as it is, and use and 
control it from Python, using all the nice tools that Python provides for 
quickly writing anything from distributed code and test suites to graphical 
user interfaces for visualising your data. Since you specifically asked 
about plotting, don't miss out on matplotlib.

Stefan

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web