Path: csiph.com!usenet.pasdenom.info!aioe.org!news.stack.nl!newsfeed.xs4all.nl!newsfeed3.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.020 X-Spam-Evidence: '*H*': 0.96; '*S*': 0.00; 'python.': 0.02; 'python,': 0.02; 'scripts': 0.03; 'subject:Python': 0.06; 'perl,': 0.07; 'advance': 0.07; 'assuming': 0.09; 'in-house': 0.09; 'solution,': 0.09; 'cc:addr:python-list': 0.11; 'python': 0.11; 'archive': 0.14; 'thread': 0.14; 'above?': 0.16; 'perl.': 0.16; 'portable': 0.16; 'python-based': 0.16; 'rewriting': 0.16; 'scalable,': 0.16; 'throw': 0.16; 'units,': 0.16; 'wrote:': 0.18; 'wed,': 0.18; 'numerical': 0.19; 'email addr:gmail.com>': 0.22; 'cc:addr:python.org': 0.22; 'cc:2**0': 0.24; 'options': 0.25; '>': 0.26; 'solutions.': 0.26; 'least': 0.26; 'asking': 0.27; 'header:In-Reply-To:1': 0.27; 'idea': 0.28; 'am,': 0.29; 'message- id:@mail.gmail.com': 0.30; 'url:mailman': 0.30; 'easier': 0.31; "skip:' 10": 0.31; 'accomplished': 0.31; 'ahead.': 0.31; 'initiate': 0.31; 'loads': 0.31; 'perl': 0.31; 'lists': 0.32; 'url:python': 0.33; 'community': 0.33; 'actual': 0.34; "can't": 0.35; 'but': 0.35; 'received:google.com': 0.35; 'there': 0.35; 'really': 0.36; 'ibm': 0.36; 'url:listinfo': 0.36; 'thanks': 0.36; 'url:org': 0.36; 'employee': 0.37; 'two': 0.37; 'system,': 0.38; 'pm,': 0.38; 'that,': 0.38; 'url:mail': 0.40; 'read': 0.60; 'new': 0.61; 'high': 0.63; 'developed': 0.63; 'such': 0.63; 'today': 0.64; 'our': 0.64; 'more': 0.64; 'situation': 0.65; 'to:addr:gmail.com': 0.65; 'jobs': 0.68; 'mar': 0.68; '2-3': 0.68; '3-4': 0.68; 'advantages': 0.68; 'opinions': 0.70; 'analysis': 0.75; '4:29': 0.84; 'julia': 0.84; 'old,': 0.85; 'belt': 0.91; 'joel': 0.91; 'rusi': 0.91; 'have.': 0.93; 'colleagues': 0.97; 'picture': 0.97; 'subject:From': 0.97; '2013': 0.98 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=aYdx8dIX/VNUHLAswWoa2lUgM/PEvTg2MZq///0cED8=; b=bkaeeFRnEUYyqZ95YIaXUObV43mIM6Y3kfMMUWKBznwOePECWsL1Ey2TKyboBj0Tvr j7Gc/z6pB/bxMu/Bm01O5IJ5zAbgTEm9vPRo20U2rXJVoNq7jgN3cbVTZ3JQ/s/sanMd QHnVuOch/WPFFRNgGJo+M1Ozo9eBaEnN8liZ7KyryUaz2lpayFUZdgRx228Dj/JEbYZW vbXOe3uALB9UpL1J99Hg0JZFuo2+xd7sEW/cIAeOMuiTy2c0wCzLgAShp/PNUCljpDo0 39/4O3v199UagiImxW9rDMqHKYiIZ4uUF5CvRjPCaSeTGAnw61p6SYn8W5tGJSCZdJo6 2UfQ== MIME-Version: 1.0 X-Received: by 10.58.12.69 with SMTP id w5mr23069556veb.16.1364387035982; Wed, 27 Mar 2013 05:23:55 -0700 (PDT) In-Reply-To: <8a301e20-9de8-4563-bcd9-478b0a9a20f6@z4g2000pbw.googlegroups.com> References: <8a301e20-9de8-4563-bcd9-478b0a9a20f6@z4g2000pbw.googlegroups.com> Date: Wed, 27 Mar 2013 08:23:55 -0400 Subject: Re: From Perl to Python: restructuring a HPC workflow From: Joel Goldstick To: rusi Content-Type: multipart/alternative; boundary=047d7b2e4ab49db61204d8e71cec Cc: "python-list@python.org" X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 175 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1364387044 news.xs4all.nl 6924 [2001:888:2000:d::a6]:42682 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:42009 --047d7b2e4ab49db61204d8e71cec Content-Type: text/plain; charset=UTF-8 On Wed, Mar 27, 2013 at 7:51 AM, rusi wrote: > On Mar 27, 4:29 pm, neurino wrote: > > In the need for restructuring our daily workflow, i think it might be a > > good idea to ask the Python community and hopefully initiate a thread > > about pros and cons. > > > > We are a small group of people (approx. 10), working separetely on > > their own projects (each employee manages approx. 2-3 projects). We > > deal with high loads of data everyday. > > > > While the processing is accomplished with fortran and C programs mainly > > on three systems (one cluster, two standalone IBM HPCs, 8852 and p770, > > all managed by a grid-Engine), networking, pre/postprocessing, jobs > > queue administration and numerical analysis have been accomplished with > > Perl. > > > > This workflow has been flawless now for at least 15 years. New > > generations of employees have been given Perl scripts and they > > developed the tools further. > So for 15 years the workflow is flawless, yet you want to throw it all out? I don't understand that > > > > If i think at the actual situation of Perl, i can't see a shiny time > > ahead. Perl 6 is far to be a reliable solution, the CPAN archive is > > slowing down. My idea is to persuade my colleagues to move toward > > Python-based solutions. > So you want to change things but your colleagues don't? > But our concerns are that, in 3-4 years from > > now, the tools we are going to develop must be still scalable, > > mantainable, portable and of high-performance. > > > > > We don't have any solid in-house know-how on Python. We just have to > > start everything from scracth. Where do you see advantages and > > drawbacks in switching from Perl to Python, given the work picture > > above? > > > > Thanks in advance for any opinions you might have. > > Python is a lot easier to read than perl. > Switching is always a con; see > http://www.joelonsoftware.com/articles/fog0000000069.html > Assuming you have that under your belt > - if python is the way to go, asking on the scipy/numpy and ipython > lists may give you more specific answers. > - And if the 'rewrite-bug' has really got you, remember that if perl > is old, C/Fortran are older. > There are options today for rewriting the whole system, such as > haskell and julia http://julialang.org/ > > WARNING: If the Spolsky warning above for perl->python is X units, > take it 2X for Haskell and 4X for Julia! > -- > http://mail.python.org/mailman/listinfo/python-list > -- Joel Goldstick http://joelgoldstick.com --047d7b2e4ab49db61204d8e71cec Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable



On Wed, Mar 27, 2013 at 7:51 AM, rusi <rustompmody@gmail.com> wrote:
On M= ar 27, 4:29=C2=A0pm, neurino <lelli.l...@googlemail.com> wrote:
> In the need for restructuring our daily workflow, i think it might be = a
> good idea to ask the Python community and hopefully initiate a thread<= br> > about pros and cons.
>
> We are a small group of people (approx. 10), working separetely on
> their own projects (each employee manages approx. 2-3 projects). We > deal with high loads of data everyday.
>
> While the processing is accomplished with fortran and C programs mainl= y
> on three systems (one cluster, two standalone IBM HPCs, =C2=A08852 and= p770,
> all managed by a grid-Engine), networking, pre/postprocessing, jobs > queue administration and numerical analysis have been accomplished wit= h
> Perl.
>
> This workflow has been flawless now for at least 15 years. New
> generations of employees have been given Perl scripts and they
> developed the tools further.

So for 15 years the workflow is flawless, yet you want to throw it a= ll out?=C2=A0 I don't understand that
>
> If i think at the actual situation of Perl, i can't see a shiny ti= me
> ahead. Perl 6 is far to be a reliable solution, the CPAN archive is > slowing down. My idea is to persuade my colleagues to move toward
> Python-based solutions.

S= o you want to change things but your colleagues don't?
= =C2=A0
But our concerns are that, in 3-4 y= ears from
> now, the tools we are going to develop must be still scalable,
> mantainable, portable and of high-performance.

=C2=A0
>
> We don't have any solid in-house know-how on Python. We just have = to
> start everything from scracth. Where do you see advantages and
> drawbacks in switching from Perl to Python, given the work picture
> above?
>
> Thanks in advance for any opinions you might have.

Python is a lot easier to read than perl.=
=C2=A0
Switching is always a con; see http://www.joelonso= ftware.com/articles/fog0000000069.html
Assuming you have that under your belt
- if python is the way to go, asking on the scipy/numpy and ipython
lists may give you more specific answers.
- And if the 'rewrite-bug' has really got you, remember that if per= l
is old, C/Fortran are older.
There are options today for rewriting the whole system, such as
haskell and julia http:= //julialang.org/

WARNING: If the Spolsky warning above for perl->python is X units,
take it 2X for Haskell and 4X for Julia!
--
http://mail.python.org/mailman/listinfo/python-list



--
--047d7b2e4ab49db61204d8e71cec--