Path: csiph.com!usenet.pasdenom.info!goblin2!goblin.stu.neva.ru!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.084 X-Spam-Evidence: '*H*': 0.83; '*S*': 0.00; 'python.': 0.02; 'subject:Python': 0.06; 'scaling': 0.07; 'subject:using': 0.09; 'python': 0.11; 'advance!': 0.16; 'c++.': 0.16; 'code?': 0.16; 'datasets': 0.16; 'language)': 0.16; 'libraries.': 0.16; 'models,': 0.16; 'operational': 0.16; 'path.': 0.16; 'flexibility': 0.16; 'appropriate': 0.16; 'language': 0.16; 'code.': 0.18; 'library': 0.18; 'wed,': 0.18; 'trying': 0.19; 'projects,': 0.19; 'written': 0.21; 'comfortable': 0.22; 'python?': 0.22; 'to:name:python-list@python.org': 0.22; 'driven': 0.24; 'prototype': 0.24; 'received:65.55.116': 0.24; 'looks': 0.24; '(or': 0.24; "i've": 0.25; 'source': 0.25; 'header:In-Reply- To:1': 0.27; 'skip:- 40': 0.29; '[1]': 0.29; 'compared': 0.30; "i'm": 0.30; 'url:mailman': 0.30; 'code': 0.31; 'libraries': 0.31; 'anyone': 0.31; 'probably': 0.32; 'stuff': 0.32; 'interface': 0.32; 'url:python': 0.33; 'raw': 0.33; 'date:': 0.34; "i'd": 0.34; 'something': 0.35; 'computing': 0.35; 'convert': 0.35; 'but': 0.35; 'building': 0.35; 'there': 0.35; 'c++': 0.36; 'opposed': 0.36; 'subject:data': 0.36; 'url:listinfo': 0.36; 'thanks': 0.36; 'subject:?': 0.36; 'similar': 0.36; 'url:org': 0.36; 'application': 0.37; 'operating': 0.37; 'email addr:python.org': 0.37; 'project': 0.37; 'level': 0.37; 'handle': 0.38; 'to:addr :python-list': 0.38; 'little': 0.38; 'does': 0.39; 'subject:': 0.39; 'heard': 0.39; 'to:addr:python.org': 0.39; 'url:mail': 0.40; 'how': 0.40; 'read': 0.60; 'making': 0.63; 'email addr:gmail.com': 0.63; 'high': 0.63; 'kind': 0.63; 'such': 0.63; 'skip:n 10': 0.64; 'email name:python-list': 0.65; 'occur': 0.65; 'worth': 0.66; 'here': 0.66; 'it!': 0.67; 'incorporate': 0.68; 'wish': 0.70; 'serial': 0.72; 'application?': 0.84; 'assimilation': 0.84; 'calls,': 0.84; 'gps': 0.84; 'horrible': 0.84; 'investigated': 0.84; 'received:65.55.116.40': 0.84; 'received:blu0-omc1-s29.blu0.hotmail.com': 0.84; 'relate': 0.84; 'subject:system': 0.84; 'careful': 0.91; 'trend': 0.91; 'hand,': 0.93; '2013': 0.98 X-TMN: [FVBa72KaelxAsqRHiJELkiT8XziL60D3] X-Originating-Email: [carlosnepomuceno@outlook.com] From: Carlos Nepomuceno To: "python-list@python.org" Subject: RE: Building a HPC data assimilation system using Python? Date: Fri, 31 May 2013 01:34:26 +0300 Importance: Normal In-Reply-To: <676e9ba9-c54a-404a-8602-fa078cf19c32@googlegroups.com> References: <676e9ba9-c54a-404a-8602-fa078cf19c32@googlegroups.com> Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginalArrivalTime: 30 May 2013 22:34:27.0106 (UTC) FILETIME=[D7B10820:01CE5D85] X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 60 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1369953275 news.xs4all.nl 15902 [2001:888:2000:d::a6]:58112 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:46550 Hi Matthew! I'm on a similar quest!=0A= =0A= I'm still learning the basics of Python so I may not be a good source of in= formation.=0A= =0A= I'm reading a lot of stuff about how to use Python for the parallelization = of code and data and found BSP[1] to be very interesting and perhaps worth = the time to learn it! =3B)=0A= =0A= =0A= [1] http://www.multicorebsp.com/=0A= =0A= =0A= =0A= ----------------------------------------=0A= > Date: Wed=2C 29 May 2013 21:27:54 -0700=0A= > Subject: Building a HPC data assimilation system using Python?=0A= > From: mattjamesfrancis@gmail.com=0A= > To: python-list@python.org=0A= >=0A= > I have a prototype data assimilation code ( an ionospheric nowcast/foreca= st model driven by GPS data ) that is written in IDL (interactive data lang= uage) which is a horrible language choice for scaling the application up to= large datasets as IDL is serial and slow (interpreted).=0A= >=0A= > I am embarking on a project to convert this prototype into an operational= parallel HPC code. In the past I've used C++ for this kind of project and = am comfortable using MPI. On the other hand=2C I've recently started using = python and appreciate the flexibility and speed of development using python= compared with C++. I have read that there is a trend to use python as the = high level 'glue' for these kind of large number crunching projects=2C so i= t would seem appropriate to go down that path. There are a number of C++ an= d FORTRAN(!) libraries I'd need to incorporate that handle things such as t= he processing of raw GPS data and computing ionospheric models=2C so I'd ne= ed to be able to make the appropriate interface for these into python.=0A= >=0A= > If anyone uses python is this way=2C I'd appreciate any tips=2C hints=2C = things to be careful about and in general any war stories you can relate th= at you wish you'd heard before making some mistake.=0A= >=0A= > Here are the things I have investigated that it looks like I'd probably n= eed to use:=0A= >=0A= > * scipy/numpy/matplotlib=0A= > * Cython (or pyrex?) for speeding up any bottlenecks that occur in python= code (as opposed to C++/FORTRAN libraries)=0A= > * MPI for Python (mpi4py). Does this play nice with Cython?=0A= > * Something to interface python with other language libraries. ctypes=2C = swig=2C boost? Which would be best for this application?=0A= > * Profiling. profile/cprofile are straightforward to use=2C but how do th= ey cope with a parallel (mpi4py) code?=0A= > * If a C++ library call has its own MPI calls=2C does that work smoothly = with mpi4py operating in the python part of the code?=0A= >=0A= > Sorry if some of this is a little basic=2C I'm trying to get up to speed = on this a quick as I can.=0A= >=0A= > Thanks in advance!=0A= > --=0A= > http://mail.python.org/mailman/listinfo/python-list =