Re: Pickle based workflow - looking for advice

Path	csiph.com!usenet.pasdenom.info!news.redatomik.org!newsfeed.xs4all.nl!newsfeed3a.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path	<python-python-list@m.gmane.org>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.002
X-Spam-Evidence	'H': 1.00; 'S': 0.00; 'resulting': 0.04; 'output': 0.05; 'makefile': 0.09; 'received:80.91': 0.09; 'received:80.91.229': 0.09; 'received:gmane.org': 0.09; 'received:list': 0.09; 'scripts,': 0.09; 'csv': 0.16; 'determines': 0.16; 'distinct': 0.16; 'forth.': 0.16; 'invoking': 0.16; 'json,': 0.16; 'received:80.91.229.3': 0.16; 'received:dip0.t-ipconnect.de': 0.16; 'received:plane.gmane.org': 0.16; 'received:t-ipconnect.de': 0.16; 'subject:based': 0.16; 'size,': 0.16; 'wrote:': 0.18; 'all,': 0.19; 'input': 0.22; 'header:User-Agent:1': 0.23; '(such': 0.24; 'software.': 0.24; 'script': 0.25; 'define': 0.26; 'first,': 0.26; 'header:X -Complaints-To:1': 0.27; 'testing': 0.29; 'respective': 0.29; 'comments': 0.31; 'piece': 0.31; 'know.': 0.32; 'run': 0.32; 'quite': 0.32; 'checking': 0.33; 'plain': 0.33; 'tool': 0.35; 'computing': 0.35; 'operations': 0.35; 'but': 0.35; 'there': 0.35; 'detail': 0.37; 'step': 0.37; 'mapping': 0.38; 'writes': 0.38; 'to:addr:python-list': 0.38; 'to:addr:python.org': 0.39; 'received:org': 0.40; 'how': 0.40; 'easy': 0.60; 'eventually': 0.60; 'course': 0.61; 'different': 0.65; 'series': 0.66; 'reads': 0.68; 'have?': 0.84; 'iterative': 0.84; 'producers': 0.84; 'subject:looking': 0.91
X-Injected-Via-Gmane	http://gmane.org/
To	python-list@python.org
From	Peter Otten <__peter__@web.de>
Subject	Re: Pickle based workflow - looking for advice
Date	Mon, 13 Apr 2015 19:08:50 +0200
Organization	None
References	<mgglho$tpc$1@speranza.aioe.org>
Mime-Version	1.0
Content-Type	text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding	7Bit
X-Gmane-NNTP-Posting-Host	p57bd8dc8.dip0.t-ipconnect.de
User-Agent	KNode/4.13.3
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.20
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups	comp.lang.python
Message-ID	<mailman.277.1428944943.12925.python-list@python.org> (permalink)
Lines	29
NNTP-Posting-Host	2001:888:2000:d::a6
X-Trace	1428944943 news.xs4all.nl 2865 [2001:888:2000:d::a6]:44621
X-Complaints-To	abuse@xs4all.nl
Xref	csiph.com comp.lang.python:88928

Show key headers only | View raw

Fabien wrote:

> I am writing a quite extensive piece of scientific software. Its
> workflow is quite easy to explain. The tool realizes series of
> operations on watersheds (such as mapping data on it, geostatistics and
> more). There are thousands of independent watersheds of different size,
> and the size determines the computing time spent on each of them.
> 
> Say I have the operations A, B, C and D. B and C are completely
> independent but they need A to be run first, D needs B and C, and so
> forth. Eventually the whole operations A, B, C and D will run once for
> all, but of course the whole development is an iterative process and I
> rerun all operations many times.

> 4. Other comments you might have?

How about a file-based workflow?

Write distinct scripts, e. g.

a2b.py that reads from *.a and writes to *.b

and so on. Then use a plain old makefile to define the dependencies.
Whether .a uses pickle, .b uses json, and .z uses csv is but an 
implementation detail that only its producers and consumers need to know. 
Testing an arbitrary step is as easy as invoking the respective script with 
some prefabricated input and checking the resulting output file(s).

Thread

Pickle based workflow - looking for advice Fabien <fabien.maussion@gmail.com> - 2015-04-13 16:58 +0200
  Re: Pickle based workflow - looking for advice Devin Jeanpierre <jeanpierreda@gmail.com> - 2015-04-13 11:45 -0400
    Re: Pickle based workflow - looking for advice Fabien <fabien.maussion@gmail.com> - 2015-04-13 19:39 +0200
  Re: Pickle based workflow - looking for advice Robin Becker <robin@reportlab.com> - 2015-04-13 16:53 +0100
  Re: Pickle based workflow - looking for advice Dave Angel <davea@davea.name> - 2015-04-13 12:25 -0400
    Re: Pickle based workflow - looking for advice Fabien <fabien.maussion@gmail.com> - 2015-04-13 19:30 +0200
  Re: Pickle based workflow - looking for advice Peter Otten <__peter__@web.de> - 2015-04-13 19:08 +0200
    Re: Pickle based workflow - looking for advice Fabien <fabien.maussion@gmail.com> - 2015-04-13 19:35 +0200
      Re: Pickle based workflow - looking for advice Chris Angelico <rosuav@gmail.com> - 2015-04-14 14:05 +1000
        Re: Pickle based workflow - looking for advice Fabien <fabien.maussion@gmail.com> - 2015-04-14 09:58 +0200
          Re: Pickle based workflow - looking for advice Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-04-14 23:08 +1000
            Re: Pickle based workflow - looking for advice Chris Angelico <rosuav@gmail.com> - 2015-04-14 23:45 +1000
              Re: Pickle based workflow - looking for advice Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-04-15 00:14 +1000
                Re: Pickle based workflow - looking for advice Chris Angelico <rosuav@gmail.com> - 2015-04-15 00:30 +1000

csiph-web