Re: Pickle based workflow - looking for advice

Path	csiph.com!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.xs4all.nl!newsfeed1.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path	<davea@davea.name>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.031
X-Spam-Evidence	'H': 0.94; 'S': 0.00; 'method.': 0.07; 'rest,': 0.07; 'suddenly': 0.07; 'name?': 0.09; 'stored': 0.12; 'a()': 0.16; 'determines': 0.16; 'dynamic,': 0.16; 'empty,': 0.16; 'fetch': 0.16; 'folks,': 0.16; 'forth.': 0.16; 'preserve': 0.16; 'separated': 0.16; 'subject:based': 0.16; 'size,': 0.16; 'wrote:': 0.18; 'all,': 0.19; 'bit': 0.19; 'seems': 0.21; 'creating': 0.23; 'header:User-Agent:1': 0.23; '(such': 0.24; 'directory.': 0.24; 'tells': 0.24; 'software.': 0.24; 'first,': 0.26; 'mention': 0.26; 'second': 0.26; 'header:In-Reply-To:1': 0.27; 'function': 0.29; 'external': 0.29; 'am,': 0.29; 'database,': 0.30; 'apparently': 0.31; 'commonly': 0.31; 'pickle': 0.31; 'piece': 0.31; 'question:': 0.31; 'them?': 0.31; 'file': 0.32; 'run': 0.32; 'another': 0.32; 'quite': 0.32; 'totally': 0.33; 'comment': 0.34; "can't": 0.35; 'except': 0.35; 'problem.': 0.35; 'tool': 0.35; 'computing': 0.35; 'operations': 0.35; 'but': 0.35; 'there': 0.35; 'choosing': 0.36; 'data,': 0.36; 'method': 0.36; 'possible': 0.36; 'clear': 0.37; 'mapping': 0.38; 'question,': 0.38; 'to:addr :python-list': 0.38; 'to:addr:python.org': 0.39; 'enough': 0.39; 'either': 0.39; 'easy': 0.60; 'eventually': 0.60; 'most': 0.60; 'course': 0.61; 'first': 0.61; "you've": 0.63; 'name': 0.63; 'more': 0.64; 'different': 0.65; 'charset:windows-1252': 0.65; 'series': 0.66; 'between': 0.67; 'received:74.208': 0.68; 'comment.': 0.84; 'id,': 0.84; 'iterative': 0.84; 'persistent': 0.84; 'received:74.208.4.194': 0.84; 'subject:looking': 0.91
Date	Mon, 13 Apr 2015 12:25:38 -0400
From	Dave Angel <davea@davea.name>
User-Agent	Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0
MIME-Version	1.0
To	python-list@python.org
Subject	Re: Pickle based workflow - looking for advice
References	<mgglho$tpc$1@speranza.aioe.org>
In-Reply-To	<mgglho$tpc$1@speranza.aioe.org>
Content-Type	text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding	7bit
X-Provags-ID	V03:K0:A2m6ao5DumgkyayHSsUpP5TMtMd0AiCpEWIUvMOmCVrFyu21Q8d /DU6+04snNYSXOdhGzm0TXmCjw6zqWFmnANCZ5aTvLWx9z54OfgJvzxsybOjcS2STuaoq+3 nswOhdn1lZNhH6OxtbvdRIvG2u/qAfVPGZAlCO5WG7rC8zqmfKnxNlpk4SFFiFSH8n2smex EYRo0SyW/fTfG0rd8PH9g==
X-UI-Out-Filterresults	notjunk:1;
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.20
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups	comp.lang.python
Message-ID	<mailman.276.1428942352.12925.python-list@python.org> (permalink)
Lines	58
NNTP-Posting-Host	2001:888:2000:d::a6
X-Trace	1428942352 news.xs4all.nl 2901 [2001:888:2000:d::a6]:33625
X-Complaints-To	abuse@xs4all.nl
Xref	csiph.com comp.lang.python:88927

Show key headers only | View raw

On 04/13/2015 10:58 AM, Fabien wrote:
> Folks,
>

A comment.  Pickle is a method of creating persistent data, most 
commonly used to preserve data between runs.  A database is another 
method.  Although either one can also be used with multiprocessing, you 
seem to be worrying more about the mechanism, and not enough about the 
problem.

> I am writing a quite extensive piece of scientific software. Its
> workflow is quite easy to explain. The tool realizes series of
> operations on watersheds (such as mapping data on it, geostatistics and
> more). There are thousands of independent watersheds of different size,
> and the size determines the computing time spent on each of them.

First question:  what is the name or "identity" of a watershed? 
Apparently it's named by a directory.  But you mention ID as well.  You 
write a function A() that takes only a directory name. Is that the name 
of the watershed?  One per directory?  And you can derive the ID from 
the directory name?

Second question, is there any communication between watersheds, or are 
they totally independent?

Third:  this "external data", is it dynamic, do you have to fetch it in 
a particular order, is it separated by watershed id, or what?

Fourth:  when the program starts, are the directories all empty, so the 
presence of a pickle file tells you that A() has run?  Or is there some 
other meaning for those files?

>
> Say I have the operations A, B, C and D. B and C are completely
> independent but they need A to be run first, D needs B and C, and so
> forth. Eventually the whole operations A, B, C and D will run once for
> all,

For all what?

> but of course the whole development is an iterative process and I
> rerun all operations many times.

Based on what?  Is the external data changing, and you have to rerun 
functions to update what you've already stored about them?  Or do you 
just mean you call the A() function on every possible watershed?

(I suddenly have to go out, so I can't comment on the rest, except that 
choosing to pickle, or to marshall, or to database, or to 
custom-serialize seems a bit premature.  You may have it all clear in 
your head, but I can't see what the interplay between all these calls to 
one-letter-named functions is intended to be.)

-- 
DaveA

Thread

Pickle based workflow - looking for advice Fabien <fabien.maussion@gmail.com> - 2015-04-13 16:58 +0200
  Re: Pickle based workflow - looking for advice Devin Jeanpierre <jeanpierreda@gmail.com> - 2015-04-13 11:45 -0400
    Re: Pickle based workflow - looking for advice Fabien <fabien.maussion@gmail.com> - 2015-04-13 19:39 +0200
  Re: Pickle based workflow - looking for advice Robin Becker <robin@reportlab.com> - 2015-04-13 16:53 +0100
  Re: Pickle based workflow - looking for advice Dave Angel <davea@davea.name> - 2015-04-13 12:25 -0400
    Re: Pickle based workflow - looking for advice Fabien <fabien.maussion@gmail.com> - 2015-04-13 19:30 +0200
  Re: Pickle based workflow - looking for advice Peter Otten <__peter__@web.de> - 2015-04-13 19:08 +0200
    Re: Pickle based workflow - looking for advice Fabien <fabien.maussion@gmail.com> - 2015-04-13 19:35 +0200
      Re: Pickle based workflow - looking for advice Chris Angelico <rosuav@gmail.com> - 2015-04-14 14:05 +1000
        Re: Pickle based workflow - looking for advice Fabien <fabien.maussion@gmail.com> - 2015-04-14 09:58 +0200
          Re: Pickle based workflow - looking for advice Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-04-14 23:08 +1000
            Re: Pickle based workflow - looking for advice Chris Angelico <rosuav@gmail.com> - 2015-04-14 23:45 +1000
              Re: Pickle based workflow - looking for advice Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-04-15 00:14 +1000
                Re: Pickle based workflow - looking for advice Chris Angelico <rosuav@gmail.com> - 2015-04-15 00:30 +1000

csiph-web