Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #88924 > unrolled thread

Pickle based workflow - looking for advice

Back to article view | Back to comp.lang.python

Contents

  Pickle based workflow - looking for advice Fabien <fabien.maussion@gmail.com> - 2015-04-13 16:58 +0200
    Re: Pickle based workflow - looking for advice Devin Jeanpierre <jeanpierreda@gmail.com> - 2015-04-13 11:45 -0400
      Re: Pickle based workflow - looking for advice Fabien <fabien.maussion@gmail.com> - 2015-04-13 19:39 +0200
    Re: Pickle based workflow - looking for advice Robin Becker <robin@reportlab.com> - 2015-04-13 16:53 +0100
    Re: Pickle based workflow - looking for advice Dave Angel <davea@davea.name> - 2015-04-13 12:25 -0400
      Re: Pickle based workflow - looking for advice Fabien <fabien.maussion@gmail.com> - 2015-04-13 19:30 +0200
    Re: Pickle based workflow - looking for advice Peter Otten <__peter__@web.de> - 2015-04-13 19:08 +0200
      Re: Pickle based workflow - looking for advice Fabien <fabien.maussion@gmail.com> - 2015-04-13 19:35 +0200
        Re: Pickle based workflow - looking for advice Chris Angelico <rosuav@gmail.com> - 2015-04-14 14:05 +1000
          Re: Pickle based workflow - looking for advice Fabien <fabien.maussion@gmail.com> - 2015-04-14 09:58 +0200
            Re: Pickle based workflow - looking for advice Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-04-14 23:08 +1000
              Re: Pickle based workflow - looking for advice Chris Angelico <rosuav@gmail.com> - 2015-04-14 23:45 +1000
                Re: Pickle based workflow - looking for advice Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-04-15 00:14 +1000
                  Re: Pickle based workflow - looking for advice Chris Angelico <rosuav@gmail.com> - 2015-04-15 00:30 +1000

#88924 — Pickle based workflow - looking for advice

Folks,

I am writing a quite extensive piece of scientific software. Its 
workflow is quite easy to explain. The tool realizes series of 
operations on watersheds (such as mapping data on it, geostatistics and 
more). There are thousands of independent watersheds of different size, 
and the size determines the computing time spent on each of them.

Say I have the operations A, B, C and D. B and C are completely 
independent but they need A to be run first, D needs B and C, and so 
forth. Eventually the whole operations A, B, C and D will run once for 
all, but of course the whole development is an iterative process and I 
rerun all operations many times.

Currently my workflow is defined as follows:

Define a unique ID and file directory for each watershed, and define A 
and B:

def A(watershed_dir):
	# read some external data
	# do stuff
	# Store the stuff in a Watershed object
	# save it
	f_pickle = os.path.join(watershed_dir, 'watershed.p')
	with open(f_pickle, 'wb') as f:
		pickle.dump(watershed, f)

def B(watershed_directory):
	w = pickle.read()
	f_pickle = os.path.join(watershed_dir, 'watershed.p')
	with open(f_pickle, 'rb') as f:
		watershed = pickle.load(f)
	# do new stuff
	# store it in watershed and save
	with open(f_pickle, 'wb') as f:
		pickle.dump(watershed, f)

So the watershed object is a data container which grows in content. The 
pickle that stores the info can reach a few Mb of size. I chose this 
strategy because A, B, C and D are independent, but they can share their 
results through the pickle. The functions have a single argument (the 
path to the working directory), which means that when I run the 
thousands catchments I can use the multiprocessing pool:

	import multiprocessing as mp
	poolargs = [list of directories]
         pool = mp.Pool()
         poolout = pool.map(A, poolargs, chunksize=1)
         poolout = pool.map(B, poolargs, chunksize=1)
	etc.

I can easily choose to rerun just B without rerunning A. Reading and 
writing pickle times is real slow in comparison to the other stuffs to 
do (running B or C on a single catchment can take seconds for example).

Now, to my questions:
1. Does that seem reasonable?
2. Should Watershed be an object or should it be a simple dictionary? I 
thought that an object could be nice, because it could take care of some 
operations such as plotting and logging. Currently I defined a class 
Watershed, but its attributes are defined and filled by A, B and C (this 
seems a bit wrong to me). I could give more responsibilities to this 
class but it might become way too big: since the whole purpose of the 
tool is to work on watersheds, making a Watershed class actually sounds 
like a code smell (http://en.wikipedia.org/wiki/God_object)
3. The operation A opens an external file, reads data out of it and 
writes it in Watershed object. Is it a bad idea to multiprocess this? (I 
guess it is, since the file might be read twice at the same time)
4. Other comments you might have?

Sorry for the lengthy mail but thanks for any tip.

Fabien




	

[toc] | [next] | [standalone]

#88925

On Mon, Apr 13, 2015 at 10:58 AM, Fabien <fabien.maussion@gmail.com> wrote:
> Now, to my questions:
> 1. Does that seem reasonable?

A big issue is the use of pickle, which is:

* Often suboptimal performance wise (e.g. you can't load only subsets
of the data)
* Makes forwards/backwards compatibility very difficult
* Can make python 2/3 migrations harder
* Creates data files which are difficult to analyze/fix by hand if
they get broken
* Is schemaless, and can accidentally include irrelevant data you
didn't mean to store, making all of the above worse.
* Means you have to be very careful who wrote the pickles, or you open
a remote code execution vulnerability. It's common for people to
forget that code is unsafe, and get themselves pwned. Security is
always better if you don't do anything bad in the first place, than if
you do something bad but try to manage the context in which the bad
thing is done.

Cap'n Proto might be a decent alternatives that gives you good
performance, by letting you process only the bits of the file you want
to. It is also not a walking security nightmare.

> 2. Should Watershed be an object or should it be a simple dictionary? I
> thought that an object could be nice, because it could take care of some
> operations such as plotting and logging. Currently I defined a class
> Watershed, but its attributes are defined and filled by A, B and C (this
> seems a bit wrong to me).

It is usually very confusing for attributes to be defined anywhere
other than __init__. It's very really confusing for them to be defined
by some random other function living somewhere else.

> I could give more responsibilities to this class
> but it might become way too big: since the whole purpose of the tool is to
> work on watersheds, making a Watershed class actually sounds like a code
> smell (http://en.wikipedia.org/wiki/God_object)

Whether they are methods or not doesn't make this any more or less of
a god object -- if it stores all this data used by all these different
things, it is already a bit off.

> 3. The operation A opens an external file, reads data out of it and writes
> it in Watershed object. Is it a bad idea to multiprocess this? (I guess it
> is, since the file might be read twice at the same time)

That does sound like a bad idea, for the reason you gave. It might be
possible to read it once, and share it among many processes.

-- Devin

[toc] | [prev] | [next] | [standalone]

#88931

On 13.04.2015 17:45, Devin Jeanpierre wrote:
> On Mon, Apr 13, 2015 at 10:58 AM, Fabien<fabien.maussion@gmail.com>  wrote:
>> >Now, to my questions:
>> >1. Does that seem reasonable?
> A big issue is the use of pickle, which is:
>
> * Often suboptimal performance wise (e.g. you can't load only subsets
> of the data)
> * Makes forwards/backwards compatibility very difficult
> * Can make python 2/3 migrations harder
> * Creates data files which are difficult to analyze/fix by hand if
> they get broken
> * Is schemaless, and can accidentally include irrelevant data you
> didn't mean to store, making all of the above worse.
> * Means you have to be very careful who wrote the pickles, or you open
> a remote code execution vulnerability. It's common for people to
> forget that code is unsafe, and get themselves pwned. Security is
> always better if you don't do anything bad in the first place, than if
> you do something bad but try to manage the context in which the bad
> thing is done.
>
> Cap'n Proto might be a decent alternatives that gives you good
> performance, by letting you process only the bits of the file you want
> to. It is also not a walking security nightmare.

Thanks for your thoughts. All these concerns are rather secondary for 
the kind of tool I am working on, with the exception of speed. I will 
have a look at Proto


>
>> >2. Should Watershed be an object or should it be a simple dictionary? I
>> >thought that an object could be nice, because it could take care of some
>> >operations such as plotting and logging. Currently I defined a class
>> >Watershed, but its attributes are defined and filled by A, B and C (this
>> >seems a bit wrong to me).
> It is usually very confusing for attributes to be defined anywhere
> other than __init__. It's very really confusing for them to be defined
> by some random other function living somewhere else.

Yes, OK. I will stop that.

>> >I could give more responsibilities to this class
>> >but it might become way too big: since the whole purpose of the tool is to
>> >work on watersheds, making a Watershed class actually sounds like a code
>> >smell (http://en.wikipedia.org/wiki/God_object)
> Whether they are methods or not doesn't make this any more or less of
> a god object -- if it stores all this data used by all these different
> things, it is already a bit off.

Yes, but I see no other way. The "god" container will probably be the 
watershed's directory with the data in it. The rest will specialize.


>> >3. The operation A opens an external file, reads data out of it and writes
>> >it in Watershed object. Is it a bad idea to multiprocess this? (I guess it
>> >is, since the file might be read twice at the same time)
> That does sound like a bad idea, for the reason you gave. It might be
> possible to read it once, and share it among many processes.

Yes. Thanks!

[toc] | [prev] | [next] | [standalone]

#88926

for what it's worth I believe that marshal is a faster method for storing simple 
python objects. So if your information can be stored using simple python things 
eg strings, floats, integers, lists and dicts then storage using marshal is 
faster than pickle/cpickle. If you want to persist the objects for years then 
the pickle protocol is probably better as it is not python version dependant.
-- 
Robin Becker

[toc] | [prev] | [next] | [standalone]

#88927

On 04/13/2015 10:58 AM, Fabien wrote:
> Folks,
>

A comment.  Pickle is a method of creating persistent data, most 
commonly used to preserve data between runs.  A database is another 
method.  Although either one can also be used with multiprocessing, you 
seem to be worrying more about the mechanism, and not enough about the 
problem.

> I am writing a quite extensive piece of scientific software. Its
> workflow is quite easy to explain. The tool realizes series of
> operations on watersheds (such as mapping data on it, geostatistics and
> more). There are thousands of independent watersheds of different size,
> and the size determines the computing time spent on each of them.

First question:  what is the name or "identity" of a watershed? 
Apparently it's named by a directory.  But you mention ID as well.  You 
write a function A() that takes only a directory name. Is that the name 
of the watershed?  One per directory?  And you can derive the ID from 
the directory name?

Second question, is there any communication between watersheds, or are 
they totally independent?

Third:  this "external data", is it dynamic, do you have to fetch it in 
a particular order, is it separated by watershed id, or what?

Fourth:  when the program starts, are the directories all empty, so the 
presence of a pickle file tells you that A() has run?  Or is there some 
other meaning for those files?

>
> Say I have the operations A, B, C and D. B and C are completely
> independent but they need A to be run first, D needs B and C, and so
> forth. Eventually the whole operations A, B, C and D will run once for
> all,

For all what?

> but of course the whole development is an iterative process and I
> rerun all operations many times.

Based on what?  Is the external data changing, and you have to rerun 
functions to update what you've already stored about them?  Or do you 
just mean you call the A() function on every possible watershed?



(I suddenly have to go out, so I can't comment on the rest, except that 
choosing to pickle, or to marshall, or to database, or to 
custom-serialize seems a bit premature.  You may have it all clear in 
your head, but I can't see what the interplay between all these calls to 
one-letter-named functions is intended to be.)


-- 
DaveA

[toc] | [prev] | [next] | [standalone]

#88929

On 13.04.2015 18:25, Dave Angel wrote:
> On 04/13/2015 10:58 AM, Fabien wrote:
>> Folks,
>>
>
> A comment.  Pickle is a method of creating persistent data, most
> commonly used to preserve data between runs.  A database is another
> method.  Although either one can also be used with multiprocessing, you
> seem to be worrying more about the mechanism, and not enough about the
> problem.
>
>> I am writing a quite extensive piece of scientific software. Its
>> workflow is quite easy to explain. The tool realizes series of
>> operations on watersheds (such as mapping data on it, geostatistics and
>> more). There are thousands of independent watersheds of different size,
>> and the size determines the computing time spent on each of them.
>
> First question:  what is the name or "identity" of a watershed?
> Apparently it's named by a directory.  But you mention ID as well.  You
> write a function A() that takes only a directory name. Is that the name
> of the watershed?  One per directory?  And you can derive the ID from
> the directory name?
>
> Second question, is there any communication between watersheds, or are
> they totally independent?
>
> Third:  this "external data", is it dynamic, do you have to fetch it in
> a particular order, is it separated by watershed id, or what?
>
> Fourth:  when the program starts, are the directories all empty, so the
> presence of a pickle file tells you that A() has run?  Or is there some
> other meaning for those files?
>
>>
>> Say I have the operations A, B, C and D. B and C are completely
>> independent but they need A to be run first, D needs B and C, and so
>> forth. Eventually the whole operations A, B, C and D will run once for
>> all,
>
> For all what?
>
>> but of course the whole development is an iterative process and I
>> rerun all operations many times.
>
> Based on what?  Is the external data changing, and you have to rerun
> functions to update what you've already stored about them?  Or do you
> just mean you call the A() function on every possible watershed?
>
>
>
> (I suddenly have to go out, so I can't comment on the rest, except that
> choosing to pickle, or to marshall, or to database, or to
> custom-serialize seems a bit premature.  You may have it all clear in
> your head, but I can't see what the interplay between all these calls to
> one-letter-named functions is intended to be.)


Thanks Dave for your interest. I'll make an example:

external files:
- watershed outlines (single file)
- global topography (single file)
- climate data (single file)

Each watershed has an ID. Each watershed is completely independant.

So the function A for example will take one ID as argument, open the 
watershed file and extract its outlines, make a local map, open the 
topography file, extract a part of it, make a watershed object and store 
the watersheds local data in it.

Function B will open the watershed pickle, take the local information it 
needs (like local topography, already cropped to the region of interest) 
and map climate data on it.

And so forth, so that each function A, B, C, ... builds upon the 
information of the others and adds it's own "service" in terms of data.

Currently, all data (numpy arrays and vecor objects mostly) are stored 
as object attributes, which is I guess bad practice. It's kind of a 
"database for dummies": read topography of watershed ID 0128 will be:
- open watershed.p in the '0128' directory
- read the watershed.topography attribute

I think that I like Peter's idea to follow a file based workflow 
instead, and forget about my watershed object for now.

But I'd still be interested in your comments if you find time for it.

Fabien

[toc] | [prev] | [next] | [standalone]

#88928

Fabien wrote:

> I am writing a quite extensive piece of scientific software. Its
> workflow is quite easy to explain. The tool realizes series of
> operations on watersheds (such as mapping data on it, geostatistics and
> more). There are thousands of independent watersheds of different size,
> and the size determines the computing time spent on each of them.
> 
> Say I have the operations A, B, C and D. B and C are completely
> independent but they need A to be run first, D needs B and C, and so
> forth. Eventually the whole operations A, B, C and D will run once for
> all, but of course the whole development is an iterative process and I
> rerun all operations many times.

> 4. Other comments you might have?

How about a file-based workflow?

Write distinct scripts, e. g.

a2b.py that reads from *.a and writes to *.b

and so on. Then use a plain old makefile to define the dependencies.
Whether .a uses pickle, .b uses json, and .z uses csv is but an 
implementation detail that only its producers and consumers need to know. 
Testing an arbitrary step is as easy as invoking the respective script with 
some prefabricated input and checking the resulting output file(s).

[toc] | [prev] | [next] | [standalone]

#88930

On 13.04.2015 19:08, Peter Otten wrote:
> How about a file-based workflow?
>
> Write distinct scripts, e. g.
>
> a2b.py that reads from *.a and writes to *.b
>
> and so on. Then use a plain old makefile to define the dependencies.
> Whether .a uses pickle, .b uses json, and .z uses csv is but an
> implementation detail that only its producers and consumers need to know.
> Testing an arbitrary step is as easy as invoking the respective script with
> some prefabricated input and checking the resulting output file(s).

I think I like the idea because it is more durable. The data I 
manipulate comes with specific formats which are very efficient. With 
the pickle I was kind of "lazy" and, well, saved a couple of read/write 
routines.

Still, your idea is probably more elegant.

With multiprocessing, do I have to care about processes writing 
simultaneously in *different* files? I guess the OS takes good care of 
this stuff but I'm not an expert.

Tahnks,

Fabien

[toc] | [prev] | [next] | [standalone]

#88939

On Tue, Apr 14, 2015 at 3:35 AM, Fabien <fabien.maussion@gmail.com> wrote:
> With multiprocessing, do I have to care about processes writing
> simultaneously in *different* files? I guess the OS takes good care of this
> stuff but I'm not an expert.

Not sure what you mean, here. Any given file will be written by
exactly one process? No possible problem. Multiprocessing within one
application doesn't change that.

ChrisA

[toc] | [prev] | [next] | [standalone]

#88941

On 14.04.2015 06:05, Chris Angelico wrote:
> Not sure what you mean, here. Any given file will be written by
> exactly one process? No possible problem. Multiprocessing within one
> application doesn't change that.

yes that's what I meant. Thanks!

[toc] | [prev] | [next] | [standalone]

#88949

On Tue, 14 Apr 2015 05:58 pm, Fabien wrote:

> On 14.04.2015 06:05, Chris Angelico wrote:
>> Not sure what you mean, here. Any given file will be written by
>> exactly one process? No possible problem. Multiprocessing within one
>> application doesn't change that.
> 
> yes that's what I meant. Thanks!

It's not that simple though. If you require files to be written in precisely
a certain order, then parallel processing requires synchronisation.

Suppose you write A, then B, then C, then D, each in it's own process (or
thread). So the B process has to wait for A to finish, the C process has to
wait for B to finish, and so on. Otherwise you could find yourself with C
reading the data from B before B is finished writing it.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#88958

On Tue, Apr 14, 2015 at 11:08 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> On Tue, 14 Apr 2015 05:58 pm, Fabien wrote:
>
>> On 14.04.2015 06:05, Chris Angelico wrote:
>>> Not sure what you mean, here. Any given file will be written by
>>> exactly one process? No possible problem. Multiprocessing within one
>>> application doesn't change that.
>>
>> yes that's what I meant. Thanks!
>
> It's not that simple though. If you require files to be written in precisely
> a certain order, then parallel processing requires synchronisation.
>
> Suppose you write A, then B, then C, then D, each in it's own process (or
> thread). So the B process has to wait for A to finish, the C process has to
> wait for B to finish, and so on. Otherwise you could find yourself with C
> reading the data from B before B is finished writing it.

Sure, which is a matter of writer/reader conflicts on a single file -
nothing to do with "writing multiple files simultaneously" which was
the question raised.

ChrisA

[toc] | [prev] | [next] | [standalone]

#88963

On Tue, 14 Apr 2015 11:45 pm, Chris Angelico wrote:

> On Tue, Apr 14, 2015 at 11:08 PM, Steven D'Aprano
> <steve+comp.lang.python@pearwood.info> wrote:
>> On Tue, 14 Apr 2015 05:58 pm, Fabien wrote:
>>
>>> On 14.04.2015 06:05, Chris Angelico wrote:
>>>> Not sure what you mean, here. Any given file will be written by
>>>> exactly one process? No possible problem. Multiprocessing within one
>>>> application doesn't change that.
>>>
>>> yes that's what I meant. Thanks!
>>
>> It's not that simple though. If you require files to be written in
>> precisely a certain order, then parallel processing requires
>> synchronisation.
>>
>> Suppose you write A, then B, then C, then D, each in it's own process (or
>> thread). So the B process has to wait for A to finish, the C process has
>> to wait for B to finish, and so on. Otherwise you could find yourself
>> with C reading the data from B before B is finished writing it.
> 
> Sure, which is a matter of writer/reader conflicts on a single file -
> nothing to do with "writing multiple files simultaneously" which was
> the question raised.

Fabien: "So I'm trying to crack open an old grenade I found, and I was
wondering if I need a ball-peen hammer or whether a regular hammer will be
okay."

You: "Oh, a regular hammer will be fine."

Me: "Just a minute. You're hitting a grenade with a hammer hard enough to
crack the case. That could be bad. It might explode."

You: "Sure, but the OP never asked about that. He just asked if the kind of
hammer makes a difference."

:-P


Seriously though, the OP did specify in his first post that there is at
least one dependency of the "B depends on A finishing first" kind. I
understood that A writes to a file, B reads that file and writes to a new
file, C reads that file and writes to yet another file, and so on. In which
case, *writing* the files is the least of his problems, it's the exploding
grenade, er, synchronisation problems that will get him.

:-)


Apart from "embarrassingly parallel" problems, thread- and
multiprocessing-based workflows are often trickier than they may seen ahead
of time, and may even be slower than a purely sequential algorithm:

http://en.wikipedia.org/wiki/Parallel_slowdown
http://en.wikipedia.org/wiki/Embarrassingly_parallel



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#88965

On Wed, Apr 15, 2015 at 12:14 AM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> On Tue, 14 Apr 2015 11:45 pm, Chris Angelico wrote:
>
>> On Tue, Apr 14, 2015 at 11:08 PM, Steven D'Aprano
>> <steve+comp.lang.python@pearwood.info> wrote:
>>> On Tue, 14 Apr 2015 05:58 pm, Fabien wrote:
>>>
>>>> On 14.04.2015 06:05, Chris Angelico wrote:
>>>>> Not sure what you mean, here. Any given file will be written by
>>>>> exactly one process? No possible problem. Multiprocessing within one
>>>>> application doesn't change that.
>>>>
>>>> yes that's what I meant. Thanks!
>>>
>>> It's not that simple though. If you require files to be written in
>>> precisely a certain order, then parallel processing requires
>>> synchronisation.
>>>
>>> Suppose you write A, then B, then C, then D, each in it's own process (or
>>> thread). So the B process has to wait for A to finish, the C process has
>>> to wait for B to finish, and so on. Otherwise you could find yourself
>>> with C reading the data from B before B is finished writing it.
>>
>> Sure, which is a matter of writer/reader conflicts on a single file -
>> nothing to do with "writing multiple files simultaneously" which was
>> the question raised.
>
> Fabien: "So I'm trying to crack open an old grenade I found, and I was
> wondering if I need a ball-peen hammer or whether a regular hammer will be
> okay."
>
> You: "Oh, a regular hammer will be fine."
>
> Me: "Just a minute. You're hitting a grenade with a hammer hard enough to
> crack the case. That could be bad. It might explode."
>
> You: "Sure, but the OP never asked about that. He just asked if the kind of
> hammer makes a difference."
>
> :-P

Heh, fair point. This list is superb at answering the questions people
never even knew to ask.

> Seriously though, the OP did specify in his first post that there is at
> least one dependency of the "B depends on A finishing first" kind. I
> understood that A writes to a file, B reads that file and writes to a new
> file, C reads that file and writes to yet another file, and so on. In which
> case, *writing* the files is the least of his problems, it's the exploding
> grenade, er, synchronisation problems that will get him.
>
> :-)
>
> Apart from "embarrassingly parallel" problems, thread- and
> multiprocessing-based workflows are often trickier than they may seen ahead
> of time, and may even be slower than a purely sequential algorithm:

Yep. The way I read the OP's problem, it's easiest thought of as a
generic request-response system - same as most internet servers.
Basically, you have a piece of code that reacts to an incoming
request, and produces some form of response, then goes back and looks
for the next request. Whether you actually code along those lines or
not, it's a reasonable way to get your head around it.

If you _do_ code it that way, one big benefit is that you effectively
have a multiprocessable state machine; you can fork out to N processes
to take advantage of your CPU cores, or run in a single process for
debugging, and none of the code cares in the slightest.

ChrisA

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web