Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder4.news.weretis.net!feeder2.ecngs.de!ecngs!feeder.ecngs.de!xlned.com!feeder1.xlned.com!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
In-Reply-To: <5018E687.9000105@shopzeus.com>
References: <CAF_E5JY5FgEJub7tNNzZEj0tri8Z7oPZUSLOLtD+5f0t7Yv+Yg@mail.gmail.com> <CAF_E5JY2sEsauXM-teW4hTHePqFCYK6z+9oLMZcjVWFyYs+gKg@mail.gmail.com> <5017EFB0.6080608@shopzeus.com> <CAF_E5JaEHxsjKoHRhZLzh+YdPU8UqKHae9efCsP7zEvm_37+2w@mail.gmail.com> <5018E687.9000105@shopzeus.com>
Date: Wed, 1 Aug 2012 10:50:38 +0100
Subject: Re: Pass data to a subprocess
From: andrea crotti <andrea.crotti.0@gmail.com>
To: Laszlo Nagy <gandalf@shopzeus.com>
Content-Type: text/plain; charset=ISO-8859-1
Cc: python-list@python.org
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.2811.1343814640.4697.python-list@python.org>
Lines: 71
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:26340

2012/8/1 Laszlo Nagy <gandalf@shopzeus.com>:
>> I was just surprised that it worked better than I expected even
>> without Pipes and Queues, but now I understand why..
>>
>> Anyway now I would like to be able to detach subprocesses to avoid the
>> nasty code reloading that I was talking about in another thread, but
>> things get more tricky, because I can't use queues and pipes to
>> communicate with a running process that it's noit my child, correct?
>>
> Yes, I think that is correct. Instead of detaching a child process, you can
> create independent processes and use other frameworks for IPC. For example,
> Pyro.  It is not as effective as multiprocessing.Queue, but in return, you
> will have the option to run your service across multiple servers.
>
> The most effective IPC is usually through shared memory. But there is no OS
> independent standard Python module that can communicate over shared memory.
> Except multiprocessing of course, but AFAIK it can only be used to
> communicate between fork()-ed processes.


Thanks, there is another thing which is able to interact with running
processes in theory:
https://github.com/lmacken/pyrasite

I don't know though if it's a good idea to use a similar approach for
production code, as far as I understood it uses gdb..  In theory
though I could be able to set up every subprocess with all the data
they need, so I might not even need to share data between them.

Anyway now I had another idea to avoid to be able to stop the main
process without killing the subprocesses, using multiple forks.  Does
the following makes sense?  I don't really need these subprocesses to
be daemons since they should quit when done, but is there anything
that can go wrong with this approach?

from os import fork
from time import sleep
from itertools import count
from sys import exit

from multiprocessing import Process, Queue

class LongProcess(Process):
    def __init__(self, idx, queue):
        Process.__init__(self)
        # self.daemon = True
        self.queue = queue
        self.idx = idx

    def run(self):
        for i in count():
            self.queue.put("%d: %d"  % (self.idx, i))
            print("adding %d: %d"  % (self.idx, i))
            sleep(2)


if __name__ == '__main__':
    qu = Queue()

    # how do I do a multiple fork?
    for i in range(5):
        pid = fork()
        # if I create here all the data structures I should still be
able to do things
        if pid == 0:
            lp = LongProcess(1, qu)
            lp.start()
            lp.join()
            exit(0)
        else:
            print("started subprocess with pid ", pid)