Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder4.news.weretis.net!ecngs!feeder2.ecngs.de!newsfeed.freenet.ag!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
Sender: kushal.kumaran@gmail.com
In-Reply-To: <CACCLA54K9HV1qPhuT2i_dSjUk+ey3TdFN2+9x8_3Aay7de3Snw@mail.gmail.com>
References: <CACCLA54K9HV1qPhuT2i_dSjUk+ey3TdFN2+9x8_3Aay7de3Snw@mail.gmail.com>
From: Kushal Kumaran <kushal.kumaran+python@gmail.com>
Date: Fri, 20 Jul 2012 14:18:51 +0530
Subject: Re: Let child process to run while parent is out (multiprocessing)
To: John Wong <gokoproject@gmail.com>
Content-Type: text/plain; charset=UTF-8
Cc: python-list@python.org
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.2338.1342774158.4697.python-list@python.org>
Lines: 84
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:25684

On Fri, Jul 20, 2012 at 2:04 AM, John Wong <gokoproject@gmail.com> wrote:
> def main(...):
>     build_id = create_build_id(...)
>     build_stuff
>     return build_id
>
> Suppose build_stuff compiles a C program. It could take days to finish, and
> notify users their builds are ready. I was thinking about using
> mutliprocessing to handle the build_stuff.
>
> So here is a sample:
>
> #!/usr/bin/python
>
> import multiprocessing as mp
> import time
>
> def build():
>     print 'I am building HUGE things'
>     time.sleep(10)
>
> def main():
>     build_p = mp.Process(name='build process', target=build)
>     build_p.start()
>     return 'abcd12345'
>
> if __name__ == '__main__':
>
>     v = main()
>     print v
>     print 'done'
>
> Here is output:
> yeukhon@fermat:~$ python c2.py
> abcd12345
> done  [now hangs for 10 seconds]
> I build things
>
> When I looked at `ps -elf|grep python`, I can see two processes running, and
> one of the python c2.py process is `wait`.  But this is not ideal,
> especially this is a web app. I can't implement any advanced queue / event
> system right now (I use Pylon, I believe I have gevent installed). But just
> with multiprocessing, is it possible to send the id first, while running
> child in the backgroud?
>
> Right now it hangs there as long as the child process is alive. Any
> solutions?
>

>From the documentation, there does not seem to be any way of
"detaching" a multiprocessing Process.  But it is doable by using the
underlying os.fork directly (CAUTION: not ready for being invoked from
a web app):

#!/usr/bin/python

import os
import time

def build_and_send_email():
    print 'I am building HUGE things'
    time.sleep(10)

def main():
    child_pid = os.fork()
    if child_pid == 0:
        build_and_send_email()
        os._exit(0)
    return 'abcd12345'

if __name__ == '__main__':
    v = main()
    print v
    print 'done'

To make it work correctly with web app will require a bit more work.
At the least, you will have to close all file descriptors to make sure
the request processing finishes.  You can turn it into a proper
background process (a daemon) using the python-daemon library with
very little code, I think.

-- 
regards,
kushal