Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #74406 > unrolled thread

Multiprocessing question

Back to article view | Back to comp.lang.python

Contents

  Multiprocessing question Paul LaFollette <paul.lafollette@gmail.com> - 2014-07-13 19:53 -0400
    Re: Multiprocessing question Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-07-14 02:44 +0000
      Re: Multiprocessing question Roy Smith <roy@panix.com> - 2014-07-14 08:59 -0400

#74406 — Multiprocessing question

[Multipart message — attachments visible in raw view] — view raw

Kind people,
I have thrown together a little C/UNIX program that forks a child process,
then proceeds to let the child and parent alternate.  Either can run until
it pauses itself and wakes the other.

I would like to know if there be a way to create the same behavior in
Python 3, preferably in a non-platform dependent fashion.  I would prefer
to use processes rather than threads, but could live with threads if I had
to.  I've studied the documentation for the multiprocessing and thread
modules, but I can't see an easy way to do what I want to do.  I need
minimal communication between processes beyond what i have described, so
creating queues or pipes seems like overkill.  Unlike what I have written
here, I will want to exec the child rather than write the whole thing in
the else clause.  Is there a reasonably simple way to do this?  A reference
to useful documentation would be appreciated.  Sample code even more so, of
course.
Thank you
Paul

-------------------------
Paul S. LaFollette, Jr
CIS Department
Temple University
+1 215 204 6822
paul.lafollette@temple.edu
http://knight.cis.temple.edu/~lafollet

#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
#include <signal.h>
#include <errno.h>

int main(int argc, char **argv)
{
  void handler(int);
  pid_t pid;

  signal(SIGCONT, handler);

  pid = fork();
  if (pid < 0) //failure
  {
    printf("Unable to fork\n");
    exit(1);
  }
  else if (pid > 0) // parent
  {
    while (1)
    {
      printf("Parent waiting for child to do something\n");
      pause();
      printf("Parent doing nifty stuff.\n");
      sleep(1);
      printf("Parent waking child\n");
      errno = 0;
      if (kill(pid, SIGCONT) < 0)
        perror("Parent failed to SIGCONT child.");
    }
  }
  else //pid == 0 so child
  {
    while (1)
    {
      printf ("                                    Child doing useful
work.\n");
      sleep(1);
      printf("                                     Child waking parent\n");
      if (kill(getppid(), SIGCONT) < 0)
        perror("Child failed to SIGCONT parent.");
      printf("                                     Child waiting for parent
to do something\n");
      pause();
    }
  }
}

void handler(int signum)
{
}
===============================================================================
Output:
Parent waiting for child to do something
                                    Child doing useful work.
                                     Child waking parent
                                     Child waiting for parent to do
something
Parent doing nifty stuff.
Parent waking child
Parent waiting for child to do something
                                    Child doing useful work.
                                     Child waking parent
                                     Child waiting for parent to do
something
Parent doing nifty stuff.
Parent waking child
Parent waiting for child to do something
                                    Child doing useful work.
                                     Child waking parent
                                     Child waiting for parent to do
something

[toc] | [next] | [standalone]

#74412

On Sun, 13 Jul 2014 19:53:09 -0400, Paul LaFollette wrote:

> I have thrown together a little C/UNIX program that forks a child
> process, then proceeds to let the child and parent alternate.  Either
> can run until it pauses itself and wakes the other.
> 
> I would like to know if there be a way to create the same behavior in
> Python 3, preferably in a non-platform dependent fashion.

The most direct way of duplicating the Unix idiom of forking a child 
process is to use the os.fork function, which is (I believe) just a thin 
wrapper around the C fork function.

https://docs.python.org/3/library/os.html#os.fork


But this is not platform-independent, it is Unix only.

Alternatively, you can look at the various os.exec* functions, which are 
available on Windows and Unix, and see if any of them are useful.

But the best way to solve this in a platform independent way is to use 
one of the concurrency modules:

https://docs.python.org/3/library/concurrency.html

such as multiprocessing. Although these are written for Python 2 rather 
than 3, you may find them useful:

http://pymotw.com/2/multiprocessing/index.html#module-multiprocessing

https://www.ibm.com/developerworks/aix/library/au-multiprocessing/


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#74420

In article <53c34400$0$9505$c3e8da3$5496439d@news.astraweb.com>,
 Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:

> On Sun, 13 Jul 2014 19:53:09 -0400, Paul LaFollette wrote:
> 
> > I have thrown together a little C/UNIX program that forks a child
> > process, then proceeds to let the child and parent alternate.  Either
> > can run until it pauses itself and wakes the other.
> > 
> > I would like to know if there be a way to create the same behavior in
> > Python 3, preferably in a non-platform dependent fashion.

[...]
> But the best way to solve this in a platform independent way is to use 
> one of the concurrency modules:
> 
> https://docs.python.org/3/library/concurrency.html
> 
> such as multiprocessing. Although these are written for Python 2 rather 
> than 3, you may find them useful:
> 
> http://pymotw.com/2/multiprocessing/index.html#module-multiprocessing
> 
> https://www.ibm.com/developerworks/aix/library/au-multiprocessing/

Let me toss out one other possibility.  If you are going to build some 
kind of message queue between two processes, you might want to look at 
an external queue mechanism such as http://kr.github.io/beanstalkd/ or 
http://www.celeryproject.org/.  The advantage of these over rolling your 
own using the built-in Python modules is they handle a lot of queueing, 
locking, and persistence problems which you might end up having to deal 
with yourself.

They also give you the ability to locate the two processes on different 
machines.  You might not want to do that today, but if your application 
grows, it might be how you want to scale your processing power.  It is 
easy, for example, to have one process pushing tasks onto the queue, and 
a large number of workers pulling them off.  The producer doesn't have 
to know anything about how many workers there are (nor does there need 
to be a fixed number).

They are also language independent.  You might discover at some point in 
the future that you want to rewrite one side of the queue in a different 
language for some reason (perhaps to take advantage of a library that's 
not available in Python).  A queue like Beanstalk or Celery makes that 
easy.  Of course, to do this, you would need to format your messages in 
some language-neutral way (JSON, MessagePack, etc).

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web