Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #26339 > unrolled thread

Re: Pass data to a subprocess

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Pass data to a subprocess Laszlo Nagy <gandalf@shopzeus.com> - 2012-08-01 10:19 +0200
    Re: Pass data to a subprocess Roy Smith <roy@panix.com> - 2012-08-01 06:59 -0400
      Re: Pass data to a subprocess Laszlo Nagy <gandalf@shopzeus.com> - 2012-08-01 13:07 +0200
      Re: Pass data to a subprocess Laszlo Nagy <gandalf@shopzeus.com> - 2012-08-01 13:26 +0200
      Re: Pass data to a subprocess andrea crotti <andrea.crotti.0@gmail.com> - 2012-08-01 14:25 +0100
      Re: Pass data to a subprocess Roy Smith <roy@panix.com> - 2012-08-01 15:07 -0400
    Re: Pass data to a subprocess Grant Edwards <invalid@invalid.invalid> - 2012-08-01 14:16 +0000
      Re: Pass data to a subprocess Laszlo Nagy <gandalf@shopzeus.com> - 2012-08-01 16:32 +0200
        Re: Pass data to a subprocess Grant Edwards <invalid@invalid.invalid> - 2012-08-01 19:48 +0000
          Re: Pass data to a subprocess Laszlo Nagy <gandalf@shopzeus.com> - 2012-08-02 08:10 +0200
            Re: Pass data to a subprocess Grant Edwards <invalid@invalid.invalid> - 2012-08-02 14:29 +0000
          Re: Pass data to a subprocess Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2012-08-02 13:21 -0400
      Re: Pass data to a subprocess Laszlo Nagy <gandalf@shopzeus.com> - 2012-08-01 16:42 +0200
      Re: Pass data to a subprocess andrea crotti <andrea.crotti.0@gmail.com> - 2012-08-01 16:24 +0100

#26339 — Re: Pass data to a subprocess

>
> As I wrote "I found many nice things (Pipe, Manager and so on), but
> actually even
> this seems to work:" yes I did read the documentation.
Sorry, I did not want be offensive.
>
> I was just surprised that it worked better than I expected even
> without Pipes and Queues, but now I understand why..
>
> Anyway now I would like to be able to detach subprocesses to avoid the
> nasty code reloading that I was talking about in another thread, but
> things get more tricky, because I can't use queues and pipes to
> communicate with a running process that it's noit my child, correct?
>
Yes, I think that is correct. Instead of detaching a child process, you 
can create independent processes and use other frameworks for IPC. For 
example, Pyro.  It is not as effective as multiprocessing.Queue, but in 
return, you will have the option to run your service across multiple 
servers.

The most effective IPC is usually through shared memory. But there is no 
OS independent standard Python module that can communicate over shared 
memory. Except multiprocessing of course, but AFAIK it can only be used 
to communicate between fork()-ed processes.

[toc] | [next] | [standalone]

#26348

In article <mailman.2809.1343809166.4697.python-list@python.org>,
 Laszlo Nagy <gandalf@shopzeus.com> wrote:

> Yes, I think that is correct. Instead of detaching a child process, you 
> can create independent processes and use other frameworks for IPC. For 
> example, Pyro.  It is not as effective as multiprocessing.Queue, but in 
> return, you will have the option to run your service across multiple 
> servers.

You might want to look at beanstalk (http://kr.github.com/beanstalkd/).  
We've been using it in production for the better part of two years.  At 
a 30,000 foot level, it's an implementation of queues over named pipes 
over TCP, but it takes care of a zillion little details for you.

Setup is trivial, and there's clients for all sorts of languages.  For a 
Python client, go with beanstalkc (pybeanstalk appears to be 
abandonware).
> 
> The most effective IPC is usually through shared memory. But there is no 
> OS independent standard Python module that can communicate over shared 
> memory.

It's true that shared memory is faster than serializing objects over a 
TCP connection.  On the other hand, it's hard to imagine anything 
written in Python where you would notice the difference.

[toc] | [prev] | [next] | [standalone]

#26349

>> The most effective IPC is usually through shared memory. But there is no
>> OS independent standard Python module that can communicate over shared
>> memory.
> It's true that shared memory is faster than serializing objects over a
> TCP connection.  On the other hand, it's hard to imagine anything
> written in Python where you would notice the difference.
Well, except in response times. ;-)

The TCP stack likes to wait after you call send() on a socket. Yes, you 
can use setsockopt/TCP_NOWAIT, but my experience is that response times 
with TCP can be long, especially when you have to do many 
request-response pairs.

It also depends on the protocol design - if you can reduce the number of 
request-response pairs then it helps a lot.

[toc] | [prev] | [next] | [standalone]

#26351

On 2012-08-01 12:59, Roy Smith wrote:
> In article <mailman.2809.1343809166.4697.python-list@python.org>,
>   Laszlo Nagy <gandalf@shopzeus.com> wrote:
>
>> Yes, I think that is correct. Instead of detaching a child process, you
>> can create independent processes and use other frameworks for IPC. For
>> example, Pyro.  It is not as effective as multiprocessing.Queue, but in
>> return, you will have the option to run your service across multiple
>> servers.
> You might want to look at beanstalk (http://kr.github.com/beanstalkd/).
> We've been using it in production for the better part of two years.  At
> a 30,000 foot level, it's an implementation of queues over named pipes
> over TCP, but it takes care of a zillion little details for you.
Looks very simple to use. Too bad that it doesn't work on Windows systems.

[toc] | [prev] | [next] | [standalone]

#26357

2012/8/1 Roy Smith <roy@panix.com>:
> In article <mailman.2809.1343809166.4697.python-list@python.org>,
>  Laszlo Nagy <gandalf@shopzeus.com> wrote:
>
>> Yes, I think that is correct. Instead of detaching a child process, you
>> can create independent processes and use other frameworks for IPC. For
>> example, Pyro.  It is not as effective as multiprocessing.Queue, but in
>> return, you will have the option to run your service across multiple
>> servers.
>
> You might want to look at beanstalk (http://kr.github.com/beanstalkd/).
> We've been using it in production for the better part of two years.  At
> a 30,000 foot level, it's an implementation of queues over named pipes
> over TCP, but it takes care of a zillion little details for you.
>
> Setup is trivial, and there's clients for all sorts of languages.  For a
> Python client, go with beanstalkc (pybeanstalk appears to be
> abandonware).
>>
>> The most effective IPC is usually through shared memory. But there is no
>> OS independent standard Python module that can communicate over shared
>> memory.
>
> It's true that shared memory is faster than serializing objects over a
> TCP connection.  On the other hand, it's hard to imagine anything
> written in Python where you would notice the difference.
> --
> http://mail.python.org/mailman/listinfo/python-list


That does look nice and I would like to have something like that..
But since I have to convince my boss of another external dependency I
think it might be worth
to try out zeromq instead, which can also do similar things and looks
more powerful, what do you think?

[toc] | [prev] | [next] | [standalone]

#26377

On Aug 1, 2012, at 9:25 AM, andrea crotti wrote:

> [beanstalk] does look nice and I would like to have something like that..
> But since I have to convince my boss of another external dependency I
> think it might be worth
> to try out zeromq instead, which can also do similar things and looks
> more powerful, what do you think?

I'm afraid I have no experience with zeromq, so I can't offer an opinion.

--
Roy Smith
roy@panix.com

[toc] | [prev] | [next] | [standalone]

#26360

On 2012-08-01, Laszlo Nagy <gandalf@shopzeus.com> wrote:
>>
>> As I wrote "I found many nice things (Pipe, Manager and so on), but
>> actually even
>> this seems to work:" yes I did read the documentation.
> Sorry, I did not want be offensive.
>>
>> I was just surprised that it worked better than I expected even
>> without Pipes and Queues, but now I understand why..
>>
>> Anyway now I would like to be able to detach subprocesses to avoid the
>> nasty code reloading that I was talking about in another thread, but
>> things get more tricky, because I can't use queues and pipes to
>> communicate with a running process that it's noit my child, correct?
>>
> Yes, I think that is correct.

I don't understand why detaching a child process on Linux/Unix would
make IPC stop working.  Can somebody explain?

-- 
Grant Edwards               grant.b.edwards        Yow! My vaseline is
                                  at               RUNNING...
                              gmail.com            

[toc] | [prev] | [next] | [standalone]

#26362

>>> things get more tricky, because I can't use queues and pipes to
>>> communicate with a running process that it's noit my child, correct?
>>>
>> Yes, I think that is correct.
> I don't understand why detaching a child process on Linux/Unix would
> make IPC stop working.  Can somebody explain?
>
It is implemented with shared memory. I think (although I'm not 100% 
sure) that shared memory is created *and freed up* (shm_unlink() system 
call) by the parent process. It makes sense, because the child processes 
will surely die with the parent. If you detach a child process, then it 
won't be killed with its original parent. But the shared memory will be 
freed by the original parent process anyway. I suspect that the child 
that has mapped that shared memory segment will try to access a freed up 
resource, do a segfault or something similar.

[toc] | [prev] | [next] | [standalone]

#26378

On 2012-08-01, Laszlo Nagy <gandalf@shopzeus.com> wrote:
>
>>>> things get more tricky, because I can't use queues and pipes to
>>>> communicate with a running process that it's noit my child, correct?
>>>>
>>> Yes, I think that is correct.
>> I don't understand why detaching a child process on Linux/Unix would
>> make IPC stop working.  Can somebody explain?
>
> It is implemented with shared memory. I think (although I'm not 100% 
> sure) that shared memory is created *and freed up* (shm_unlink() system 
> call) by the parent process. It makes sense, because the child processes 
> will surely die with the parent. If you detach a child process, then it 
> won't be killed with its original parent. But the shared memory will be 
> freed by the original parent process anyway. I suspect that the child 
> that has mapped that shared memory segment will try to access a freed up 
> resource, do a segfault or something similar.

I still don't get it.  shm_unlink() works the same way unlink() does.
The resource itself doesn't cease to exist until all open file handles
are closed. From the shm_unlink() man page on Linux:

       The operation of shm_unlink() is analogous to unlink(2): it
       removes a shared memory object name, and, once all processes
       have unmapped the object, de-allocates and destroys the
       contents of the associated memory region. After a successful
       shm_unlink(), attempts to shm_open() an object with the same
       name will fail (unless O_CREAT was specified, in which case a
       new, distinct object is created).
       
Even if the parent calls shm_unlink(), the shared-memory resource will
continue to exist (and be usable) until all processes that are holding
open file handles unmap/close them.  So not only will detached
children not crash, they'll still be able to use the shared memory
objects to talk to each other.
       
-- 
Grant Edwards               grant.b.edwards        Yow! Why is it that when
                                  at               you DIE, you can't take
                              gmail.com            your HOME ENTERTAINMENT
                                                   CENTER with you??

[toc] | [prev] | [next] | [standalone]

#26381

> I still don't get it.  shm_unlink() works the same way unlink() does.
> The resource itself doesn't cease to exist until all open file handles
> are closed. From the shm_unlink() man page on Linux:
>
>         The operation of shm_unlink() is analogous to unlink(2): it
>         removes a shared memory object name, and, once all processes
>         have unmapped the object, de-allocates and destroys the
>         contents of the associated memory region. After a successful
>         shm_unlink(), attempts to shm_open() an object with the same
>         name will fail (unless O_CREAT was specified, in which case a
>         new, distinct object is created).
>         
> Even if the parent calls shm_unlink(), the shared-memory resource will
> continue to exist (and be usable) until all processes that are holding
> open file handles unmap/close them.  So not only will detached
> children not crash, they'll still be able to use the shared memory
> objects to talk to each other.
>         
I stand corrected. It should still be examined, what kind shared memory 
is used under non-linux systems. System V on AIX? And what about 
Windows? So maybe the general answer is still no. But I guess that the 
OP wanted this to work on a specific system.

Dear Andrea Crotti! Please try to detach two child processes, exit from 
the main process, and communicate over a multiprocessing queue. It will 
possibly work. Sorry for my bad advice.

[toc] | [prev] | [next] | [standalone]

#26409

On 2012-08-02, Laszlo Nagy <gandalf@shopzeus.com> wrote:
>
>> I still don't get it.  shm_unlink() works the same way unlink() does.
>> The resource itself doesn't cease to exist until all open file
>> handles are closed. From the shm_unlink() man page on Linux:
>>
>>         The operation of shm_unlink() is analogous to unlink(2): it
>>         removes a shared memory object name, and, once all processes
>>         have unmapped the object, de-allocates and destroys the
>>         contents of the associated memory region. After a successful
>>         shm_unlink(), attempts to shm_open() an object with the same
>>         name will fail (unless O_CREAT was specified, in which case a
>>         new, distinct object is created).
>>         
>> Even if the parent calls shm_unlink(), the shared-memory resource
>> will continue to exist (and be usable) until all processes that are
>> holding open file handles unmap/close them.  So not only will
>> detached children not crash, they'll still be able to use the shared
>> memory objects to talk to each other.

Note that when I say the detached children will still be able to talk
to each other using shared memory after the parent calls shm_unlink()
and exit(), I'm talking about the general case -- not specifically
about the multiprocessing module. There may be something else going on
with the multiprocessing module.

> I stand corrected. It should still be examined, what kind shared
> memory is used under non-linux systems. System V on AIX? And what
> about Windows? So maybe the general answer is still no. But I guess
> that the OP wanted this to work on a specific system.
>
> Dear Andrea Crotti! Please try to detach two child processes, exit
> from the main process, and communicate over a multiprocessing queue.
> It will possibly work. Sorry for my bad advice.

I'm not claiming it will work, since I don't know how the IPC in the
multiprocessing module works.  It may indeed break when a child
process is detatched (which I'm assuming means being removed from the
process group and/or detached from the controlling tty).

But, I'm not aware of any underlying Unix IPC mechanism that breaks
when a child is detached, so I was curious about what would cause
multiprocessing's IPC to break.

-- 
Grant Edwards               grant.b.edwards        Yow! I didn't order any
                                  at               WOO-WOO ... Maybe a YUBBA
                              gmail.com            ... But no WOO-WOO!

[toc] | [prev] | [next] | [standalone]

#26412

On Thu, 02 Aug 2012 08:10:59 +0200, Laszlo Nagy <gandalf@shopzeus.com>
declaimed the following in gmane.comp.python.general:


> I stand corrected. It should still be examined, what kind shared memory 
> is used under non-linux systems. System V on AIX? And what about 
> Windows? So maybe the general answer is still no. But I guess that the 
> OP wanted this to work on a specific system.
> 
	From an ancient MSDN disk (goes back to VB6):

"""
Shared Memory
The Win32 API uses a special case of file mapping to provide shared
memory access between processes. If you specify the system swap file
when creating a file-mapping object, the file-mapping object is treated
as a shared memory block. Other processes can access the same block of
memory by opening the same file-mapping object (see File Mapping).
Because shared memory is implemented with file mapping, it supports
security access attributes and can operate only between processes
running on the same computer. 

For information about shared memory, see "General Library" in the
"Windows Base Services" section of the Microsoft Platform SDK.
"""

	There is also the MFC CSharedFile class.

	.NET prior to Framework v4 did not implement mapped files.

	I believe Python exposes this (the Win32 aspect) via mmap.

-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
        wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [next] | [standalone]

#26363

>>> Yes, I think that is correct.
>> I don't understand why detaching a child process on Linux/Unix would
>> make IPC stop working.  Can somebody explain?
>>
> It is implemented with shared memory. I think (although I'm not 100% 
> sure) that shared memory is created *and freed up* (shm_unlink() 
> system call) by the parent process. It makes sense, because the child 
> processes will surely die with the parent. If you detach a child 
> process, then it won't be killed with its original parent. But the 
> shared memory will be freed by the original parent process anyway. I 
> suspect that the child that has mapped that shared memory segment will 
> try to access a freed up resource, do a segfault or something similar.
So detaching the child process will not make IPC stop working. But 
exiting from the original parent process will. (And why else would you 
detach the child?)

[toc] | [prev] | [next] | [standalone]

#26365

2012/8/1 Laszlo Nagy <gandalf@shopzeus.com>:
>
> So detaching the child process will not make IPC stop working. But exiting
> from the original parent process will. (And why else would you detach the
> child?)
>
> --
> http://mail.python.org/mailman/listinfo/python-list


Well it makes perfect sense if it stops working to me, so or
- I use zeromq or something similar to communicate
- I make every process independent without the need to further
communicate with the parent..

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web