Path: csiph.com!usenet.pasdenom.info!aioe.org!news.stack.nl!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
To: python-list@python.org
From: Dennis Lee Bieber <wlfraed@ix.netcom.com>
Subject: Re: How to properly implement worker processes
Date: Wed, 22 Aug 2012 16:09:56 -0400
Organization: > Bestiaria Support Staff <
References: <db59479c-198f-468d-9e73-40b1b992895c@googlegroups.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.3685.1345666196.4697.python-list@python.org>
Lines: 76
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:27682

On Wed, 22 Aug 2012 10:29:49 -0700 (PDT), Dennis Jacobfeuerborn
<djacobfeuerborn@gmail.com> declaimed the following in
gmane.comp.python.general:

> How can I get around this problem and receive status updates from all children efficiently without a shared queue and with the ability to simply kill the child process when it's no longer needed?
> 

	How much actual processing is done during the "check"? 

	Your description makes it sound like these are I/O bound operations
(combined with sleep()) -- and plain old threading tends to work fine
for I/O bound systems.

	If you used threading, you could signal a thread to die via simply
setting a property in the thread:

	t[x].die = True.

The thread would wrap a loop of the form:

while not self.die:
	#do URL check
	resque.put(status of check)
	time.sleep()

Thereby doing away with many of your queues -- you'd only need the
result/status queue, and the thread would only exit at a clean point.

	The next step up, depending on the overhead of spawning processes,
would be to still use control threads and a local result queue, but have
"do URL check" create the check process each time -- you could probably
use "proc.communicate()" to obtain the status via the process stdout
[and pass the URL via stdin]. The rest of the control thread remains the
same.

	The third step: Still use controller threads, but the controller
thread would create a queue pair (to-process, from-process) on
initialization, and then spawn the process. You might even be able to
remove the time.sleep() from the thread level. Actually, checking the
docs, forget about the Queue... Use a Pipe

self.from, self.to = multiprocessing.Pipe()
self.p = multiprocessing.Process(target = worker, 
				args = (self.from, self.to, URL) )
workerdead = False
while not self.die:
	try:
		status = self.from.recv()	#blocks until data
		resque.put(status)	#local Queue collecting results
	except EOFError:
		# whatever for unexpected shutdown
		workerdead = True
if not workerdead:
	self.to.send("SHUTDOWN")
	while True:
		status = self.from.recv()
		if status == "SHUTTING DOWN": break
		resque.put(status)	#might have had a last cycle


	The worker should poll rather than sleep.

def worker(pto, pfrom, URL):	#note reverse of to/from connections
	while True:
		#do URL check
		pto.send(status)
		data = pfrom.poll(1.0)		#sleep until command or time-out
		if data:
			command = pfrom.recv()
			if command == "SHUTDOWN":
				pto.send("SHUTTING DOWN")
				break
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
        wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/