Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #27682

Re: How to properly implement worker processes

Show key headers only | View raw

On Wed, 22 Aug 2012 10:29:49 -0700 (PDT), Dennis Jacobfeuerborn
<djacobfeuerborn@gmail.com> declaimed the following in
gmane.comp.python.general:

> How can I get around this problem and receive status updates from all children efficiently without a shared queue and with the ability to simply kill the child process when it's no longer needed?
> 

	How much actual processing is done during the "check"? 

	Your description makes it sound like these are I/O bound operations
(combined with sleep()) -- and plain old threading tends to work fine
for I/O bound systems.

	If you used threading, you could signal a thread to die via simply
setting a property in the thread:

	t[x].die = True.

The thread would wrap a loop of the form:

while not self.die:
	#do URL check
	resque.put(status of check)
	time.sleep()

Thereby doing away with many of your queues -- you'd only need the
result/status queue, and the thread would only exit at a clean point.

	The next step up, depending on the overhead of spawning processes,
would be to still use control threads and a local result queue, but have
"do URL check" create the check process each time -- you could probably
use "proc.communicate()" to obtain the status via the process stdout
[and pass the URL via stdin]. The rest of the control thread remains the
same.

	The third step: Still use controller threads, but the controller
thread would create a queue pair (to-process, from-process) on
initialization, and then spawn the process. You might even be able to
remove the time.sleep() from the thread level. Actually, checking the
docs, forget about the Queue... Use a Pipe

self.from, self.to = multiprocessing.Pipe()
self.p = multiprocessing.Process(target = worker, 
				args = (self.from, self.to, URL) )
workerdead = False
while not self.die:
	try:
		status = self.from.recv()	#blocks until data
		resque.put(status)	#local Queue collecting results
	except EOFError:
		# whatever for unexpected shutdown
		workerdead = True
if not workerdead:
	self.to.send("SHUTDOWN")
	while True:
		status = self.from.recv()
		if status == "SHUTTING DOWN": break
		resque.put(status)	#might have had a last cycle


	The worker should poll rather than sleep.

def worker(pto, pfrom, URL):	#note reverse of to/from connections
	while True:
		#do URL check
		pto.send(status)
		data = pfrom.poll(1.0)		#sleep until command or time-out
		if data:
			command = pfrom.recv()
			if command == "SHUTDOWN":
				pto.send("SHUTTING DOWN")
				break
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
        wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

Back to comp.lang.python | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

How to properly implement worker processes Dennis Jacobfeuerborn <djacobfeuerborn@gmail.com> - 2012-08-22 10:29 -0700
  Re: How to properly implement worker processes Ian Kelly <ian.g.kelly@gmail.com> - 2012-08-22 11:46 -0600
    Re: How to properly implement worker processes Dennis Jacobfeuerborn <djacobfeuerborn@gmail.com> - 2012-08-22 12:40 -0700
    Re: How to properly implement worker processes Dennis Jacobfeuerborn <djacobfeuerborn@gmail.com> - 2012-08-22 12:40 -0700
      Re: How to properly implement worker processes Ian Kelly <ian.g.kelly@gmail.com> - 2012-08-22 15:15 -0600
        Re: How to properly implement worker processes Dennis Jacobfeuerborn <djacobfeuerborn@gmail.com> - 2012-08-22 19:28 -0700
        Re: How to properly implement worker processes Dennis Jacobfeuerborn <djacobfeuerborn@gmail.com> - 2012-08-22 19:28 -0700
  Re: How to properly implement worker processes Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2012-08-22 16:09 -0400

csiph-web