Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #81371 > unrolled thread

Recurring Task

Back to article view | Back to comp.lang.python

Contents

  Recurring Task tim@timothyarnold.co.uk - 2014-11-23 07:23 -0800
    Re: Recurring Task Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2014-11-23 13:35 -0500
    Re: Recurring Task Dan Stromberg <drsalists@gmail.com> - 2014-11-23 11:48 -0800
    Re: Recurring Task Dan Stromberg <drsalists@gmail.com> - 2014-11-23 11:56 -0800

#81371 — Recurring Task

Hi All
Looking for some advice. I'm creating a small netwok poller and wondered what people recommend to use? Will be polling up to 100 hosts every ten reconds or so

Options I can see

Home grown using worker threads with Queue and dispatcher based on timestamp

Twisted timer service

Python Scheduler

Has anyone done anything similar and can share some insight?

Thanks!
Tim

[toc] | [next] | [standalone]

#81398

On Sun, 23 Nov 2014 07:23:43 -0800 (PST), tim@timothyarnold.co.uk declaimed
the following:

>Hi All
>Looking for some advice. I'm creating a small netwok poller and wondered what people recommend to use? Will be polling up to 100 hosts every ten reconds or so
>

	That's only 10 per second -- presuming scattered access is allowed...
How much overhead is there in making the connection and transferring the
data? That is -- what is your worst case transaction time (account for
failure to connect time-outs).

	Since you're likely I/O bound, plain Python threads are probably
usable. You worst case analysis should indicate how many threads are
desirable (if 9 of the 10 per second take the full second, you likely want
10 and maybe a few extra threads).

	Trickiest part would be maintaining the time-ordered queue (has Python
created a priority queue yet? -- Ah, apparently it has -- so one could use
the desired clock value for the priority, and when the thread finishes it
requeues incrementing the clock value by the rescan interval... -- you'd
need to implement a sleep_until if the thread grabs an entry before its
time)
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
    wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [next] | [standalone]

#81399

On Sun, Nov 23, 2014 at 7:23 AM,  <tim@timothyarnold.co.uk> wrote:
> Hi All
> Looking for some advice. I'm creating a small netwok poller and wondered what people recommend to use? Will be polling up to 100 hosts every ten reconds or so
>
> Options I can see
>
> Home grown using worker threads with Queue and dispatcher based on timestamp

This might be good, as might the multiprocessing module.

> Twisted timer service

I wouldn't wish a large Twisted project on my worst enemy.  If you
need async, you might consider gevent.

> Python Scheduler

Perhaps - I have no experience with this.

I'd probably go the multiprocessing route.  multithreading is fine
with CPython if you're 100% I/O bound, but as soon as you add one
CPU-bound thread, the multitasking gets all messed up - unless you
move to jython.

[toc] | [prev] | [next] | [standalone]

#81400

On Sun, Nov 23, 2014 at 10:35 AM, Dennis Lee Bieber
<wlfraed@ix.netcom.com> wrote:
> On Sun, 23 Nov 2014 07:23:43 -0800 (PST), tim@timothyarnold.co.uk declaimed
> the following:
>
>>Hi All
>>Looking for some advice. I'm creating a small netwok poller and wondered what people recommend to use? Will be polling up to 100 hosts every ten reconds or so
>>
>
>         Trickiest part would be maintaining the time-ordered queue (has Python
> created a priority queue yet? -- Ah, apparently it has -- so one could use
> the desired clock value for the priority, and when the thread finishes it
> requeues incrementing the clock value by the rescan interval... -- you'd
> need to implement a sleep_until if the thread grabs an entry before its
> time)

The standard library way: https://docs.python.org/3/library/queue.html
Here's another way - a class I ported from Java:
https://pypi.python.org/pypi/fibonacci-heap-mod

The Fibonacci Heap is theoretically nice because it's lazy and
supports some interesting operations like O(1) merge and O(1)
decrease-key, but it has a large constant on its O(logn) dequeue-min.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web