Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #90107 > unrolled thread

Multi-threaded TCP server class for general use

Back to article view | Back to comp.lang.python

Contents

  Multi-threaded TCP server class for general use mark.r.bannister@googlemail.com - 2015-05-07 15:20 -0700
    Re: Multi-threaded TCP server class for general use Chris Angelico <rosuav@gmail.com> - 2015-05-08 11:53 +1000

#90107 — Multi-threaded TCP server class for general use

Hi,

I needed to develop a highly scalable multi-threaded TCP server in Python and when I started writing it in 2013 I could not find a suitable library that would scale the way I needed but also easy to use.

So I invented one - it's called Pyloom.  If you want to take a look, it's part of my DBIS project at the moment: https://sourceforge.net/p/dbis/code/ci/default/tree/src/pyloom

Question: does anyone see the value of this library?  Would you like me to fork it as a separate project?  Is SF.net good enough or do people prefer Python libraries to be hosted somewhere else?

In a nutshell, Pyloom is a multi-threaded TCP server class which you overload in your own program.  The library provides:

* Connection management.

* 1 or more marshal threads that listen and pick up new connections, passing to a pool of dedicated worker threads, each of which can manage multiple sessions.

* Methods that you override for handling various states: new connection, data received, send data, close connection.

* Can track custom sockets and wake up a session when there is I/O on the socket.

* Has a notification service so that one session can wait for data to be processed/collected by a different session, and can be woken up when the data is ready.

Let me know if anyone is interested in re-using this library.  Tested ok on Linux and Solaris, not tried on Windows yet.

Best regards,
Mark.

[toc] | [next] | [standalone]

#90111

On Fri, May 8, 2015 at 8:20 AM,  <mark.r.bannister@googlemail.com> wrote:
> I needed to develop a highly scalable multi-threaded TCP server in Python and when I started writing it in 2013 I could not find a suitable library that would scale the way I needed but also easy to use.
>
> So I invented one - it's called Pyloom.  If you want to take a look, it's part of my DBIS project at the moment: https://sourceforge.net/p/dbis/code/ci/default/tree/src/pyloom
>
> Question: does anyone see the value of this library?

I haven't looked at your actual code, but one thing I would suggest
considering is the new asyncio facilities that are coming in Python
3.5 - or the existing asyncio that came in provisionally with 3.4. For
ultimate scalability, you may find yourself wanting less threads and
more asynchronicity, and this is something that's looking pretty cool
and awesome.

(It's also looking like over a thousand, maybe over two thousand,
posts on python-ideas. I have not, I regret to say, been reading every
single post.)

ChrisA

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web