Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.001 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'url:pypi': 0.03; 'url:sourceforge': 0.03; 'amounts': 0.07; 'assignment': 0.07; 'dynamically': 0.07; 'extent': 0.07; 'apis': 0.09; 'chime': 0.09; 'feasible.': 0.09; 'instances.': 0.09; 'keys,': 0.09; 'promising': 0.09; 'subject:design': 0.09; 'mailman': 0.10; 'cc:addr:python- list': 0.11; 'python': 0.11; 'stored': 0.12; 'thread': 0.14; 'actors': 0.16; 'centric,': 0.16; 'charles': 0.16; 'dict': 0.16; 'instances,': 0.16; 'less,': 0.16; 'loops': 0.16; 'mean,': 0.16; 'message-id:@earthlink.net': 0.16; 'otoh,': 0.16; 'received:dsl.mindspring.com': 0.16; 'subject:item': 0.16; 'tuples,': 0.16; 'url:kamaelia': 0.16; 'index': 0.16; 'wrote:': 0.18; 'bit': 0.19; 'implementing': 0.19; 'seems': 0.21; '(in': 0.22; 'cc:addr:python.org': 0.22; 'header:User-Agent:1': 0.23; 'instance,': 0.24; "shouldn't": 0.24; 'subject:problem': 0.24; 'looks': 0.24; 'cc:2**0': 0.24; "i've": 0.25; '>': 0.26; 'post': 0.26; 'least': 0.26; 'url:edu': 0.26; 'header:In-Reply- To:1': 0.27; 'appear': 0.29; 'fixed': 0.29; 'chris': 0.29; 'external': 0.29; 'thus': 0.29; 'database,': 0.30; "i'm": 0.30; 'code': 0.31; 'posting': 0.31; '(perhaps': 0.31; 'cells': 0.31; 'continually': 0.31; 'libraries': 0.31; 'post.': 0.31; 'stuff': 0.32; 'url:python': 0.33; 'running': 0.33; '(e.g.': 0.33; 'implemented': 0.33; "i'd": 0.34; 'could': 0.34; 'problem': 0.35; 'something': 0.35; 'operations': 0.35; 'but': 0.35; 'there': 0.35; '14,': 0.36; 'ram': 0.36; 'shorter': 0.36; "i'll": 0.36; 'url:org': 0.36; 'too': 0.37; 'list': 0.37; 'list,': 0.38; 'planning': 0.38; 'pm,': 0.38; 'rather': 0.38; 'either': 0.39; 'space': 0.40; 'how': 0.40; 'even': 0.60; 'read': 0.60; 'around.': 0.60; 'removing': 0.60; 'length': 0.61; 'details.': 0.61; 'simple': 0.61; 'save': 0.62; 'guarantee': 0.63; 'costs': 0.63; 'more': 0.64; 'different': 0.65; 'occur': 0.65; 'within': 0.65; 'mailbox': 0.68; 'study': 0.69; 'avoids': 0.84; 'dict.': 0.84; 'fortunately': 0.84; 'intending': 0.84; 'locked': 0.84; 'or...': 0.84; 'rebert': 0.84; 'url:contents': 0.84; 'forgotten': 0.91; 'url:latest': 0.91; 'from.': 0.93; '2013': 0.98 Date: Mon, 15 Apr 2013 10:14:14 -0700 From: Charles Hixson User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.12) Gecko/20130116 Icedove/10.0.12 MIME-Version: 1.0 To: Chris Rebert Subject: Re: Threadpool item mailboxes design problem References: <516B37CB.1020607@earthlink.net> In-Reply-To: Content-Type: multipart/alternative; boundary="------------040709000002040608050208" Cc: "comp.lang.python" X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 175 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1366046315 news.xs4all.nl 2603 [2001:888:2000:d::a6]:50674 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:43629 This is a multi-part message in MIME format. --------------040709000002040608050208 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 04/14/2013 07:32 PM, Chris Rebert wrote: > > On Apr 14, 2013 4:27 PM, "Charles Hixson" > wrote: > > > > What is the best approach to implementing actors that accept and > post messages (and have no other external contacts). > > You might look at how some of the existing Python actor libraries are > implemented (perhaps one of these might even save you from reinventing > the wheel): > > http://www.pykka.org/en/latest/ > http://www.kamaelia.org/Docs/Axon/Axon.html > https://pypi.python.org/pypi/pulsar > > Kinda old: > http://candygram.sourceforge.net/contents.html > http://osl.cs.uiuc.edu/parley/ > Candygram looks interesting. I'd forgotten about it. The others look either a bit limited (in different ways), or overly general, with the costs that that brings. I'll need to study Candygram a bit more. However, even Candygram seems to have a RAM centric model that I'd need to work around. (Well, the mailbox synchronization must clearly be RAM centric, but the storage shouldn't be.) > > > So far what I've come up with is something like: > > actors = {} > > mailboxs = {} > > > > Stuff actors with actor instances, mailboxes with > multiprocessing.queue instances. (Actors and mailboxes will have > identical keys, which are id#, but it's got to be a dict rather than a > list, because too many are rolled out to disk.) And I'm planning of > having the actors running simultaneously and continually in a > threadpool that just loops through the actors that are assigned to > each thread of the pool. > > > It would, however, be better if the mailbox could be specific to the > threadpool instance, so less space would be wasted. Or if the queues > could dynamically resize. Or if there was a threadsafe dict. Or... > But I don't know that any of these are feasible. (I mean, yes, I > could write all the mail to a database, but is that a better answer, > or even a good one?) > > My recollection is that the built-in collection types are threadsafe > at least to the limited extent that the operations exposed by their > APIs (e.g. dict.setdefault) are atomic. > Perhaps someone will be able to chime in with more details. > If list operations were threadsafe, why would multiprocessing.queue have been created? I don't recall any claim that they were. Still, I've found an assertion on StackOverflow that they are...at least for simple assignment and reading. And the same would appear to be true of dicts from that post. This *does* require that either the index be constant, and the stored value be constant, or that the code section be locked during the access. Fortunately I'm intending to have id#s be unchangable, and the messages to be tuples, and thus constant. OTOH, to use this approach I'll need to find some way to guarantee that removing messages and posting messages don't occur at the same time. So that still means I'll need to lock each access. The answer that seems best is for each thread to have a mailbox that cells within the thread post and read messages from. This will automatically deal with internal to thread synchronization. Then I'll need a mailman thread that... This seems a promising approach, that avoids the problem of fixed length queues, but I'll still need to do a lot of synchronization. Still, it's a lot less, and each thread would be locked for shorter amounts of time. -- Charles Hixson --------------040709000002040608050208 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit On 04/14/2013 07:32 PM, Chris Rebert wrote:

On Apr 14, 2013 4:27 PM, "Charles Hixson" <charleshixsn@earthlink.net> wrote:
>
> What is the best approach to implementing actors that accept and post messages (and have no other external contacts).

You might look at how some of the existing Python actor libraries are implemented (perhaps one of these might even save you from reinventing the wheel):

http://www.pykka.org/en/latest/
http://www.kamaelia.org/Docs/Axon/Axon.html
https://pypi.python.org/pypi/pulsar

Kinda old:
http://candygram.sourceforge.net/contents.html
http://osl.cs.uiuc.edu/parley/

Candygram looks interesting.  I'd forgotten about it.  The others look either a bit limited (in different ways), or overly general, with the costs that that brings.  I'll need to study Candygram a bit more.  However, even Candygram seems to have a RAM centric model that I'd need to work around.  (Well, the mailbox synchronization must clearly be RAM centric, but the storage shouldn't be.)

> So far what I've come up with is something like:
> actors     = {}
> mailboxs = {}
>
> Stuff actors with actor instances, mailboxes with multiprocessing.queue instances.   (Actors and mailboxes will have identical keys, which are id#, but it's got to be a dict rather than a list, because too many are rolled out to disk.)  And I'm planning of having the actors running simultaneously and continually in a threadpool that just loops through the actors that are assigned to each thread of the pool.
<snip>
> It would, however, be better if the mailbox could be specific to the threadpool instance, so less space would be wasted.  Or if the queues could dynamically resize.  Or if there was a threadsafe dict.  Or...  But I don't know that any of these are feasible.  (I mean, yes, I could write all the mail to a database, but is that a better answer, or even a good one?)

My recollection is that the built-in collection types are threadsafe at least to the limited extent that the operations exposed by their APIs (e.g. dict.setdefault) are atomic.
Perhaps someone will be able to chime in with more details.

If list operations were threadsafe, why would multiprocessing.queue have been created?  I don't recall any claim that they were.  Still, I've found an assertion on StackOverflow that they are...at least for simple assignment and reading.  And the same would appear to be true of dicts from that post.  This *does* require that either the index be constant, and the stored value be constant, or that the code section be locked during the access.  Fortunately I'm intending to have id#s be unchangable, and the messages to be tuples, and thus constant.

OTOH, to use this approach I'll need to find some way to guarantee that removing messages and posting messages don't occur at the same time.  So that still means I'll need to lock each access.  The answer that seems best is for each thread to have a mailbox that cells within the thread post and read messages from.  This will automatically deal with internal to thread synchronization.  Then I'll need a mailman thread that...

This seems a promising approach, that avoids the problem of fixed length queues, but I'll still need to do a lot of synchronization.  Still, it's a lot less, and each thread would be locked for shorter amounts of time.
-- 
Charles Hixson
--------------040709000002040608050208--