Return-Path: <gandalf@shopzeus.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.006
X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; 'python,': 0.02; 'arguments': 0.07; 'space.': 0.07; 'python': 0.09; '(it': 0.09; 'buffer.': 0.09; 'deletion': 0.09; 'stl': 0.09; 'windows,': 0.09; '"free"': 0.16; '4-byte': 0.16; 'arrays.': 0.16; 'blocks': 0.16; 'did,': 0.16; 'processes.': 0.16; 'segment': 0.16; 'bytes': 0.17; 'integer': 0.17; 'memory': 0.18; 'requests': 0.18; 'module': 0.19; 'amounts': 0.22; 'libraries': 0.22; 'names.': 0.22; 'to:2**1': 0.23; 'header:In-Reply-To:1': 0.25; 'header:User-Agent:1': 0.26; 'creating': 0.26; 'values': 0.26; 'guess': 0.27; 'disk': 0.27; 'structures': 0.27; "doesn't": 0.28; 'arrays': 0.29; "i'm": 0.29; 'file': 0.32; 'operate': 0.32; 'space,': 0.32; 'running': 0.32; 'could': 0.32; "aren't": 0.33; 'to:addr:python-list': 0.33; 'another': 0.33; 'minimum': 0.34; 'server': 0.35; 'list': 0.35; 'faster': 0.35; 'locations': 0.35; 'doing': 0.35; 'there': 0.35; 'but': 0.36; 'operating': 0.36; 'uses': 0.37; 'subject:: ': 0.38; 'files': 0.38; 'object': 0.38; 'some': 0.38; 'sure': 0.38; 'system.': 0.39; 'to:addr:python.org': 0.39; 'received:192': 0.39; 'space': 0.39; 'received:192.168': 0.40; 'your': 0.60; 'address': 0.60; 'free': 0.61; 'more.': 0.62; 'provide': 0.62; 'different': 0.63; 'times': 0.63; 'within': 0.64; 'become': 0.65; 'details,': 0.65; 'received:204': 0.72; 'yourself': 0.77; 'conducive': 0.84; 'routines': 0.84; 'prone': 0.91
Date: Thu, 06 Sep 2012 12:00:10 +0200
From: Laszlo Nagy <gandalf@shopzeus.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20120827 Thunderbird/15.0
MIME-Version: 1.0
To: Aaron Brady <castironpi@gmail.com>, python-list@python.org
Subject: Re: Looking for an IPC solution
References: <mailman.9.1346440878.27098.python-list@python.org> <40ab9c6f-737e-47e3-950b-558472ad4a09@googlegroups.com>
In-Reply-To: <40ab9c6f-737e-47e3-950b-558472ad4a09@googlegroups.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.285.1346925623.27098.python-list@python.org>
Lines: 11
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1346925623 news.xs4all.nl 6880 [2001:888:2000:d::a6]:42871
X-Complaints-To: abuse@xs4all.nl
Path: csiph.com!usenet.pasdenom.info!news.stben.net!border3.nntp.ams.giganews.com!border1.nntp.ams.giganews.com!nntp.giganews.com!usenetcore.com!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Xref: csiph.com comp.lang.python:28567


> Hi Laszlo,
>
> There aren't a lot of ways to create a Python object in an "mmap" buffer.  "mmap" is conducive to arrays of arrays.  For variable-length structures like strings and lists, you need "dynamic allocation".  The C functions "malloc" and "free" allocate memory space, and file creation and deletion routines operate on disk space.  However "malloc" doesn't allow you to allocate memory space within memory that's already allocated.  Operating systems don't provide that capability, and doing it yourself amounts to creating your own file system.  If you did, you still might not be able to use existing libraries like the STL or Python, because one address might refer to different locations in different processes.
>
> One solution is to keep a linked list of free blocks within your "mmap" buffer.  It is prone to slow access times and segment fragmentation.  Another solution is to create many small files with fixed-length names.  The minimum file size on your system might become prohibitive depending on your constraints, since a 4-byte integer could occupy 4096 bytes on disk or more.  Or you can serialize the arguments and return values of your functions, and make requests to a central process.
I'm not sure about the technical details, but I was said that 
multiprocessing module uses mmap() under windows. And it is faster than 
TCP/IP. So I guess the same thing could be used from zmq, under Windows. 
(It is not a big concern, I plan to operate server on Unix. Some clients 
might be running on Windows, but they will use TCP/IP.)