Path: csiph.com!usenet.pasdenom.info!gegeweb.org!news.ecp.fr!feeder1.cambriumusenet.nl!feed.tweaknews.nl!194.109.133.84.MISMATCH!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Newsgroups: comp.lang.python
Date: Sat, 1 Sep 2012 19:10:43 -0700 (PDT)
In-Reply-To: <mailman.9.1346440878.27098.python-list@python.org>
Complaints-To: groups-abuse@google.com
Injection-Info: glegroupsg2000goo.googlegroups.com; posting-host=67.184.78.189; posting-account=MQ3pigoAAACeFzUFjVAePnOjOJMNlvq9
References: <mailman.9.1346440878.27098.python-list@python.org>
User-Agent: G2/1.0
MIME-Version: 1.0
Subject: Re: Looking for an IPC solution
From: Aaron Brady <castironpi@gmail.com>
To: comp.lang.python@googlegroups.com
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Cc: python-list@python.org
Precedence: list
Message-ID: <mailman.53.1346551846.27098.python-list@python.org>
Lines: 62
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:28228

On Friday, August 31, 2012 2:22:00 PM UTC-5, Laszlo Nagy wrote:
> There are just so many IPC modules out there. I'm looking for a solution=
=20
>=20
> for developing a new a multi-tier application. The core application will=
=20
>=20
> be running on a single computer, so the IPC should be using shared=20
>=20
> memory (or mmap) and have very short response times. But there will be a=
=20
>=20
> tier that will hold application state for clients, and there will be=20
>=20
> lots of clients. So that tier needs to go to different computers. E.g.=20
>=20
> the same IPC should also be accessed over TCP/IP. Most messages will be=
=20
>=20
> simple data structures, nothing complicated. The ability to run on PyPy=
=20
>=20
> would, and also to run on both Windows and Linux would be a plus.
>=20
>=20
>=20
> I have seen a stand alone cross platform IPC server before that could=20
>=20
> serve "channels", and send/receive messages using these channels. But I=
=20
>=20
> don't remember its name and now I cannot find it. Can somebody please hel=
p?
>=20
>=20
>=20
> Thanks,
>=20
>=20
>=20
>     Laszlo

Hi Laszlo,

There aren't a lot of ways to create a Python object in an "mmap" buffer.  =
"mmap" is conducive to arrays of arrays.  For variable-length structures li=
ke strings and lists, you need "dynamic allocation".  The C functions "mall=
oc" and "free" allocate memory space, and file creation and deletion routin=
es operate on disk space.  However "malloc" doesn't allow you to allocate m=
emory space within memory that's already allocated.  Operating systems don'=
t provide that capability, and doing it yourself amounts to creating your o=
wn file system.  If you did, you still might not be able to use existing li=
braries like the STL or Python, because one address might refer to differen=
t locations in different processes.

One solution is to keep a linked list of free blocks within your "mmap" buf=
fer.  It is prone to slow access times and segment fragmentation.  Another =
solution is to create many small files with fixed-length names.  The minimu=
m file size on your system might become prohibitive depending on your const=
raints, since a 4-byte integer could occupy 4096 bytes on disk or more.  Or=
 you can serialize the arguments and return values of your functions, and m=
ake requests to a central process.