Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #73497

Re: Python ORM library for distributed mostly-read-only objects?

Show all headers | View raw

In article <85659fdd-511b-4aea-9c4b-17a4bbb88662@googlegroups.com>,
 smurfix@gmail.com wrote:

> My problem: I have a large database of interconnected objects which I need to 
> process with a combination of short- and long-lived workers. These objects 
> are mostly read-only (i.e. any of them can be changed/marked-as-deleted, but 
> that happens infrequently). The workers may or may not be within one Python 
> process, or even on one system.
> 
> I've been doing this with a "classic" session-based SQLAlchemy ORM, approach, 
> but that ends up way too slow and memory intense, as each thread gets its own 
> copy of every object it needs. I don't want that.
> 
> My existing code does object loading and traversal by simple attribute 
> access; I'd like to keep that if at all possible.
> 
> Ideally, what I'd like to have is an object server which mediates write 
> access to the database and then sends change/invalidation notices to the 
> workers. (Changes are infrequent enough that I don't care if a worker gets a 
> notice it's not interested in.)
> 
> I don't care if updates are applied immediately or are only visible to the 
> local process until committed. I also don't need fancy indexing or query 
> abilities; if necessary I can go to the storage backend for that. (That 
> should be SQL, though a NoSQL back-end would be nice to have.)
> 
> Does something like this already exist, somewhere out there, or do I need to 
> write this, or does somebody know of an alternate solution?

If you want to go NoSQL, I think what you're describing is a MongoDB 
replica set (http://docs.mongodb.org/manual/replication/).  One of the 
replicas is the primary, to which all writes are directed.  You can have 
some number of secondaries, which get all the changes applied to the 
primary, and spread out the load for read access.  If you want a vaguely 
SQLAlchemy flavored ORM, there's mongoengine (http://mongoengine.org/).

On the other hand, this may be overkill for what you're trying to do.  
Can you give us some more quantitative idea of your requirements?  How 
many objects?  How much total data is being stored?  How many queries 
per second, and what is the acceptable latency for a query?

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Python ORM library for distributed mostly-read-only objects? smurfix@gmail.com - 2014-06-22 02:46 -0700
  Re: Python ORM library for distributed mostly-read-only objects? Roy Smith <roy@panix.com> - 2014-06-22 09:49 -0400
    Re: Python ORM library for distributed mostly-read-only objects? smurfix@gmail.com - 2014-06-22 21:26 -0700
      Re: Python ORM library for distributed mostly-read-only objects? William Ray Wing <wrw@mac.com> - 2014-06-23 09:43 -0400
        Re: Python ORM library for distributed mostly-read-only objects? Roy Smith <roy@panix.com> - 2014-06-23 11:11 -0400
          Re: Python ORM library for distributed mostly-read-only objects? smurfix@gmail.com - 2014-06-23 11:00 -0700
      Re: Python ORM library for distributed mostly-read-only objects? Matthias Urlichs <matthias@urlichs.de> - 2014-06-23 19:42 +0200
  Re: Python ORM library for distributed mostly-read-only objects? Lie Ryan <lie.1296@gmail.com> - 2014-06-23 16:54 +0100
    Re: Python ORM library for distributed mostly-read-only objects? smurfix@gmail.com - 2014-06-23 11:05 -0700
      Re: Python ORM library for distributed mostly-read-only objects? Lie Ryan <lie.1296@gmail.com> - 2014-06-24 00:16 +0100

csiph-web