Path: csiph.com!usenet.pasdenom.info!gegeweb.org!aioe.org!news.stack.nl!newsfeed.xs4all.nl!newsfeed4a.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path: <kushal@locationd.net>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.010
X-Spam-Evidence: '*H*': 0.98; '*S*': 0.00; 'duplicate': 0.07; 'sql.': 0.07; 'indexes': 0.09; 'python': 0.11; 'changes': 0.15; 'bars': 0.16; 'chris,': 0.16; 'dictionaries': 0.16; 'in-memory': 0.16; 'subject:tabular': 0.16; 'code.': 0.18; 'hacking': 0.19; 'fit': 0.20; 'memory': 0.22; 'import': 0.22; 'header:User-Agent:1': 0.23; 'simpler': 0.24; 'query': 0.26; 'references': 0.26; 'switch': 0.26; 'header:In-Reply-To:1': 0.27; 'getting': 0.31; 'option.': 0.31; 'skip:q 20': 0.31; 'writes:': 0.31; 'could': 0.34; 'skip:s 30': 0.35; 'but': 0.35; 'there': 0.35; 'interface,': 0.36; 'right?': 0.36; 'subject:data': 0.36; 'too': 0.37; 'to:addr :python-list': 0.38; 'to:addr:python.org': 0.39; 'enough': 0.39; 'simply': 0.61; 'back': 0.62; 'kind': 0.63; 'inherent': 0.84; 'presumably': 0.84
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=locationd.net; s=mail; t=1421689473; bh=FP/W7U0v3OQx7VC4lO00V1s32qU6CEDV+z7/ze/mEzA=; h=From:To:Subject:References:Date:In-Reply-To:From; b=gpNIAP4fQ2Y3S9Tpn01GbcP+Ru+/qWy7vEVuD/xb9EDe9kLRa+W1TMKXdMQKNKn6B /dLTU+Xydm2B763M2nUQh836HszuEmomjqEp6/KixaFFew50vKW5r4iqKCzpYOtS8q nDv5TSn8CfHvQV68VbZ3AozjVG4AJfpBiPIZC8BE=
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on calcium.locationd.net
X-Spam-Level: 
X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,NO_RELAYS, T_DKIM_INVALID autolearn=ham version=3.3.2
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=locationd.net; s=mail; t=1421689473; bh=FP/W7U0v3OQx7VC4lO00V1s32qU6CEDV+z7/ze/mEzA=; h=From:To:Subject:References:Date:In-Reply-To:From; b=gpNIAP4fQ2Y3S9Tpn01GbcP+Ru+/qWy7vEVuD/xb9EDe9kLRa+W1TMKXdMQKNKn6B /dLTU+Xydm2B763M2nUQh836HszuEmomjqEp6/KixaFFew50vKW5r4iqKCzpYOtS8q nDv5TSn8CfHvQV68VbZ3AozjVG4AJfpBiPIZC8BE=
From: Kushal Kumaran <kushal@locationd.net>
To: python-list@python.org
Subject: Re: Hashed lookups for tabular data
References: <84c6e1d5671842038e81994478fb5476@exch.activenetwerx.com> <CAPTjJmr1YXTXGi+YE0yjUxgdoy2iOWDq_th4u6ghiQ+QOouNvg@mail.gmail.com> <1421687345419.39659@activenetwerx.com>
Date: Mon, 19 Jan 2015 23:14:31 +0530
In-Reply-To: <1421687345419.39659@activenetwerx.com> (Joseph L. Casale's message of "Mon, 19 Jan 2015 17:09:05 +0000")
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.3 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Mailman-Approved-At: Mon, 19 Jan 2015 19:39:03 +0100
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.17858.1421692744.18130.python-list@python.org>
Lines: 39
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1421692744 news.xs4all.nl 2915 [2001:888:2000:d::a6]:47267
X-Complaints-To: abuse@xs4all.nl
Xref: csiph.com comp.lang.python:84023

"Joseph L. Casale" <jcasale@activenetwerx.com> writes:

>> So presumably your data's small enough to fit into memory, right? If
>> it isn't, going back to the database every time would be the best
>> option. But if it is, can you simply keep three dictionaries in sync?
>
> Hi Chris,
> Yeah the data can fit in memory and hence the desire to avoid a trip here.
>
>> row = (foo, bar, quux) # there could be duplicate quuxes but not foos or bars
>> foo_dict = {}
>> bar_dict = {}
>> quux_dict = collections.defaultdict(set)
>> 
>> foo_dict[row[0]] = row
>> bar_dict[row[1]] = row
>> quux_dict[row[2]].add(row)
>
> This is actually far simpler than I had started imagining, however the row data
> is duplicated. I am hacking away at an attempt with references to one copy of
> the row.
>
> Its kind of hard to recreate an sql like object in Python with indexes and the
> inherent programmability against a single copy of data.
>

If you want an sql-like interface, you can simply create an in-memory
sqlite3 database.

 import sqlite3
 db = sqlite3.Connection(':memory:')

You can create indexes as you need, and query using SQL.  Later, if you
find the data getting too big to fit in memory, you can switch to using
an on-disk database instead without significant changes to the code.

-- 
regards,
kushal