Re: Hashed lookups for tabular data

Path	csiph.com!usenet.pasdenom.info!aioe.org!news.stack.nl!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path	<jcasale@activenetwerx.com>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.060
X-Spam-Evidence	'H': 0.88; 'S': 0.00; 'duplicate': 0.07; 'indexes': 0.09; 'python': 0.11; 'bars': 0.16; 'chris,': 0.16; 'dictionaries': 0.16; 'received:172.18.0': 0.16; 'subject:tabular': 0.16; 'hacking': 0.19; 'fit': 0.20; 'memory': 0.22; 'to:name:python-list@python.org': 0.22; 'simpler': 0.24; 'references': 0.26; 'header:In-Reply-To:1': 0.27; 'option.': 0.31; 'skip:q 20': 0.31; 'thanks!': 0.32; 'could': 0.34; 'but': 0.35; 'there': 0.35; 'right?': 0.36; 'subject:data': 0.36; "i'll": 0.36; 'to:addr:python-list': 0.38; 'to:addr:python.org': 0.39; 'enough': 0.39; 'received:unknown': 0.61; 'simply': 0.61; 'back': 0.62; 'kind': 0.63; 'inherent': 0.84; 'mock': 0.84; 'presumably': 0.84; 'profiles': 0.91
X-Cloudmark-SP-Filtered	true
X-Cloudmark-SP-Result	v=1.1 cv=dwjLhcNAn+N65iW9oApC92yF2BXVJAhWBlCn9pRWT9g= c=1 sm=1 a=P90J6pEA2ccA:10 a=BLceEmwcHowA:10 a=8nJEP1OIZ-IA:10 a=xqWC_Br6kY4A:10 a=g3mLq75WYuDrh3Lt0JSDww==:17 a=Z79uS4uhH7_HAJvJhYwA:9 a=wPNLvfGTeEIA:10 a=HpAAvcLHHh0Zw7uRqdWCyQ==:117
X-Spam-Checker-Version	SpamAssassin 3.3.2 (2011-06-06) on mail.activenetwerx.com
X-Spam-Level
X-Spam-Status	No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00, T_RP_MATCHES_RCVD autolearn=ham version=3.3.2
From	"Joseph L. Casale" <jcasale@activenetwerx.com>
To	"python-list@python.org" <python-list@python.org>
Subject	Re: Hashed lookups for tabular data
Thread-Topic	Hashed lookups for tabular data
Thread-Index	AdAz8ITsOlDb5rdsT/6XGbGmQKKOvAAPhuYA//+35wE=
Date	Mon, 19 Jan 2015 17:09:05 +0000
References	<CAPTjJmr1YXTXGi+YE0yjUxgdoy2iOWDq_th4u6ghiQ+QOouNvg@mail.gmail.com>
In-Reply-To	<CAPTjJmr1YXTXGi+YE0yjUxgdoy2iOWDq_th4u6ghiQ+QOouNvg@mail.gmail.com>
Accept-Language	en-US
Content-Language	en-US
X-MS-Has-Attach
X-MS-TNEF-Correlator
x-ms-exchange-transport-fromentityheader	Hosted
x-originating-ip	[172.18.0.4]
Content-Type	text/plain; charset="iso-8859-1"
Content-Transfer-Encoding	quoted-printable
MIME-Version	1.0
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.15
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups	comp.lang.python
Message-ID	<mailman.17855.1421687418.18130.python-list@python.org> (permalink)
Lines	31
NNTP-Posting-Host	2001:888:2000:d::a6
X-Trace	1421687418 news.xs4all.nl 2957 [2001:888:2000:d::a6]:56403
X-Complaints-To	abuse@xs4all.nl
Xref	csiph.com comp.lang.python:84019

Show key headers only | View raw

> So presumably your data's small enough to fit into memory, right? If
> it isn't, going back to the database every time would be the best
> option. But if it is, can you simply keep three dictionaries in sync?

Hi Chris,
Yeah the data can fit in memory and hence the desire to avoid a trip here.

> row = (foo, bar, quux) # there could be duplicate quuxes but not foos or bars
> foo_dict = {}
> bar_dict = {}
> quux_dict = collections.defaultdict(set)
> 
> foo_dict[row[0]] = row
> bar_dict[row[1]] = row
> quux_dict[row[2]].add(row)

This is actually far simpler than I had started imagining, however the row data
is duplicated. I am hacking away at an attempt with references to one copy of
the row.

Its kind of hard to recreate an sql like object in Python with indexes and the
inherent programmability against a single copy of data.

I'll mock this up and see what it profiles like.
Thanks!
jlc

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

Re: Hashed lookups for tabular data "Joseph L. Casale" <jcasale@activenetwerx.com> - 2015-01-19 17:09 +0000

csiph-web