Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!goblin1!goblin2!goblin.stu.neva.ru!newsfeed.xs4all.nl!newsfeed4.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.000 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'else:': 0.03; 'example:': 0.03; 'memory.': 0.05; 'params': 0.07; 'subject:two': 0.07; 'though:': 0.07; '[1,': 0.09; 'collections': 0.09; 'dict': 0.09; 'logic': 0.09; '{})': 0.09; 'cc:addr:python-list': 0.10; 'yet.': 0.13; '(set': 0.16; '[3],': 0.16; 'cc:name:python list': 0.16; 'iterator': 0.16; 'iterators': 0.16; 'subject:through': 0.16; 'wrote:': 0.17; '>>>': 0.18; 'bit': 0.21; 'import': 0.21; 'cc:2**0': 0.23; 'matching': 0.23; 'cc:addr:python.org': 0.25; 'header:In-Reply-To:1': 0.25; 'skip:[ 10': 0.26; 'message- id:@mail.gmail.com': 0.27; 'consisting': 0.29; 'idea,': 0.29; "skip:' 10": 0.30; 'saves': 0.30; 'basic': 0.30; '(and': 0.32; 'loading': 0.33; 'times.': 0.33; 'received:google.com': 0.34; 'list': 0.35; 'subject:?': 0.35; 'continue': 0.35; 'received:209.85': 0.35; 'received:209': 0.37; 'subject:: ': 0.38; 'files': 0.38; 'easier': 0.38; 'instead': 0.39; 'build': 0.39; 'little': 0.39; 'header:Received:5': 0.40; 'think': 0.40; 'kind': 0.61; 'needing': 0.62; 'more': 0.63; 'results': 0.65; 'to:addr:yahoo.com': 0.80; '2013': 0.84; 'oscar': 0.84; 'transactions': 0.91 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=EwAiUyahLfnz0lQ3DG/p2AVUF0TCOcYQWhl+ZY8xXV8=; b=II+i+PGSk9Dn45ayAGKJyvVb7hWv019jfOhJUExwe/d9i2Q3eGu5C6p3JwBSrPE+BK SwIqbgEkZHuXgUoOzjUlC+lJZQi0vZs+GbgqQNq0qooIoMMFeuYzWMLDFRQpy2a+hFOn cyxweX8X9OTu+D6l3ASOCfBSMwL9aeLVsHDDLCVZ0D9raRpBYx0lSpBUWUmyhwa3w1YN Cf6l18rxQdtgQOpFLyOmAeOsec37B87+yZBjW1gre+QmT3YAQVtseslhxmUridq9igfx vUy5QNbhJlPedvLlIwXKLRtOiFDqohlrgxNM8qpJMPFYOx0zWvmj1/WZt7T4YqIZLdWT ap3g== MIME-Version: 1.0 In-Reply-To: <7662762a-1dca-4e9d-8a43-ccd34051fc8e@b8g2000yqh.googlegroups.com> References: <8990fb07-fbd2-47ab-8b83-97d34580ebe3@googlegroups.com> <7662762a-1dca-4e9d-8a43-ccd34051fc8e@b8g2000yqh.googlegroups.com> Date: Tue, 8 Jan 2013 23:40:01 +0000 Subject: Re: Searching through two logfiles in parallel? From: Oscar Benjamin To: darnold Content-Type: text/plain; charset=ISO-8859-1 Cc: Python List X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 45 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1357688409 news.xs4all.nl 6877 [2001:888:2000:d::a6]:48093 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:36458 On 8 January 2013 19:16, darnold wrote: > i don't think in iterators (yet), so this is a bit wordy. > same basic idea, though: for each message (set of parameters), build a > list of transactions consisting of matching send/receive times. The advantage of an iterator based solution is that we can avoid loading all of both log files into memory. [SNIP] > > results = {} > > for line in sendData.split('\n'): > if not line.strip(): > continue > > timestamp, params = parse(line) > if params not in results: > results[params] = [{'sendTime': timestamp, 'receiveTime': > None}] > else: > results[params].append({'sendTime': timestamp, 'receiveTime': > None}) [SNIP] This kind of logic is made a little easier (and more efficient) if you use a collections.defaultdict instead of a dict since it saves needing to check if the key is in the dict yet. Example: >>> import collections >>> results = collections.defaultdict(list) >>> results defaultdict(, {}) >>> results['asd'].append(1) >>> results defaultdict(, {'asd': [1]}) >>> results['asd'].append(2) >>> results defaultdict(, {'asd': [1, 2]}) >>> results['qwe'].append(3) >>> results defaultdict(, {'qwe': [3], 'asd': [1, 2]}) Oscar