Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #36458

Re: Searching through two logfiles in parallel?

Show all headers | View raw

On 8 January 2013 19:16, darnold <darnold992000@yahoo.com> wrote:
> i don't think in iterators (yet), so this is a bit wordy.
> same basic idea, though: for each message (set of parameters), build a
> list of transactions consisting of matching send/receive times.

The advantage of an iterator based solution is that we can avoid
loading all of both log files into memory.

[SNIP]
>
> results = {}
>
> for line in sendData.split('\n'):
>     if not line.strip():
>         continue
>
>     timestamp, params = parse(line)
>     if params not in results:
>         results[params] = [{'sendTime': timestamp, 'receiveTime':
> None}]
>     else:
>         results[params].append({'sendTime': timestamp, 'receiveTime':
> None})
[SNIP]

This kind of logic is made a little easier (and more efficient) if you
use a collections.defaultdict instead of a dict since it saves needing
to check if the key is in the dict yet. Example:

>>> import collections
>>> results = collections.defaultdict(list)
>>> results
defaultdict(<type 'list'>, {})
>>> results['asd'].append(1)
>>> results
defaultdict(<type 'list'>, {'asd': [1]})
>>> results['asd'].append(2)
>>> results
defaultdict(<type 'list'>, {'asd': [1, 2]})
>>> results['qwe'].append(3)
>>> results
defaultdict(<type 'list'>, {'qwe': [3], 'asd': [1, 2]})


Oscar

Back to comp.lang.python | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

Searching through two logfiles in parallel? Victor Hooi <victorhooi@gmail.com> - 2013-01-07 14:10 -0800
  Re: Searching through two logfiles in parallel? Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-01-07 22:58 +0000
    Re: Searching through two logfiles in parallel? Victor Hooi <victorhooi@gmail.com> - 2013-01-07 15:41 -0800
      Re: Searching through two logfiles in parallel? Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-01-08 00:33 +0000
    Re: Searching through two logfiles in parallel? Victor Hooi <victorhooi@gmail.com> - 2013-01-07 15:41 -0800
  Re: Searching through two logfiles in parallel? darnold <darnold992000@yahoo.com> - 2013-01-08 11:16 -0800
    Re: Searching through two logfiles in parallel? Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-01-08 23:40 +0000

csiph-web