Path: csiph.com!newsfeed.hal-mli.net!feeder3.hal-mli.net!newsfeed.hal-mli.net!feeder1.hal-mli.net!newsreader4.netcologne.de!news.netcologne.de!newsfeed.freenet.ag!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path: <oscar.j.benjamin@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.001
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'python.': 0.02; 'else:': 0.03; 'problem?': 0.07; 'subject:two': 0.07; "'no": 0.09; 'impose': 0.09; 'iterate': 0.09; 'threshold': 0.09; 'timestamp': 0.09; 'cc:addr:python-list': 0.10; 'def': 0.10; 'assume': 0.11; '6.2,': 0.16; '9.9': 0.16; 'cc:name:python list': 0.16; 'pair.': 0.16; 'pythonic': 0.16; 'subject:through': 0.16; 'variable.': 0.16; 'later': 0.16; 'wrote:': 0.17; 'yield': 0.17; 'obviously': 0.18; 'file.': 0.20; 'trying': 0.21; 'assuming': 0.22; 'received:209.85.216.181': 0.22; 'cc:2**0': 0.23; 'matching': 0.23; 'idea': 0.24; 'cc:addr:python.org': 0.25; 'header:In-Reply- To:1': 0.25; 'message-id:@mail.gmail.com': 0.27; 'lines': 0.28; 'comparison': 0.29; 'probably': 0.29; "i'm": 0.29; 'could': 0.32; 'message.': 0.33; 'certain': 0.33; 'right?': 0.33; 'safely': 0.33; 'hi,': 0.33; 'received:google.com': 0.34; 'recieve': 0.35; 'subject:?': 0.35; 'continue': 0.35; 'received:209.85': 0.35; 'something': 0.35; 'there': 0.35; 'next': 0.35; 'compare': 0.36; 'two': 0.37; 'being': 0.37; 'sent:': 0.37; 'quite': 0.37; 'received:209': 0.37; 'received:209.85.216': 0.37; 'subject:: ': 0.38; 'files': 0.38; 'some': 0.38; 'header:Received:5': 0.40; 'lost': 0.60; 'chance': 0.61; 'kind': 0.61; 'between': 0.63; 'maximum': 0.63; 'direct': 0.69; 'goal': 0.74; '2013': 0.84; 'idiom': 0.84; 'oscar': 0.84; 'timestamps,': 0.84; 'victor': 0.84; 'stamp': 0.91
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=J0TwANFzMWHcaMGagQvhRBm4S+tJruOXpJhcuyuajLI=; b=sKvzmy6i7h0rPxB2Iwafz2BB7I5kyzHM8gtj/qy+lP6eEOi/gGLMrJ6ey6dVpFXkwx eeyJ3tAUGfFGu7kaEDoUbaDyc5fEJOyMTfDwnjOrPVR3fHgVmSSasKq1yijs+M/Capti 8UA3UU/DIIsH99ik2cP43UNwtw8FUIrLNLid5GE/EtMi7WgjBytZdHP6dKSZmejWL3Fa T7Vhblh7DFOKxWB+5alzn4ApCEBcXDF09vRPF+3+KaTnfH9KSII4xblsIV7yHCf20cm+ h2BDFA9GXAc//VRMo8RjAJh36PHC08MVApOChwBiyMEvFJl8jkrjggaofgxjd9QGmAC4 N8/Q==
MIME-Version: 1.0
In-Reply-To: <8990fb07-fbd2-47ab-8b83-97d34580ebe3@googlegroups.com>
References: <8990fb07-fbd2-47ab-8b83-97d34580ebe3@googlegroups.com>
Date: Mon, 7 Jan 2013 22:58:36 +0000
Subject: Re: Searching through two logfiles in parallel?
From: Oscar Benjamin <oscar.j.benjamin@gmail.com>
To: Victor Hooi <victorhooi@gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Cc: Python List <python-list@python.org>
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.240.1357599526.2939.python-list@python.org>
Lines: 49
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1357599526 news.xs4all.nl 6873 [2001:888:2000:d::a6]:40785
X-Complaints-To: abuse@xs4all.nl
Xref: csiph.com comp.lang.python:36378

On 7 January 2013 22:10, Victor Hooi <victorhooi@gmail.com> wrote:
> Hi,
>
> I'm trying to compare two logfiles in Python.
>
> One logfile will have lines recording the message being sent:
>
>     05:00:06 Message sent - Value A: 5.6, Value B: 6.2, Value C: 9.9
>
> the other logfile has line recording the message being received
>
>     05:00:09 Message received - Value A: 5.6, Value B: 6.2, Value C: 9.9
>
> The goal is to compare the time stamp between the two - we can safely assume the timestamp on the message being received is later than the timestamp on transmission.
>
> If it was a direct line-by-line, I could probably use itertools.izip(), right?
>
> However, it's not a direct line-by-line comparison of the two files - the lines I'm looking for are interspersed among other loglines, and the time difference between sending/receiving is quite variable.
>
> So the idea is to iterate through the sending logfile - then iterate through the receiving logfile from that timestamp forwards, looking for the matching pair. Obviously I want to minimise the amount of back-forth through the file.
>
> Also, there is a chance that certain messages could get lost - so I assume there's a threshold after which I want to give up searching for the matching received message, and then just try to resync to the next sent message.
>
> Is there a Pythonic way, or some kind of idiom that I can use to approach this problem?

Assuming that you can impose a maximum time between the send and
recieve timestamps, something like the following might work
(untested):

def find_matching(logfile1, logfile2, maxdelta):
    buf = {}
    logfile2 = iter(logfile2)
    for msg1 in logfile1:
        if msg1.key in buf:
            yield msg1, buf.pop(msg1.key)
            continue
        maxtime = msg1.time + maxdelta
        for msg2 in logfile2:
            if msg2.key == msg1.key:
                yield msg1, msg2
                break
            buf[msg2.key] = msg2
            if msg2.time > maxtime:
                break
        else:
            yield msg1, 'No match'


Oscar