Path: csiph.com!x330-a1.tempe.blueboxinc.net!newsfeed.hal-mli.net!feeder1.hal-mli.net!border3.nntp.dca.giganews.com!Xl.tags.giganews.com!border1.nntp.dca.giganews.com!nntp.giganews.com!local2.nntp.dca.giganews.com!nntp.earthlink.com!news.earthlink.com.POSTED!not-for-mail
NNTP-Posting-Date: Mon, 23 May 2011 08:17:28 -0500
Date: Mon, 23 May 2011 06:17:24 -0700
From: Patricia Shanahan <pats@acm.org>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.17) Gecko/20110414 Thunderbird/3.1.10
MIME-Version: 1.0
Newsgroups: comp.lang.java.programmer
Subject: Re: analysis of java application logs
References: <bd933ace-5641-4711-9105-4e949a602b87@c1g2000yqe.googlegroups.com>
In-Reply-To: <bd933ace-5641-4711-9105-4e949a602b87@c1g2000yqe.googlegroups.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Message-ID: <4t6dnUyMGer0w0fQnZ2dnUVZ_j2dnZ2d@earthlink.com>
Lines: 28
X-Usenet-Provider: http://www.giganews.com
NNTP-Posting-Host: 75.8.126.96
X-Trace: sv3-vLO9ingnGA2lL3EcNPOOn13BzEoei9DqUoTLG5ciqEqPZVKgHulhJ5wwSLrVvnr51jiYtaCGuSPoucB!2HLwDRXyobU4LKqZmgwh576lOGuIStPQyTUPOu58mNhksLomO2airWiVh8tejsmjUds/J3gt4g0x!eLy18pfga1zw5DvmgJVYetyUqijb+6ivamoijrAQRrY=
X-Abuse-and-DMCA-Info: Please be sure to forward a copy of ALL headers
X-Abuse-and-DMCA-Info: Otherwise we will be unable to process your complaint properly
X-Postfilter: 1.3.40
X-Original-Bytes: 2514
Xref: x330-a1.tempe.blueboxinc.net comp.lang.java.programmer:4450

On 5/23/2011 12:50 AM, Ulrich Scholz wrote:
> Hi,
>
> I'm looking for an approach to the problem of analyzing application
> log files.
>
> I need to analyse Java log files from applications (i.e., not logs of
> web servers). These logs contain Java exceptions, thread dumps, and
> free-form log4j messages issued by log statements inserted by
> programmers during development. Right now, these man-made log entries
> do not have any specific format.
>
> What I'm looking for is a tool and/or strategy that supports in lexing/
> parsing, tagging, and analysing the log entries. Because there is only
> little defined syntax and grammar - and because you might not know
> what you are looking for - the task requires the quick issuing of
> queries against the log data base. Some sort of visualization would be
> nice, too.
>
> Pointers to existing tools and approaches as well as appropriate tools/
> algorithms to develop the required system would be welcome.

I would use Perl, and begin by recognizing some of the more important
formats, such as thread dumps. I agree with the desirability of
introducing some organized formatting into the log messages, but an
ad-hoc Perl program can often get useful data out of a disorganized log.

Patricia