Path: csiph.com!x330-a1.tempe.blueboxinc.net!newsfeed.hal-mli.net!feeder1.hal-mli.net!feeder.news-service.com!fu-berlin.de!uni-berlin.de!individual.net!not-for-mail From: Robert Klemme Newsgroups: comp.lang.java.programmer Subject: Re: analysis of java application logs Date: Mon, 23 May 2011 20:33:07 +0200 Lines: 40 Message-ID: <93vnjeFovuU1@mid.individual.net> References: <4t6dnUyMGer0w0fQnZ2dnUVZ_j2dnZ2d@earthlink.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Trace: individual.net nRQgMJsjdkLok6vOsL3/mAUS76RwKyNCj2/GmLhqr/WLZ/apI= Cancel-Lock: sha1:a78BUeHpJEQvrZfdLQ392P+5LfM= User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.17) Gecko/20110414 Lightning/1.0b2 Thunderbird/3.1.10 In-Reply-To: <4t6dnUyMGer0w0fQnZ2dnUVZ_j2dnZ2d@earthlink.com> X-Antivirus: avast! (VPS 110523-1, 23.05.2011), Outbound message X-Antivirus-Status: Clean Xref: x330-a1.tempe.blueboxinc.net comp.lang.java.programmer:4479 On 23.05.2011 15:17, Patricia Shanahan wrote: > On 5/23/2011 12:50 AM, Ulrich Scholz wrote: >> I'm looking for an approach to the problem of analyzing application >> log files. >> >> I need to analyse Java log files from applications (i.e., not logs of >> web servers). These logs contain Java exceptions, thread dumps, and >> free-form log4j messages issued by log statements inserted by >> programmers during development. Right now, these man-made log entries >> do not have any specific format. >> >> What I'm looking for is a tool and/or strategy that supports in lexing/ >> parsing, tagging, and analysing the log entries. Because there is only >> little defined syntax and grammar - and because you might not know >> what you are looking for - the task requires the quick issuing of >> queries against the log data base. Some sort of visualization would be >> nice, too. >> >> Pointers to existing tools and approaches as well as appropriate tools/ >> algorithms to develop the required system would be welcome. > > I would use Perl, and begin by recognizing some of the more important > formats, such as thread dumps. I agree with the desirability of > introducing some organized formatting into the log messages, but an > ad-hoc Perl program can often get useful data out of a disorganized log. Only that Perl is so awful - YMMV of course. But for these kinds of tasks (more correctly: for *any* task) I very much prefer to use Ruby because of its cleaner OO and cleaner syntax. In these cases where the basic format is fixed I place general parsing code in a library (a single file really) and then I can write ad hoc scripts which do arbitrary processing of the data. That's very productive. Kind regards robert -- remember.guy do |as, often| as.you_can - without end http://blog.rubybestpractices.com/