Date: Mon, 23 May 2011 11:45:09 +0200 From: jlp User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; fr; rv:1.9.2.17) Gecko/20110414 Thunderbird/3.1.10 MIME-Version: 1.0 Newsgroups: comp.lang.java.programmer Subject: Re: analysis of java application logs References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit Lines: 48 Message-ID: <4dda2ca1$0$30771$ba4acef3@reader.news.orange.fr> Organization: les newsgroups par Orange NNTP-Posting-Date: 23 May 2011 11:45:05 CEST NNTP-Posting-Host: 86.217.228.79 X-Trace: 1306143905 reader.news.orange.fr 30771 86.217.228.79:2108 X-Complaints-To: abuse@orange.fr Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!gegeweb.org!feed.ac-versailles.fr!news.in2p3.fr!in2p3.fr!proxad.net!feeder1-2.proxad.net!193.252.117.184.MISMATCH!feeder.news.orange.fr!not-for-mail Xref: x330-a1.tempe.blueboxinc.net comp.lang.java.programmer:4445 Le 23/05/2011 09:50, Ulrich Scholz a écrit : > Hi, > > I'm looking for an approach to the problem of analyzing application > log files. > > I need to analyse Java log files from applications (i.e., not logs of > web servers). These logs contain Java exceptions, thread dumps, and > free-form log4j messages issued by log statements inserted by > programmers during development. Right now, these man-made log entries > do not have any specific format. > > What I'm looking for is a tool and/or strategy that supports in lexing/ > parsing, tagging, and analysing the log entries. Because there is only > little defined syntax and grammar - and because you might not know > what you are looking for - the task requires the quick issuing of > queries against the log data base. Some sort of visualization would be > nice, too. > > Pointers to existing tools and approaches as well as appropriate tools/ > algorithms to develop the required system would be welcome. > > Ulrich At work, so it is not free, with a colleague we have developped a such tool. The colleague has developped the Viewer of CSV file with the library JFreeChart. The csv files are time series ( date are for example in format YYYY/MM/DD:HH:mm:ss ) I have developped my own parser that translates native logs => csv files. In java i have used the java regexp patterns. In a file, we have to find the beginning and the end of an enregistrement ( it can be a multi-lines enregistrement). I can exclude/include enregistrements with java regexp patterns. We have to match the pattern of the date ( regexp and java dateFormat pattern). For every enregistrement, we can extract usefull values by pattern matching ( I use a two passes matching to simplify the patterns) the values can be bound to a filter ( http URL for example) All this is embedded in swing components. I can parse acces logs ( Apache, tomcat, weblogic), log4J logs, Verbse GC of JVM ( IBM JVM, Open JDK 7 ..), java Threads dumps, hibernate sql logs, Tuxedo logs and more generally all implicit or explicit dated enregistrements. That are the main ways ... I take me a long time, an still in developpement ... but we have not found any other tool.