Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #4447

Re: analysis of java application logs

Show all headers | View raw

Ulrich Scholz wrote:
> I'm looking for an approach to the problem of analyzing application
> log files.
>
> I need to analyse Java log files from applications (i.e., not logs of
> web servers). These logs contain Java exceptions, thread dumps, and
> free-form log4j messages issued by log statements inserted by
> programmers during development. Right now, these man-made log entries
> do not have any specific format.
>
> What I'm looking for is a tool and/or strategy that supports in lexing/
> parsing, tagging, and analysing the log entries. Because there is only
> little defined syntax and grammar - and because you might not know
> what you are looking for - the task requires the quick issuing of
> queries against the log data base. Some sort of visualization would be
> nice, too.
>
> Pointers to existing tools and approaches as well as appropriate tools/
> algorithms to develop the required system would be welcome.

It helps if you have a logging strategy that mandates a consistent logging 
format, specific information in particular positions or marked by particular 
markup, logging levels and other such so that your analysis tool isn't faced 
with a completely open-ended input.  What you describe requires a general 
text-analysis approach, as you indicate that you can make no guarantees about 
the format.  Based on that, your best tool is "less" or equivalent text-file 
reader.

What is a tool supposed to do, read your mind?

It's really hard to extract information from a garbage can where people just 
randomly dumped whatever they individually felt like dumping without regard 
for operational needs.  You can't build a skyscraper on a bad foundation, and 
you can't build a good log analysis off a crappy log.

Fix the logging system, then the analysis problem will be tractable.

-- 
Lew
Honi soit qui mal y pense.
http://upload.wikimedia.org/wikipedia/commons/c/cf/Friz.jpg

Back to comp.lang.java.programmer | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

analysis of java application logs Ulrich Scholz <d7@thispla.net> - 2011-05-23 00:50 -0700
  Re: analysis of java application logs Robert Klemme <shortcutter@googlemail.com> - 2011-05-23 02:20 -0700
  Re: analysis of java application logs jlp <jlp@jlp.com> - 2011-05-23 11:45 +0200
  Re: analysis of java application logs Lew <noone@lewscanon.com> - 2011-05-23 09:11 -0400
    Re: analysis of java application logs Daniele Futtorovic <da.futt.news@laposte-dot-net.invalid> - 2011-05-23 19:16 +0200
      Re: analysis of java application logs Lew <noone@lewscanon.com> - 2011-05-23 15:02 -0400
        Re: analysis of java application logs Daniele Futtorovic <da.futt.news@laposte-dot-net.invalid> - 2011-05-23 22:03 +0200
          Re: analysis of java application logs Michael Wojcik <mwojcik@newsguy.com> - 2011-05-26 14:43 -0400
        Re: analysis of java application logs Arved Sandstrom <asandstrom3minus1@eastlink.ca> - 2011-05-23 18:32 -0300
          Re: analysis of java application logs Ulrich Scholz <d7@thispla.net> - 2011-05-25 06:00 -0700
            Re: analysis of java application logs Arved Sandstrom <asandstrom3minus1@eastlink.ca> - 2011-05-25 19:04 -0300
        Re: analysis of java application logs Martin Gregorie <martin@address-in-sig.invalid> - 2011-05-23 22:25 +0000
          Re: analysis of java application logs Nigel Wade <nmw-news@ion.le.ac.uk> - 2011-05-24 12:26 +0100
            Re: analysis of java application logs Martin Gregorie <martin@address-in-sig.invalid> - 2011-05-24 12:29 +0000
              Re: analysis of java application logs Lew <noone@lewscanon.com> - 2011-05-24 08:49 -0400
                Re: analysis of java application logs Martin Gregorie <martin@address-in-sig.invalid> - 2011-05-24 14:37 +0000
                Re: analysis of java application logs Lew <noone@lewscanon.com> - 2011-05-24 12:26 -0400
                Re: analysis of java application logs Jim Gibson <jimsgibson@gmail.com> - 2011-05-24 11:00 -0700
                Re: analysis of java application logs Lew <noone@lewscanon.com> - 2011-05-24 14:35 -0400
                Re: analysis of java application logs Nigel Wade <nmw-news@ion.le.ac.uk> - 2011-05-25 09:53 +0100
            Re: analysis of java application logs Daniele Futtorovic <da.futt.news@laposte-dot-net.invalid> - 2011-05-24 19:12 +0200
  Re: analysis of java application logs Patricia Shanahan <pats@acm.org> - 2011-05-23 06:17 -0700
    Re: analysis of java application logs Robert Klemme <shortcutter@googlemail.com> - 2011-05-23 20:33 +0200
      Re: analysis of java application logs Martin Gregorie <martin@address-in-sig.invalid> - 2011-05-23 19:07 +0000
  Re: analysis of java application logs CncShipper <anon@nowhere.com> - 2011-05-23 14:56 +0000
    Re: analysis of java application logs Lew <noone@lewscanon.com> - 2011-05-23 11:43 -0400
      Re: analysis of java application logs jlp <jlp@jlp.com> - 2011-05-23 18:00 +0200
        Re: analysis of java application logs Lew <noone@lewscanon.com> - 2011-05-23 12:20 -0400
          Re: analysis of java application logs Daniele Futtorovic <da.futt.news@laposte-dot-net.invalid> - 2011-05-23 19:06 +0200
            Re: analysis of java application logs Robert Klemme <shortcutter@googlemail.com> - 2011-05-23 20:27 +0200
              Re: analysis of java application logs Daniele Futtorovic <da.futt.news@laposte-dot-net.invalid> - 2011-05-23 21:02 +0200
              Re: analysis of java application logs Lew <noone@lewscanon.com> - 2011-05-23 15:06 -0400
                Re: analysis of java application logs Robert Klemme <shortcutter@googlemail.com> - 2011-05-23 22:10 +0200
                Re: analysis of java application logs Lew <noone@lewscanon.com> - 2011-05-23 17:04 -0400
  Re: analysis of java application logs Tom Anderson <twic@urchin.earth.li> - 2011-05-24 15:04 +0100
    Re: analysis of java application logs Martin Gregorie <martin@address-in-sig.invalid> - 2011-05-24 14:50 +0000
      Re: analysis of java application logs Michael Wojcik <mwojcik@newsguy.com> - 2011-05-26 14:58 -0400
      Re: analysis of java application logs Lawrence D'Oliveiro <ldo@geek-central.gen.new_zealand> - 2011-05-30 16:23 +1200
        Re: analysis of java application logs Lew <noone@lewscanon.com> - 2011-05-30 01:08 -0400

csiph-web