Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #4472

Re: analysis of java application logs

Show all headers | View raw

On 23/05/2011 15:11, Lew allegedly wrote:
> Ulrich Scholz wrote:
>> I'm looking for an approach to the problem of analyzing application
>> log files.
>>
>> I need to analyse Java log files from applications (i.e., not logs of
>> web servers). These logs contain Java exceptions, thread dumps, and
>> free-form log4j messages issued by log statements inserted by
>> programmers during development. Right now, these man-made log entries
>> do not have any specific format.
>>
>> What I'm looking for is a tool and/or strategy that supports in lexing/
>> parsing, tagging, and analysing the log entries. Because there is only
>> little defined syntax and grammar - and because you might not know
>> what you are looking for - the task requires the quick issuing of
>> queries against the log data base. Some sort of visualization would be
>> nice, too.
>>
>> Pointers to existing tools and approaches as well as appropriate tools/
>> algorithms to develop the required system would be welcome.
>
> It helps if you have a logging strategy that mandates a consistent
> logging format, specific information in particular positions or marked
> by particular markup, logging levels and other such so that your
> analysis tool isn't faced with a completely open-ended input. What you
> describe requires a general text-analysis approach, as you indicate that
> you can make no guarantees about the format. Based on that, your best
> tool is "less" or equivalent text-file reader.
>
> What is a tool supposed to do, read your mind?
>
> It's really hard to extract information from a garbage can where people
> just randomly dumped whatever they individually felt like dumping
> without regard for operational needs. You can't build a skyscraper on a
> bad foundation, and you can't build a good log analysis off a crappy log.
>
> Fix the logging system, then the analysis problem will be tractable.
>

I would argue around the same lines.

I've been faced a while ago with a situation where some orthogonal
organisational unit wanted to exploit my logs. I told them to GTFO.

My logs are my logs. I put in it what I consider necessary. I often
improve them as I step through the code. I might change the message, fix
the level, &c. I don't want to have them set in stone. Neither do I
generally have enough confidence in them to allow them to be used for
analysis.

"The solution, then, is simple", I told them, "spec out the exact
messages and arguments you want, and the exact situations you want them
logged in, and I'll add them for you. But leave me my precious debugging
logs."

Let me emphasize: IMHO debugging logs and logs for analysis are two
different things and should be kept strictly separated -- possibly
logged to a different target respectively.

-- 
DF.
An escaped convict once said to me:
"Alcatraz is the place to be"

Back to comp.lang.java.programmer | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

analysis of java application logs Ulrich Scholz <d7@thispla.net> - 2011-05-23 00:50 -0700
  Re: analysis of java application logs Robert Klemme <shortcutter@googlemail.com> - 2011-05-23 02:20 -0700
  Re: analysis of java application logs jlp <jlp@jlp.com> - 2011-05-23 11:45 +0200
  Re: analysis of java application logs Lew <noone@lewscanon.com> - 2011-05-23 09:11 -0400
    Re: analysis of java application logs Daniele Futtorovic <da.futt.news@laposte-dot-net.invalid> - 2011-05-23 19:16 +0200
      Re: analysis of java application logs Lew <noone@lewscanon.com> - 2011-05-23 15:02 -0400
        Re: analysis of java application logs Daniele Futtorovic <da.futt.news@laposte-dot-net.invalid> - 2011-05-23 22:03 +0200
          Re: analysis of java application logs Michael Wojcik <mwojcik@newsguy.com> - 2011-05-26 14:43 -0400
        Re: analysis of java application logs Arved Sandstrom <asandstrom3minus1@eastlink.ca> - 2011-05-23 18:32 -0300
          Re: analysis of java application logs Ulrich Scholz <d7@thispla.net> - 2011-05-25 06:00 -0700
            Re: analysis of java application logs Arved Sandstrom <asandstrom3minus1@eastlink.ca> - 2011-05-25 19:04 -0300
        Re: analysis of java application logs Martin Gregorie <martin@address-in-sig.invalid> - 2011-05-23 22:25 +0000
          Re: analysis of java application logs Nigel Wade <nmw-news@ion.le.ac.uk> - 2011-05-24 12:26 +0100
            Re: analysis of java application logs Martin Gregorie <martin@address-in-sig.invalid> - 2011-05-24 12:29 +0000
              Re: analysis of java application logs Lew <noone@lewscanon.com> - 2011-05-24 08:49 -0400
                Re: analysis of java application logs Martin Gregorie <martin@address-in-sig.invalid> - 2011-05-24 14:37 +0000
                Re: analysis of java application logs Lew <noone@lewscanon.com> - 2011-05-24 12:26 -0400
                Re: analysis of java application logs Jim Gibson <jimsgibson@gmail.com> - 2011-05-24 11:00 -0700
                Re: analysis of java application logs Lew <noone@lewscanon.com> - 2011-05-24 14:35 -0400
                Re: analysis of java application logs Nigel Wade <nmw-news@ion.le.ac.uk> - 2011-05-25 09:53 +0100
            Re: analysis of java application logs Daniele Futtorovic <da.futt.news@laposte-dot-net.invalid> - 2011-05-24 19:12 +0200
  Re: analysis of java application logs Patricia Shanahan <pats@acm.org> - 2011-05-23 06:17 -0700
    Re: analysis of java application logs Robert Klemme <shortcutter@googlemail.com> - 2011-05-23 20:33 +0200
      Re: analysis of java application logs Martin Gregorie <martin@address-in-sig.invalid> - 2011-05-23 19:07 +0000
  Re: analysis of java application logs CncShipper <anon@nowhere.com> - 2011-05-23 14:56 +0000
    Re: analysis of java application logs Lew <noone@lewscanon.com> - 2011-05-23 11:43 -0400
      Re: analysis of java application logs jlp <jlp@jlp.com> - 2011-05-23 18:00 +0200
        Re: analysis of java application logs Lew <noone@lewscanon.com> - 2011-05-23 12:20 -0400
          Re: analysis of java application logs Daniele Futtorovic <da.futt.news@laposte-dot-net.invalid> - 2011-05-23 19:06 +0200
            Re: analysis of java application logs Robert Klemme <shortcutter@googlemail.com> - 2011-05-23 20:27 +0200
              Re: analysis of java application logs Daniele Futtorovic <da.futt.news@laposte-dot-net.invalid> - 2011-05-23 21:02 +0200
              Re: analysis of java application logs Lew <noone@lewscanon.com> - 2011-05-23 15:06 -0400
                Re: analysis of java application logs Robert Klemme <shortcutter@googlemail.com> - 2011-05-23 22:10 +0200
                Re: analysis of java application logs Lew <noone@lewscanon.com> - 2011-05-23 17:04 -0400
  Re: analysis of java application logs Tom Anderson <twic@urchin.earth.li> - 2011-05-24 15:04 +0100
    Re: analysis of java application logs Martin Gregorie <martin@address-in-sig.invalid> - 2011-05-24 14:50 +0000
      Re: analysis of java application logs Michael Wojcik <mwojcik@newsguy.com> - 2011-05-26 14:58 -0400
      Re: analysis of java application logs Lawrence D'Oliveiro <ldo@geek-central.gen.new_zealand> - 2011-05-30 16:23 +1200
        Re: analysis of java application logs Lew <noone@lewscanon.com> - 2011-05-30 01:08 -0400

csiph-web