Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #4442 > unrolled thread

analysis of java application logs

Back to article view | Back to comp.lang.java.programmer

Contents

  analysis of java application logs Ulrich Scholz <d7@thispla.net> - 2011-05-23 00:50 -0700
    Re: analysis of java application logs Robert Klemme <shortcutter@googlemail.com> - 2011-05-23 02:20 -0700
    Re: analysis of java application logs jlp <jlp@jlp.com> - 2011-05-23 11:45 +0200
    Re: analysis of java application logs Lew <noone@lewscanon.com> - 2011-05-23 09:11 -0400
      Re: analysis of java application logs Daniele Futtorovic <da.futt.news@laposte-dot-net.invalid> - 2011-05-23 19:16 +0200
        Re: analysis of java application logs Lew <noone@lewscanon.com> - 2011-05-23 15:02 -0400
          Re: analysis of java application logs Daniele Futtorovic <da.futt.news@laposte-dot-net.invalid> - 2011-05-23 22:03 +0200
            Re: analysis of java application logs Michael Wojcik <mwojcik@newsguy.com> - 2011-05-26 14:43 -0400
          Re: analysis of java application logs Arved Sandstrom <asandstrom3minus1@eastlink.ca> - 2011-05-23 18:32 -0300
            Re: analysis of java application logs Ulrich Scholz <d7@thispla.net> - 2011-05-25 06:00 -0700
              Re: analysis of java application logs Arved Sandstrom <asandstrom3minus1@eastlink.ca> - 2011-05-25 19:04 -0300
          Re: analysis of java application logs Martin Gregorie <martin@address-in-sig.invalid> - 2011-05-23 22:25 +0000
            Re: analysis of java application logs Nigel Wade <nmw-news@ion.le.ac.uk> - 2011-05-24 12:26 +0100
              Re: analysis of java application logs Martin Gregorie <martin@address-in-sig.invalid> - 2011-05-24 12:29 +0000
                Re: analysis of java application logs Lew <noone@lewscanon.com> - 2011-05-24 08:49 -0400
                  Re: analysis of java application logs Martin Gregorie <martin@address-in-sig.invalid> - 2011-05-24 14:37 +0000
                    Re: analysis of java application logs Lew <noone@lewscanon.com> - 2011-05-24 12:26 -0400
                      Re: analysis of java application logs Jim Gibson <jimsgibson@gmail.com> - 2011-05-24 11:00 -0700
                        Re: analysis of java application logs Lew <noone@lewscanon.com> - 2011-05-24 14:35 -0400
                          Re: analysis of java application logs Nigel Wade <nmw-news@ion.le.ac.uk> - 2011-05-25 09:53 +0100
              Re: analysis of java application logs Daniele Futtorovic <da.futt.news@laposte-dot-net.invalid> - 2011-05-24 19:12 +0200
    Re: analysis of java application logs Patricia Shanahan <pats@acm.org> - 2011-05-23 06:17 -0700
      Re: analysis of java application logs Robert Klemme <shortcutter@googlemail.com> - 2011-05-23 20:33 +0200
        Re: analysis of java application logs Martin Gregorie <martin@address-in-sig.invalid> - 2011-05-23 19:07 +0000
    Re: analysis of java application logs CncShipper <anon@nowhere.com> - 2011-05-23 14:56 +0000
      Re: analysis of java application logs Lew <noone@lewscanon.com> - 2011-05-23 11:43 -0400
        Re: analysis of java application logs jlp <jlp@jlp.com> - 2011-05-23 18:00 +0200
          Re: analysis of java application logs Lew <noone@lewscanon.com> - 2011-05-23 12:20 -0400
            Re: analysis of java application logs Daniele Futtorovic <da.futt.news@laposte-dot-net.invalid> - 2011-05-23 19:06 +0200
              Re: analysis of java application logs Robert Klemme <shortcutter@googlemail.com> - 2011-05-23 20:27 +0200
                Re: analysis of java application logs Daniele Futtorovic <da.futt.news@laposte-dot-net.invalid> - 2011-05-23 21:02 +0200
                Re: analysis of java application logs Lew <noone@lewscanon.com> - 2011-05-23 15:06 -0400
                  Re: analysis of java application logs Robert Klemme <shortcutter@googlemail.com> - 2011-05-23 22:10 +0200
                    Re: analysis of java application logs Lew <noone@lewscanon.com> - 2011-05-23 17:04 -0400
    Re: analysis of java application logs Tom Anderson <twic@urchin.earth.li> - 2011-05-24 15:04 +0100
      Re: analysis of java application logs Martin Gregorie <martin@address-in-sig.invalid> - 2011-05-24 14:50 +0000
        Re: analysis of java application logs Michael Wojcik <mwojcik@newsguy.com> - 2011-05-26 14:58 -0400
        Re: analysis of java application logs Lawrence D'Oliveiro <ldo@geek-central.gen.new_zealand> - 2011-05-30 16:23 +1200
          Re: analysis of java application logs Lew <noone@lewscanon.com> - 2011-05-30 01:08 -0400

Page 2 of 2 — ← Prev page 1 [2]

#4542

On 24/05/2011 13:26, Nigel Wade allegedly wrote:
> On 23/05/11 23:25, Martin Gregorie wrote:
>> I tend to use at least two logging streams: debugging and
>> operational. I leave debugging statements in production code: its
>> normally off (of course) but can be turned on if needed.
>
> There is one caveat to leaving debug logging in production code; it
> may affect performance. Even with output disabled the string
> arguments are still constructed, unless they are constant. If logging
> is located in a tight loop, or critical section, it might become
> significant.
>
> In one particular application I profiled I found this to be eating
> up 80% of the CPU time. Admittedly, this was a rather special case,
> but it's still something which ought to be borne in mind if you leave
> debug logging code in production software.
>

Use a print-eff'ing logger and you'll have next to no overhead. If
evaluating the arguments is too heavy, use the ugly
   if( logger.isXXXEnabled() ){}
jump.

-- 
DF.
An escaped convict once said to me:
"Alcatraz is the place to be"

[toc] | [prev] | [next] | [standalone]

#4450

On 5/23/2011 12:50 AM, Ulrich Scholz wrote:
> Hi,
>
> I'm looking for an approach to the problem of analyzing application
> log files.
>
> I need to analyse Java log files from applications (i.e., not logs of
> web servers). These logs contain Java exceptions, thread dumps, and
> free-form log4j messages issued by log statements inserted by
> programmers during development. Right now, these man-made log entries
> do not have any specific format.
>
> What I'm looking for is a tool and/or strategy that supports in lexing/
> parsing, tagging, and analysing the log entries. Because there is only
> little defined syntax and grammar - and because you might not know
> what you are looking for - the task requires the quick issuing of
> queries against the log data base. Some sort of visualization would be
> nice, too.
>
> Pointers to existing tools and approaches as well as appropriate tools/
> algorithms to develop the required system would be welcome.

I would use Perl, and begin by recognizing some of the more important
formats, such as thread dumps. I agree with the desirability of
introducing some organized formatting into the log messages, but an
ad-hoc Perl program can often get useful data out of a disorganized log.

Patricia

[toc] | [prev] | [next] | [standalone]

#4479

On 23.05.2011 15:17, Patricia Shanahan wrote:
> On 5/23/2011 12:50 AM, Ulrich Scholz wrote:
>> I'm looking for an approach to the problem of analyzing application
>> log files.
>>
>> I need to analyse Java log files from applications (i.e., not logs of
>> web servers). These logs contain Java exceptions, thread dumps, and
>> free-form log4j messages issued by log statements inserted by
>> programmers during development. Right now, these man-made log entries
>> do not have any specific format.
>>
>> What I'm looking for is a tool and/or strategy that supports in lexing/
>> parsing, tagging, and analysing the log entries. Because there is only
>> little defined syntax and grammar - and because you might not know
>> what you are looking for - the task requires the quick issuing of
>> queries against the log data base. Some sort of visualization would be
>> nice, too.
>>
>> Pointers to existing tools and approaches as well as appropriate tools/
>> algorithms to develop the required system would be welcome.
>
> I would use Perl, and begin by recognizing some of the more important
> formats, such as thread dumps. I agree with the desirability of
> introducing some organized formatting into the log messages, but an
> ad-hoc Perl program can often get useful data out of a disorganized log.

Only that Perl is so awful - YMMV of course.  But for these kinds of 
tasks (more correctly: for *any* task) I very much prefer to use Ruby 
because of its cleaner OO and cleaner syntax.  In these cases where the 
basic format is fixed I place general parsing code in a library (a 
single file really) and then I can write ad hoc scripts which do 
arbitrary processing of the data.  That's very productive.

Kind regards

	robert

-- 
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

[toc] | [prev] | [next] | [standalone]

#4485

On Mon, 23 May 2011 20:33:07 +0200, Robert Klemme wrote:

> On 23.05.2011 15:17, Patricia Shanahan wrote:
>> On 5/23/2011 12:50 AM, Ulrich Scholz wrote:
>>> I'm looking for an approach to the problem of analyzing application
>>> log files.
>>>
>>> I need to analyse Java log files from applications (i.e., not logs of
>>> web servers). These logs contain Java exceptions, thread dumps, and
>>> free-form log4j messages issued by log statements inserted by
>>> programmers during development. Right now, these man-made log entries
>>> do not have any specific format.
>>>
>>> What I'm looking for is a tool and/or strategy that supports in
>>> lexing/ parsing, tagging, and analysing the log entries. Because there
>>> is only little defined syntax and grammar - and because you might not
>>> know what you are looking for - the task requires the quick issuing of
>>> queries against the log data base. Some sort of visualization would be
>>> nice, too.
>>>
>>> Pointers to existing tools and approaches as well as appropriate
>>> tools/ algorithms to develop the required system would be welcome.
>>
>> I would use Perl, and begin by recognizing some of the more important
>> formats, such as thread dumps. I agree with the desirability of
>> introducing some organized formatting into the log messages, but an
>> ad-hoc Perl program can often get useful data out of a disorganized
>> log.
> 
> Only that Perl is so awful - YMMV of course.  But for these kinds of
> tasks (more correctly: for *any* task) I very much prefer to use Ruby
> because of its cleaner OO and cleaner syntax.
>
I do the same, but use gawk rather than Perl: I have the same objections 
to Perl as you, while gawk is pretty straight forward if you understand 
regexes and can write C.

So far, using gawk to extract the information I've needed from Linux 
system logs has been rather straight forward. Besides, I generally find 
gawk to be more concise and readable than Perl, for this type of job, 
anyway.


-- 
martin@   | Martin Gregorie
gregorie. | Essex, UK
org       |

[toc] | [prev] | [next] | [standalone]

#4455

I wrote one of these and thought about Open Sourcing it, but lost 
interest.  I parsed the logs into a db, and assigned id's to the 
various fields.

You could then search by Type, ( WARNING, SEVERE, etc... )
You could search a range of times
It could handle multiple log files into one run
could Sync on an event and stop analyzing on another trigger
Graphs to count trends, events, exceptions
Used Reg-Ex a heck of a lot of work.. Sorted all the transactions in 
the logs, so you could also display by package name, really helped 
me solve a lot of problems when I was working .. took me nearly two 
years to complete everything to where it is today..

I never found a package that even came close to it.. which is why I 
wrote it



-- 
--------------------------------- --- -- -
Posted with NewsLeecher v5.0 Beta 1
Web @ http://www.newsleecher.com/?usenet
------------------- ----- ---- -- -

[toc] | [prev] | [next] | [standalone]

#4462

CncShipper wrote:
> I wrote one of these and thought about Open Sourcing it, but lost

"open sourcing"

> interest.  I parsed the logs into a db, and assigned id's to the

"DB, "IDs" (no greengrocer's apostrophe, and "id" is a different word from 
"ID", although the meaning you imputed by the substitution is poetic and 
interesting)

> various fields.
>
> You could then search by Type, ( WARNING, SEVERE, etc... )

"type"

> You could search a range of times
> It could handle multiple log files into one run
> could Sync on an event and stop analyzing on another trigger

"synch"

> Graphs to count trends, events, exceptions

"graphs"

> Used Reg-Ex a heck of a lot of work.. Sorted all the transactions in

"used" "regex"

> the logs, so you could also display by package name, really helped
> me solve a lot of problems when I was working .. took me nearly two
> years to complete everything to where it is today..

Double-dot, or two consecutive periods, is not legitimate punctuation in lieu 
of a comma or full stop.

> I never found a package that even came close to it.. which is why I
> wrote it

You have made an important and useful point.  Covering for a bad log format is 
a freeform-text parsing problem, inherently difficult and heuristic and 
probably never perfect.  I wonder if your effort would have been better spent 
converting to a log format that is parser-friendly, as the OP should do.

-- 
Lew
Honi soit qui mal y pense.
http://upload.wikimedia.org/wikipedia/commons/c/cf/Friz.jpg

[toc] | [prev] | [next] | [standalone]

#4465

Le 23/05/2011 17:43, Lew a écrit :
  [SNIP]
> You have made an important and useful point. Covering for a bad log
> format is a freeform-text parsing problem, inherently difficult and
> heuristic and probably never perfect. I wonder if your effort would have
> been better spent converting to a log format that is parser-friendly, as
> the OP should do.
>
I agree with you, Lee, it is what i did with my own tool. Native logs 
are converted in CSV files. But some logs are not simple to convert :
- java exceptions
- java threads dumps ( different for every JVM : Sun/Oracle, JRockit, 
IBM ...)
- java heap dump summary ( same remark)
- verbose GC logs (same remark)
- multi-lines log enregistrement (xml logs ...)

Others are more simple :
- acces logs that are Common Log Format ( CLF) or CLF extended compliant 
( Apache, Tomcat, IIS, WebLogic, Websphere ...)
- Log4J

[toc] | [prev] | [next] | [standalone]

#4469

jlp wrote:
> I agree with you, Lee, it is what i [sic] did with my own tool. Native logs are

Who's Lee?

-- 
Lew
The first-person singular pronoun in English is spelled "I", not "i". It's 
only one letter long, so it should be possible to spell it correctly.  This is 
one of the first lessons in an EFL course, so it should come as no surprise.

[toc] | [prev] | [next] | [standalone]

#4471

On 23/05/2011 18:20, Lew allegedly wrote:
> jlp wrote:
>> I agree with you, Lee, it is what i [sic] did with my own tool. Native
>> logs are
>
> Who's Lee?
>

You're Lee now, Lee.

[toc] | [prev] | [next] | [standalone]

#4477

On 23.05.2011 19:06, Daniele Futtorovic wrote:
> On 23/05/2011 18:20, Lew allegedly wrote:
>> jlp wrote:
>>> I agree with you, Lee, it is what i [sic] did with my own tool. Native
>>> logs are
>>
>> Who's Lee?
>>
>
> You're Lee now, Lee.

Did you mean to say "Bruce"?
;-)

	robert

-- 
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

[toc] | [prev] | [next] | [standalone]

#4483

On 23/05/2011 20:27, Robert Klemme allegedly wrote:
> On 23.05.2011 19:06, Daniele Futtorovic wrote:
>> On 23/05/2011 18:20, Lew allegedly wrote:
>>> jlp wrote:
>>>> I agree with you, Lee, it is what i [sic] did with my own tool. Native
>>>> logs are
>>>
>>> Who's Lee?
>>>
>>
>> You're Lee now, Lee.
>
> Did you mean to say "Bruce"?
> ;-)

Wait. I thought Lee was a Yank, not an Aussie? :)

[toc] | [prev] | [next] | [standalone]

#4484

On 05/23/2011 02:27 PM, Robert Klemme wrote:
> On 23.05.2011 19:06, Daniele Futtorovic wrote:
>> On 23/05/2011 18:20, Lew allegedly wrote:
>>> jlp wrote:
>>>> I agree with you, Lee, it is what i [sic] did with my own tool. Native
>>>> logs are
>>>
>>> Who's Lee?
>>>
>>
>> You're Lee now, Lee.
>
> Did you mean to say "Bruce"?
> ;-)

Just call me Lew Lee.  Then you can sing, "Lew Lee.  Lew, lay, thou little 
tiny child.  Bye-bye, Lew Lee.  Lew, lay!"

-- 
Lew
Honi soit qui mal y pense.
http://upload.wikimedia.org/wikipedia/commons/c/cf/Friz.jpg

[toc] | [prev] | [next] | [standalone]

#4492

On 23.05.2011 21:06, Lew wrote:
> On 05/23/2011 02:27 PM, Robert Klemme wrote:
>> On 23.05.2011 19:06, Daniele Futtorovic wrote:
>>> On 23/05/2011 18:20, Lew allegedly wrote:
>>>> jlp wrote:
>>>>> I agree with you, Lee, it is what i [sic] did with my own tool. Native
>>>>> logs are
>>>>
>>>> Who's Lee?
>>>>
>>>
>>> You're Lee now, Lee.
>>
>> Did you mean to say "Bruce"?
>> ;-)
>
> Just call me Lew Lee. Then you can sing, "Lew Lee. Lew, lay, thou little
> tiny child. Bye-bye, Lew Lee. Lew, lay!"

Aye!

Which reminds me of a guy who called himself "LL Cool J". ;-)

Associative memory...

	robert

-- 
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

[toc] | [prev] | [next] | [standalone]

#4495

On 05/23/2011 04:10 PM, Robert Klemme wrote:
> On 23.05.2011 21:06, Lew wrote:
>> On 05/23/2011 02:27 PM, Robert Klemme wrote:
>>> On 23.05.2011 19:06, Daniele Futtorovic wrote:
>>>> On 23/05/2011 18:20, Lew allegedly wrote:
>>>>> jlp wrote:
>>>>>> I agree with you, Lee, it is what i [sic] did with my own tool. Native
>>>>>> logs are
>>>>>
>>>>> Who's Lee?
>>>>>
>>>>
>>>> You're Lee now, Lee.
>>>
>>> Did you mean to say "Bruce"?
>>> ;-)
>>
>> Just call me Lew Lee. Then you can sing, "Lew Lee. Lew, lay, thou little
>> tiny child. Bye-bye, Lew Lee. Lew, lay!"
>
> Aye!
>
> Which reminds me of a guy who called himself "LL Cool J". ;-)
>
> Associative memory...

Wow, I feel lucky.  I was so afraid I'd be put in Coventry for that pun.

-- 
Lew
Honi soit qui mal y pense.
http://upload.wikimedia.org/wikipedia/commons/c/cf/Friz.jpg

[toc] | [prev] | [next] | [standalone]

#4535

On Mon, 23 May 2011, Ulrich Scholz wrote:

> I'm looking for an approach to the problem of analyzing application
> log files.
>
> I need to analyse Java log files from applications (i.e., not logs of
> web servers). These logs contain Java exceptions, thread dumps, and
> free-form log4j messages issued by log statements inserted by
> programmers during development. Right now, these man-made log entries
> do not have any specific format.
>
> What I'm looking for is a tool and/or strategy that supports in lexing/
> parsing, tagging, and analysing the log entries. Because there is only
> little defined syntax and grammar - and because you might not know
> what you are looking for - the task requires the quick issuing of
> queries against the log data base. Some sort of visualization would be
> nice, too.
>
> Pointers to existing tools and approaches as well as appropriate tools/
> algorithms to develop the required system would be welcome.

grep.

For particularly thorny bits of analysis, sed.

There are those who advocate awk for this sort of thing, but frankly that 
seems like overkill.

tom

-- 
shouting drunkenly about 6502 assembler at parties

[toc] | [prev] | [next] | [standalone]

#4537

On Tue, 24 May 2011 15:04:41 +0100, Tom Anderson wrote:

> There are those who advocate awk for this sort of thing, but frankly
> that seems like overkill.
> 
Depends on your background, I suppose. 

I was using FIND 2, NCC Filetab and (shudder) RPG3 long before I met awk, 
so I was familiar with the sort of scriptable file processing utilities 
that have a built-in record reading loop and a means of triggering 
actions that match particular record types. As a result I felt right at 
home with awk, which is about the easiest to use of all the ones I've 
tried.

Besides, I very often find I need to calculate maxima, minima, averages, 
etc as part of the log analysis and find that very easy to do with awk.


-- 
martin@   | Martin Gregorie
gregorie. | Essex, UK
org       |

[toc] | [prev] | [next] | [standalone]

#4613

Martin Gregorie wrote:
> On Tue, 24 May 2011 15:04:41 +0100, Tom Anderson wrote:
> 
>> There are those who advocate awk for this sort of thing, but frankly
>> that seems like overkill.
> 
> Besides, I very often find I need to calculate maxima, minima, averages, 
> etc as part of the log analysis and find that very easy to do with awk.

Yes, sometimes it's useful to maintain state while analyzing logs, and
that's more convenient with awk (or any of a thousand other scripting
languages) than it is with sed.

A couple of months back I tracked down some tricky locking issues
(multiple multi-threaded processes, with multiple components written
by multiple developers in multiple languages) by analyzing lock files
with an awk script; it made it easy to keep track of contention,
reentrant locking, number of waiters, etc on a per-lock basis.

Back in '91 I demonstrated a bug in AIX 3.1 libc's implementation of
free by using awk to turn a malloc/free log into a series of cut-down
C programs that duplicated the allocation access pattern, until I had
a minimal test program that demonstrated the problem.

Come to think of it, just a few weeks ago I was using a little awk
script to churn through the output from my various
sentiment-determination UIMA annotators[1] and compute precision,
recall, and F1 values.

Obviously, I could do any of those tasks in Perl, or Python, or Java,
or COBOL, or what have you. I only use awk because I've been using it
for more than 20 years now and so it has the advantage of familiarity.


[1] ObCLJP: The annotators are written in Java.

-- 
Michael Wojcik
Micro Focus
Rhetoric & Writing, Michigan State University

[toc] | [prev] | [next] | [standalone]

#4712

In message <irggkc$k9d$2@localhost.localdomain>, Martin Gregorie wrote:

> Besides, I very often find I need to calculate maxima, minima, averages,
> etc as part of the log analysis and find that very easy to do with awk.

perl can do everything awk can do, and more, and do it just as concisely.

[toc] | [prev] | [next] | [standalone]

#4713

On 05/30/2011 12:23 AM, Lawrence D'Oliveiro wrote:
> In message<irggkc$k9d$2@localhost.localdomain>, Martin Gregorie wrote:
>
>> Besides, I very often find I need to calculate maxima, minima, averages,
>> etc as part of the log analysis and find that very easy to do with awk.
>
> perl can do everything awk can do, and more, and do it just as concisely.

Assembler can do anything perl or awk can do, and do it much more efficiently, 
Lawrence.

Just out of curiosity, Lawrence, what's the percentage for you to continually 
try to trigger these Language Wars?  What do you get out of it, Lawrence?

Especially since you know as well as everyone here that perl and awk are toy 
languages compared to the One True Language, right, Lawrence?

Come on, Lawrence, the gauntlet has been thrown, Lawrence.  Let's see what 
you've got, Lawrence.  The One True Language vs. perl and awk combined in the 
tag-team Match of the Century(r) (tm) (k).  Are you man enough to take it, 
Lawrence?

I advise you to just give up now, Lawrence, and face the fact that the One 
True Language is so far and away superior to awk, perl, and yes, I'm sorry, 
Lawrence, but even Python - despite your passeltongue fluency, The One True 
Language rules, Lawrence, and your favorite languages drool, Lawrence..

So, Lawrence, man up and respond, Lawrence, as the world awaits your wisdom.

Lawrence.

-- 
Lew
Honi soit qui mal y pense.
http://upload.wikimedia.org/wikipedia/commons/c/cf/Friz.jpg

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.lang.java.programmer

csiph-web