Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #30018 > unrolled thread

log4j socket appender

Back to article view | Back to comp.lang.java.programmer

Contents

  log4j socket appender doug.farrell@gmail.com - 2014-09-19 13:34 -0700
    Re: log4j socket appender Eric Sosman <esosman@comcast-dot-net.invalid> - 2014-09-19 16:48 -0400
      Re: log4j socket appender markspace <markspace@nospam.nospam> - 2014-09-19 17:17 -0700
        Re: log4j socket appender doug.farrell@gmail.com - 2014-09-22 08:29 -0700
          Re: log4j socket appender Nigel Wade <nmw@ion.le.ac.uk> - 2014-09-23 12:35 +0100
            Re: log4j socket appender doug.farrell@gmail.com - 2014-10-02 06:23 -0700
    Re: log4j socket appender Marcel Mueller <news.5.maazl@spamgourmet.org> - 2014-09-20 11:46 +0200
      Re: log4j socket appender Kevin McMurtrie <mcmurtrie@pixelmemory.us> - 2014-09-20 10:17 -0700
        Re: log4j socket appender markspace <markspace@nospam.nospam> - 2014-09-20 13:29 -0700
          Re: log4j socket appender Jeff Higgins <jeff@invalid.invalid> - 2014-09-20 23:05 -0400

#30018 — log4j socket appender

Hi all,

I'm not a Java developer (work primarily in Python and JavaScript), but I need some help. I've written a logging server that handles log records sent via UDP, and I'd like to do the same thing with Java. I've got a Java test harness built that looks like this:

import org.apache.logging.log4j.Logger;
import org.apache.logging.log4j.LogManager;

public class HelloWorld {
	
	/* get an actual logger */
	private static final Logger logger = LogManager.getLogger("HelloWorld");
		
	public static void main(String[] args) {
		logger.info("Info level message");
	}
}

and has a log4j.xml configuration file that looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
  <Appenders>
    <Console name="STDOUT" target="SYSTEM_OUT">
      <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
    </Console>
    <Socket name="SOCKET" host="localhost" port="9023" protocol="UDP">
    	<SerializedLayout />
    </Socket>
  </Appenders>
  <Loggers>
    <Root level="info">
      <AppenderRef ref="STDOUT"/>
      <AppenderRef ref="SOCKET"/>
    </Root>
  </Loggers>
</Configuration>

This works and sends log records from the Java test program to my Python logging server. However, I can't figure out what the format of the data received by the Python logging server is? This is the data I get back in my handler:

'\xac\xed\x00\x05sr\x00>org.apache.logging.log4j.core.impl.Log4jLogEvent$LogEventProxy\x9c\xed\x0f]p\xda*\x16\x02\x00\rZ\x00\x0cisEndOfBatchZ\x00\x12isLocationRequiredJ\x00\ntimeMillisL\x00\ncontextMapt\x00\x0fLjava/util/Map;L\x00\x0ccontextStackt\x005Lorg/apache/logging/log4j/ThreadContext$ContextStack;L\x00\x05levelt\x00 Lorg/apache/logging/log4j/Level;L\x00\nloggerFQCNt\x00\x12Ljava/lang/String;L\x00\nloggerNameq\x00~\x00\x04L\x00\x06markert\x00!Lorg/apache/logging/log4j/Marker;L\x00\x07messaget\x00*Lorg/apache/logging/log4j/message/Message;L\x00\x06sourcet\x00\x1dLjava/lang/StackTraceElement;L\x00\nthreadNameq\x00~\x00\x04L\x00\x0bthrownProxyt\x003Lorg/apache/logging/log4j/core/impl/ThrowableProxy;xp\x00\x01\x00\x00\x01H\x8f\x9d\xfb>sr\x00\x1ejava.util.Collections$EmptyMapY6\x14\x85Z\xdc\xe7\xd0\x02\x00\x00xpsr\x00>org.apache.logging.log4j.ThreadContext$EmptyThreadContextStack\x00\x00\x00\x00\x00\x00\x00\x01\x02\x00\x00xpsr\x00\x1eorg.apache.logging.log4j.Level\x00\x00\x00\x00\x00\x18 \x1a\x02\x00\x03I\x00\x08intLevelL\x00\x04nameq\x00~\x00\x04L\x00\rstandardLevelt\x00,Lorg/apache/logging/log4j/spi/StandardLevel;xp\x00\x00\x01\x90t\x00\x04INFO~r\x00*org.apache.logging.log4j.spi.StandardLevel\x00\x00\x00\x00\x00\x00\x00\x00\x12\x00\x00xr\x00\x0ejava.lang.Enum\x00\x00\x00\x00\x00\x00\x00\x00\x12\x00\x00xpt\x00\x04INFOt\x00+org.apache.logging.log4j.spi.AbstractLoggert\x00\nHelloWorldpsr\x00.org.apache.logging.log4j.message.SimpleMessage\x8btM0`\xb7\xa2\xa8\x02\x00\x01L\x00\x07messageq\x00~\x00\x04xpt\x00\x12Info level messagesr\x00\x1bjava.lang.StackTraceElementa\t\xc5\x9a&6\xdd\x85\x02\x00\x04I\x00\nlineNumberL\x00\x0edeclaringClassq\x00~\x00\x04L\x00\x08fileNameq\x00~\x00\x04L\x00\nmethodNameq\x00~\x00\x04xp\x00\x00\x00\x0cq\x00~\x00\x17t\x00\x0fHelloWorld.javat\x00\x04maint\x00\x04mainpy'

I can see the data I want in there, but don't know how to parse this. Any hints, pointers or suggestions would be most welcome!

Thanks in advance,
Doug

[toc] | [next] | [standalone]

#30019

On 9/19/2014 4:34 PM, doug.farrell@gmail.com wrote:
> Hi all,
>
> I'm not a Java developer (work primarily in Python and JavaScript), but I need some help. I've written a logging server that handles log records sent via UDP, and I'd like to do the same thing with Java. I've got a Java test harness built that looks like this:
>
> import org.apache.logging.log4j.Logger;
> import org.apache.logging.log4j.LogManager;
>
> public class HelloWorld {
> 	
> 	/* get an actual logger */
> 	private static final Logger logger = LogManager.getLogger("HelloWorld");
> 		
> 	public static void main(String[] args) {
> 		logger.info("Info level message");
> 	}
> }
>
> and has a log4j.xml configuration file that looks like this:
>
> <?xml version="1.0" encoding="UTF-8"?>
> <Configuration status="WARN">
>    <Appenders>
>      <Console name="STDOUT" target="SYSTEM_OUT">
>        <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
>      </Console>
>      <Socket name="SOCKET" host="localhost" port="9023" protocol="UDP">
>      	<SerializedLayout />
>      </Socket>
>    </Appenders>
>    <Loggers>
>      <Root level="info">
>        <AppenderRef ref="STDOUT"/>
>        <AppenderRef ref="SOCKET"/>
>      </Root>
>    </Loggers>
> </Configuration>
>
> This works and sends log records from the Java test program to my Python logging server. However, I can't figure out what the format of the data received by the Python logging server is? This is the data I get back in my handler:
>
> '\xac\xed\x00\x05sr\x00>org.apache.logging.log4j.core.impl.Log4jLogEvent$LogEventProxy\x9c\xed\x0f]p\xda*\x16\x02\x00\rZ\x00\x0cisEndOfBatchZ\x00\x12isLocationRequiredJ\x00\ntimeMillisL\x00\ncontextMapt\x00\x0fLjava/util/Map;L\x00\x0ccontextStackt\x005Lorg/apache/logging/log4j/ThreadContext$ContextStack;L\x00\x05levelt\x00 Lorg/apache/logging/log4j/Level;L\x00\nloggerFQCNt\x00\x12Ljava/lang/String;L\x00\nloggerNameq\x00~\x00\x04L\x00\x06markert\x00!Lorg/apache/logging/log4j/Marker;L\x00\x07messaget\x00*Lorg/apache/logging/log4j/message/Message;L\x00\x06sourcet\x00\x1dLjava/lang/StackTraceElement;L\x00\nthreadNameq\x00~\x00\x04L\x00\x0bthrownProxyt\x003Lorg/apache/logging/log4j/core/impl/ThrowableProxy;xp\x00\x01\x00\x00\x01H\x8f\x9d\xfb>sr\x00\x1ejava.util.Collections$EmptyMapY6\x14\x85Z\xdc\xe7\xd0\x02\x00\x00xpsr\x00>org.apache.logging.log4j.ThreadContext$EmptyThreadContextStack\x00\x00\x00\x00\x00\x00\x00\x01\x02\x00\x00xpsr\x00\x1eorg.apache.logging.log4j.Level\x00\x00\x00\x00
\x00\x18 \x1a\x02\x00\x03I\x00\x08intLevelL\x00\x04nameq\x00~\x00\x04L\x00\rstandardLevelt\x00,Lorg/apache/logging/log4j/spi/StandardLevel;xp\x00\x00\x01\x90t\x00\x04INFO~r\x00*org.apache.logging.log4j.spi.StandardLevel\x00\x00\x00\x00\x00\x00\x00\x00\x12\x00\x00xr\x00\x0ejava.lang.Enum\x00\x00\x00\x00\x00\x00\x00\x00\x12\x00\x00xpt\x00\x04INFOt\x00+org.apache.logging.log4j.spi.AbstractLoggert\x00\nHelloWorldpsr\x00.org.apache.logging.log4j.message.SimpleMessage\x8btM0`\xb7\xa2\xa8\x02\x00\x01L\x00\x07messageq\x00~\x00\x04xpt\x00\x12Info level messagesr\x00\x1bjava.lang.StackTraceElementa\t\xc5\x9a&6\xdd\x85\x02\x00\x04I\x00\nlineNumberL\x00\x0edeclaringClassq\x00~\x00\x04L\x00\x08fileNameq\x00~\x00\x04L\x00\nmethodNameq\x00~\x00\x04xp\x00\x00\x00\x0cq\x00~\x00\x17t\x00\x0fHelloWorld.javat\x00\x04maint\x00\x04mainpy'
>
> I can see the data I want in there, but don't know how to parse this. Any hints, pointers or suggestions would be most welcome!

     (Disclaimer: I'm no expert on log4j, so read at your own risk...)

     You don't even want to think about parsing this data with anything
short of a full Java implementation.  This is "serialized" data, that
is, a data stream into which one JVM can pickle entire objects (and
their sub-objects, recursively) so another JVM can reconstruct clones
of the originals.  Serialization is mostly used for things like remote
procedure calls, where a JVM on machine 1 ships objects to a JVM on
machine 2, which operates on them and ships back results in a similar
fashion.  You don't want to parse this stuff.

     What you want, I think, is something other than SerializedLayout:
JsonLayout, maybe, or XmlLayout.

-- 
esosman@comcast-dot-net.invalid

[toc] | [prev] | [next] | [standalone]

#30020

On 9/19/2014 1:48 PM, Eric Sosman wrote:

>
>      (Disclaimer: I'm no expert on log4j, so read at your own risk...)

Ditto.

>
>      You don't even want to think about parsing this data with anything
> short of a full Java implementation.  This is "serialized" data, that
> is, a data stream into which one JVM can pickle entire objects (and
> their sub-objects, recursively) so another JVM can reconstruct clones
> of the originals.  Serialization is mostly used for things like remote
> procedure calls, where a JVM on machine 1 ships objects to a JVM on
> machine 2, which operates on them and ships back results in a similar
> fashion.  You don't want to parse this stuff.
>

Totally agree.


>      What you want, I think, is something other than SerializedLayout:
> JsonLayout, maybe, or XmlLayout.
>


I think I found the most recent documentation on these layouts here:

<http://logging.apache.org/log4j/2.x/manual/layouts.html>

Double check your version of log4j, there's still a lot of 1.x stuff 
floating around out there.

[toc] | [prev] | [next] | [standalone]

#30026

Hi all,

Thanks for the responses, I'm a complete Java novice, so struggling to follow along. However, the JsonLayout provided by the jackson-core JAR seems to apply to creating and reading configuration files in JSON format, not streaming output from a log4j (version 2.*) to a socket appender. Am I mistaken about this?

Thanks again,
Doug

[toc] | [prev] | [next] | [standalone]

#30028

On 22/09/14 16:29, doug.farrell@gmail.com wrote:
> Hi all,
>
> Thanks for the responses, I'm a complete Java novice, so struggling to follow along. However, the JsonLayout provided by the jackson-core JAR seems to apply to creating and reading configuration files in JSON format, not streaming output from a log4j (version 2.*) to a socket appender. Am I mistaken about this?
>
> Thanks again,
> Doug
>


I've never used the SocketAppender, and from looking at the API docs. I presume you are using log4j version 2,
previously it seems as though SocketAppender would only send serialized messages over TCP.

What format is your logging server expecting the log messages to be in? If you want a text format try using the same 
PatternLayout as you do for the ConsoleAppender.

-- 
Nigel Wade

[toc] | [prev] | [next] | [standalone]

#30051

Hi all,

This is the OP here. Sorry for some of the misinformation, yes, I'm using log4j version 2. I was also wrong about the Socket appender, it can in fact use various serializers to send data across the network. I settled on using JSON serialized output as that's fairly universal, and certainly easy enough for a Python program to consume. Here is my test harness that generates messages:


import org.apache.logging.log4j.Logger;
import org.apache.logging.log4j.LogManager;

public class HelloWorld {
	
	/* get an actual logger */
	private static final Logger logger = LogManager.getLogger("HelloWorld");
		
	public static void main(String[] args) {
		//logger.error("Hello World");
		logger.trace("Trace level message");
		logger.debug("Debug level message");
		logger.info("Info level message");
		logger.warn("Warn level message");
		logger.error("Error level message");
		logger.fatal("Fatal level message");
	}
}

And here is the configuration file that ties things together:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
  <Appenders>
    <Console name="STDOUT" target="SYSTEM_OUT">
      <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
    </Console>
    <Socket name="SOCKET" host="myservername.com" port="9023" protocol="UDP">
    	<JSONLayout charset="UTF-8" compact="true" />
    </Socket>
  </Appenders>
  <Loggers>
    <Root level="info">
      <AppenderRef ref="STDOUT"/>
      <AppenderRef ref="SOCKET"/>
    </Root>
  </Loggers>
</Configuration>


Part of my problem was knowing all of the jar files necessary to get the jackson-core system up and working. I needed jackson-core, jackson-annotations and jackson-databind attached to my project in order to get this working. This might have seemed obvious to a Java developer, but it was obtuse to me.

Thanks for your help!!
Doug

[toc] | [prev] | [next] | [standalone]

#30021

On 19.09.14 22.34, doug.farrell@gmail.com wrote:
> and has a log4j.xml configuration file that looks like this:
>
> <?xml version="1.0" encoding="UTF-8"?>
> <Configuration status="WARN">
>    <Appenders>
>      <Console name="STDOUT" target="SYSTEM_OUT">
>        <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
>      </Console>
>      <Socket name="SOCKET" host="localhost" port="9023" protocol="UDP">
>      	<SerializedLayout />
          ^
This is probably not what you want ...

> However, I can't figure out what the format of the data received by the Python logging server is? This is the data I get back in my handler:
>
> '\xac\xed\x00\x05sr\x00>org.apache.logging.log4j.core.impl.Log4jLogEvent$LogEventProxy...

... because of this.

If you are unhappy with log4j SocketAppender and you cannot depend on 
log4j 2, simply write your own appender. This is quite simple. You 
basically need to derive from AppenderSkeleton and implement the method 
append (who'd a thunk). There you can do whatever you like with the 
LogEvent passed as an argument.
Alternatively you might extend WriterAppender and pass an 
UDPOutputStream (Apache license as log4j is) to the constructor.

This also works if you do not own the logging application as long as you 
can add your class to the JVM classpath. BTDT.


Marcel

[toc] | [prev] | [next] | [standalone]

#30023

In article <lvjiei$djk$1@gwaiyur.mb-net.net>,
 Marcel Mueller <news.5.maazl@spamgourmet.org> wrote:

> On 19.09.14 22.34, doug.farrell@gmail.com wrote:
> > and has a log4j.xml configuration file that looks like this:
> >
> > <?xml version="1.0" encoding="UTF-8"?>
> > <Configuration status="WARN">
> >    <Appenders>
> >      <Console name="STDOUT" target="SYSTEM_OUT">
> >        <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - 
> >        %msg%n"/>
> >      </Console>
> >      <Socket name="SOCKET" host="localhost" port="9023" protocol="UDP">
> >      	<SerializedLayout />
>           ^
> This is probably not what you want ...
> 
> > However, I can't figure out what the format of the data received by the 
> > Python logging server is? This is the data I get back in my handler:
> >
> > '\xac\xed\x00\x05sr\x00>org.apache.logging.log4j.core.impl.Log4jLogEvent$Log
> > EventProxy...
> 
> ... because of this.
> 
> If you are unhappy with log4j SocketAppender and you cannot depend on 
> log4j 2, simply write your own appender. This is quite simple. You 
> basically need to derive from AppenderSkeleton and implement the method 
> append (who'd a thunk). There you can do whatever you like with the 
> LogEvent passed as an argument.
> Alternatively you might extend WriterAppender and pass an 
> UDPOutputStream (Apache license as log4j is) to the constructor.
> 
> This also works if you do not own the logging application as long as you 
> can add your class to the JVM classpath. BTDT.
> 
> 
> Marcel

I'm not a fan of Log4J.  I haven't seen it perform well and configuring 
it to work as you wish becomes complicated.  Formatting log entries with 
hostile content (control characters, toString() throws, etc.) is 
definitely a weak spot.  I need a custom formatter that won't trash the 
logs if somebody injects binary data into a text workflow.

I've been using the Java logging framework.  It's trivial to implement 
and it lets you take advantage of high performance/high concurrency 
libraries that you may already be using.  Since handlers are easy to 
configure runtime, you can do neat stuff like temporarily change log 
levels and replicate a log path to a WebSocket.

Sun's auto configuration system is a poorly documented wreck but it's 
easy enough to read in a list of Handlers and log paths from a config 
file yourself.

-- 
I will not see posts from astraweb, theremailer, dizum, or google
because they host Usenet flooders.

[toc] | [prev] | [next] | [standalone]

#30024

On 9/20/2014 10:17 AM, Kevin McMurtrie wrote:

> Sun's auto configuration system is a poorly documented wreck but it's
> easy enough to read in a list of Handlers and log paths from a config
> file yourself.
>


Hmm, have you made any progress understanding Sun's auto configuration 
system?  (I'm not even 100% sure what you are referring to.)  If you 
have some information, post it.  Other folks might be able to fill in 
additional details.

I've poked at the logging configuration myself, but I've never used it 
on a large project, so I'm sure what parts would be useful to elaborate on.

[toc] | [prev] | [next] | [standalone]

#30025

On 09/20/2014 04:29 PM, markspace wrote:
> On 9/20/2014 10:17 AM, Kevin McMurtrie wrote:
>
>> Sun's auto configuration system is a poorly documented wreck but it's
>> easy enough to read in a list of Handlers and log paths from a config
>> file yourself.
>>
>
>
> Hmm, have you made any progress understanding Sun's auto configuration
> system?  (I'm not even 100% sure what you are referring to.)

My guess would be:
<http://docs.oracle.com/javase/8/docs/api/java/util/logging/LogManager.html>

> If you
> have some information, post it.  Other folks might be able to fill in
> additional details.
>
> I've poked at the logging configuration myself, but I've never used it
> on a large project, so I'm sure what parts would be useful to elaborate on.
>
>

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.java.programmer

csiph-web