Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #96173 > unrolled thread

From logging to files to a better solution: syslog, Sentry, Logstash, ....

Back to article view | Back to comp.lang.python

Contents

  From logging to files to a better solution: syslog, Sentry, Logstash, .... Thomas Güttler <hv@tbz-pariv.de> - 2015-09-09 01:33 -0700
    Re: From logging to files to a better solution: syslog, Sentry, Logstash, .... dieter <dieter@handshake.de> - 2015-09-10 08:41 +0200
      Re: From logging to files to a better solution: syslog, Sentry, Logstash, .... Thomas Güttler <hv@tbz-pariv.de> - 2015-09-11 00:22 -0700
        Re: From logging to files to a better solution: syslog, Sentry, Logstash, .... marco.nawijn@colosso.nl - 2015-09-11 01:17 -0700
          Re: From logging to files to a better solution: syslog, Sentry, Logstash, .... Thomas Güttler <hv@tbz-pariv.de> - 2015-09-15 02:37 -0700
        Re: From logging to files to a better solution: syslog, Sentry, Logstash, .... jmp <jeanmichel@sequans.com> - 2015-09-11 11:02 +0200
          Re: From logging to files to a better solution: syslog, Sentry, Logstash, .... Thomas Güttler <hv@tbz-pariv.de> - 2015-09-15 02:35 -0700
            Re: From logging to files to a better solution: syslog, Sentry, Logstash, .... jmp <jeanmichel@sequans.com> - 2015-09-15 16:36 +0200
            Re: From logging to files to a better solution: syslog, Sentry, Logstash, .... dieter <dieter@handshake.de> - 2015-09-16 07:57 +0200
              Re: From logging to files to a better solution: syslog, Sentry, Logstash, .... Thomas Güttler <hv@tbz-pariv.de> - 2015-09-16 01:31 -0700
        Re: From logging to files to a better solution: syslog, Sentry, Logstash, .... dieter <dieter@handshake.de> - 2015-09-12 08:36 +0200

#96173 — From logging to files to a better solution: syslog, Sentry, Logstash, ....

Up to now we use simple logging to files.

We don't use syslog or an other server based solution.

I am unsure which architecture works for our environment.

Our environment:

 - django based applications
 - a lot of batch/cron jobs (non web gui) processing
 - One linux server runs several systems. Each system has its own virtualenv.
 - servers are running in the intranet of our customers. 
 - We have access via VPN, but it is slow. And sometimes down for some minutes.
 - 40MByte logs per day per server.

Why we are unhappy with logging to files:

 - filtering: We don't want to get INFO messages over the VPN.
   Background: Sometimes a there is too much logging and we don't want 
   to pull gigabytes over the VPN.

 - Rotating: Rotating files is possible, but somehow cumbersome.

 - Support structured logging of values (json) in the future.

I have not locked at  Sentry or Logstash in detail.

Which solution could fit for our environment?

Please ask if you have question!

Regards,
  Thomas Güttler

[toc] | [next] | [standalone]

#96244

Thomas Güttler <hv@tbz-pariv.de> writes:
> ...
> Why we are unhappy with logging to files:
>
>  - filtering: We don't want to get INFO messages over the VPN.

You can quite easily control at what level messages are logged with
the standard Python logging framework. Each handler has a level
and will ignore messages at a lower level.

>  - Rotating: Rotating files is possible, but somehow cumbersome.

There are standard tools to rotate logfiles.

>  - Support structured logging of values (json) in the future.

Again, the Python logging framework is quite flexible with
respect to the format of logged messages.

> ...
> Which solution could fit for our environment?

I work for a customer with a similar environment (he uses "Zope" instead
of "Django") - and he uses logfiles. The logfiles are automatically
rotated and there are in the order of half a dozen to a dozen logfiles
per day.

When I have to analyse a problem with the help of the logfiles,
I do not copy them via VPN but do the filtering remotely and only
copy the filtered portion, if necessary.

[toc] | [prev] | [next] | [standalone]

#96328

Am Donnerstag, 10. September 2015 08:42:47 UTC+2 schrieb dieter:
> Thomas Güttler writes:
> > ...
> > Why we are unhappy with logging to files:
> >
> >  - filtering: We don't want to get INFO messages over the VPN.
> 
> You can quite easily control at what level messages are logged with
> the standard Python logging framework. Each handler has a level
> and will ignore messages at a lower level.


I want INFO to be logged and stored on the remote host.
Therefore I must not filter INFO messages.

I don't want to pull INFO messages over the VPN.

Ergo, the filtering at Python level does not help in my use case.
Or I am missing something.

And now I have an ugly soup. 

The ugly soup is a text file with not well defined syntax. It looks line
based. But sometimes there are log messages which span multiple lines ....

Remember: "Life is too short to (re)write parsers"

Yes, there are tools to parse that soup. But why create this soup in 
the first place?

That's why I want to move from file based logging to a different solution.

Unfortunately there too many choices (graylog, logstash, sentry, ...) :-(



> >  - Rotating: Rotating files is possible, but somehow cumbersome.
> 
> There are standard tools to rotate logfiles.
> 
> >  - Support structured logging of values (json) in the future.
> 
> Again, the Python logging framework is quite flexible with
> respect to the format of logged messages.
> 
> > ...
> > Which solution could fit for our environment?
> 
> I work for a customer with a similar environment (he uses "Zope" instead
> of "Django") - and he uses logfiles. The logfiles are automatically
> rotated and there are in the order of half a dozen to a dozen logfiles
> per day.
> 
> When I have to analyse a problem with the help of the logfiles,
> I do not copy them via VPN but do the filtering remotely and only
> copy the filtered portion, if necessary.

Good to know that I am not the only one running servers in remote intranets.

Regards,
  Thomas Güttler

[toc] | [prev] | [next] | [standalone]

#96331

On Friday, September 11, 2015 at 9:22:42 AM UTC+2, Thomas Güttler wrote:
> Am Donnerstag, 10. September 2015 08:42:47 UTC+2 schrieb dieter:
> > Thomas Güttler writes:
> > > ...
> > > Why we are unhappy with logging to files:
> > >
> > >  - filtering: We don't want to get INFO messages over the VPN.
> > 
> > You can quite easily control at what level messages are logged with
> > the standard Python logging framework. Each handler has a level
> > and will ignore messages at a lower level.
> 
> 
> I want INFO to be logged and stored on the remote host.
> Therefore I must not filter INFO messages.
> 
> I don't want to pull INFO messages over the VPN.
> 
> Ergo, the filtering at Python level does not help in my use case.
> Or I am missing something.
> 
> And now I have an ugly soup. 
> 
> The ugly soup is a text file with not well defined syntax. It looks line
> based. But sometimes there are log messages which span multiple lines ....
> 
> Remember: "Life is too short to (re)write parsers"
> 
> Yes, there are tools to parse that soup. But why create this soup in 
> the first place?
> 
> That's why I want to move from file based logging to a different solution.
> 
> Unfortunately there too many choices (graylog, logstash, sentry, ...) :-(
> 
> 
> 
> > >  - Rotating: Rotating files is possible, but somehow cumbersome.
> > 
> > There are standard tools to rotate logfiles.
> > 
> > >  - Support structured logging of values (json) in the future.
> > 
> > Again, the Python logging framework is quite flexible with
> > respect to the format of logged messages.
> > 
> > > ...
> > > Which solution could fit for our environment?
> > 
> > I work for a customer with a similar environment (he uses "Zope" instead
> > of "Django") - and he uses logfiles. The logfiles are automatically
> > rotated and there are in the order of half a dozen to a dozen logfiles
> > per day.
> > 
> > When I have to analyse a problem with the help of the logfiles,
> > I do not copy them via VPN but do the filtering remotely and only
> > copy the filtered portion, if necessary.
> 
> Good to know that I am not the only one running servers in remote intranets.
> 
> Regards,
>   Thomas Güttler

So, if logging to json is on the horizon anyway, why don't you create
something like a MongoDb handler in the standard Python logging framework 
and run a MongoDb server in the client intranet? You could then connect
over VPN to MongoDb, filter the warning/error messages there on the server
side and fetch them to your local systems for analysis.

Marco

[toc] | [prev] | [next] | [standalone]

#96624

Am Freitag, 11. September 2015 10:18:11 UTC+2 schrieb marco....@colosso.nl:
> On Friday, September 11, 2015 at 9:22:42 AM UTC+2, Thomas Güttler wrote:
> > Am Donnerstag, 10. September 2015 08:42:47 UTC+2 schrieb dieter:
> > > Thomas Güttler writes:
> > > > ...
> > > > Why we are unhappy with logging to files:
> > > >
> > > >  - filtering: We don't want to get INFO messages over the VPN.
> > > 
> > > You can quite easily control at what level messages are logged with
> > > the standard Python logging framework. Each handler has a level
> > > and will ignore messages at a lower level.
> > 
> > 
> > I want INFO to be logged and stored on the remote host.
> > Therefore I must not filter INFO messages.
> > 
> > I don't want to pull INFO messages over the VPN.
> > 
> > Ergo, the filtering at Python level does not help in my use case.
> > Or I am missing something.
> > 
> > And now I have an ugly soup. 
> > 
> > The ugly soup is a text file with not well defined syntax. It looks line
> > based. But sometimes there are log messages which span multiple lines ....
> > 
> > Remember: "Life is too short to (re)write parsers"
> > 
> > Yes, there are tools to parse that soup. But why create this soup in 
> > the first place?
> > 
> > That's why I want to move from file based logging to a different solution.
> > 
> > Unfortunately there too many choices (graylog, logstash, sentry, ...) :-(
> > 
> > 
> > 
> > > >  - Rotating: Rotating files is possible, but somehow cumbersome.
> > > 
> > > There are standard tools to rotate logfiles.
> > > 
> > > >  - Support structured logging of values (json) in the future.
> > > 
> > > Again, the Python logging framework is quite flexible with
> > > respect to the format of logged messages.
> > > 
> > > > ...
> > > > Which solution could fit for our environment?
> > > 
> > > I work for a customer with a similar environment (he uses "Zope" instead
> > > of "Django") - and he uses logfiles. The logfiles are automatically
> > > rotated and there are in the order of half a dozen to a dozen logfiles
> > > per day.
> > > 
> > > When I have to analyse a problem with the help of the logfiles,
> > > I do not copy them via VPN but do the filtering remotely and only
> > > copy the filtered portion, if necessary.
> > 
> > Good to know that I am not the only one running servers in remote intranets.
> > 
> > Regards,
> >   Thomas Güttler
> 
> So, if logging to json is on the horizon anyway, why don't you create
> something like a MongoDb handler in the standard Python logging framework 
> and run a MongoDb server in the client intranet? You could then connect
> over VPN to MongoDb, filter the warning/error messages there on the server
> side and fetch them to your local systems for analysis.


Yes I could set up a MongoDB server. But I would like to reuse and not to reinvent. 

I would like to go a proven solid road.

[toc] | [prev] | [next] | [standalone]

#96334

On 09/11/2015 09:22 AM, Thomas Güttler wrote:
>
> I want INFO to be logged and stored on the remote host.
> Therefore I must not filter INFO messages.
>
> I don't want to pull INFO messages over the VPN.
>
> Ergo, the filtering at Python level does not help in my use case.
> Or I am missing something.

Probably,

Your logger should have

   * a remote host handler
   * and a VPN handler.

You can set filters and log levels separately for each handler.
More info here
https://docs.python.org/2/library/logging.html#handler-objects
https://docs.python.org/2/howto/logging-cookbook.html#logging-to-multiple-destinations

Something like (python 2.7)

import logging

logCfg = {
     'remote':(
         logging.StreamHandler(),
         logging.Formatter('Remote - %(levelname)s - %(message)s'),
         logging.INFO,
         ),
     'vpn':(
         logging.StreamHandler(),
         logging.Formatter('VPN - %(levelname)s - %(message)s'),
         logging.ERROR,
         ),
}

log = logging.getLogger()
log.setLevel(logging.DEBUG)

for handler, formatter, level in logCfg.itervalues():
     handler.setFormatter(formatter)
     handler.setLevel(level)
     log.addHandler(handler)

log.info('This is an info')
log.error('This is error')


and the result:

Remote - INFO - This is an info
VPN - ERROR - This is error
Remote - ERROR - This is error


JM

[toc] | [prev] | [next] | [standalone]

#96622

Am Freitag, 11. September 2015 11:03:52 UTC+2 schrieb jmp:
> On 09/11/2015 09:22 AM, Thomas Güttler wrote:
> >
> > I want INFO to be logged and stored on the remote host.
> > Therefore I must not filter INFO messages.
> >
> > I don't want to pull INFO messages over the VPN.
> >
> > Ergo, the filtering at Python level does not help in my use case.
> > Or I am missing something.
> 
> Probably,
> 
> Your logger should have
> 
>    * a remote host handler
>    * and a VPN handler.
> 
> You can set filters and log levels separately for each handler.
> More info here
> https://docs.python.org/2/library/logging.html#handler-objects
> https://docs.python.org/2/howto/logging-cookbook.html#logging-to-multiple-destinations
> 
> Something like (python 2.7)
> 
> import logging
> 
> logCfg = {
>      'remote':(
>          logging.StreamHandler(),
>          logging.Formatter('Remote - %(levelname)s - %(message)s'),
>          logging.INFO,
>          ),
>      'vpn':(
>          logging.StreamHandler(),
>          logging.Formatter('VPN - %(levelname)s - %(message)s'),
>          logging.ERROR,
>          ),
> }


.... Yes, I could do it this way.

But somehow I am not happy with this solution.

I think the filtering should be outside of python.

I like DevOp, but sometimes clear responsibilities help to
cut big problems into smaller ones.

I would like to handle the log message filtering outside of python.

Can you understand my concerns?

  Thomas Güttler

[toc] | [prev] | [next] | [standalone]

#96638

On 09/15/2015 11:35 AM, Thomas Güttler wrote:
>
> .... Yes, I could do it this way.
>
> But somehow I am not happy with this solution.
>
> I think the filtering should be outside of python.
[snip]
> Can you understand my concerns?
>
>    Thomas Güttler
>

No, not really.

I showed you how it can be done in python using standard logging 
technics. Others have also provided suggestions and you've also 
dismissed those with a "I don't feel like it".

Remember this is a python mailing list hence if you're looking for help 
about a non python log infrastructure, you probably find better answers 
in those said log infrastructures mailing lists.

JM

[toc] | [prev] | [next] | [standalone]

#96657

Thomas Güttler <hv@tbz-pariv.de> writes:
> Am Freitag, 11. September 2015 11:03:52 UTC+2 schrieb jmp:
> ...
>> Something like (python 2.7)
>> 
>> import logging
>> 
>> logCfg = {
>>      'remote':(
>>          logging.StreamHandler(),
>>          logging.Formatter('Remote - %(levelname)s - %(message)s'),
>>          logging.INFO,
>>          ),
>>      'vpn':(
>>          logging.StreamHandler(),
>>          logging.Formatter('VPN - %(levelname)s - %(message)s'),
>>          logging.ERROR,
>>          ),
>> }
>
>
> .... Yes, I could do it this way.
>
> But somehow I am not happy with this solution.
>
> I think the filtering should be outside of python.

Do you think, it will be easier there?

You could also use the "syslog" handler and use "syslog"
configuration features to separate the various message levels.
>From my point of view, this will not be easier - but outside of Python :-)

And you can develop your own Python logging handler delegating logging to
your favorite external logging subsystem and then configure that.
Likely the hardest approach...

[toc] | [prev] | [next] | [standalone]

#96664

Am Mittwoch, 16. September 2015 07:58:11 UTC+2 schrieb dieter:
> Thomas Güttler <hv@tbz-pariv.de> writes:
> > Am Freitag, 11. September 2015 11:03:52 UTC+2 schrieb jmp:
> > ...
> >> Something like (python 2.7)
> >> 
> >> import logging
> >> 
> >> logCfg = {
> >>      'remote':(
> >>          logging.StreamHandler(),
> >>          logging.Formatter('Remote - %(levelname)s - %(message)s'),
> >>          logging.INFO,
> >>          ),
> >>      'vpn':(
> >>          logging.StreamHandler(),
> >>          logging.Formatter('VPN - %(levelname)s - %(message)s'),
> >>          logging.ERROR,
> >>          ),
> >> }
> >
> >
> > .... Yes, I could do it this way.
> >
> > But somehow I am not happy with this solution.
> >
> > I think the filtering should be outside of python.
> 
> Do you think, it will be easier there?
> 
> You could also use the "syslog" handler and use "syslog"
> configuration features to separate the various message levels.
> >From my point of view, this will not be easier - but outside of Python :-)
> 
> And you can develop your own Python logging handler delegating logging to
> your favorite external logging subsystem and then configure that.
> Likely the hardest approach...

Yes, this is a python list. I like python programming.

But I don't want to solve everything with one tool.

I wanted to know how python folks handle their logs.

I think filtering mails should be done outside the python interpreter.

Strange that no one seems to use one of the mentioned tools for log handling.

Regards,
  Thomas

[toc] | [prev] | [next] | [standalone]

#96417

Thomas Güttler <hv@tbz-pariv.de> writes:

> Am Donnerstag, 10. September 2015 08:42:47 UTC+2 schrieb dieter:
>> Thomas Güttler writes:
>> > ...
>> > Why we are unhappy with logging to files:
>> >
>> >  - filtering: We don't want to get INFO messages over the VPN.
>> 
>> You can quite easily control at what level messages are logged with
>> the standard Python logging framework. Each handler has a level
>> and will ignore messages at a lower level.
>
>
> I want INFO to be logged and stored on the remote host.
> Therefore I must not filter INFO messages.
>
> I don't want to pull INFO messages over the VPN.

Thus, you could define one handler which logs at INFO level and
another one which logs at WARNING level. The latter handler
would log only WARNING and more serious events.

> ...
> Yes, there are tools to parse that soup. But why create this soup in 
> the first place?
>
> That's why I want to move from file based logging to a different solution.

As you have explained above, you need this "soup" (both INFO messages
as well as messages above WARNING level - but handled differently).

Define an additional handler which handles "WARNING" (and above).
This is easy with Zope (I do not know for Django).


> Unfortunately there too many choices (graylog, logstash, sentry, ...) :-(

I do not know any of them - but I doubt that they will help you much to
untangle your "soup".

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web