From logging to files to a better solution: syslog, Sentry, Logstash, ....

Newsgroups	comp.lang.python
Date	2015-09-09 01:33 -0700
Message-ID	<5c1f52f5-59d2-48dd-b2e8-3a907a517261@googlegroups.com> (permalink)
Subject	From logging to files to a better solution: syslog, Sentry, Logstash, ....
From	Thomas Güttler <hv@tbz-pariv.de>

Show all headers | View raw

Up to now we use simple logging to files.

We don't use syslog or an other server based solution.

I am unsure which architecture works for our environment.

Our environment:

 - django based applications
 - a lot of batch/cron jobs (non web gui) processing
 - One linux server runs several systems. Each system has its own virtualenv.
 - servers are running in the intranet of our customers. 
 - We have access via VPN, but it is slow. And sometimes down for some minutes.
 - 40MByte logs per day per server.

Why we are unhappy with logging to files:

 - filtering: We don't want to get INFO messages over the VPN.
   Background: Sometimes a there is too much logging and we don't want 
   to pull gigabytes over the VPN.

 - Rotating: Rotating files is possible, but somehow cumbersome.

 - Support structured logging of values (json) in the future.

I have not locked at  Sentry or Logstash in detail.

Which solution could fit for our environment?

Please ask if you have question!

Regards,
  Thomas Güttler

Back to comp.lang.python | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

From logging to files to a better solution: syslog, Sentry, Logstash, .... Thomas Güttler <hv@tbz-pariv.de> - 2015-09-09 01:33 -0700
  Re: From logging to files to a better solution: syslog, Sentry, Logstash, .... dieter <dieter@handshake.de> - 2015-09-10 08:41 +0200
    Re: From logging to files to a better solution: syslog, Sentry, Logstash, .... Thomas Güttler <hv@tbz-pariv.de> - 2015-09-11 00:22 -0700
      Re: From logging to files to a better solution: syslog, Sentry, Logstash, .... marco.nawijn@colosso.nl - 2015-09-11 01:17 -0700
        Re: From logging to files to a better solution: syslog, Sentry, Logstash, .... Thomas Güttler <hv@tbz-pariv.de> - 2015-09-15 02:37 -0700
      Re: From logging to files to a better solution: syslog, Sentry, Logstash, .... jmp <jeanmichel@sequans.com> - 2015-09-11 11:02 +0200
        Re: From logging to files to a better solution: syslog, Sentry, Logstash, .... Thomas Güttler <hv@tbz-pariv.de> - 2015-09-15 02:35 -0700
          Re: From logging to files to a better solution: syslog, Sentry, Logstash, .... jmp <jeanmichel@sequans.com> - 2015-09-15 16:36 +0200
          Re: From logging to files to a better solution: syslog, Sentry, Logstash, .... dieter <dieter@handshake.de> - 2015-09-16 07:57 +0200
            Re: From logging to files to a better solution: syslog, Sentry, Logstash, .... Thomas Güttler <hv@tbz-pariv.de> - 2015-09-16 01:31 -0700
      Re: From logging to files to a better solution: syslog, Sentry, Logstash, .... dieter <dieter@handshake.de> - 2015-09-12 08:36 +0200

csiph-web