Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #35308

Re: Not closing connections and statements

Show all headers | View raw

On 8/13/2016 2:47 AM, Marcel Mueller wrote:
> In our company we ended up to do almost all logging into simple CSV
> files with a robust format that cannot be compromised if someone
> accidentally logs binary content.
> We wrote a small library for this purpose and ported it to at least
> half-dozen programming languages. So any application can log in a
> thread-safe and atomic way to a log file (or in fact anything else as
> well), even if two applications or instances write to the same log for
> some reason. It is similar to log4j and in fact also usable as appender
> but with more features.
> The library also handles switching the logs from time to time to limit
> their size as well as the cleanup of old logs. So no disk will get full
> after years, because no one cleaned up the logs. In contrast to log4j
> files will never get renamed because this could confuse concurrently
> reading applications or even cause locks (depending on the OS and file
> system). Instead we use (partial) time-stamps in the log name, e.g. the
> date in yyyymmdd format.
>
> A reader application can browse this log files similarly to a database,
> including group by operations, parametric search and Regex search. The
> reader uses a small in memory database with deduplication, so even many
> logs with millions of entries can be searched in only a few seconds
> without much system resources.
>
> Because of the common format of all these logs it turned out to be
> straight forward to write another small applications which scans all
> servers for theses logs and this way get simple usage statistics about
> all these applications and an overview of all used instances with a link
> to their logs using the reader above.

The decoupling of log producing and log consumption & analysis
is good.

CSV format is better than the traditional readable formats.

But I am more than a little skeptical about the concept of
inventing a custom tool set.

Standard loggers can log in CSV format.

For small data then CSV can be loaded into a RDBMS and
analyzed using standard tools.

For large data there are NoSQL based solutions out there
as well.

I believe a common stack for this is Flume+HDFS+SOLR.

Arne

Back to comp.lang.java.programmer | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Not closing connections and statements Cecil Westerhof <Cecil@decebal.nl> - 2016-08-12 13:59 +0200
  Re: Not closing connections and statements Arne Vajhøj <arne@vajhoej.dk> - 2016-08-12 09:15 -0400
    Re: Not closing connections and statements Arne Vajhøj <arne@vajhoej.dk> - 2016-08-12 09:39 -0400
    Re: Not closing connections and statements Marcel Mueller <news.5.maazl@spamgourmet.org> - 2016-08-12 22:31 +0200
      Re: Not closing connections and statements Arne Vajhøj <arne@vajhoej.dk> - 2016-08-12 22:04 -0400
        Re: Not closing connections and statements Marcel Mueller <news.5.maazl@spamgourmet.org> - 2016-08-13 07:55 +0200
  Re: Not closing connections and statements Martin Gregorie <martin@address-in-sig.invalid> - 2016-08-12 19:01 +0000
    Re: Not closing connections and statements Marcel Mueller <news.5.maazl@spamgourmet.org> - 2016-08-12 22:35 +0200
      Re: Not closing connections and statements Arne Vajhøj <arne@vajhoej.dk> - 2016-08-12 21:52 -0400
    Re: Not closing connections and statements Cecil Westerhof <Cecil@decebal.nl> - 2016-08-12 23:55 +0200
      Re: Not closing connections and statements Eric Sosman <esosman@comcast-dot-net.invalid> - 2016-08-12 18:16 -0400
        Re: Not closing connections and statements Cecil Westerhof <Cecil@decebal.nl> - 2016-08-13 01:19 +0200
          Re: Not closing connections and statements Martin Gregorie <martin@address-in-sig.invalid> - 2016-08-13 00:21 +0000
            Re: Not closing connections and statements Gunter Herrmann <gunter.herrmann@earthlink.net> - 2016-08-17 16:38 -0400
      Re: Not closing connections and statements Arne Vajhøj <arne@vajhoej.dk> - 2016-08-12 21:58 -0400
        Re: Not closing connections and statements Cecil Westerhof <Cecil@decebal.nl> - 2016-08-13 12:50 +0200
      Re: Not closing connections and statements Arne Vajhøj <arne@vajhoej.dk> - 2016-08-12 21:59 -0400
      Re: Not closing connections and statements Marcel Mueller <news.5.maazl@spamgourmet.org> - 2016-08-13 08:47 +0200
        Re: Not closing connections and statements Andreas Leitgeb <avl@logic.at> - 2016-08-13 08:05 +0000
          Re: Not closing connections and statements Cecil Westerhof <Cecil@decebal.nl> - 2016-08-13 12:21 +0200
          Re: Not closing connections and statements Arne Vajhøj <arne@vajhoej.dk> - 2016-08-13 09:54 -0400
        Re: Not closing connections and statements Cecil Westerhof <Cecil@decebal.nl> - 2016-08-13 12:19 +0200
        Re: Not closing connections and statements Arne Vajhøj <arne@vajhoej.dk> - 2016-08-13 10:04 -0400
          log4j RollingFileAppender (was: Not closing connections and statements) Marcel Mueller <news.5.maazl@spamgourmet.org> - 2016-08-14 22:24 +0200
            Re: log4j RollingFileAppender Arne Vajhøj <arne@vajhoej.dk> - 2016-08-14 21:01 -0400
        Re: Not closing connections and statements Arne Vajhøj <arne@vajhoej.dk> - 2016-08-13 11:58 -0400
          logging (was: Not closing connections and statements) Marcel Mueller <news.5.maazl@spamgourmet.org> - 2016-08-14 23:46 +0200
    Re: Not closing connections and statements Arne Vajhøj <arne@vajhoej.dk> - 2016-08-12 21:44 -0400
  Re: Not closing connections and statements Florian Weimer <fw@deneb.enyo.de> - 2016-08-13 16:33 +0200
    Re: Not closing connections and statements Cecil Westerhof <Cecil@decebal.nl> - 2016-08-13 17:34 +0200

csiph-web