Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #35268 > unrolled thread

Not closing connections and statements

Back to article view | Back to comp.lang.java.programmer

Contents

  Not closing connections and statements Cecil Westerhof <Cecil@decebal.nl> - 2016-08-12 13:59 +0200
    Re: Not closing connections and statements Arne Vajhøj <arne@vajhoej.dk> - 2016-08-12 09:15 -0400
      Re: Not closing connections and statements Arne Vajhøj <arne@vajhoej.dk> - 2016-08-12 09:39 -0400
      Re: Not closing connections and statements Marcel Mueller <news.5.maazl@spamgourmet.org> - 2016-08-12 22:31 +0200
        Re: Not closing connections and statements Arne Vajhøj <arne@vajhoej.dk> - 2016-08-12 22:04 -0400
          Re: Not closing connections and statements Marcel Mueller <news.5.maazl@spamgourmet.org> - 2016-08-13 07:55 +0200
    Re: Not closing connections and statements Martin Gregorie <martin@address-in-sig.invalid> - 2016-08-12 19:01 +0000
      Re: Not closing connections and statements Marcel Mueller <news.5.maazl@spamgourmet.org> - 2016-08-12 22:35 +0200
        Re: Not closing connections and statements Arne Vajhøj <arne@vajhoej.dk> - 2016-08-12 21:52 -0400
      Re: Not closing connections and statements Cecil Westerhof <Cecil@decebal.nl> - 2016-08-12 23:55 +0200
        Re: Not closing connections and statements Eric Sosman <esosman@comcast-dot-net.invalid> - 2016-08-12 18:16 -0400
          Re: Not closing connections and statements Cecil Westerhof <Cecil@decebal.nl> - 2016-08-13 01:19 +0200
            Re: Not closing connections and statements Martin Gregorie <martin@address-in-sig.invalid> - 2016-08-13 00:21 +0000
              Re: Not closing connections and statements Gunter Herrmann <gunter.herrmann@earthlink.net> - 2016-08-17 16:38 -0400
        Re: Not closing connections and statements Arne Vajhøj <arne@vajhoej.dk> - 2016-08-12 21:58 -0400
          Re: Not closing connections and statements Cecil Westerhof <Cecil@decebal.nl> - 2016-08-13 12:50 +0200
        Re: Not closing connections and statements Arne Vajhøj <arne@vajhoej.dk> - 2016-08-12 21:59 -0400
        Re: Not closing connections and statements Marcel Mueller <news.5.maazl@spamgourmet.org> - 2016-08-13 08:47 +0200
          Re: Not closing connections and statements Andreas Leitgeb <avl@logic.at> - 2016-08-13 08:05 +0000
            Re: Not closing connections and statements Cecil Westerhof <Cecil@decebal.nl> - 2016-08-13 12:21 +0200
            Re: Not closing connections and statements Arne Vajhøj <arne@vajhoej.dk> - 2016-08-13 09:54 -0400
          Re: Not closing connections and statements Cecil Westerhof <Cecil@decebal.nl> - 2016-08-13 12:19 +0200
          Re: Not closing connections and statements Arne Vajhøj <arne@vajhoej.dk> - 2016-08-13 10:04 -0400
            log4j RollingFileAppender (was: Not closing connections and statements) Marcel Mueller <news.5.maazl@spamgourmet.org> - 2016-08-14 22:24 +0200
              Re: log4j RollingFileAppender Arne Vajhøj <arne@vajhoej.dk> - 2016-08-14 21:01 -0400
          Re: Not closing connections and statements Arne Vajhøj <arne@vajhoej.dk> - 2016-08-13 11:58 -0400
            logging (was: Not closing connections and statements) Marcel Mueller <news.5.maazl@spamgourmet.org> - 2016-08-14 23:46 +0200
      Re: Not closing connections and statements Arne Vajhøj <arne@vajhoej.dk> - 2016-08-12 21:44 -0400
    Re: Not closing connections and statements Florian Weimer <fw@deneb.enyo.de> - 2016-08-13 16:33 +0200
      Re: Not closing connections and statements Cecil Westerhof <Cecil@decebal.nl> - 2016-08-13 17:34 +0200

Page 2 of 2 — ← Prev page 1 [2]

#35303

On 8/13/2016 4:05 AM, Andreas Leitgeb wrote:
> Marcel Mueller <news.5.maazl@spamgourmet.org> wrote:
>> However, it might or might not be a good advice to intentionally keep a
>> DB connection idle for an hour.
>
> I've seen too many firewalls shut off connections after an hour or
> even earlier to leave the "might"-option uncontradicted.
>
> Well, it "might" be that somewhere else there is no such firewall,
> but judging it as a piece of advice, I see no way it could possibly
> be a good one.

In general no.

But in the cases where a single long lasting connection is a good idea,
then maybe.

The single long lasting connection is often an adhoc internal thing. For
production with separate database server and firewall between app server
and db server, then open/close and a connection pool is the norm.

Note that a connection pool actually also tries to keep connections
open, but it is usually configured to verify the connection before
returning it and do a discard and connect again if necessary.

Arne

[toc] | [prev] | [next] | [standalone]

#35300

On Saturday 13 Aug 2016 08:47 CEST, Marcel Mueller wrote:

> On 12.08.16 23.55, Cecil Westerhof wrote:
>> That is what I use yes. I just write one record an hour to log the
>> memory usage of the application. But maybe I am going to log more
>> in the future.
>
> If it is just for logging with auto commit I would not care much.
> However, it might or might not be a good advice to intentionally
> keep a DB connection idle for an hour. I would prefer to close the
> connection after each log write. In this scenario your question is
> pointless.

I have thought about opening and closing every time, but for the
moment I decided against it.


> But as already mentioned, log4j might be a good advice. But from my
> experience any logging to a database is at most the second-best
> solution. In critical situations it might not be reliable enough and
> you loose exactly the log records that you really need. Of course,
> it depends on your use case. Logging only once an hour might not
> fall into this category.

At the moment it is just to have some easily accessible information
about the memory usage of the applications. I think it would be rare
to loose a record, but even when that happens that would not be very
big disaster.

I did work a little bit with log4j, but it never hurts to deepen my
knowledge of it.


>> It works great: it is much better to get you log information out of
>> a (SQLite) database as from different files.
>
> As soon as your program runs twice for some reason, you might think
> different. ;-)

That is why I also but the PID in the record. ;-)


> However, appending to the same file from different JVMs portably and
> atomically is also a challenge, albeit possible in Java.

I just used Bash redirection. And once it went wrong because I used
the same file.


> The library also handles switching the logs from time to time to
> limit their size as well as the cleanup of old logs. So no disk will
> get full after years, because no one cleaned up the logs. In

That is very easy to do with SQLite. ;-)

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof

[toc] | [prev] | [next] | [standalone]

#35304

On 8/13/2016 2:47 AM, Marcel Mueller wrote:
> But as already mentioned, log4j might be a good advice.
> But from my experience any logging to a database is at most the
> second-best solution. In critical situations it might not be reliable
> enough and you loose exactly the log records that you really need. Of
> course, it depends on your use case. Logging only once an hour might not
> fall into this category.

Very relevant point.

But one of the possibilities with a highly configurable logging
framework is that it is possible to log all the usual crap to
a DB and log major problems with the logging to a flat file.

> The library also handles switching the logs from time to time to limit
> their size as well as the cleanup of old logs. So no disk will get full
> after years, because no one cleaned up the logs. In contrast to log4j
> files will never get renamed because this could confuse concurrently
> reading applications or even cause locks (depending on the OS and file
> system). Instead we use (partial) time-stamps in the log name, e.g. the
> date in yyyymmdd format.

I believe that is what log4j RollingFileAppender does as well.

Arne

[toc] | [prev] | [next] | [standalone]

#35345 — log4j RollingFileAppender (was: Not closing connections and statements)

On 13.08.16 16.04, Arne Vajhøj wrote:
>> In contrast to log4j
>> files will never get renamed because this could confuse concurrently
>> reading applications or even cause locks (depending on the OS and file
>> system). Instead we use (partial) time-stamps in the log name, e.g. the
>> date in yyyymmdd format.
>
> I believe that is what log4j RollingFileAppender does as well.

AFAIK it uses rename, similar to syslog. See documentation of rollOver.
This is unusable for concurrent (unsynchronized) read access.


Marcel

[toc] | [prev] | [next] | [standalone]

#35347 — Re: log4j RollingFileAppender

On 8/14/2016 4:24 PM, Marcel Mueller wrote:
> On 13.08.16 16.04, Arne Vajhøj wrote:
>>> In contrast to log4j
>>> files will never get renamed because this could confuse concurrently
>>> reading applications or even cause locks (depending on the OS and file
>>> system). Instead we use (partial) time-stamps in the log name, e.g. the
>>> date in yyyymmdd format.
>>
>> I believe that is what log4j RollingFileAppender does as well.
>
> AFAIK it uses rename, similar to syslog. See documentation of rollOver.
> This is unusable for concurrent (unsynchronized) read access.

org.apache.log4j.RollingFileAppender rename.

But org.apache.log4j.rolling.RollingFileAppender should not rename.

Arne

[toc] | [prev] | [next] | [standalone]

#35308

On 8/13/2016 2:47 AM, Marcel Mueller wrote:
> In our company we ended up to do almost all logging into simple CSV
> files with a robust format that cannot be compromised if someone
> accidentally logs binary content.
> We wrote a small library for this purpose and ported it to at least
> half-dozen programming languages. So any application can log in a
> thread-safe and atomic way to a log file (or in fact anything else as
> well), even if two applications or instances write to the same log for
> some reason. It is similar to log4j and in fact also usable as appender
> but with more features.
> The library also handles switching the logs from time to time to limit
> their size as well as the cleanup of old logs. So no disk will get full
> after years, because no one cleaned up the logs. In contrast to log4j
> files will never get renamed because this could confuse concurrently
> reading applications or even cause locks (depending on the OS and file
> system). Instead we use (partial) time-stamps in the log name, e.g. the
> date in yyyymmdd format.
>
> A reader application can browse this log files similarly to a database,
> including group by operations, parametric search and Regex search. The
> reader uses a small in memory database with deduplication, so even many
> logs with millions of entries can be searched in only a few seconds
> without much system resources.
>
> Because of the common format of all these logs it turned out to be
> straight forward to write another small applications which scans all
> servers for theses logs and this way get simple usage statistics about
> all these applications and an overview of all used instances with a link
> to their logs using the reader above.

The decoupling of log producing and log consumption & analysis
is good.

CSV format is better than the traditional readable formats.

But I am more than a little skeptical about the concept of
inventing a custom tool set.

Standard loggers can log in CSV format.

For small data then CSV can be loaded into a RDBMS and
analyzed using standard tools.

For large data there are NoSQL based solutions out there
as well.

I believe a common stack for this is Flume+HDFS+SOLR.

Arne

[toc] | [prev] | [next] | [standalone]

#35346 — logging (was: Not closing connections and statements)

On 13.08.16 17.58, Arne Vajhøj wrote:
[logging library]
> The decoupling of log producing and log consumption & analysis
> is good.
>
> CSV format is better than the traditional readable formats.
>
> But I am more than a little skeptical about the concept of
> inventing a custom tool set.

Well, it's quite old and it's origin was in C++. I just ported the code.

> Standard loggers can log in CSV format.

Of course.

But it is quite difficult to find one that can safely write concurrently 
to the same file from different processes. This happens e.g. when a 
server application is replaced by a newer one and the old instance is 
still running to complete its current requests.

It is also difficult to find a one that writes the files in a way that 
concurrently running scanner applications can parse each logged entry 
exactly once in order even with rolling logfiles.

> For small data then CSV can be loaded into a RDBMS and
> analyzed using standard tools.

Also true. However, in memory processing turned out to be much better. 
Loading 0,5GB of log files usually takes only a few dozen MB RAM because 
of the high redundancies in typical logfiles. And the import performance 
is I/O bound. After a few seconds you can do almost any analysis against 
the data. That's enough for the most purposes.

We forward only important events into a RDBMS. And for alerting we have 
an old Java program that compiles the set of alerting rules into a 
decision tree. Checking an event against all rules turns into constant 
time this way.

> For large data there are NoSQL based solutions out there
> as well.
> I believe a common stack for this is Flume+HDFS+SOLR.

Never heard about the latter so far.


Marcel

[toc] | [prev] | [next] | [standalone]

#35286

On 8/12/2016 3:01 PM, Martin Gregorie wrote:
> On Fri, 12 Aug 2016 13:59:37 +0200, Cecil Westerhof wrote:
>> As I understood it you should close your SQL connections and statements
>> before quitting a program. How bad is this when that does not happen?
>>
> If you're going to use an SQL connection, it should be opened during
> initialisation and closed as one of the last actions of the process:

Yes and no.

For a standalone Java SE application with single threaded database
access then this strategy of open at start and close at finish works
well.

But for Java EE applications, Java SE application with multi threaded
database access the strategy should be:
- open just before it is needed for an operation
- close as soon as operation is done

> opening and closing DB connections is generally pretty slow.

If that is a problem, then one use a connection pool.

Arne

[toc] | [prev] | [next] | [standalone]

#35305

* Cecil Westerhof:

> At the moment I am writing a general function to log memory usage to a
> SQLite database to be used by different applications. How bad would it
> be if a program would exit without cleanly closing the connection and
> statement?

Is the application long-running?  How many statement objects do you
expect to leak this way?

[toc] | [prev] | [next] | [standalone]

#35307

On Saturday 13 Aug 2016 16:33 CEST, Florian Weimer wrote:

> * Cecil Westerhof:
>
>> At the moment I am writing a general function to log memory usage
>> to a SQLite database to be used by different applications. How bad
>> would it be if a program would exit without cleanly closing the
>> connection and statement?
>
> Is the application long-running?  How many statement objects do you
> expect to leak this way?

I am still no master in defining problems. :'-(


It could be short running, but I expect it could also run for weeks.


There is just one connection object and one statement object. No
leaking is happening here. Normally I have an init and a deinit
function. In the init function I create them and in the deinit
function I close them.

But I now wrote a function to log the memory usage. It is used by
calling during the initialisation of you program:
    freeMemoryThread(this.getClass().getSimpleName());

Every statement, prepared statement and resultset is consumed
immediately. That and the reason the updating is done as long as the
program is running was the reason for my question.

I rewrote the code a little so that all statement, preparedstatement
and resultset are closed after being used. So that leaves only the
connection. This is needed as long as the program is running and is in
its own tread. As it looks like there is no real problem that it is
not closed immediately before the program closes, I am accepting the
situation as it is for now.

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.lang.java.programmer

csiph-web