Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #33955 > unrolled thread

Troubles with informix jdbc driver ...

Back to article view | Back to comp.lang.java.programmer

Contents

  Troubles with informix jdbc driver ... Andreas Leitgeb <avl@logic.at> - 2016-03-14 13:54 +0000
    Re: Troubles with informix jdbc driver ... Marcel Mueller <news.5.maazl@spamgourmet.org> - 2016-03-14 21:23 +0100
      Re: Troubles with informix jdbc driver ... Andreas Leitgeb <avl@logic.at> - 2016-03-14 21:30 +0000
    Re: Troubles with informix jdbc driver ... Arne Vajhøj <arne@vajhoej.dk> - 2016-03-14 18:16 -0400
      Re: Troubles with informix jdbc driver ... Marcel Mueller <news.5.maazl@spamgourmet.org> - 2016-03-15 21:57 +0100
        Re: Troubles with informix jdbc driver ... Arne Vajhøj <arne@vajhoej.dk> - 2016-03-15 21:18 -0400
      Re: Troubles with informix jdbc driver ... Andreas Leitgeb <avl@logic.at> - 2016-03-15 22:25 +0000
        Re: Troubles with informix jdbc driver ... Arne Vajhøj <arne@vajhoej.dk> - 2016-03-15 21:25 -0400
    Re: Troubles with informix jdbc driver ... Kevin McMurtrie <mcmurtrie@pixelmemory.us> - 2016-03-14 21:26 -0700
      Re: Troubles with informix jdbc driver ... Andreas Leitgeb <avl@logic.at> - 2016-03-15 22:09 +0000
        Re: Troubles with informix jdbc driver ... Andreas Leitgeb <avl@logic.at> - 2016-03-16 21:37 +0000
          Re: Troubles with informix jdbc driver ... Kevin McMurtrie <mcmurtrie@pixelmemory.us> - 2016-03-16 22:36 -0700
            Re: Troubles with informix jdbc driver ... -- workaround found Andreas Leitgeb <avl@logic.at> - 2016-03-17 17:23 +0000
              Re: Troubles with informix jdbc driver ... -- workaround found Andreas Leitgeb <avl@logic.at> - 2016-03-18 11:13 +0000

#33955 — Troubles with informix jdbc driver ...

A previous version of Informix jdbc ("3.70.JC5DE") was very
strict with "thread safety", in that it governed any connection
to any db server with a global lock.  If one DB server was
temporarily unreachable, then it would block also connections
to any other db server (attempted from other threads).

Some newer version of their jdbc driver (4.10.JC5) seems to
have that fixed, but we now observe that Connections obtained
from calling DriverManager.getConnection(...) casually get
mixed up!  As a result, we observe threads retrieving data
from wrong servers - not always, but "moderately reproducible".

In my application the threads share little to no data, and there
are definitely no static fields involved in my code related to 
the keeping of jdbc-related objects.  I can't vouch for the
driver, though.

Has anyone else observed similar effects with Informix jdbc drivers
and a bunch of independent threads opening connections to different
db servers?

[toc] | [next] | [standalone]

#33958

On 14.03.16 14.54, Andreas Leitgeb wrote:
> A previous version of Informix jdbc ("3.70.JC5DE") was very
> strict with "thread safety", in that it governed any connection
> to any db server with a global lock.  If one DB server was
> temporarily unreachable, then it would block also connections
> to any other db server (attempted from other threads).

I would call this a bug.

> Some newer version of their jdbc driver (4.10.JC5) seems to
> have that fixed, but we now observe that Connections obtained
> from calling DriverManager.getConnection(...) casually get
> mixed up!

Sure that you did not introduce a race condition in your code?

Did you write a test case that opens and verifies different connections 
concurrently? If it fails you could contact the vendor.


Marcel

[toc] | [prev] | [next] | [standalone]

#33962

Marcel Mueller <news.5.maazl@spamgourmet.org> wrote:
> On 14.03.16 14.54, Andreas Leitgeb wrote:
>> A previous version of Informix jdbc ("3.70.JC5DE") was very
>> strict with "thread safety", in that it governed any connection
>> to any db server with a global lock.  If one DB server was
>> temporarily unreachable, then it would block also connections
>> to any other db server (attempted from other threads).
> I would call this a bug.

So did I, and apparently so did IBM when they fixed that one ;-)

>> Some newer version of their jdbc driver (4.10.JC5) seems to
>> have that fixed, but we now observe that Connections obtained
>> from calling DriverManager.getConnection(...) casually get
>> mixed up!
> Sure that you did not introduce a race condition in your code?

How can one ever be sure about such a thing? ;-)
That's why I asked for others' experience. 

My code wasn't changed for the new driver (I didn't even know they were
using a newer driver, until they started to report connection mixups.)

Never using static fields to hold jdbc-related objects, and never passing
such objects around from thread to thread makes me quite sure that the
mix up doesn't happen in my code, though.

> Did you write a test case that opens and verifies different connections 
> concurrently?

We have such patterns happening in production. We had the same patterns
already with old driver, but that one's connect-lock might have covered
that particular bug (be it in our code or in the driver).

> If it fails you could contact the vendor.

I think our customers are already walking that path...

I just hoped for others reporting similar observations.

[toc] | [prev] | [next] | [standalone]

#33963

On 3/14/2016 9:54 AM, Andreas Leitgeb wrote:
> A previous version of Informix jdbc ("3.70.JC5DE") was very
> strict with "thread safety", in that it governed any connection
> to any db server with a global lock.  If one DB server was
> temporarily unreachable, then it would block also connections
> to any other db server (attempted from other threads).
>
> Some newer version of their jdbc driver (4.10.JC5) seems to
> have that fixed, but we now observe that Connections obtained
> from calling DriverManager.getConnection(...) casually get
> mixed up!  As a result, we observe threads retrieving data
> from wrong servers - not always, but "moderately reproducible".
>
> In my application the threads share little to no data, and there
> are definitely no static fields involved in my code related to
> the keeping of jdbc-related objects.  I can't vouch for the
> driver, though.
>
> Has anyone else observed similar effects with Informix jdbc drivers
> and a bunch of independent threads opening connections to different
> db servers?

I have never seen such a problem before.

But let us investigate a bit.

So essentially you have

thread 1:

Connection cona = DriverManager.getConnection(constr2a);

thread 2:

Connection conb = DriverManager.getConnection(constr2b);

and sometimes you end up with the Connection object actually having
the connection to the wrong database (different connection string)?

As if there somewhere deep down in the driver code were a static field
being overwritten.

That would be easy to test by just putting:

synchronized(DriverManagager.class) {
}

around all the getConnection calls.

If that removes the problem, then I would say that it is clear that
the problem is in the driver code.

Arne

[toc] | [prev] | [next] | [standalone]

#33976

On 14.03.16 23.16, Arne Vajhøj wrote:
> That would be easy to test by just putting:
>
> synchronized(DriverManagager.class) {
> }
>
> around all the getConnection calls.
>
> If that removes the problem, then I would say that it is clear that
> the problem is in the driver code.

Not necessarily. The change may significantly reduce parallelism and 
this may hide a race condition.

Some race-conditions are that unlikely to happen that it may take 
billions of years to get even one fault. I remember that I have written 
a high efficiency Java like string library for C++ some years ago. It 
used a trick to steal 2 lower bits from the (32 bit) string pointer for 
a temporary reference counter (for strong thread safety). I.e. no more 
than 3 threads could use this field concurrently. However, with test 
cases I failed to get even 3 threads at the same time into this short 
code block (checked with an assertion). Although I had hundreds of 
parallel threads which did nothing else than accessing a single (lock 
free) string object.


Marcel

[toc] | [prev] | [next] | [standalone]

#33980

On 3/15/2016 4:57 PM, Marcel Mueller wrote:
> On 14.03.16 23.16, Arne Vajhøj wrote:
>> That would be easy to test by just putting:
>>
>> synchronized(DriverManagager.class) {
>> }
>>
>> around all the getConnection calls.
>>
>> If that removes the problem, then I would say that it is clear that
>> the problem is in the driver code.
>
> Not necessarily. The change may significantly reduce parallelism and
> this may hide a race condition.

It could happen.

So let me change "clear" to "another indication".

Arne

[toc] | [prev] | [next] | [standalone]

#33978

Arne Vajhøj <arne@vajhoej.dk> wrote:
> On 3/14/2016 9:54 AM, Andreas Leitgeb wrote:
>> A previous version of Informix jdbc ("3.70.JC5DE") was very
>> strict with "thread safety", in that it governed any connection
>> to any db server with a global lock.  If one DB server was
>> temporarily unreachable, then it would block also connections
>> to any other db server (attempted from other threads).
>>
>> Some newer version of their jdbc driver (4.10.JC5) seems to
>> have that fixed, but we now observe that Connections obtained
>> from calling DriverManager.getConnection(...) casually get
>> mixed up!  As a result, we observe threads retrieving data
>> from wrong servers - not always, but "moderately reproducible".
>
> I have never seen such a problem before.
> But let us investigate a bit.
> So essentially you have
>
> thread 1:
> Connection cona = DriverManager.getConnection(constr2a);
>
> thread 2:
> Connection conb = DriverManager.getConnection(constr2b);
>
> and sometimes you end up with the Connection object actually having
> the connection to the wrong database (different connection string)?

Yes, that's my interpretation of what's happening at the lower levels,
based on observing the higher levels...  I'll first need to come up with
an <sccee> to run on my PC, then I can add further debugging aids to
determine if two threads eventually get references to the same connection,
or whether they get two separate connections for the wrongly same db-url.

> As if there somewhere deep down in the driver code were a static field
> being overwritten.
> That would be easy to test by just putting:
> synchronized(DriverManagager.class) {
> }
> around all the getConnection calls.

Like Marcel in his follow-up, I can't quite see how that could
help me locate the bug.  It would probably "fix" the bug, but 
get me back to what I had with the old driver, namely the problem
that a single bad connect will - until timeout of the tcp-connect - 
block other threads from connecting even to other&alive servers.

Thanks, anyway.

[toc] | [prev] | [next] | [standalone]

#33981

On 3/15/2016 6:25 PM, Andreas Leitgeb wrote:
> Arne Vajhøj <arne@vajhoej.dk> wrote:
>> On 3/14/2016 9:54 AM, Andreas Leitgeb wrote:
>>> A previous version of Informix jdbc ("3.70.JC5DE") was very
>>> strict with "thread safety", in that it governed any connection
>>> to any db server with a global lock.  If one DB server was
>>> temporarily unreachable, then it would block also connections
>>> to any other db server (attempted from other threads).
>>>
>>> Some newer version of their jdbc driver (4.10.JC5) seems to
>>> have that fixed, but we now observe that Connections obtained
>>> from calling DriverManager.getConnection(...) casually get
>>> mixed up!  As a result, we observe threads retrieving data
>>> from wrong servers - not always, but "moderately reproducible".
>>
>> I have never seen such a problem before.
>> But let us investigate a bit.
>> So essentially you have
>>
>> thread 1:
>> Connection cona = DriverManager.getConnection(constr2a);
>>
>> thread 2:
>> Connection conb = DriverManager.getConnection(constr2b);
>>
>> and sometimes you end up with the Connection object actually having
>> the connection to the wrong database (different connection string)?
>
> Yes, that's my interpretation of what's happening at the lower levels,
> based on observing the higher levels...  I'll first need to come up with
> an <sccee> to run on my PC, then I can add further debugging aids to
> determine if two threads eventually get references to the same connection,
> or whether they get two separate connections for the wrongly same db-url.

Unless you are using a faulty connection pool or Connection is
non-local then I don't see the first as being likely.

>> As if there somewhere deep down in the driver code were a static field
>> being overwritten.
>> That would be easy to test by just putting:
>> synchronized(DriverManagager.class) {
>> }
>> around all the getConnection calls.
>
> Like Marcel in his follow-up, I can't quite see how that could
> help me locate the bug.

It may help convince IBM that it indeed it is a bug in their
code.

>                             It would probably "fix" the bug, but
> get me back to what I had with the old driver, namely the problem
> that a single bad connect will - until timeout of the tcp-connect -
> block other threads from connecting even to other&alive servers.

True.

But if you have an unreliable connection between the application and
the database, then I think you should fix that. Because it would not
just impact connect but everything else as well. It will never work
well.

Arne

PS: Have you considered biting the bullet and switching to using a
     connection pool? It would probably fix all your problems!

[toc] | [prev] | [next] | [standalone]

#33969

In article <slrnnedgqn.456.avl@login.logic.tuwien.ac.at>,
 Andreas Leitgeb <avl@logic.at> wrote:

> A previous version of Informix jdbc ("3.70.JC5DE") was very
> strict with "thread safety", in that it governed any connection
> to any db server with a global lock.  If one DB server was
> temporarily unreachable, then it would block also connections
> to any other db server (attempted from other threads).
> 
> Some newer version of their jdbc driver (4.10.JC5) seems to
> have that fixed, but we now observe that Connections obtained
> from calling DriverManager.getConnection(...) casually get
> mixed up!  As a result, we observe threads retrieving data
> from wrong servers - not always, but "moderately reproducible".
> 
> In my application the threads share little to no data, and there
> are definitely no static fields involved in my code related to 
> the keeping of jdbc-related objects.  I can't vouch for the
> driver, though.
> 
> Has anyone else observed similar effects with Informix jdbc drivers
> and a bunch of independent threads opening connections to different
> db servers?

There's too much legacy cruft and unreliable magic in DriverManager.  I 
strongly recommend finding your driver using Class.forName(classname), 
creating an instance, then calling it directly through its 
java.sql.Driver interface.

You must call Driver.connect(String, Properties) with a new Properties 
instance each time.  Many drivers put parameters in there that can't be 
shared between connections.  This could be how you're seeing mixups.

If the Informix JDBC driver is still really terrible at multi-threading, 
you can use your class reference to create a pool of instances of the 
driver.  I'd really hope that this is no longer necessary, but it had to 
be done with their driver a decade ago.

-- 
I will not see posts from astraweb, theremailer, dizum, or google
because they host Usenet flooders.

[toc] | [prev] | [next] | [standalone]

#33977

Kevin McMurtrie <mcmurtrie@pixelmemory.us> wrote:
> In article <slrnnedgqn.456.avl@login.logic.tuwien.ac.at>,
>  Andreas Leitgeb <avl@logic.at> wrote:
>> Some newer version of their jdbc driver (4.10.JC5) seems to
>> have that fixed, but we now observe that Connections obtained
>> from calling DriverManager.getConnection(...) casually get
>> mixed up!  As a result, we observe threads retrieving data
>> from wrong servers - not always, but "moderately reproducible".

> There's too much legacy cruft and unreliable magic in DriverManager.
> I strongly recommend finding your driver using Class.forName(classname), 
> creating an instance, then calling it directly through its 
> java.sql.Driver interface.

Thanks!

I put this on my todo-list of things I'll experiment with.
For now, a coworker will prepare a simple class that will connect
to a selected db server (one of a few) and pick a row from some table
that will allow us to check whether the row comes from the right server.
Once I can reproduce the bug with simpler code than that pretty complex 
productive code, then I expect to either be able to cut it down to
an <sccee>, or actually find a bug in my code. Either way ... ;-)

> You must call Driver.connect(String, Properties) with a new Properties 
> instance each time.

I definitely use a fresh Properties instance for each getConnection()
at the moment.

>   Many drivers put parameters in there that can't be 
> shared between connections.  This could be how you're seeing mixups.

Yeah, reusing Properties instances would be pretty negligent.

> If the Informix JDBC driver is still really terrible at multi-threading, 
Well, I can't prove that, yet.

> you can use your class reference to create a pool of instances of the 
> driver.
>  I'd really hope that this is no longer necessary, but it had to 
> be done with their driver a decade ago.

I don't fully understand this. So far I asked the DriverManager for 
Connections. If I were to load the informix driver class explicitly,
then would I keep one driver instance for the whole VM, or a "throw
away" one for each new connection, or just a couple of driver objects
to be used "round robin" just to create connections and given back
afterwards?  I could try such a driver-pool eventually (once I can
reproduce the bug on my own PC).  What I cannot do is switch to a
*connection*-pool. My threads' db usage patterns are just not compatible
with connection pooling, mostly because most of the threads have their
private credentials for their connections and keep them occupied for
extended periods of time.

It wouldn't have occurred to me to suspect DriverManager (isn't that 
a class of the JRE, rather than of the specific driver?), so thanks
for that heads-up.

[toc] | [prev] | [next] | [standalone]

#33998

Andreas Leitgeb <avl@logic.at> wrote:
> Kevin McMurtrie <mcmurtrie@pixelmemory.us> wrote:
>> There's too much legacy cruft and unreliable magic in DriverManager.
>> I strongly recommend finding your driver using Class.forName(classname), 
>> creating an instance, then calling it directly through its 
>> java.sql.Driver interface.
> I put this on my todo-list of things I'll experiment with.
> For now, a coworker will prepare a simple class that will connect
> to a selected db server (one of a few) and pick a row from some table
> that will allow us to check whether the row comes from the right server.

I'm now that one step further, and with that test program (which is still
far away from an <sccee>) I can now reproduce cases of mixup from within
eclipse debugger.

Informix's Connection instance has an internal field "connInfo", which
seems to be some kind of map that contains Properties and other stuff.

Most notably, for the thread I trapped on the wrong DB, it contained a pair
for the original (and correct) DB-URL:
  USERURL=jdbc:informix-sqli://machine:12345/dbA:informixserver=ifxserver
but it also contained another pair:
  DATABASE=dbB
which contained a wrong database name. This one probably got mixed in from
some other concurrent attempt to connect to dbB on the same informixserver.
That other thread targetting dbB got its own connection (to dbB) just fine.
Each of the threads had got an individual instance of Connection.

> It wouldn't have occurred to me to suspect DriverManager (isn't that 
> a class of the JRE, rather than of the specific driver?), so thanks
> for that heads-up.

Tomorrow I'll locally change the code to avoid DriverManager and instead
deal with instances of IfxDriver directly - just for testing.

Btw., so far I need about 5 to 20 bursts (of about 10 concurrent connection
attempts each time) to get to see the misbehaviour happening.

Thanks for caring.

[toc] | [prev] | [next] | [standalone]

#34010

In article <slrnnejkme.456.avl@login.logic.tuwien.ac.at>,
 Andreas Leitgeb <avl@logic.at> wrote:

> Andreas Leitgeb <avl@logic.at> wrote:
> > Kevin McMurtrie <mcmurtrie@pixelmemory.us> wrote:
> >> There's too much legacy cruft and unreliable magic in DriverManager.
> >> I strongly recommend finding your driver using Class.forName(classname), 
> >> creating an instance, then calling it directly through its 
> >> java.sql.Driver interface.
> > I put this on my todo-list of things I'll experiment with.
> > For now, a coworker will prepare a simple class that will connect
> > to a selected db server (one of a few) and pick a row from some table
> > that will allow us to check whether the row comes from the right server.
> 
> I'm now that one step further, and with that test program (which is still
> far away from an <sccee>) I can now reproduce cases of mixup from within
> eclipse debugger.
> 
> Informix's Connection instance has an internal field "connInfo", which
> seems to be some kind of map that contains Properties and other stuff.
> 
> Most notably, for the thread I trapped on the wrong DB, it contained a pair
> for the original (and correct) DB-URL:
>   USERURL=jdbc:informix-sqli://machine:12345/dbA:informixserver=ifxserver
> but it also contained another pair:
>   DATABASE=dbB
> which contained a wrong database name. This one probably got mixed in from
> some other concurrent attempt to connect to dbB on the same informixserver.
> That other thread targetting dbB got its own connection (to dbB) just fine.
> Each of the threads had got an individual instance of Connection.
> 
> > It wouldn't have occurred to me to suspect DriverManager (isn't that 
> > a class of the JRE, rather than of the specific driver?), so thanks
> > for that heads-up.
> 
> Tomorrow I'll locally change the code to avoid DriverManager and instead
> deal with instances of IfxDriver directly - just for testing.
> 
> Btw., so far I need about 5 to 20 bursts (of about 10 concurrent connection
> attempts each time) to get to see the misbehaviour happening.
> 
> Thanks for caring.

Wow, I guess the Informix driver still sucks a decade later.
Here's some totally untested code to start with :)


import java.lang.ref.WeakReference;
import java.sql.Connection;
import java.sql.Driver;
import java.sql.DriverPropertyInfo;
import java.sql.SQLException;
import java.util.Properties;

public class IfxDriver implements Driver
{
   private final Class<Driver> driverClass;
   private final ThreadLocal<WeakReference<Driver>> driverCache = new ThreadLocal<WeakReference<Driver>>();

   public IfxDriver() throws SQLException
   {
      try
      {
         driverClass = (Class<Driver>) Class.forName("com.informix.jdbc.IfxDriver");
      }
      catch (ClassNotFoundException e)
      {
         throw new SQLException("Missing driver", e);
      }
   }

   private Driver getDriver() throws SQLException
   {
      final WeakReference<Driver> driverRef = driverCache.get();
      Driver driver = (driverRef != null) ? driverRef.get() : null;

      if (driver == null)
      {
         try
         {
            driver = driverClass.newInstance();
         }
         catch (InstantiationException e)
         {
            throw new SQLException(e.getMessage(), e.getCause());
         }
         catch (IllegalAccessException e)
         {
            throw new SQLException("Wrong driver class", e);
         }
         driverCache.set(new WeakReference<Driver>(driver));
      }
      return driver;
   }

   @Override
   public boolean acceptsURL(String url) throws SQLException
   {
      return getDriver().acceptsURL(url);
   }

   @Override
   public Connection connect(String url, Properties info) throws SQLException
   {
      return getDriver().connect(url, info);
   }

   @Override
   public int getMajorVersion()
   {
      try
      {
         return getDriver().getMajorVersion();
      }
      catch (SQLException e)
      {
         throw new RuntimeException(e);
      }
   }

   @Override
   public int getMinorVersion()
   {
      try
      {
         return getDriver().getMajorVersion();
      }
      catch (SQLException e)
      {
         throw new RuntimeException(e);
      }
   }

   @Override
   public DriverPropertyInfo[] getPropertyInfo(String url, Properties info) throws SQLException
   {
      return getDriver().getPropertyInfo(url, info);
   }

   @Override
   public boolean jdbcCompliant()
   {
      try
      {
         return getDriver().jdbcCompliant();
      }
      catch (SQLException e)
      {
         throw new RuntimeException(e);
      }
   }
}

-- 
I will not see posts from astraweb, theremailer, dizum, or google
because they host Usenet flooders.

[toc] | [prev] | [next] | [standalone]

#34014 — Re: Troubles with informix jdbc driver ... -- workaround found

Kevin McMurtrie <mcmurtrie@pixelmemory.us> wrote:
> Wow, I guess the Informix driver still sucks a decade later.

At least it sucks in a somewhat different way now. :-)

** work-around **
I found out that adding a property "DATABASE" with a copy of
the value from the URL to the Properties given to the call to
getConnection(url,props) appears to work around the bug.
**

Nice thing is, that they really solved the old problem with
concurrent connection attempts being principially serialized.
I gave it some urls with invalid ip-addresses and some fine ones,
and even while it was waiting for the timeout on the bogus one,
it immediately connected the fine one.

DriverManager itself isn't all that complicated... it just passes
the url to all registered drivers and the first one to return a
non-null result (or exception) is the chosen one.

[toc] | [prev] | [next] | [standalone]

#34015 — Re: Troubles with informix jdbc driver ... -- workaround found

Andreas Leitgeb <avl@logic.at> wrote:
> Kevin McMurtrie <mcmurtrie@pixelmemory.us> wrote:
>> Wow, I guess the Informix driver still sucks a decade later.
>
> At least it sucks in a somewhat different way now. :-)
>
> ** work-around **
> I found out that adding a property "DATABASE" with a copy of
> the value from the URL

Just the db name part of the URL, of course, not the whole URL.

> to the Properties given to the call to
> getConnection(url,props) appears to work around the bug.
> **
>
> Nice thing is, that they really solved the old problem with
> concurrent connection attempts being principially serialized.
> I gave it some urls with invalid ip-addresses and some fine ones,
> and even while it was waiting for the timeout on the bogus one,
> it immediately connected the fine one.
>
> DriverManager itself isn't all that complicated... it just passes
> the url to all registered drivers and the first one to return a
> non-null result (or exception) is the chosen one.
>

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.java.programmer

csiph-web