Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #32966 > unrolled thread

Performance JPA

Back to article view | Back to comp.lang.java.programmer

Contents

  Performance JPA Cecil Westerhof <Cecil@decebal.nl> - 2015-11-02 09:47 +0100
    Re: Performance JPA Silvio <silvio@internet.com> - 2015-11-02 14:23 +0100
      Re: Performance JPA Cecil Westerhof <Cecil@decebal.nl> - 2015-11-03 07:52 +0100
        Re: Performance JPA Martin Gregorie <martin@address-in-sig.invalid> - 2015-11-03 11:16 +0000
          Re: Performance JPA Cecil Westerhof <Cecil@decebal.nl> - 2015-11-03 19:11 +0100
            Re: Performance JPA Martin Gregorie <martin@address-in-sig.invalid> - 2015-11-03 19:15 +0000
              Re: Performance JPA Cecil Westerhof <Cecil@decebal.nl> - 2015-11-04 09:28 +0100
                Re: Performance JPA Cecil Westerhof <Cecil@decebal.nl> - 2015-11-10 08:07 +0100
                  Re: Performance JPA Silvio <silvio@internet.com> - 2015-11-10 10:35 +0100
                    Re: Performance JPA Cecil Westerhof <Cecil@decebal.nl> - 2015-11-11 06:20 +0100
                      Re: Performance JPA Arne Vajhøj <arne@vajhoej.dk> - 2015-11-11 09:32 -0500
                  Re: Performance JPA Arne Vajhøj <arne@vajhoej.dk> - 2015-11-10 19:56 -0500
                    Re: Performance JPA Cecil Westerhof <Cecil@decebal.nl> - 2015-11-11 06:50 +0100
                      Re: Performance JPA Martin Gregorie <martin@address-in-sig.invalid> - 2015-11-11 12:17 +0000
                      Re: Performance JPA Arne Vajhøj <arne@vajhoej.dk> - 2015-11-11 09:30 -0500
                        Re: Performance JPA Cecil Westerhof <Cecil@decebal.nl> - 2015-11-11 23:20 +0100
                    Re: Performance JPA Cecil Westerhof <Cecil@decebal.nl> - 2015-11-11 23:11 +0100
                      Re: Performance JPA Arne Vajhøj <arne@vajhoej.dk> - 2015-11-11 19:10 -0500
                        Re: Performance JPA Cecil Westerhof <Cecil@decebal.nl> - 2015-11-12 07:27 +0100
                          Re: Performance JPA Cecil Westerhof <Cecil@decebal.nl> - 2015-11-13 08:30 +0100
                            Re: Performance JPA Silvio <silvio@internet.com> - 2015-11-13 10:45 +0100
                              Re: Performance JPA Cecil Westerhof <Cecil@decebal.nl> - 2015-11-13 11:06 +0100
                                Re: Performance JPA Silvio <silvio@internet.com> - 2015-11-13 13:36 +0100
                                  Re: Performance JPA Cecil Westerhof <Cecil@decebal.nl> - 2015-11-20 09:04 +0100
                            Re: Performance JPA Arne Vajhøj <arne@vajhoej.dk> - 2015-11-13 21:46 -0500
                              Re: Performance JPA Cecil Westerhof <Cecil@decebal.nl> - 2015-11-20 09:09 +0100
            Re: Performance JPA Arne Vajhøj <arne@vajhoej.dk> - 2015-11-03 20:35 -0500
              Re: Performance JPA Cecil Westerhof <Cecil@decebal.nl> - 2015-11-04 09:35 +0100
                Re: Performance JPA Arne Vajhøj <arne@vajhoej.dk> - 2015-11-07 22:09 -0500
        Re: Performance JPA Silvio <silvio@internet.com> - 2015-11-03 12:29 +0100
          Re: Performance JPA Cecil Westerhof <Cecil@decebal.nl> - 2015-11-03 19:23 +0100
          Re: Performance JPA Arne Vajhøj <arne@vajhoej.dk> - 2015-11-03 20:44 -0500
            Re: Performance JPA Cecil Westerhof <Cecil@decebal.nl> - 2015-11-04 09:48 +0100
            Re: Performance JPA Silvio <silvio@internet.com> - 2015-11-04 10:23 +0100
              Re: Performance JPA Cecil Westerhof <Cecil@decebal.nl> - 2015-11-04 11:27 +0100
                Re: Performance JPA Silvio <silvio@internet.com> - 2015-11-04 12:42 +0100
                  Re: Performance JPA Arne Vajhøj <arne@vajhoej.dk> - 2015-11-07 22:18 -0500
                    Re: Performance JPA Silvio <silvio@internet.com> - 2015-11-09 10:43 +0100
                      Re: Performance JPA Cecil Westerhof <Cecil@decebal.nl> - 2015-11-20 09:13 +0100
              Re: Performance JPA Arne Vajhøj <arne@vajhoej.dk> - 2015-11-07 22:12 -0500
                Re: Performance JPA Silvio <silvio@internet.com> - 2015-11-09 10:22 +0100
                  Re: Performance JPA Arne Vajhøj <arne@vajhoej.dk> - 2015-11-10 21:35 -0500
                    Re: Performance JPA Silvio <silvio@internet.com> - 2015-11-11 11:29 +0100
        Re: Performance JPA Arne Vajhøj <arne@vajhoej.dk> - 2015-11-03 20:42 -0500
          Re: Performance JPA Cecil Westerhof <Cecil@decebal.nl> - 2015-11-04 11:19 +0100
            Re: Performance JPA Arne Vajhøj <arne@vajhoej.dk> - 2015-11-07 22:05 -0500
    Re: Performance JPA Arne Vajhøj <arne@vajhoej.dk> - 2015-11-02 20:08 -0500
      Re: Performance JPA Cecil Westerhof <Cecil@decebal.nl> - 2015-11-03 07:55 +0100
        Re: Performance JPA Arne Vajhøj <arne@vajhoej.dk> - 2015-11-03 20:45 -0500
    Re: Performance JPA Lew <lewbloch@gmail.com> - 2015-11-16 14:43 -0800

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

#33117

On 11/13/2015 08:30 AM, Cecil Westerhof wrote:
>
> Problem solved. I removed openjpa.ConnectionDriverName and
> openjpa.jdbc.DBDictionary.
>
> The unit starts with:
>      <persistence-unit name="cvu-persistence-test" transaction-type="JTA">
>      <description>Persistence Testen</description>
>      <jta-data-source>jdbc/cvu/datasource</jta-data-source>
>
>      <exclude-unlisted-classes>false</exclude-unlisted-classes>
>      <validation-mode>AUTO</validation-mode>
>      <properties>
>
> I added transaction-type, jta-data-source and validation-mode.
>
> The validation-mode needs to be entered just before properties,
> otherwise it does not work.
>
> The performance is now comparable with the performance of JDBC. So I
> proved the naysayers wrong.

Good to hear and congratulations for solving the issue!

And sorry for aiding in hijacking your thread for a more principal and 
theoretical discussion.

>
> OneToMany relations are also going to be implemented, so maybe the JPA
> version will be even faster as the JDBC version. ;-)

One must always remain optimistic...

>
>
> One question: I removed
>      <property name="openjpa.jdbc.DBDictionary" value="org.apache.openjpa.jdbc.sql.DB2Dictionary" />
>
> It works, but I was told that I could better put it back, because it
> was used to optimise complex queries. Is this true?
>

I have done quite some testing with both and to be honest I could never 
reliably measure any difference either way. But as always: YMMV. My case 
was a medium size (70G) Postgresql database.

[toc] | [prev] | [next] | [standalone]

#33118

On Friday 13 Nov 2015 10:45 CET, silvio@internet.com wrote:

> On 11/13/2015 08:30 AM, Cecil Westerhof wrote:
>>
>> Problem solved. I removed openjpa.ConnectionDriverName and
>> openjpa.jdbc.DBDictionary.
>>
>> The unit starts with: <persistence-unit name="cvu-persistence-test"
>> transaction-type="JTA"> <description>Persistence
>> Testen</description>
>> <jta-data-source>jdbc/cvu/datasource</jta-data-source>
>>
>> <exclude-unlisted-classes>false</exclude-unlisted-classes>
>> <validation-mode>AUTO</validation-mode>
>> <properties>
>>
>> I added transaction-type, jta-data-source and validation-mode.
>>
>> The validation-mode needs to be entered just before properties,
>> otherwise it does not work.
>>
>> The performance is now comparable with the performance of JDBC. So
>> I proved the naysayers wrong.
>
> Good to hear and congratulations for solving the issue!

Thank you.


> And sorry for aiding in hijacking your thread for a more principal
> and theoretical discussion.

No problem. I like principals an theories: I always find that we spend
to little time on them.


>> OneToMany relations are also going to be implemented, so maybe the
>> JPA version will be even faster as the JDBC version. ;-)
>
> One must always remain optimistic...

Well at the moment the performance is about the same. When the
relations are fetched in one call instead one by one as it is done in
JDBC I would expect to see a speed improvement. Is of-course depending
how big the sets to be retrieved are and the percentage of the queries
where sets are retrieved.


>> One question: I removed <property name="openjpa.jdbc.DBDictionary"
>> value="org.apache.openjpa.jdbc.sql.DB2Dictionary" />
>>
>> It works, but I was told that I could better put it back, because
>> it was used to optimise complex queries. Is this true?
>>
>
> I have done quite some testing with both and to be honest I could
> never reliably measure any difference either way. But as always:
> YMMV. My case was a medium size (70G) Postgresql database.

For the moment I do not put it back then.

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof

[toc] | [prev] | [next] | [standalone]

#33120

On 11/13/2015 11:06 AM, Cecil Westerhof wrote:
>
> Well at the moment the performance is about the same. When the
> relations are fetched in one call instead one by one as it is done in
> JDBC I would expect to see a speed improvement. Is of-course depending
> how big the sets to be retrieved are and the percentage of the queries
> where sets are retrieved.
>

Don't forget JPA will use JDBC in the end so it can by definition never 
be faster than plain JDBC. The ORM can outsmart the JDBC programmer but 
that should raise questions about his ability. But to be fair it is not 
at all uncommon that poor JDBC code replaced by ORM code results in a 
better system. The opposite is just also not uncommon. It all depends on 
the quality of the old JDBC code and how good a fit the ORM is.

Fetching 1+N records can be done in several ways.

Most obvious is do a 1-row query for the 1-end and an N row query for 
the N-end. Two queries and minimal data traffic.

You can also join both tables and select the 1-end N times in the join. 
This fetches redundant data for the 1-end but has the advantage that you 
only need to do one query. This trade-off can easily be positive but 
that totally depends. I have seen it improve things dramatically (40%) 
on a remote Postgresql server behind a slowish connection and also 
double the time needed on an in-process H2 database.
Also dependent of the relative record sizes in both tables, of course.

More dramatic patters are not only imaginable but frequently spotted in 
the wild...

However, nothing more costly than eager fetching of the N-side when you 
won't actually use the data. So be very meticulous in what to use when.

[toc] | [prev] | [next] | [standalone]

#33146

On Friday 13 Nov 2015 13:36 CET, silvio@internet.com wrote:

> On 11/13/2015 11:06 AM, Cecil Westerhof wrote:
>>
>> Well at the moment the performance is about the same. When the
>> relations are fetched in one call instead one by one as it is done
>> in JDBC I would expect to see a speed improvement. Is of-course
>> depending how big the sets to be retrieved are and the percentage
>> of the queries where sets are retrieved.
>>
>
> Don't forget JPA will use JDBC in the end so it can by definition
> never be faster than plain JDBC. The ORM can outsmart the JDBC
> programmer but that should raise questions about his ability. But to
> be fair it is not at all uncommon that poor JDBC code replaced by
> ORM code results in a better system. The opposite is just also not
> uncommon. It all depends on the quality of the old JDBC code and how
> good a fit the ORM is.

True in principal, but it is a bit like: Java code will at the lowest
level be executed as machine code instructions, so it is better to
program in assembler. ;-)

Also I understood that JPA uses caching which could speed things up.

But anyway: the performance is about the same while the code is a lot
cleaner. So mission accomplished.

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof

[toc] | [prev] | [next] | [standalone]

#33127

On 11/13/2015 2:30 AM, Cecil Westerhof wrote:
> On Thursday 12 Nov 2015 07:27 CET, Cecil Westerhof wrote:
>> On Thursday 12 Nov 2015 01:10 CET, Arne Vajhøj wrote:
>>> The technique may work fine for a local database with a simple
>>> login protocol but with a remote database and complex protocol,
>>> then it could create a huge overhead.
>>>
>>> But we would need to see the rest of the persistence,xml to be sure
>>> (the standard part actually determines whether it is driver or
>>> data source).
>>>
>>> Try see:
>>>
>>> http://stackoverflow.com/questions/3526434/ibm-websphere-jpa-configuration-how-to-update-persistence-xml
>>>
>>> for an example of a persistence.xml that looks way
>>> more I would expect (no driver, data source, all managed).
>>
>> I have been trying some things, but until now without success. I
>> need to go to the office again and from there I do not have
>> newsgroup access, so some silence on my part again. I will
>> communicate my progress when I am back home.
>
> Problem solved. I removed openjpa.ConnectionDriverName and
> openjpa.jdbc.DBDictionary.

> The performance is now comparable with the performance of JDBC. So I
> proved the naysayers wrong.

:-)

> OneToMany relations are also going to be implemented, so maybe the JPA
> version will be even faster as the JDBC version. ;-)

Not necessarily faster but you may save even more code.

Note that when doing this then you will need to decide on eager vs
lazy loading.

> One question: I removed
>      <property name="openjpa.jdbc.DBDictionary" value="org.apache.openjpa.jdbc.sql.DB2Dictionary" />
>
> It works, but I was told that I could better put it back, because it
> was used to optimise complex queries. Is this true?

You are telling OpenJPA to use standard SQL and not any DB2 specific SQL
constructs.

It could mean a difference in some cases.

But in most cases no difference as what you usually do via ORM are
simple stuff that is covered fine by standard SQL.

I would add it back though. It does not hurt.

Arne

[toc] | [prev] | [next] | [standalone]

#33148

On Saturday 14 Nov 2015 03:46 CET, Arne Vajhøj wrote:

> On 11/13/2015 2:30 AM, Cecil Westerhof wrote:
>> On Thursday 12 Nov 2015 07:27 CET, Cecil Westerhof wrote:
>>> On Thursday 12 Nov 2015 01:10 CET, Arne Vajhøj wrote:
>>>> The technique may work fine for a local database with a simple
>>>> login protocol but with a remote database and complex protocol,
>>>> then it could create a huge overhead.
>>>>
>>>> But we would need to see the rest of the persistence,xml to be
>>>> sure (the standard part actually determines whether it is driver
>>>> or data source).
>>>>
>>>> Try see:
>>>>
>>>> http://stackoverflow.com/questions/3526434/ibm-websphere-jpa-configuration-how-to-update-persistence-xml
>>>>
>>>> for an example of a persistence.xml that looks way
>>>> more I would expect (no driver, data source, all managed).
>>>
>>> I have been trying some things, but until now without success. I
>>> need to go to the office again and from there I do not have
>>> newsgroup access, so some silence on my part again. I will
>>> communicate my progress when I am back home.
>>
>> Problem solved. I removed openjpa.ConnectionDriverName and
>> openjpa.jdbc.DBDictionary.
>
>> The performance is now comparable with the performance of JDBC. So
>> I proved the naysayers wrong.
>
> :-)
>
>> OneToMany relations are also going to be implemented, so maybe the
>> JPA version will be even faster as the JDBC version. ;-)
>
> Not necessarily faster but you may save even more code.

Depending on the many side. Fetching 100 records in one swoop, should
be faster as 100 records one at a time. But the cleaner code is more
important.


> Note that when doing this then you will need to decide on eager vs
> lazy loading.

At the moment they choose eager loading. Performance is good, so I
doubt it will be changed, but when necessary it van be changed to
lazy.

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof

[toc] | [prev] | [next] | [standalone]

#32989

On 11/3/2015 1:11 PM, Cecil Westerhof wrote:
> In principal the JPA should do about the same queries as the JDBC. I
> am busy trying to get the logging of the queries at the database side
> enabled. I hope I will succeed.

You should also be able to log SQL statements on the Java/JPA side.

Hibernate can do it. Various .NET ORM's can do it.

Arne

[toc] | [prev] | [next] | [standalone]

#32994

On Wednesday  4 Nov 2015 02:35 CET, Arne Vajhøj wrote:

> On 11/3/2015 1:11 PM, Cecil Westerhof wrote:
>> In principal the JPA should do about the same queries as the JDBC.
>> I am busy trying to get the logging of the queries at the database
>> side enabled. I hope I will succeed.
>
> You should also be able to log SQL statements on the Java/JPA side.
>
> Hibernate can do it. Various .NET ORM's can do it.

It probably would be possible, but it would be best I think to have
the queries that are executed when using JDBC and when using JPA. The
only sure way to get this information is to run both and let the
database log the queries.

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof

[toc] | [prev] | [next] | [standalone]

#33019

On 11/4/2015 3:35 AM, Cecil Westerhof wrote:
> On Wednesday  4 Nov 2015 02:35 CET, Arne Vajhøj wrote:
>> On 11/3/2015 1:11 PM, Cecil Westerhof wrote:
>>> In principal the JPA should do about the same queries as the JDBC.
>>> I am busy trying to get the logging of the queries at the database
>>> side enabled. I hope I will succeed.
>>
>> You should also be able to log SQL statements on the Java/JPA side.
>>
>> Hibernate can do it. Various .NET ORM's can do it.
>
> It probably would be possible, but it would be best I think to have
> the queries that are executed when using JDBC and when using JPA. The
> only sure way to get this information is to run both and let the
> database log the queries.

If the database supports it and the DBA's are willing to enable it.

Otherwise logging generated SQL may be the way to go.

(and the JDBC SQL is directly visible in the code)

Arne

[toc] | [prev] | [next] | [standalone]

#32983

On 11/03/2015 07:52 AM, Cecil Westerhof wrote:
> On Monday  2 Nov 2015 14:23 CET, silvio@internet.com wrote:
>
>> On 11/02/2015 09:47 AM, Cecil Westerhof wrote:
>>> We needed to change from JDBC to JPA. It looks like the performance
>>> is 10 to 20 times as slow as it was with JDBC. I could understand
>>> JPA to be slower as JDBC, but not with this kind of difference. Are
>>> there any common error that would explain this kind of penalty?
>>>
>>
>> I remember you came to us here a couple of months ago with the
>> intent of moving an application from JDBC to JPA. You gave us little
>> to go on for advice. I also remember telling you this day would
>> probably come...
>>
>> But let's put that aside and see if we can fix your situation.
>> Please tell us what the typical database access from your
>> application looks like and which things are particularly slow in
>> comparison to how it was?
>
> At the moment I am told that everything is slower. I am told that a
> query that took at most 5 seconds now takes one to two minutes. But I
> asked them to measure the old situation and until now they did not do
> this. I hope to get today real data about this. But several pages now
> give a timeout.
>
>
>> JDBC and JPA are incomparable. JDBC is an API to execute SQL
>> statements against an RDBMS (with some very limited added
>> abstractions to standardize access to meta-data). JPA is an object
>> persistence implementation that uses an RDBMS for storage (via JDBC,
>> btw). It has some quirky options to execute raw SQL but that is just
>> plain stupid.
>>
>> UNLESS you where already doing the second (persisting objects)
>> manually using JDBC, the performance hit after switching to JPA is
>> inevitable. Database access can be very quick, using JDBC or any
>> other library.
>
> OK, JDBC was just using rowmapper and copying the fields. No
> persistence at all. So maybe the difference is less strange as I
> thought.
>
>
>> But having an object persistence layer sitting between your
>> application and the database can make thing stupendously inefficient
>> for one application and it can also be a nice match with little
>> added overhead for the other. It just depends on what your
>> application is.
>>
>> Please feel free to elaborate...
>
> As I understood it until now, there should be a little overhead, but
> it can be worse I understand now. As said in the above rowmapper was
> used. Now the application uses em.find to read. For displaying a page
> only reads are done, so using em.find instead rs.get... seems to make
> a very big difference. Is there a reason for this and can there be
> something done about this?
>
> By the way we use a generic function because a lot of classes are
> extended. We use:
>      public T read(K id) {
>          return em.find(entityClass, id);
>      }
>
> We fill entityClass in the constructor with:
>      entityClass = (Class<t>) genericSuperClass.getActualTypeArguments()[0];
>
> In the create, update and delete functions we use an em.flush beside
> the em.persist, em.merge and em.remove.
>

If you look at the executed SQL you will probably see a huge amount of 
lookup-by-pk statements executed where in the past these where 
relatively rare since they where mostly multi-row selects (usually 
foreign key conditions possibly combined with other filters).

This is typical for an application that has undergone a naive (in the 
one-to-one sense) port to an ORM layer.

You can optimize most if not all of these specific cases with JPA as 
well but that will take extra work. Things will still be slower but the 
numbers should e a lot closer to what you where used to.

Just out of curiosity: we know the demand for JPA was beyond your 
influence but there must have been reasons for it. Did the move deliver 
any of the projected goals or has the only result been a lot of work to 
get a slower system?

Cheers,

Silvio

[toc] | [prev] | [next] | [standalone]

#32986

On Tuesday  3 Nov 2015 12:29 CET, silvio@internet.com wrote:

> If you look at the executed SQL you will probably see a huge amount
> of lookup-by-pk statements executed where in the past these where
> relatively rare since they where mostly multi-row selects (usually
> foreign key conditions possibly combined with other filters).

Trying to convince that I need this information.


> This is typical for an application that has undergone a naive (in
> the one-to-one sense) port to an ORM layer.

Well, I am afraid that it was certainly a naive port.


> Just out of curiosity: we know the demand for JPA was beyond your
> influence but there must have been reasons for it. Did the move
> deliver any of the projected goals or has the only result been a lot
> of work to get a slower system?

It is a bit complicated. It is a very large organisation. The project
I am on is one of the pilots. In one broad sweep several frameworks
need to be changed. And JPA proved to be the hardest. What will be
learned in this project should be used as input in the following
migrations.

In the current situation there is a gallimaufry of used things. Some
projects use JDBC, some projects use Hibernate. What they want is that
‘all’ projects are going to use JPA, with openJPA as implementation.
And that they can switch from openJPA to Hibernate, EclipseLink, … in
‘15 minutes’.

There are certainly positive points. Found several error in the code.
Found errors in Unit Tests. So code quality and test quality has been
improved. Also changing of tables should be a lot easier now. In the
past a change in the table would result in changes in the code all
over the place. Now only the class has to be changed and the places
where the new info is used.

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof

[toc] | [prev] | [next] | [standalone]

#32991

On 11/3/2015 6:29 AM, Silvio wrote:
> On 11/03/2015 07:52 AM, Cecil Westerhof wrote:
>> As I understood it until now, there should be a little overhead, but
>> it can be worse I understand now. As said in the above rowmapper was
>> used. Now the application uses em.find to read. For displaying a page
>> only reads are done, so using em.find instead rs.get... seems to make
>> a very big difference. Is there a reason for this and can there be
>> something done about this?

> If you look at the executed SQL you will probably see a huge amount of
> lookup-by-pk statements executed where in the past these where
> relatively rare since they where mostly multi-row selects (usually
> foreign key conditions possibly combined with other filters).
>
> This is typical for an application that has undergone a naive (in the
> one-to-one sense) port to an ORM layer.

Really??

I have never seen such a problem.

em.createQuery is easy and generate an equivalent SQL to the one
to chose if manually written.

Arne

[toc] | [prev] | [next] | [standalone]

#32996

On Wednesday  4 Nov 2015 02:44 CET, Arne Vajhøj wrote:

> On 11/3/2015 6:29 AM, Silvio wrote:
>> On 11/03/2015 07:52 AM, Cecil Westerhof wrote:
>>> As I understood it until now, there should be a little overhead,
>>> but it can be worse I understand now. As said in the above
>>> rowmapper was used. Now the application uses em.find to read. For
>>> displaying a page only reads are done, so using em.find instead
>>> rs.get... seems to make a very big difference. Is there a reason
>>> for this and can there be something done about this?
>
>> If you look at the executed SQL you will probably see a huge amount
>> of lookup-by-pk statements executed where in the past these where
>> relatively rare since they where mostly multi-row selects (usually
>> foreign key conditions possibly combined with other filters).
>>
>> This is typical for an application that has undergone a naive (in
>> the one-to-one sense) port to an ORM layer.
>
> Really??
>
> I have never seen such a problem.
>
> em.createQuery is easy and generate an equivalent SQL to the one
> to chose if manually written.

For read, create, update and delete we use find, persist, merge and
remove. For the other queries mostly createQuery is used. On a few
places criteria is used.

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof

[toc] | [prev] | [next] | [standalone]

#32997

On 11/04/2015 02:44 AM, Arne Vajhøj wrote:
>
> Really??
>
> I have never seen such a problem.

I have seen tons of them. Code that used to work with a logical filter 
that was easily represented as a WHERE-clause for SELECT, UPDATE or 
DELETE statements was converted to something that was working with some 
Set of entities. Every action on the complete set was then translated to 
N calls on the individual entities. Sweet.


>
> em.createQuery is easy and generate an equivalent SQL to the one
> to chose if manually written.
>

Sure, that is why I said it could be fixed by changing the code. But 
IMHO it is a bit of a wart on the whole idea of having entities 
represented as individual instances (which I generally consider a stupid 
idea in the first place).

[toc] | [prev] | [next] | [standalone]

#32998

On Wednesday  4 Nov 2015 10:23 CET, silvio@internet.com wrote:

> On 11/04/2015 02:44 AM, Arne Vajhøj wrote:
>>
>> Really??
>>
>> I have never seen such a problem.
>
> I have seen tons of them. Code that used to work with a logical
> filter that was easily represented as a WHERE-clause for SELECT,
> UPDATE or DELETE statements was converted to something that was
> working with some Set of entities. Every action on the complete set
> was then translated to N calls on the individual entities. Sweet.

That is funny: that is exactly what I was thinking, but was brushed
of.

The application uses a lot of lists and they acknowledge that
something goes wrong with list fetching. But because ‘only’ the DAO
changed this could not be the problem. How do I pinpoint this problem?

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof

[toc] | [prev] | [next] | [standalone]

#33000

On 11/04/2015 11:27 AM, Cecil Westerhof wrote:
>
> That is funny: that is exactly what I was thinking, but was brushed
> of.
>
> The application uses a lot of lists and they acknowledge that
> something goes wrong with list fetching. But because ‘only’ the DAO
> changed this could not be the problem. How do I pinpoint this problem?
>

Well, that depends on what you can look at. The executed SQL would be a 
dead giveaway but it appears you can not get to that easily.

List of entities can be populated in different ways. I have seen cases 
where an initial select would only fetch a whole bunch of primary keys 
and then the entities where fetched by key. Of course this is the result 
of bad application code but it is not at all uncommon, especially if the 
code was written by people with less experience using ORMs.

I have seen similar patterns occur when a large set of entities needed 
to be updated or deleted. The whole entity list was simply iterated to 
set a timestamp attribute to the current time and that resulted in a lot 
of update-by-pk statements where the "old" code just did something like

UPDATE abc SET ts = ? WHERE fk=?

The end result in both cases was a couple of hundred updated records but 
the time spent was incomparable.

As said elsewhere in this thread: this can all be fixed in the 
application code but first you have to pinpoint exactly where the 
offending stuff is.

I just hope (for your sake) the application code is written quite flat 
on the JPA layer. If some smarty pants has created 
yet-another-intelligent-be-it-redundant-persistence-layer on top of that 
it might be a lot harder to fix things.

Many programmers have a hard time resisting the urge to create redundant 
abstraction layers. Once in place they will defend them fiercely.

[toc] | [prev] | [next] | [standalone]

#33021

On 11/4/2015 6:42 AM, Silvio wrote:
> On 11/04/2015 11:27 AM, Cecil Westerhof wrote:
>> The application uses a lot of lists and they acknowledge that
>> something goes wrong with list fetching. But because ‘only’ the DAO
>> changed this could not be the problem. How do I pinpoint this problem?
>>
>
> Well, that depends on what you can look at. The executed SQL would be a
> dead giveaway but it appears you can not get to that easily.
>
> List of entities can be populated in different ways. I have seen cases
> where an initial select would only fetch a whole bunch of primary keys
> and then the entities where fetched by key. Of course this is the result
> of bad application code but it is not at all uncommon, especially if the
> code was written by people with less experience using ORMs.

I have not seen that happen since entity beans with BMP.

> I have seen similar patterns occur when a large set of entities needed
> to be updated or deleted. The whole entity list was simply iterated to
> set a timestamp attribute to the current time and that resulted in a lot
> of update-by-pk statements where the "old" code just did something like
>
> UPDATE abc SET ts = ? WHERE fk=?
>
> The end result in both cases was a couple of hundred updated records but
> the time spent was incomparable.

You can code that right and wrong using JDBC - and you can code that 
right and wrong using ORM.

> I just hope (for your sake) the application code is written quite flat
> on the JPA layer. If some smarty pants has created
> yet-another-intelligent-be-it-redundant-persistence-layer on top of that
> it might be a lot harder to fix things.

????

If the construct is used only once then direct JPA and a layer
will both require on change.

If the construct is used more than once then direct JPA will
require multiple changes and a layer will only require one change.

Arne

[toc] | [prev] | [next] | [standalone]

#33037

On 11/08/2015 04:18 AM, Arne Vajhøj wrote:
>
> You can code that right and wrong using JDBC - and you can code that
> right and wrong using ORM.
>

Yes, but if you do it the wrong way using JDBC you have to code it in 
the least obvious way. That just demonstrates a lack of knowledge about 
how to work with an RDBMS in general and is not caused (or can be fixed) 
by JDBC.

The ORM code that caused this particular behavior looked very clean and 
neat to a whole team of programmers that had gone over it dozens of 
times to see if they could explain the poor performance.
The problem was that they did not have enough knowledge about RDBMSs AND 
data abstraction layer inner workings to fully grasp what was going on 
behind the scenes.

They thought they did not have to know that since the ORM would take 
care of that. Just doing things the "proper OOP way" was all they 
thought they had to do.

>
> ????
>
> If the construct is used only once then direct JPA and a layer
> will both require on change.
>
> If the construct is used more than once then direct JPA will
> require multiple changes and a layer will only require one change.
>

Spoken like a true believer in abstraction layers. There is a third 
possibility: the construct is used more than once (often very many times 
more) but the underlying scenarios are so different that they need to be 
fixed in different ways. That can then only be fixed by extending the 
abstraction layer and changing a lot of application code.

As a bonus in time the abstraction layer keeps getting bigger and more 
complex because all the hammers need replacement for the lack of proper 
nails.

[toc] | [prev] | [next] | [standalone]

#33147

On Monday  9 Nov 2015 10:43 CET, silvio@internet.com wrote:

> On 11/08/2015 04:18 AM, Arne Vajhøj wrote:
>>
>> You can code that right and wrong using JDBC - and you can code
>> that right and wrong using ORM.
>>
>
> Yes, but if you do it the wrong way using JDBC you have to code it
> in the least obvious way. That just demonstrates a lack of knowledge
> about how to work with an RDBMS in general and is not caused (or can
> be fixed) by JDBC.
>
> The ORM code that caused this particular behavior looked very clean
> and neat to a whole team of programmers that had gone over it dozens
> of times to see if they could explain the poor performance. The
> problem was that they did not have enough knowledge about RDBMSs AND
> data abstraction layer inner workings to fully grasp what was going
> on behind the scenes.

Is it possible to share what was happening? It had nothing to do with
my case, but I always like to learn.

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof

[toc] | [prev] | [next] | [standalone]

#33020

On 11/4/2015 4:23 AM, Silvio wrote:
> On 11/04/2015 02:44 AM, Arne Vajhøj wrote:
>> Really??
>>
>> I have never seen such a problem.
>
> I have seen tons of them. Code that used to work with a logical filter
> that was easily represented as a WHERE-clause for SELECT, UPDATE or
> DELETE statements was converted to something that was working with some
> Set of entities. Every action on the complete set was then translated to
> N calls on the individual entities. Sweet.

Can you provide an example?

>> em.createQuery is easy and generate an equivalent SQL to the one
>> to chose if manually written.
>>
>
> Sure, that is why I said it could be fixed by changing the code. But
> IMHO it is a bit of a wart on the whole idea of having entities
> represented as individual instances (which I generally consider a stupid
> idea in the first place).

????

Having a method that returns a collection of objects is basic
feature in almost all data access code.

Arne

[toc] | [prev] | [next] | [standalone]

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

Back to top | Article view | comp.lang.java.programmer

csiph-web