Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #21338 > unrolled thread

Threads, waiting for last one to finish

Back to article view | Back to comp.lang.java.programmer

Contents

  Threads, waiting for last one to finish Roedy Green <see_website@mindprod.com.invalid> - 2013-01-11 13:56 -0800
    Re: Threads, waiting for last one to finish markspace <markspace@nospam.nospam> - 2013-01-11 14:05 -0800
      Re: Threads, waiting for last one to finish Roedy Green <see_website@mindprod.com.invalid> - 2013-01-11 14:14 -0800
        Re: Threads, waiting for last one to finish markspace <markspace@nospam.nospam> - 2013-01-11 15:11 -0800
          Re: Threads, waiting for last one to finish Arne Vajhøj <arne@vajhoej.dk> - 2013-01-11 20:02 -0500
          Re: Threads, waiting for last one to finish Roedy Green <see_website@mindprod.com.invalid> - 2013-01-12 01:51 -0800
          Re: Threads, waiting for last one to finish Roedy Green <see_website@mindprod.com.invalid> - 2013-01-12 02:20 -0800
            Re: Threads, waiting for last one to finish Arne Vajhøj <arne@vajhoej.dk> - 2013-01-12 09:08 -0500
          Re: Threads, waiting for last one to finish Roedy Green <see_website@mindprod.com.invalid> - 2013-01-12 10:36 -0800
            Re: Threads, waiting for last one to finish Arne Vajhøj <arne@vajhoej.dk> - 2013-01-12 16:17 -0500
      Re: Threads, waiting for last one to finish Kevin McMurtrie <mcmurtrie@pixelmemory.us> - 2013-01-12 23:39 -0800
        Re: Threads, waiting for last one to finish Arne Vajhøj <arne@vajhoej.dk> - 2013-01-13 22:35 -0500
          Re: Threads, waiting for last one to finish Daniel Pitts <newsgroup.nospam@virtualinfinity.net> - 2013-01-14 09:12 -0800
          Re: Threads, waiting for last one to finish Kevin McMurtrie <mcmurtrie@pixelmemory.us> - 2013-01-14 22:29 -0800
            Re: Threads, waiting for last one to finish markspace <markspace@nospam.nospam> - 2013-01-15 08:19 -0800
              Re: Threads, waiting for last one to finish Kevin McMurtrie <mcmurtrie@pixelmemory.us> - 2013-01-16 01:28 -0800
                Re: Threads, waiting for last one to finish Robert Klemme <shortcutter@googlemail.com> - 2013-01-20 15:21 +0100
              Re: Threads, waiting for last one to finish Kevin McMurtrie <mcmurtrie@pixelmemory.us> - 2013-01-18 21:58 -0800
        Re: Threads, waiting for last one to finish Robert Klemme <shortcutter@googlemail.com> - 2013-01-26 17:20 +0100
    Re: Threads, waiting for last one to finish Knute Johnson <nospam@knutejohnson.com> - 2013-01-11 21:19 -0800
      Re: Threads, waiting for last one to finish Robert Klemme <shortcutter@googlemail.com> - 2013-01-13 19:38 +0100
        Re: Threads, waiting for last one to finish Knute Johnson <nospam@knutejohnson.com> - 2013-01-14 08:58 -0800
        Re: Threads, waiting for last one to finish Arved Sandstrom <asandstrom2@eastlink.ca> - 2013-01-15 05:05 -0400
    Re: Threads, waiting for last one to finish Arved Sandstrom <asandstrom2@eastlink.ca> - 2013-01-12 08:42 -0400
      Re: Threads, waiting for last one to finish Roedy Green <see_website@mindprod.com.invalid> - 2013-01-12 23:56 -0800
        Re: Threads, waiting for last one to finish Arved Sandstrom <asandstrom2@eastlink.ca> - 2013-01-13 10:53 -0400
          Re: Threads, waiting for last one to finish Roedy Green <see_website@mindprod.com.invalid> - 2013-01-13 19:18 -0800
            Re: Threads, waiting for last one to finish Arne Vajhøj <arne@vajhoej.dk> - 2013-01-13 22:26 -0500
              Re: Threads, waiting for last one to finish Arved Sandstrom <asandstrom2@eastlink.ca> - 2013-01-14 05:51 -0400
                Re: Threads, waiting for last one to finish Arne Vajhøj <arne@vajhoej.dk> - 2013-01-14 19:02 -0500
          Re: Threads, waiting for last one to finish Roedy Green <see_website@mindprod.com.invalid> - 2013-01-13 19:43 -0800
    Re: Threads, waiting for last one to finish Sebastian <news@seyweiler.dyndns.org> - 2013-01-13 15:19 +0100

Page 1 of 2  [1] 2  Next page →

#21338 — Threads, waiting for last one to finish

I have 25 threads that start at once. I need to wait until the last
one completes. Is there a better way to handle that than using a
ThreadPoolExecutor which seems overkill.
-- 
Roedy Green Canadian Mind Products http://mindprod.com
Students who hire or con others to do their homework are as foolish 
as couch potatoes who hire others to go to the gym for them. 

[toc] | [next] | [standalone]

#21340

On 1/11/2013 1:56 PM, Roedy Green wrote:
> I have 25 threads that start at once. I need to wait until the last
> one completes. Is there a better way to handle that than using a
> ThreadPoolExecutor which seems overkill.
>

The Executors class has convenience methods to make thread pools easier 
to use.

   ExecutorService es = Executors.newFixedThreadPool( 25 );
   // submit jobs here
   es.shutdown();

The shutdown() will wait for all jobs to finish.  I don't think you can 
get easier than that.  It's two lines of code!

[toc] | [prev] | [next] | [standalone]

#21342

On Fri, 11 Jan 2013 14:05:10 -0800, markspace
<markspace@nospam.nospam> wrote, quoted or indirectly quoted someone
who said :

>
>   ExecutorService es = Executors.newFixedThreadPool( 25 );
>   // submit jobs here
>   es.shutdown();

 Excellent!
-- 
Roedy Green Canadian Mind Products http://mindprod.com
Students who hire or con others to do their homework are as foolish 
as couch potatoes who hire others to go to the gym for them. 

[toc] | [prev] | [next] | [standalone]

#21343

On 1/11/2013 2:14 PM, Roedy Green wrote:
> On Fri, 11 Jan 2013 14:05:10 -0800, markspace
> <markspace@nospam.nospam> wrote, quoted or indirectly quoted someone
> who said :
>
>>
>>    ExecutorService es = Executors.newFixedThreadPool( 25 );
>>    // submit jobs here
>>    es.shutdown();
>
>   Excellent!
>

Oops, sorry, it's awaitTermination() that you want, not shutdown().

[toc] | [prev] | [next] | [standalone]

#21344

On 1/11/2013 6:11 PM, markspace wrote:
> On 1/11/2013 2:14 PM, Roedy Green wrote:
>> On Fri, 11 Jan 2013 14:05:10 -0800, markspace
>> <markspace@nospam.nospam> wrote, quoted or indirectly quoted someone
>> who said :
>>
>>>
>>>    ExecutorService es = Executors.newFixedThreadPool( 25 );
>>>    // submit jobs here
>>>    es.shutdown();
>>
>>   Excellent!
>
> Oops, sorry, it's awaitTermination() that you want, not shutdown().

If the Java docs are to be believed he actually wants both.

<quote>
Blocks until all tasks have completed execution after a shutdown 
request, or
</quote>

Arne

[toc] | [prev] | [next] | [standalone]

#21347

On Fri, 11 Jan 2013 15:11:17 -0800, markspace
<markspace@nospam.nospam> wrote, quoted or indirectly quoted someone
who said :

>
>Oops, sorry, it's awaitTermination() that you want, not shutdown().

Working great. Thanks.
-- 
Roedy Green Canadian Mind Products http://mindprod.com
Students who hire or con others to do their homework are as foolish 
as couch potatoes who hire others to go to the gym for them. 

[toc] | [prev] | [next] | [standalone]

#21350

On Fri, 11 Jan 2013 15:11:17 -0800, markspace
<markspace@nospam.nospam> wrote, quoted or indirectly quoted someone
who said :

>
>Oops, sorry, it's awaitTermination()

I think there is a bug. If there are no threads submitted it hangs.
-- 
Roedy Green Canadian Mind Products http://mindprod.com
Students who hire or con others to do their homework are as foolish 
as couch potatoes who hire others to go to the gym for them. 

[toc] | [prev] | [next] | [standalone]

#21355

On 1/12/2013 5:20 AM, Roedy Green wrote:
> On Fri, 11 Jan 2013 15:11:17 -0800, markspace
> <markspace@nospam.nospam> wrote, quoted or indirectly quoted someone
> who said :
>> Oops, sorry, it's awaitTermination()
>
> I think there is a bug. If there are no threads submitted it hangs.

You meed to call shutdown first.

It is documented.

http://mindprod.com/jgloss/rtfm.html

:-)

Arne

[toc] | [prev] | [next] | [standalone]

#21365

On Fri, 11 Jan 2013 15:11:17 -0800, markspace
<markspace@nospam.nospam> wrote, quoted or indirectly quoted someone
who said :

>Oops, sorry, it's awaitTermination() that you want, not shutdown().

you need both.
-- 
Roedy Green Canadian Mind Products http://mindprod.com
Students who hire or con others to do their homework are as foolish 
as couch potatoes who hire others to go to the gym for them. 

[toc] | [prev] | [next] | [standalone]

#21369

On 1/12/2013 1:36 PM, Roedy Green wrote:
> On Fri, 11 Jan 2013 15:11:17 -0800, markspace
> <markspace@nospam.nospam> wrote, quoted or indirectly quoted someone
> who said :
>
>> Oops, sorry, it's awaitTermination() that you want, not shutdown().
>
> you need both.

So the documentation is correct. Surprise. Not!

Arne

[toc] | [prev] | [next] | [standalone]

#21372

In article <kcq2ap$m4o$1@dont-email.me>,
 markspace <markspace@nospam.nospam> wrote:

> On 1/11/2013 1:56 PM, Roedy Green wrote:
> > I have 25 threads that start at once. I need to wait until the last
> > one completes. Is there a better way to handle that than using a
> > ThreadPoolExecutor which seems overkill.
> >
> 
> The Executors class has convenience methods to make thread pools easier 
> to use.
> 
>    ExecutorService es = Executors.newFixedThreadPool( 25 );
>    // submit jobs here
>    es.shutdown();
> 
> The shutdown() will wait for all jobs to finish.  I don't think you can 
> get easier than that.  It's two lines of code!

The Executor classes are horribly buggy.  Pool size adjustment, idle 
timeouts, and shutdown sequences do NOT work.  Some of the features are 
impossible to implement efficiently so Sun instead chose to implement 
them incorrectly.

For an Executor, it would be more correct to iterate through all the 
Future or Callable objects returned and ask for their result.
-- 
I will not see posts from Google because I must filter them as spam

[toc] | [prev] | [next] | [standalone]

#21390

On 1/13/2013 2:39 AM, Kevin McMurtrie wrote:
> In article <kcq2ap$m4o$1@dont-email.me>,
>   markspace <markspace@nospam.nospam> wrote:
>> On 1/11/2013 1:56 PM, Roedy Green wrote:
>>> I have 25 threads that start at once. I need to wait until the last
>>> one completes. Is there a better way to handle that than using a
>>> ThreadPoolExecutor which seems overkill.
>>>
>>
>> The Executors class has convenience methods to make thread pools easier
>> to use.
>>
>>     ExecutorService es = Executors.newFixedThreadPool( 25 );
>>     // submit jobs here
>>     es.shutdown();
>>
>> The shutdown() will wait for all jobs to finish.  I don't think you can
>> get easier than that.  It's two lines of code!
>
> The Executor classes are horribly buggy.  Pool size adjustment, idle
> timeouts, and shutdown sequences do NOT work.  Some of the features are
> impossible to implement efficiently so Sun instead chose to implement
> them incorrectly.
>
> For an Executor, it would be more correct to iterate through all the
> Future or Callable objects returned and ask for their result.


Can you be more specific about when shutdown and await does
not work?

Arne

[toc] | [prev] | [next] | [standalone]

#21400

On 1/13/13 7:35 PM, Arne Vajhøj wrote:
> On 1/13/2013 2:39 AM, Kevin McMurtrie wrote:
>> In article <kcq2ap$m4o$1@dont-email.me>,
>>   markspace <markspace@nospam.nospam> wrote:
>>> On 1/11/2013 1:56 PM, Roedy Green wrote:
>>>> I have 25 threads that start at once. I need to wait until the last
>>>> one completes. Is there a better way to handle that than using a
>>>> ThreadPoolExecutor which seems overkill.
>>>>
>>>
>>> The Executors class has convenience methods to make thread pools easier
>>> to use.
>>>
>>>     ExecutorService es = Executors.newFixedThreadPool( 25 );
>>>     // submit jobs here
>>>     es.shutdown();
>>>
>>> The shutdown() will wait for all jobs to finish.  I don't think you can
>>> get easier than that.  It's two lines of code!
>>
>> The Executor classes are horribly buggy.  Pool size adjustment, idle
>> timeouts, and shutdown sequences do NOT work.  Some of the features are
>> impossible to implement efficiently so Sun instead chose to implement
>> them incorrectly.
>>
>> For an Executor, it would be more correct to iterate through all the
>> Future or Callable objects returned and ask for their result.
>
>
> Can you be more specific about when shutdown and await does
> not work?
>
> Arne
>
>
I am also interested. However, I will say that I would have used the 
ExecutorService and simply used the Future objects to wait for 
completion of all tasks. This has the added benefit that you can 
actually process the results of the Callables in a single thread, rather 
than have them try to update some shared data structure.  Using the 
Future result is inherently easier to get correct.

[toc] | [prev] | [next] | [standalone]

#21408

In article <50f37d14$0$293$14726298@news.sunsite.dk>,
 Arne Vajhøj <arne@vajhoej.dk> wrote:

> On 1/13/2013 2:39 AM, Kevin McMurtrie wrote:
> > In article <kcq2ap$m4o$1@dont-email.me>,
> >   markspace <markspace@nospam.nospam> wrote:
> >> On 1/11/2013 1:56 PM, Roedy Green wrote:
> >>> I have 25 threads that start at once. I need to wait until the last
> >>> one completes. Is there a better way to handle that than using a
> >>> ThreadPoolExecutor which seems overkill.
> >>>
> >>
> >> The Executors class has convenience methods to make thread pools easier
> >> to use.
> >>
> >>     ExecutorService es = Executors.newFixedThreadPool( 25 );
> >>     // submit jobs here
> >>     es.shutdown();
> >>
> >> The shutdown() will wait for all jobs to finish.  I don't think you can
> >> get easier than that.  It's two lines of code!
> >
> > The Executor classes are horribly buggy.  Pool size adjustment, idle
> > timeouts, and shutdown sequences do NOT work.  Some of the features are
> > impossible to implement efficiently so Sun instead chose to implement
> > them incorrectly.
> >
> > For an Executor, it would be more correct to iterate through all the
> > Future or Callable objects returned and ask for their result.
> 
> 
> Can you be more specific about when shutdown and await does
> not work?
> 
> Arne

There are race conditions in determining the predicted thread states.  
There's simply no way to check the work queue plus all of the states of 
all of the threads in an atomic manner without locking or using a master 
thread.  Sun's implementation has almost no locking and no master thread 
so it makes mistakes while threads are transitioning states.  The 
variable sized thread pool is the worst (glitches maybe 10% of the 
time!) but the others have rare glitches too.  These glitches are fixed 
when a new task is queued so a steady stream of tasks appears to work 
correctly.  What happened with the very last task is a gamble.  That's 
why you shouldn't use shutdown() to wait for completion.

I'm not going to walk through the code here.  It's awful stuff that's 
difficult to follow and it changes a little with each JVM release.  You 
can step through it in a debugger and see that there's some wishful 
thinking in there.  Careful placement of breakpoints can produce a fault 
that doesn't recover when all breakpoints are removed.
-- 
I will not see posts from Google because I must filter them as spam

[toc] | [prev] | [next] | [standalone]

#21414

On 1/14/2013 10:29 PM, Kevin McMurtrie wrote:

> variable sized thread pool is the worst (glitches maybe 10% of the
> time!) but the others have rare glitches too.  These glitches are fixed
> when a new task is queued so a steady stream of tasks appears to work
> correctly.  What happened with the very last task is a gamble.  That's
> why you shouldn't use shutdown() to wait for completion.
>
> I'm not going to walk through the code here.  It's awful stuff that's


Hmm, could you produce a code example that glitches 10% of the time? 
Otherwise I'm afraid that I'll have to file your claims under "internet 
b.s."  There's no way I'm going to believe that Oracle's code is that 
buggy just on hearsay.

[toc] | [prev] | [next] | [standalone]

#21431

In article <kd3vjc$cat$1@dont-email.me>,
 markspace <markspace@nospam.nospam> wrote:

> On 1/14/2013 10:29 PM, Kevin McMurtrie wrote:
> 
> > variable sized thread pool is the worst (glitches maybe 10% of the
> > time!) but the others have rare glitches too.  These glitches are fixed
> > when a new task is queued so a steady stream of tasks appears to work
> > correctly.  What happened with the very last task is a gamble.  That's
> > why you shouldn't use shutdown() to wait for completion.
> >
> > I'm not going to walk through the code here.  It's awful stuff that's
> 
> 
> Hmm, could you produce a code example that glitches 10% of the time? 
> Otherwise I'm afraid that I'll have to file your claims under "internet 
> b.s."  There's no way I'm going to believe that Oracle's code is that 
> buggy just on hearsay.

I can prove it to you when you're paying me for it.  Here are some bugs 
with the variable sized thread pool:

With short tasks on a system having many CPU cores, the time it takes a 
thread to be seen as idle becomes large compared to the task execution 
time.  The result is the needless creation of more threads  The surge of 
extra threads steals CPU time from existing threads, making the problem 
worse.  You can demonstrate this by feeding a fixed number of rapidly 
completing tasks into a thread pool and then checking the pool size.  A 
common symptom can be seen in Tomcat on very large servers, where the 
connection handler pool suddenly jumps from ~100 to thousands of threads.

The idle timeout doesn't work.  Threads are used round-robin so they're 
never idle as long as tasks keep coming at a rate of timeout/threads.  
For a timeout of 60 seconds, 100 tasks per second will keep a pool of at 
least 6000 threads alive.  Combine this with the previous bug and you 
can see that the variable sized pool is very broken.

Scheduled tasks do not expand the pool size so they starve a thread pool 
of workers.  Read the code for ScheduledThreadPoolExecutor if you'd like.

java.util.concurrent.Delayed is defined in relative time and must 
implement Comparable<Delayed>.  It's impossible to sort tasks using 
relative time so Sun hacked it.  Look at 
java.util.concurrent.ScheduledThreadPoolExecutor.ScheduledFutureTask.comp
areTo(Delayed).  Any implementation other than 
ScheduledFutureCallableLink does not work reliably.

Fixed size pools have some problems too but they're less frequent and 
unobtrusive.  You'd probably never hit them unless you're using 
shutdown() rather than Future.get() to wait for task completion.
-- 
I will not see posts from Google because I must filter them as spam

[toc] | [prev] | [next] | [standalone]

#21584

On 16.01.2013 10:28, Kevin McMurtrie wrote:

> Fixed size pools have some problems too but they're less frequent and
> unobtrusive.  You'd probably never hit them unless you're using
> shutdown() rather than Future.get() to wait for task completion.

I was going to add that most of the issues of the implementation we are 
talking about stem from the fact that the number of active threads is 
adjusted all the time, i.e. threads are created and die (or at least 
that was the intention) all the time.

For most real world problems I am aware of it is sufficient to create 
all threads upfront and let them block waiting for work.  You are 
expecting to use them anyway at some point in time (that's why you set a 
specific max) so at least when load is highest the biggest number of 
threads is active anyway.  Other resources (memory) have to be adjusted 
to that fact anyway and if the pool is idle there will be enough memory 
to keep those blocking threads around.

The only other scheme that seems to be reasonable to me is to create 
threads on demand and never let them die before pool shutdown.  That 
would at least ensure low overhead for low load situations.  But even 
this scheme has some overhead in terms of locking already.

My preferred way for shutdown of thread pools is a stop element of which 
as many instances as threads around are inserted into the queue.  Every 
worker threads which reads it terminates itself.  That way you do not 
need any additional locking and just join all active threads before you 
let the shutdown method return.  The details of proper shutdown are 
still tricky because one can introduce a number of nasty race conditions 
which will leave unprocessed items in the queue.

There are many applications though where production of work for the pool 
is stopped before the pool is shut down so shutdown race conditions are 
not an issues for those types of applications.

Kind regards

	robert


-- 
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

[toc] | [prev] | [next] | [standalone]

#21568

In article <kd3vjc$cat$1@dont-email.me>,
 markspace <markspace@nospam.nospam> wrote:

> On 1/14/2013 10:29 PM, Kevin McMurtrie wrote:
> 
> > variable sized thread pool is the worst (glitches maybe 10% of the
> > time!) but the others have rare glitches too.  These glitches are fixed
> > when a new task is queued so a steady stream of tasks appears to work
> > correctly.  What happened with the very last task is a gamble.  That's
> > why you shouldn't use shutdown() to wait for completion.
> >
> > I'm not going to walk through the code here.  It's awful stuff that's
> 
> 
> Hmm, could you produce a code example that glitches 10% of the time? 
> Otherwise I'm afraid that I'll have to file your claims under "internet 
> b.s."  There's no way I'm going to believe that Oracle's code is that 
> buggy just on hearsay.

I hit another one today and it's directly related to the original post.  
The shutdown() method causes a spin loop bug in 
ThreadPoolExecutor.getTask().

This is wrong:
if (state == SHUTDOWN)  // Help drain queue
   r = workQueue.poll();

It should be:
if (state == SHUTDOWN)  // Help drain queue
   return workQueue.poll();


This was very frustrating because CPU was at 400% but it wouldn't show 
up in any Java profiler.  I had to pause all threads and start them up 
one at a time.

shutdownNow() doesn't cause the bug.
-- 
I will not see posts from Google because I must filter them as spam

[toc] | [prev] | [next] | [standalone]

#21739

On 25.01.2013 20:46, Wanja Gayk wrote:
> In article <50f264aa$0$80119$742ec2ed@news.sonic.net>, Kevin McMurtrie
> (mcmurtrie@pixelmemory.us) says...
>
>> The Executor classes are horribly buggy.  Pool size adjustment, idle
>> timeouts, and shutdown sequences do NOT work.  Some of the features are
>> impossible to implement efficiently so Sun instead chose to implement
>> them incorrectly.
>
> Could be more specific?

See his reply from January 16. in this thread.  :-)

Kind regards

	robert


-- 
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

[toc] | [prev] | [next] | [standalone]

#21346

On 1/11/2013 1:56 PM, Roedy Green wrote:
> I have 25 threads that start at once. I need to wait until the last
> one completes. Is there a better way to handle that than using a
> ThreadPoolExecutor which seems overkill.
>

You can always join() them.

-- 

Knute Johnson

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.java.programmer

csiph-web