Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #34590 > unrolled thread

future run unlimited

Back to article view | Back to comp.lang.java.programmer

Contents

  future run unlimited Philipp Kraus <philipp.kraus.flashpixx@gmail.com> - 2016-06-10 06:30 -0700
    Re: future run unlimited Eric Sosman <esosman@comcast-dot-net.invalid> - 2016-06-10 10:17 -0400
      Re: future run unlimited Philipp Kraus <philipp.kraus.flashpixx@gmail.com> - 2016-06-10 07:34 -0700
        Re: future run unlimited Eric Sosman <esosman@comcast-dot-net.invalid> - 2016-06-10 11:48 -0400
          Re: future run unlimited Philipp Kraus <philipp.kraus.flashpixx@gmail.com> - 2016-06-10 12:49 -0700
            Re: future run unlimited Eric Sosman <esosman@comcast-dot-net.invalid> - 2016-06-11 08:22 -0400

#34590 — future run unlimited

Hello,

I would like to build an unlimited working process with an executor new-work-stealing executor.
I have a stream of callable objects and I submit them to the executor, and I will get on the submit call a future of my input, but after the object is done, I will run it again within the pool. In some pseudo-code illustrating:

mycallablestream
     .map( i->  pool.submit(i))
     .filter( i -> i is done)
     .foreach( i -> mycallable.put(i) )

How can I do this with the execution service and how I can shutdown the pool in a correct way if my user creates an event for closing (from UI)?

Hope it is clear for understanding

Thanks for help

Phil

[toc] | [next] | [standalone]

#34591

On 6/10/2016 9:30 AM, Philipp Kraus wrote:
> Hello,
>
> I would like to build an unlimited working process with an executor new-work-stealing executor.
> I have a stream of callable objects and I submit them to the executor, and I will get on the submit call a future of my input, but after the object is done, I will run it again within the pool. In some pseudo-code illustrating:
>
> mycallablestream
>      .map( i->  pool.submit(i))
>      .filter( i -> i is done)
>      .foreach( i -> mycallable.put(i) )

     If you want to call put() after each Callable completes, this won't
do it.  The filter() stage will not wait for completion; it will just
test "Has this Callable already completed?" and reject those that are
still in progress or have not started.  Result: The few (probably)
Callables that complete in a very small amount of time will pass to
the forEach() stage, and the stream will finish while the others are
still pending or in progress.

     I'm not sure exactly what you want to do, but perhaps you should
take a look at ExecutorCompletionService.

> How can I do this with the execution service and how I can shutdown the pool in a correct way if my user creates an event for closing (from UI)?

     What is the "correct way?"

     - Wait for all submitted tasks to complete, then shut down?

     - Discard the tasks that have not started, wait for those already
       in progress, then shut down?

     - Discard the un-started tasks, cancel those in progress, then
       shut down?

     - Something else?

-- 
esosman@comcast-dot-net.invalid
"Don't be afraid of work. Make work afraid of you." -- TLM

[toc] | [prev] | [next] | [standalone]

#34592

Hi,

thanks your answer, but I think I need to describe the problem a little bit better.

Am Freitag, 10. Juni 2016 16:18:00 UTC+2 schrieb Eric Sosman:

> If you want to call put() after each Callable completes, this won't
> do it.  The filter() stage will not wait for completion; it will just
> test "Has this Callable already completed?" and reject those that are
> still in progress or have not started.  Result: The few (probably)
> Callables that complete in a very small amount of time will pass to
> the forEach() stage, and the stream will finish while the others are
> still pending or in progress.
> 
>      I'm not sure exactly what you want to do, but perhaps you should
> take a look at ExecutorCompletionService.

I would like to create a "continuous worker", something like

while (true)
     pool.invokeAll( mycallablelist )
     
Like a queue, the pool gets a callable object from my queue / list and run the object, if it is done, it will put at the end of the list. I would like to replace the invokeAll with submit. I would like to create an asynchronized infinity working process.
 
>      - Discard the tasks that have not started, wait for those already
>        in progress, then shut down?

in my case I can break every time and lost the non-started objects, but the running objects should be done / finished

Thanks

Phil

[toc] | [prev] | [next] | [standalone]

#34593

On 6/10/2016 10:34 AM, Philipp Kraus wrote:
> Hi,
>
> thanks your answer, but I think I need to describe the problem a little bit better.
>
> Am Freitag, 10. Juni 2016 16:18:00 UTC+2 schrieb Eric Sosman:
>
>> If you want to call put() after each Callable completes, this won't
>> do it.  The filter() stage will not wait for completion; it will just
>> test "Has this Callable already completed?" and reject those that are
>> still in progress or have not started.  Result: The few (probably)
>> Callables that complete in a very small amount of time will pass to
>> the forEach() stage, and the stream will finish while the others are
>> still pending or in progress.
>>
>>      I'm not sure exactly what you want to do, but perhaps you should
>> take a look at ExecutorCompletionService.
>
> I would like to create a "continuous worker", something like
>
> while (true)
>      pool.invokeAll( mycallablelist )
>
> Like a queue, the pool gets a callable object from my queue / list and run the object, if it is done, it will put at the end of the list. I would like to replace the invokeAll with submit. I would like to create an asynchronized infinity working process.

     Let me restate what I think you want, in case I still misunderstand:
You have a fixed number of tasks, and each task should run over and over
again until you shut the whole thing down.  (If that's not right, then
disregard the rest ...)  Presumably, each task completion produces a
result or an effect that the overall program needs (otherwise, you'd
just write an infinite loop).  So: A task gets submitted once,
eventually produces its result, the result is consumed, and then
the task is re-submitted for another execution.

     If that's the goal, I think ExecutorCompletionService could help.
You would submit all N tasks to the ECS (a one-time "batch" operation),
take() each task as it finishes, consume its result, and re-submit.


>>      - Discard the tasks that have not started, wait for those already
>>        in progress, then shut down?
>
> in my case I can break every time and lost the non-started objects, but the running objects should be done / finished

     I don't know of a built-in way to do this.  Assuming the Executor is
actually an ExecutorService, it has a shutdown() method that will reject
further submissions but allow all submitted tasks to finish.  It also
has a shutdownNow() method that tries to cancel everything abruptly.
You seem to want something in between: Allow already-started tasks to
finish, while discarding those submitted but not yet started.

     Perhaps you could use shutdown() after setting a global "shutting
down now" flag.  Each task would check the flag before starting its
work, terminating immediately if it's set.  That way, all the tasks
would "run" but those that hadn't already started would finish "very
quickly" and drain out of the system.  (Of course, you won't re-submit
tasks while "shutting down now" is true -- if you tried, I think you'd
get RejectedExecutionException or something similar).

     Just curious: What is the nature of these over-and-over-again
tasks?  Can you describe the overall system?

-- 
esosman@comcast-dot-net.invalid
"Don't be afraid of work. Make work afraid of you." -- TLM

[toc] | [prev] | [next] | [standalone]

#34596

Am Freitag, 10. Juni 2016 17:48:52 UTC+2 schrieb Eric Sosman:
> On 6/10/2016 10:34 AM, Philipp Kraus wrote:
> > Hi,
> >
> > thanks your answer, but I think I need to describe the problem a little bit better.
> >
> > Am Freitag, 10. Juni 2016 16:18:00 UTC+2 schrieb Eric Sosman:
> >
> >> If you want to call put() after each Callable completes, this won't
> >> do it.  The filter() stage will not wait for completion; it will just
> >> test "Has this Callable already completed?" and reject those that are
> >> still in progress or have not started.  Result: The few (probably)
> >> Callables that complete in a very small amount of time will pass to
> >> the forEach() stage, and the stream will finish while the others are
> >> still pending or in progress.
> >>
> >>      I'm not sure exactly what you want to do, but perhaps you should
> >> take a look at ExecutorCompletionService.
> >
> > I would like to create a "continuous worker", something like
> >
> > while (true)
> >      pool.invokeAll( mycallablelist )
> >
> > Like a queue, the pool gets a callable object from my queue / list and run the object, if it is done, it will put at the end of the list. I would like to replace the invokeAll with submit. I would like to create an asynchronized infinity working process.
> 
>      Let me restate what I think you want, in case I still misunderstand:
> You have a fixed number of tasks, and each task should run over and over
> again until you shut the whole thing down.  (If that's not right, then
> disregard the rest ...)  Presumably, each task completion produces a
> result or an effect that the overall program needs (otherwise, you'd
> just write an infinite loop).  So: A task gets submitted once,
> eventually produces its result, the result is consumed, and then
> the task is re-submitted for another execution.
> 
>      If that's the goal, I think ExecutorCompletionService could help.
> You would submit all N tasks to the ECS (a one-time "batch" operation),
> take() each task as it finishes, consume its result, and re-submit.

Yes, I would like to do that, but is there an existing structure
for the submit - resubmit structure or should I write my own structure?
Is there an ExecutorService which gets only the Callable list (because it is in
my case an unmodifiable list) and do the whole wort for me?

> 
> >>      - Discard the tasks that have not started, wait for those already
> >>        in progress, then shut down?
> >
> > in my case I can break every time and lost the non-started objects, but the running objects should be done / finished
> 
>      I don't know of a built-in way to do this.  Assuming the Executor is
> actually an ExecutorService, it has a shutdown() method that will reject
> further submissions but allow all submitted tasks to finish.  It also
> has a shutdownNow() method that tries to cancel everything abruptly.
> You seem to want something in between: Allow already-started tasks to
> finish, while discarding those submitted but not yet started.
> 
>      Perhaps you could use shutdown() after setting a global "shutting
> down now" flag.  Each task would check the flag before starting its
> work, terminating immediately if it's set.  That way, all the tasks
> would "run" but those that hadn't already started would finish "very
> quickly" and drain out of the system.  (Of course, you won't re-submit
> tasks while "shutting down now" is true -- if you tried, I think you'd
> get RejectedExecutionException or something similar).

I have written an global flag, but is there a better way for doing the shutdown
process?

Thanks

Phil

[toc] | [prev] | [next] | [standalone]

#34601

On 6/10/2016 3:49 PM, Philipp Kraus wrote:
> Am Freitag, 10. Juni 2016 17:48:52 UTC+2 schrieb Eric Sosman:
>> [...]
>>      If that's the goal, I think ExecutorCompletionService could help.
>> You would submit all N tasks to the ECS (a one-time "batch" operation),
>> take() each task as it finishes, consume its result, and re-submit.
>
> Yes, I would like to do that, but is there an existing structure
> for the submit - resubmit structure or should I write my own structure?
> Is there an ExecutorService which gets only the Callable list (because it is in
> my case an unmodifiable list) and do the whole wort for me?

     I don't know of one.

>[...]
> I have written an global flag, but is there a better way for doing the shutdown
> process?

     I cannot answer because I do not know how you measure "better."

-- 
esosman@comcast-dot-net.invalid
"Don't be afraid of work. Make work afraid of you." -- TLM

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.java.programmer

csiph-web