Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > gnu.bash.bug > #15267 > unrolled thread

Re: How does this wait -n work to cap parallelism?

Back to article view | Back to gnu.bash.bug

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: How does this wait -n work to cap parallelism? Greg Wooledge <wooledg@eeg.ccf.org> - 2019-07-29 14:38 -0400

#15267 — Re: How does this wait -n work to cap parallelism?

On Mon, Jul 29, 2019 at 07:12:42PM +0100, Earnestly wrote:
>         #!/usr/bin/env bash
> 
>         # number of processes to run in parallel
>         num_procs=5
> 
>         # function that processes one item
>         my_job() {
>             printf 'Processing %s\n' "$1"
>             sleep "$(( RANDOM % 5 + 1 ))"
>         }
> 
>         i=0
>         while IFS= read -r line; do
>             if (( i++ >= num_procs )); then
>                 wait -n   # wait for any job to complete. New in 4.3
>             fi
>             my_job "$line" &
>         done < inputlist
>         wait # wait for the remaining processes
> 
> 
> The question is about how the example works in order to maintain
> parallelism capped at num_proc.

"inputlist" is a file that contains some sort of data, one thing per
line.  As each line is read, it is supposed to kick off one job (process)
with that line as the input parameter.

The while loop reads a line at a time, and for every line, it runs
my_job in the background.  BUT, if it has already looped 5 or more times,
then before it does that, it calls "wait -n".

wait -n waits for one job to terminate.

Let's suppose the inputlist contains:

1
2
3
4
5
6
7
8

The first time through the loop, we run my_job 1.  The second time, my_job 2,
and so on, up to my_job 5.  All of those are run without any delay.
So there are 5 instances of my_job running simultaneously in the background.

The sixth time through the loop, i is 5 (before the increment), so
we run "wait -n".  This waits for one of the 5 instances of my_job to
finish.  Once that happens, we run my_job 6, and loop again.

The same happens for my_job 7, and my_job 8.  Each one is preceded by
a wait -n, so it waits for one of the existing jobs to terminate before
the new job is launched.

[toc] | [standalone]

Back to top | Article view | gnu.bash.bug

csiph-web