Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #223961 > unrolled thread

Looping Shell Scripts and System Load

Back to article view | Back to linux.debian.user

Contents

  Looping Shell Scripts and System Load "Martin McCormick" <martin.m@suddenlink.net> - 2020-06-24 19:20 +0200
    Re: Looping Shell Scripts and System Load Roberto C. Sánchez <roberto@debian.org> - 2020-06-24 19:30 +0200
      Re: Looping Shell Scripts and System Load Greg Wooledge <wooledg@eeg.ccf.org> - 2020-06-24 19:50 +0200
    Re: Looping Shell Scripts and System Load Greg Wooledge <wooledg@eeg.ccf.org> - 2020-06-24 19:50 +0200
      Re: Looping Shell Scripts and System Load Roger Price <debian@rogerprice.org> - 2020-06-24 20:40 +0200
        Re: Looping Shell Scripts and System Load Greg Wooledge <wooledg@eeg.ccf.org> - 2020-06-24 21:00 +0200
          Re: Looping Shell Scripts and System Load "Martin McCormick" <martin.m@suddenlink.net> - 2020-06-24 22:00 +0200
      Re: Looping Shell Scripts and System Load Tixy <tixy@yxit.co.uk> - 2020-06-25 11:30 +0200
        Re: Looping Shell Scripts and System Load David <bouncingcats@gmail.com> - 2020-06-25 15:40 +0200
    Re: Looping Shell Scripts and System Load David Christensen <dpchrist@holgerdanske.com> - 2020-06-24 21:40 +0200

#223961 — Looping Shell Scripts and System Load

I wrote a shell script that unzips documents and I originally
wrote it such that it gets document #1, unzips it then gets
document #2, etc and it does that just fine so I wondered if I
could make it run faster by starting several processes at once,
each one unzipping a file.  It's certainly still running and will
eventually finish but I created a monster because it starts as
many processes as there are items to unzip.

#!/bin/sh
unarchive ()  {
 unzip $1
return 0
}
MEDIADIR=`pwd`
mountpoint /mags >/dev/null  ||mount /mags
mountpoint /mags >/dev/null || exit 1
cd /mags
#rm -r -f *
     for MEDIAFILE in `ls $MEDIADIR/*`; do
dirname=`basename $MEDIAFILE`
mkdir $dirname
cd $dirname
unarchive $MEDIAFILE &
cd ../
done
wait
cd ~
umount /mags
exit 0

	If there are 3 zipped files, it's probably going to be ok
and start 3 unzip processes.  This directory had 13 zip files and
the first 2 or 3 roared to life and then things slowed down as
they all tried to run.

	I expected this and I've been doing unix shell scripts
for literally 31 years this Summer so it is no mystery as each
new job spawns a whole new set of processes to unzip the file it
is working on while all the others are still grinding on.

	Miscreants have been known to deliberately create
loops that keep starting processes until the system crashes.

	Fortunately, this is one of my systems but this made me
wonder before I reinvent the wheel if there is a way to make a
shell script throttle itself based on current load so it keeps
slurping resources until the next iteration starts too many and
things start to bog down.  When some of the earlier or shorter
processes finish, the loop can restart and start some more unzips
until all are done.

	Right now, uptime looks like:

 11:48:07 up 26 days, 23:10,  7 users,  load average: 16.15, 15.60, 10.65

	That's pretty loaded so ideally, one could start the
looping script and it would fire up processes until things got
really busy and then not allow any more new procs to start until
some have stopped so cron and other system utilities don't stop
running which is what happens when systems get too busy.

	Thanks for any constructive suggestions.

Martin McCormick   WB5AGZ

[toc] | [next] | [standalone]

#223962

On Wed, Jun 24, 2020 at 12:19:30PM -0500, Martin McCormick wrote:
> I wrote a shell script that unzips documents and I originally
> wrote it such that it gets document #1, unzips it then gets
> document #2, etc and it does that just fine so I wondered if I
> could make it run faster by starting several processes at once,
> each one unzipping a file.  It's certainly still running and will
> eventually finish but I created a monster because it starts as
> many processes as there are items to unzip.
> 

I recommend you look at the parallel package.  It is specifically geared
toward parallelization of constructed shell command lines.  Think
something along the lines of "find .... -exec ...." but with the ability
to parallelize (in a way that considers the available CPU cores on your
system).

Regards,

-Roberto

-- 
Roberto C. Sánchez

[toc] | [prev] | [next] | [standalone]

#223965

On Wed, Jun 24, 2020 at 01:24:23PM -0400, Roberto C. Sánchez wrote:
> I recommend you look at the parallel package.  It is specifically geared
> toward parallelization of constructed shell command lines.  Think
> something along the lines of "find .... -exec ...." but with the ability
> to parallelize (in a way that considers the available CPU cores on your
> system).

In all honesty, many of us have *tried* to find a use for GNU parallel,
but it really seems to be a hammer looking for a nail.  The aggressive
self-promotion, the huge blob of nagging that it does... all of that, and
it's not even better than GNU xargs -P for 99% of the jobs that need a
tiny bit of parallelization.

[toc] | [prev] | [next] | [standalone]

#223964

On Wed, Jun 24, 2020 at 12:19:30PM -0500, Martin McCormick wrote:
> #!/bin/sh

Why?  Use bash.

> unarchive ()  {
>  unzip $1

Quotes.  <https://mywiki.wooledge.org/Quotes>

> MEDIADIR=`pwd`

Don't use all caps variable names.

Don't use backticks.  Use $() for command substitution.

Don't use $(pwd) to get the current directory.  It's in the PWD variable
already.

> mountpoint /mags >/dev/null  ||mount /mags
> mountpoint /mags >/dev/null || exit 1
> cd /mags

Check the result of cd.  Exit if it fails.  cd /mags || exit 1

> #rm -r -f *
>      for MEDIAFILE in `ls $MEDIADIR/*`; do

Do not use ls.  <https://mywiki.wooledge.org/ParsingLs>

Quotes again.  <https://mywiki.wooledge.org/Quotes>

What you want is:   for mediafile in "$mediadir"/*; do

> dirname=`basename $MEDIAFILE`
> mkdir $dirname
> cd $dirname

Quotes, quotes, quotes.  <https://mywiki.wooledge.org/Quotes>

Always check the result of a cd.  cd "$dirname" || exit 1

> unarchive $MEDIAFILE &

Quotes!  <https://mywiki.wooledge.org/Quotes>

> 	If there are 3 zipped files, it's probably going to be ok
> and start 3 unzip processes.  This directory had 13 zip files and
> the first 2 or 3 roared to life and then things slowed down as
> they all tried to run.

<https://mywiki.wooledge.org/ProcessManagement> has some examples
for writing "run n jobs at a time".  We found some newer ways as well,
and those haven't all made it to the wiki yet.

One of the better ones is:

13:40 =greybot> Run N processes in parallel (bash 4.3): i=0 n=5; for elem in 
                "${array[@]}"; do if (( i++ >= n )); then wait -n; fi; my_job 
                "$elem" & done; wait

In your script, that would be something like:

#!/bin/bash
# Requires bash 4.3 or higher.

# cd and mount and stuff

i=0 n=3
for f; do
  if ((i++ >= n)); then wait -n; fi
  unarchive "$f" &
done
wait


If you have to target older versions of bash, see the ProcessManagement
page on the wiki for alternatives.

[toc] | [prev] | [next] | [standalone]

#223966

On Wed, 24 Jun 2020, Greg Wooledge wrote:

>> MEDIADIR=`pwd`
>
> Don't use all caps variable names.

Without getting into syntax-religious wars, what is the reasoning behind this 
recommendation?  Roger

[toc] | [prev] | [next] | [standalone]

#223967

On Wed, Jun 24, 2020 at 08:23:18PM +0200, Roger Price wrote:
> On Wed, 24 Jun 2020, Greg Wooledge wrote:
> 
> > > MEDIADIR=`pwd`
> > 
> > Don't use all caps variable names.
> 
> Without getting into syntax-religious wars, what is the reasoning behind
> this recommendation?  Roger

All-caps names are reserved for environment variables (HOME, PATH),
and internal shell variables (IFS, PWD, HISTFILE).

Avoiding all-caps names allows you to avoid collisions with a variable
name that might be used for something else.  Most of the time.  This
being the Unix shell, there are *always* stupid exceptions (http_proxy
and friends on the environment side, and auto_resume and histchars in
bash).

[toc] | [prev] | [next] | [standalone]

#223971

Greg Wooledge <wooledg@eeg.ccf.org> writes:
> All-caps names are reserved for environment variables (HOME, PATH),
> and internal shell variables (IFS, PWD, HISTFILE).
> 
> Avoiding all-caps names allows you to avoid collisions with a variable
> name that might be used for something else.  Most of the time.  This
> being the Unix shell, there are *always* stupid exceptions (http_proxy
> and friends on the environment side, and auto_resume and histchars in
> bash).

Thank you.  I learned something today that I didn't
expect to learn which is why I posted the question in the first
place.  I just wasn't thinking, I guess and used the all-caps
names to indicate that they stood for files.  If one collided
with an environment variable name, it could make the script fail
in strange ways that would be totally unpredictable, depending on
which variable one preempted.

Martin

[toc] | [prev] | [next] | [standalone]

#223994

On Wed, 2020-06-24 at 13:43 -0400, Greg Wooledge wrote:
[Lots of good shell scripting advice snipped]

Thanks Greg for posting these code reviews of people's scripts, it's
not just the script authors which might learn something, but also some
of us list subscribers. :-)

-- 
Tixy

[toc] | [prev] | [next] | [standalone]

#224010

On Thu, 25 Jun 2020 at 19:23, Tixy <tixy@yxit.co.uk> wrote:
> On Wed, 2020-06-24 at 13:43 -0400, Greg Wooledge wrote:

> [Lots of good shell scripting advice snipped]

> Thanks Greg for posting these code reviews of people's scripts, it's
> not just the script authors which might learn something, but also some
> of us list subscribers. :-)

Another excellent resource for that:
https://www.shellcheck.net/

[toc] | [prev] | [next] | [standalone]

#223969

On 2020-06-24 10:19, Martin McCormick wrote:
> I wrote a shell script that unzips documents and I originally
> wrote it such that it gets document #1, unzips it then gets
> document #2, etc and it does that just fine so I wondered if I
> could make it run faster by starting several processes at once,
> each one unzipping a file.  It's certainly still running and will
> eventually finish but I created a monster because it starts as
> many processes as there are items to unzip.
> 
> #!/bin/sh
> unarchive ()  {
>   unzip $1
> return 0
> }
> MEDIADIR=`pwd`
> mountpoint /mags >/dev/null  ||mount /mags
> mountpoint /mags >/dev/null || exit 1
> cd /mags
> #rm -r -f *
>       for MEDIAFILE in `ls $MEDIADIR/*`; do
> dirname=`basename $MEDIAFILE`
> mkdir $dirname
> cd $dirname
> unarchive $MEDIAFILE &
> cd ../
> done
> wait
> cd ~
> umount /mags
> exit 0
> 
> 	If there are 3 zipped files, it's probably going to be ok
> and start 3 unzip processes.  This directory had 13 zip files and
> the first 2 or 3 roared to life and then things slowed down as
> they all tried to run.
> 
> 	I expected this and I've been doing unix shell scripts
> for literally 31 years this Summer so it is no mystery as each
> new job spawns a whole new set of processes to unzip the file it
> is working on while all the others are still grinding on.
> 
> 	Miscreants have been known to deliberately create
> loops that keep starting processes until the system crashes.
> 
> 	Fortunately, this is one of my systems but this made me
> wonder before I reinvent the wheel if there is a way to make a
> shell script throttle itself based on current load so it keeps
> slurping resources until the next iteration starts too many and
> things start to bog down.  When some of the earlier or shorter
> processes finish, the loop can restart and start some more unzips
> until all are done.
> 
> 	Right now, uptime looks like:
> 
>   11:48:07 up 26 days, 23:10,  7 users,  load average: 16.15, 15.60, 10.65
> 
> 	That's pretty loaded so ideally, one could start the
> looping script and it would fire up processes until things got
> really busy and then not allow any more new procs to start until
> some have stopped so cron and other system utilities don't stop
> running which is what happens when systems get too busy.
> 
> 	Thanks for any constructive suggestions.
> 
> Martin McCormick   WB5AGZ

GNU Parallel looks like a possibility:

https://packages.debian.org/stretch/parallel

https://www.gnu.org/software/parallel/


David

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web