Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #198646 > unrolled thread

non-blocking stdin from bash

Back to article view | Back to linux.debian.user

Contents

  non-blocking stdin from bash Jim Popovitch <jim@k4vqc.com> - 2018-08-12 00:10 +0200
    Re: non-blocking stdin from bash <tomas@tuxteam.de> - 2018-08-12 00:50 +0200
      Re: non-blocking stdin from bash Jim Popovitch <jim@k4vqc.com> - 2018-08-12 07:30 +0200
        Re: non-blocking stdin from bash <tomas@tuxteam.de> - 2018-08-12 10:40 +0200
          Re: non-blocking stdin from bash Jim Popovitch <jim@k4vqc.com> - 2018-08-12 18:40 +0200
            Re: non-blocking stdin from bash <tomas@tuxteam.de> - 2018-08-12 19:50 +0200
              Re: non-blocking stdin from bash Jim Popovitch <jim@k4vqc.com> - 2018-08-12 20:00 +0200
                Re: non-blocking stdin from bash <tomas@tuxteam.de> - 2018-08-12 22:00 +0200
                Re: non-blocking stdin from bash Richard Hector <richard@walnut.gen.nz> - 2018-08-13 00:20 +0200
                  Re: non-blocking stdin from bash Jim Popovitch <jim@k4vqc.com> - 2018-08-13 00:50 +0200
                    Re: non-blocking stdin from bash Greg Wooledge <wooledg@eeg.ccf.org> - 2018-08-13 15:00 +0200
                      Re: non-blocking stdin from bash Jim Popovitch <jim@k4vqc.com> - 2018-08-13 16:40 +0200
                        Re: non-blocking stdin from bash Greg Wooledge <wooledg@eeg.ccf.org> - 2018-08-13 16:50 +0200
                          Re: non-blocking stdin from bash Jim Popovitch <jim@k4vqc.com> - 2018-08-13 17:00 +0200
                          Re: non-blocking stdin from bash Dan Purgert <dan@djph.net> - 2018-08-13 17:10 +0200
                            Re: non-blocking stdin from bash Greg Wooledge <wooledg@eeg.ccf.org> - 2018-08-13 17:10 +0200
                              Re: non-blocking stdin from bash Dan Purgert <dan@djph.net> - 2018-08-13 17:30 +0200
                          Re: non-blocking stdin from bash Richard Hector <richard@walnut.gen.nz> - 2018-08-14 08:10 +0200

#198646 — non-blocking stdin from bash

Hello!

What's the best way to grab anything that's in stdin into a variable
inside a bash script, but won't block if stdin is null?


-Jim P.

[toc] | [next] | [standalone]

#198648

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sat, Aug 11, 2018 at 06:08:34PM -0400, Jim Popovitch wrote:
> Hello!
> 
> What's the best way to grab anything that's in stdin into a variable
> inside a bash script, but won't block if stdin is null?

I think read is your friend (at least in bash). It has an option
- -t <timeout>, which you can set to zero, for it to just grab what's
available at the moment without waiting (cf "help read" while in
the bash for more details, like setting a delimiter, etc.)

HTH
- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAltvZ8gACgkQBcgs9XrR2kZ4XwCeKD4PgANQHP+QTuB+LtMPGdfP
blYAn3ovvpi+8Cx+vlaQ/5oAafdxlgww
=BWci
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

#198652

[Multipart message — attachments visible in raw view] — view raw

On Sun, 2018-08-12 at 00:48 +0200, tomas@tuxteam.de wrote:
> On Sat, Aug 11, 2018 at 06:08:34PM -0400, Jim Popovitch wrote:
> > Hello!
> > 
> > What's the best way to grab anything that's in stdin into a
> > variable
> > inside a bash script, but won't block if stdin is null?
> 
> I think read is your friend (at least in bash). It has an option
> -t <timeout>, which you can set to zero, for it to just grab what's
> available at the moment without waiting (cf "help read" while in
> the bash for more details, like setting a delimiter, etc.)
> 
> HTH
> -- tomás

Thanks for the reply tomás.  I'm trying to avoid using read because of
the 1 sec minimum timeout.  This may seem odd, but 1 sec is 100+ times
longer than grep'ing/awk'ing/sed'ing the contents of a variable, so I'm
trying to find a faster way to read stdin.  

-Jim P.

[toc] | [prev] | [next] | [standalone]

#198656

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sun, Aug 12, 2018 at 01:26:40AM -0400, Jim Popovitch wrote:
> On Sun, 2018-08-12 at 00:48 +0200, tomas@tuxteam.de wrote:
> > On Sat, Aug 11, 2018 at 06:08:34PM -0400, Jim Popovitch wrote:
> > > Hello!
> > > 
> > > What's the best way to grab anything that's in stdin into a
> > > variable
> > > inside a bash script, but won't block if stdin is null?
> > 
> > I think read is your friend (at least in bash). It has an option
> > -t <timeout>, which you can set to zero, for it to just grab what's
> > available at the moment without waiting (cf "help read" while in
> > the bash for more details, like setting a delimiter, etc.)
> > 
> > HTH
> > -- tomás
> 
> Thanks for the reply tomás.  I'm trying to avoid using read because of
> the 1 sec minimum timeout.

Not on bash, at least. According to the docs, "-t 0" is a timeout of
zero. Experiments confirm that:

  tomas@trotzki:~$ time read -t 0

  real    0m0.000s
  user    0m0.000s
  sys     0m0.000s

(Of course, 0 is too optimistic here, but it's sub-millisecond).
Fractional times work too:

  tomas@trotzki:~$ time read -t 0.05

  real    0m0.050s
  user    0m0.000s
  sys     0m0.000s

Finally, you can use timeout to read incomplete (i.e. non-terminated)
input, like so:

  tomas@trotzki:~$ time echo -n mumble | bash -c 'read -t 0.01 foo ; echo $foo'
  mumble

  real    0m0.004s
  user    0m0.000s
  sys     0m0.000s

Note that "-t 0" won't work here: the "read" at the right is there before
the "echo" at the left, so read turns out empty...

>                             This may seem odd, but 1 sec is 100+ times
> longer than grep'ing/awk'ing/sed'ing the contents of a variable, so I'm
> trying to find a faster way to read stdin.  

No, it doesn't seem odd. Sometimes 1 sec is too much (sometimes 1msec is
too much, too :-)

But I think (used wisely) read is still your friend!

Note that I didn't check how much of this is bashism. Document your
requirements :-)

Cheers
- -- t
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAltv8XkACgkQBcgs9XrR2kZNaQCcCaWfnVktQY8CENggRODJ3/nl
+1UAnRbm/rs66wz+BplEsBX3Mmp3Ed7X
=NFpA
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

#198666

On Sun, 2018-08-12 at 10:36 +0200, tomas@tuxteam.de wrote:
> On Sun, Aug 12, 2018 at 01:26:40AM -0400, Jim Popovitch wrote:
> > On Sun, 2018-08-12 at 00:48 +0200, tomas@tuxteam.de wrote:
> > > On Sat, Aug 11, 2018 at 06:08:34PM -0400, Jim Popovitch wrote:
> > > > Hello!
> > > > 
> > > > What's the best way to grab anything that's in stdin into a
> > > > variable
> > > > inside a bash script, but won't block if stdin is null?
> > > 
> > > I think read is your friend (at least in bash). It has an option
> > > -t <timeout>, which you can set to zero, for it to just grab
> > > what's
> > > available at the moment without waiting (cf "help read" while in
> > > the bash for more details, like setting a delimiter, etc.)
> > > 
> > > HTH
> > > -- tomás
> > 
> > Thanks for the reply tomás.  I'm trying to avoid using read because
> > of
> > the 1 sec minimum timeout.
> 
> Not on bash, at least. According to the docs, "-t 0" is a timeout of
> zero. Experiments confirm that:
> 
>   tomas@trotzki:~$ time read -t 0
> 
>   real    0m0.000s
>   user    0m0.000s
>   sys     0m0.000s
> 
> (Of course, 0 is too optimistic here, but it's sub-millisecond).
> Fractional times work too:
> 
>   tomas@trotzki:~$ time read -t 0.05
> 
>   real    0m0.050s
>   user    0m0.000s
>   sys     0m0.000s
> 
> Finally, you can use timeout to read incomplete (i.e. non-terminated)
> input, like so:
> 
>   tomas@trotzki:~$ time echo -n mumble | bash -c 'read -t 0.01 foo ;
> echo $foo'
>   mumble
> 
>   real    0m0.004s
>   user    0m0.000s
>   sys     0m0.000s
> 
> Note that "-t 0" won't work here: the "read" at the right is there
> before
> the "echo" at the left, so read turns out empty...
> 
> >                             This may seem odd, but 1 sec is 100+
> > times
> > longer than grep'ing/awk'ing/sed'ing the contents of a variable, so
> > I'm
> > trying to find a faster way to read stdin.  
> 
> No, it doesn't seem odd. Sometimes 1 sec is too much (sometimes 1msec
> is
> too much, too :-)
> 
> But I think (used wisely) read is still your friend!
> 
> Note that I didn't check how much of this is bashism. Document your
> requirements :-)

:-)

Turns out I had mis-read your first reply as "-t 1", chiefly because
that was all that I had found before posting here.  What now works for
me avoids the -t param:


while read stdin_line
do
   MSG=$MSG"\n"$stdin_line
done < /dev/stdin


Thanks again,

-Jim P.

[toc] | [prev] | [next] | [standalone]

#198670

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sun, Aug 12, 2018 at 12:34:08PM -0400, Jim Popovitch wrote:

[...]

> Turns out I had mis-read your first reply as "-t 1", chiefly because

Ah, I see.

> that was all that I had found before posting here.  What now works for
> me avoids the -t param:
> 
> 
> while read stdin_line
> do
>    MSG=$MSG"\n"$stdin_line
> done < /dev/stdin

This will block in the read as long as the line isn't yet terminated.
But of course fine, if it rocks your boat :-)

> Thanks again,

Hey. You're welcome

Cheers
- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAltwcrYACgkQBcgs9XrR2kYqgwCdH9J+3t6DYuGAcS/23OaZNE1i
seYAn0i65VKWm16CNrmChrmpci01kchP
=ckC2
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

#198672

On Sun, 2018-08-12 at 19:47 +0200, tomas@tuxteam.de wrote:
> On Sun, Aug 12, 2018 at 12:34:08PM -0400, Jim Popovitch wrote:
> 
> [...]
> 
> > Turns out I had mis-read your first reply as "-t 1", chiefly
> > because
> 
> Ah, I see.
> 
> > that was all that I had found before posting here.  What now works
> > for
> > me avoids the -t param:
> > 
> > 
> > while read stdin_line
> > do
> >    MSG=$MSG"\n"$stdin_line
> > done < /dev/stdin
> 
> This will block in the read as long as the line isn't yet terminated.
> But of course fine, if it rocks your boat :-)

Interesting.  I'm using it via a cron script like so:

* * * * *  grep "unusual" /opt/logs/* | /opt/notify.sh `hostname`;

so it should always be terminated.

-Jim P.

[toc] | [prev] | [next] | [standalone]

#198679

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sun, Aug 12, 2018 at 01:57:09PM -0400, Jim Popovitch wrote:
> On Sun, 2018-08-12 at 19:47 +0200, tomas@tuxteam.de wrote:

[...]

> > This will block in the read as long as the line isn't yet terminated.
> > But of course fine, if it rocks your boat :-)
> 
> Interesting.  I'm using it via a cron script like so:
> 
> * * * * *  grep "unusual" /opt/logs/* | /opt/notify.sh `hostname`;
> 
> so it should always be terminated.

Then it'll rock your boat :)

Cheers
- -- t
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAltwkQMACgkQBcgs9XrR2kYn9wCcDFQMnC4UlWESSZIRXk5Mf1pZ
djEAnjCWc7A3MJXUuznsTV/o4gRF9tJO
=PmYn
-----END PGP SIGNATURE-----

[toc] | [prev] | [next] | [standalone]

#198681

[Multipart message — attachments visible in raw view] — view raw

On 13/08/18 05:57, Jim Popovitch wrote:
> Interesting.  I'm using it via a cron script like so:
> 
> * * * * *  grep "unusual" /opt/logs/* | /opt/notify.sh `hostname`;

I don't know what's in notify.sh, but it looks to me like you're going
to get notified every minute for all the unusual log entries, whether
you've already seen them or not?

Have you looked at logcheck or other similar existing solutions?

Richard

[toc] | [prev] | [next] | [standalone]

#198682

On Mon, 2018-08-13 at 10:13 +1200, Richard Hector wrote:
> On 13/08/18 05:57, Jim Popovitch wrote:
> > Interesting.  I'm using it via a cron script like so:
> > 
> > * * * * *  grep "unusual" /opt/logs/* | /opt/notify.sh `hostname`;
> 
> I don't know what's in notify.sh, but it looks to me like you're
> going to get notified every minute for all the unusual log entries,
> whether you've already seen them or not?

That was just an example of what I'm doing.  In reality logtail's in
there, but it made the example line too long.

> Have you looked at logcheck or other similar existing solutions?

Yes, i use logcheck religiously, despite the absolute lack of logcheck
rules maintenance that exists in most packages not installed on a
default system (looking at you nginx, postfix, bind9, cron, clamav*,
openvpn, dovecot, opendmarc, monit, and openntpd).   I'd submit the
improvements to the appropriate maintainers, but I dislike rejection. 
;-)

-Jim P.

[toc] | [prev] | [next] | [standalone]

#198700

On Sat, Aug 11, 2018 at 06:08:34PM -0400, Jim Popovitch wrote:
> What's the best way to grab anything that's in stdin into a variable
> inside a bash script, but won't block if stdin is null?

On Sun, Aug 12, 2018 at 01:57:09PM -0400, Jim Popovitch wrote:
> Interesting.  I'm using it via a cron script like so:
> 
> * * * * *  grep "unusual" /opt/logs/* | /opt/notify.sh `hostname`;

On Sun, Aug 12, 2018 at 06:45:23PM -0400, Jim Popovitch wrote:
> That was just an example of what I'm doing.  In reality logtail's in
> there, but it made the example line too long.

https://mywiki.wooledge.org/XyProblem
https://mywiki.wooledge.org/BashFAQ/064

[toc] | [prev] | [next] | [standalone]

#198711

On Mon, 2018-08-13 at 08:59 -0400, Greg Wooledge wrote:
> On Sat, Aug 11, 2018 at 06:08:34PM -0400, Jim Popovitch wrote:
> > What's the best way to grab anything that's in stdin into a
> > variable inside a bash script, but won't block if stdin is null?
> 
> On Sun, Aug 12, 2018 at 01:57:09PM -0400, Jim Popovitch wrote:
> > Interesting.  I'm using it via a cron script like so:
> > 
> > * * * * *  grep "unusual" /opt/logs/* | /opt/notify.sh `hostname`;
> 
> On Sun, Aug 12, 2018 at 06:45:23PM -0400, Jim Popovitch wrote:
> > That was just an example of what I'm doing.  In reality logtail's
> > in there, but it made the example line too long.
> 
> https://mywiki.wooledge.org/XyProblem
> https://mywiki.wooledge.org/BashFAQ/064

Come on, that's not warranted.... or you just like posting links to you
own site.  smh.

-Jim P.

[toc] | [prev] | [next] | [standalone]

#198712

On Mon, Aug 13, 2018 at 10:34:40AM -0400, Jim Popovitch wrote:
> On Mon, 2018-08-13 at 08:59 -0400, Greg Wooledge wrote:
> > On Sat, Aug 11, 2018 at 06:08:34PM -0400, Jim Popovitch wrote:
> > > What's the best way to grab anything that's in stdin into a
> > > variable inside a bash script, but won't block if stdin is null?
> > 
> > On Sun, Aug 12, 2018 at 01:57:09PM -0400, Jim Popovitch wrote:
> > > Interesting.  I'm using it via a cron script like so:
> > > 
> > > * * * * *  grep "unusual" /opt/logs/* | /opt/notify.sh `hostname`;
> > 
> > On Sun, Aug 12, 2018 at 06:45:23PM -0400, Jim Popovitch wrote:
> > > That was just an example of what I'm doing.  In reality logtail's
> > > in there, but it made the example line too long.
> > 
> > https://mywiki.wooledge.org/XyProblem
> > https://mywiki.wooledge.org/BashFAQ/064
> 
> Come on, that's not warranted.... or you just like posting links to you
> own site.  smh.

It's absolutely warranted.  You changed the question, and then you changed
the question AGAIN.

The original question was about testing whether stdin has input characters
AVAILABLE, for which the feature "read -t 0" was added in bash version 4.0.

But then the question mutated, and there was no longer any need to probe
for input characters to avoid blocking, because you were simply reading
from a piped grep.

But then the question mutated AGAIN, and in this THIRD version of the
question, there is something called "logtail" involved, and I have no
idea what that even IS, or how it affects the pipeline.  One may speculate
that it works something like "tail -f" or "tail -F", in which case we
are BACK TO THE ORIGINAL QUESTION.  Maybe.  But we can't be sure, since
not only is this a nonstandard tool, but you didn't show or describe
how it's being used.

So, yeah.  It's warranted.  Idiot.

[toc] | [prev] | [next] | [standalone]

#198713

On Mon, 2018-08-13 at 10:40 -0400, Greg Wooledge wrote:
> On Mon, Aug 13, 2018 at 10:34:40AM -0400, Jim Popovitch wrote:
> > On Mon, 2018-08-13 at 08:59 -0400, Greg Wooledge wrote:
> > > On Sat, Aug 11, 2018 at 06:08:34PM -0400, Jim Popovitch wrote:
> > > > What's the best way to grab anything that's in stdin into a
> > > > variable inside a bash script, but won't block if stdin is
> > > > null?
> > > 
> > > On Sun, Aug 12, 2018 at 01:57:09PM -0400, Jim Popovitch wrote:
> > > > Interesting.  I'm using it via a cron script like so:
> > > > 
> > > > * * * * *  grep "unusual" /opt/logs/* | /opt/notify.sh
> > > > `hostname`;
> > > 
> > > On Sun, Aug 12, 2018 at 06:45:23PM -0400, Jim Popovitch wrote:
> > > > That was just an example of what I'm doing.  In reality
> > > > logtail's
> > > > in there, but it made the example line too long.
> > > 
> > > https://mywiki.wooledge.org/XyProblem
> > > https://mywiki.wooledge.org/BashFAQ/064
> > 
> > Come on, that's not warranted.... or you just like posting links to
> > you
> > own site.  smh.
> 
> It's absolutely warranted.  You changed the question, and then you
> changed the question AGAIN.

Not at all.  Even your original slicey-dicey copy+paste shows that I
used the word "like" when describing the crontab, and my original
question never defined anything about what I was doing, rather it asked
a question about how to do something.

> The original question was about testing whether stdin has input
> characters AVAILABLE

Nope. Go back and read my original question, pls.

-Jim P.

[toc] | [prev] | [next] | [standalone]

#198714

Greg Wooledge wrote:
> [...]
> It's absolutely warranted.  You changed the question, and then you changed
> the question AGAIN.

This reminds me of bashphorisms 1 through 3.  Although, I guess they're
more related to the IRC channel.


-- 
|_|O|_| Registered Linux user #585947
|_|_|O| Github: https://github.com/dpurgert
|O|O|O| PGP: 05CA 9A50 3F2E 1335 4DC5  4AEE 8E11 DDF3 1279 A281

[toc] | [prev] | [next] | [standalone]

#198715

On Mon, Aug 13, 2018 at 02:58:54PM -0000, Dan Purgert wrote:
> Greg Wooledge wrote:
> > [...]
> > It's absolutely warranted.  You changed the question, and then you changed
> > the question AGAIN.
> 
> This reminds me of bashphorisms 1 through 3.  Although, I guess they're
> more related to the IRC channel.

It happens here too, just more slowly.

[toc] | [prev] | [next] | [standalone]

#198716

Greg Wooledge wrote:
> On Mon, Aug 13, 2018 at 02:58:54PM -0000, Dan Purgert wrote:
>> Greg Wooledge wrote:
>> > [...]
>> > It's absolutely warranted.  You changed the question, and then you
>> > changed the question AGAIN..
>> 
>> This reminds me of bashphorisms 1 through 3.  Although, I guess they're
>> more related to the IRC channel.
>
> It happens here too, just more slowly.

In either event, I usually learn something; even if it's merely "don't
do this, even if you're not sure why you shouldn't."


-- 
|_|O|_| Registered Linux user #585947
|_|_|O| Github: https://github.com/dpurgert
|O|O|O| PGP: 05CA 9A50 3F2E 1335 4DC5  4AEE 8E11 DDF3 1279 A281

[toc] | [prev] | [next] | [standalone]

#198740

[Multipart message — attachments visible in raw view] — view raw

On 14/08/18 02:40, Greg Wooledge wrote:
> So, yeah.  It's warranted.

Perhaps.

>  Idiot.

That bit isn't, though.

Richard

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web