Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #206423 > unrolled thread

Flushing all Buffers Before Exiting

Back to article view | Back to linux.debian.user

Contents

  Flushing all Buffers Before Exiting "Martin McCormick" <martin.m@suddenlink.net> - 2019-03-21 15:10 +0100
    Re: Flushing all Buffers Before Exiting Kenneth Parker <sea7kenp@gmail.com> - 2019-03-21 15:40 +0100
      Re: Flushing all Buffers Before Exiting <tomas@tuxteam.de> - 2019-03-21 16:20 +0100
        Re: Flushing all Buffers Before Exiting Curt <curty@free.fr> - 2019-03-21 19:10 +0100
          Re: Flushing all Buffers Before Exiting Greg Wooledge <wooledg@eeg.ccf.org> - 2019-03-21 19:20 +0100
          Re: Flushing all Buffers Before Exiting Luís Gomes <luismsgomes@gmail.com> - 2019-03-21 19:30 +0100
      Re: Flushing all Buffers Before Exiting "Martin McCormick" <martin.m@suddenlink.net> - 2019-03-21 17:40 +0100
        Re: Flushing all Buffers Before Exiting Greg Wooledge <wooledg@eeg.ccf.org> - 2019-03-21 17:50 +0100
          Re: Flushing all Buffers Before Exiting <tomas@tuxteam.de> - 2019-03-21 18:30 +0100
        Re: Flushing all Buffers Before Exiting David Wright <deblis@lionunicorn.co.uk> - 2019-03-21 19:40 +0100
          Re: Flushing all Buffers Before Exiting "Martin McCormick" <martin.m@suddenlink.net> - 2019-03-22 02:00 +0100
            Re: Flushing all Buffers Before Exiting deloptes <deloptes@gmail.com> - 2019-03-22 06:20 +0100
            Re: Flushing all Buffers Before Exiting "Jeremy Nicoll" <jn.ml.dbn.25@letterboxes.org> - 2019-03-22 17:40 +0100
            Re: Flushing all Buffers Before Exiting <tomas@tuxteam.de> - 2019-03-22 17:50 +0100
              Re: Flushing all Buffers Before Exiting David Wright <deblis@lionunicorn.co.uk> - 2019-03-23 16:30 +0100
                Re: Flushing all Buffers Before Exiting <tomas@tuxteam.de> - 2019-03-23 18:30 +0100
                  Re: Flushing all Buffers Before Exiting David Wright <deblis@lionunicorn.co.uk> - 2019-03-24 17:20 +0100
                    Re: Flushing all Buffers Before Exiting "Martin McCormick" <martin.m@suddenlink.net> - 2019-03-24 19:30 +0100
                      Re: Flushing all Buffers Before Exiting Gene Heskett <gheskett@shentel.net> - 2019-03-24 20:00 +0100
                      Re: Flushing all Buffers Before Exiting deloptes <deloptes@gmail.com> - 2019-03-24 21:40 +0100
                        Re: Flushing all Buffers Before Exiting "Martin McCormick" <martin.m@suddenlink.net> - 2019-03-25 03:00 +0100
                          Re: Flushing all Buffers Before Exiting deloptes <deloptes@gmail.com> - 2019-03-25 07:20 +0100
                    Re: Flushing all Buffers Before Exiting <tomas@tuxteam.de> - 2019-03-25 10:50 +0100
                      Re: Flushing all Buffers Before Exiting David Wright <deblis@lionunicorn.co.uk> - 2019-03-26 17:50 +0100
                        Re: Flushing all Buffers Before Exiting <tomas@tuxteam.de> - 2019-03-26 18:00 +0100
                          Re: Flushing all Buffers Before Exiting Curt <curty@free.fr> - 2019-03-26 18:40 +0100
                            Re: Flushing all Buffers Before Exiting <tomas@tuxteam.de> - 2019-03-26 19:00 +0100
                          Re: Flushing all Buffers Before Exiting David Wright <deblis@lionunicorn.co.uk> - 2019-03-27 16:00 +0100
                            Re: Flushing all Buffers Before Exiting <tomas@tuxteam.de> - 2019-03-27 16:20 +0100
    Re: Flushing all Buffers Before Exiting Lee <ler762@gmail.com> - 2019-03-21 19:50 +0100
      Re: Flushing all Buffers Before Exiting "Martin McCormick" <martin.m@suddenlink.net> - 2019-03-23 22:40 +0100

Page 1 of 2  [1] 2  Next page →

#206423 — Flushing all Buffers Before Exiting

	I have been using unix of various flavors for 30 years so
this is a bit of a bone-head question except that different
styles of unix handle this situation somewhat differently.

	Imagine that you run a process whose output you want to
catch so you run it as someproc >catchfile.  The process has an
end point so anything it produced gets saved in catchfile and all
is well.

	Now imagine you run someproc and it either has no end
condition or you haven't reached it yet so you kill it with
Control-C.  Some unixen like FreeBSD seem to flush all the
buffers  and you still get your output but Debian appears to not
flush the buffers and you get nothing or maybe a partial capture
with the most recent data lost.

	Is there a way to make sure we got everything that was
produced?

	I have noticed that the tee program in Debian also
appears to buffer data that get lost if you end early.

	Many thanks.

	Martin McCormick

[toc] | [next] | [standalone]

#206425

[Multipart message — attachments visible in raw view] — view raw

Have you tried the Command Line:   "sync"?

Kenneth Parker

[toc] | [prev] | [next] | [standalone]

#206426

[Multipart message — attachments visible in raw view] — view raw

On Thu, Mar 21, 2019 at 10:32:06AM -0400, Kenneth Parker wrote:
> Have you tried the Command Line:   "sync"?

That won't help in the OP's case, I think: sync is about writing out
the operating system's buffers to the file system. In the OP's case
it's about the process's I/O buffers which haven't yet gone to the
operating system.

Cheers
-- t

[toc] | [prev] | [next] | [standalone]

#206434

On 2019-03-21, <tomas@tuxteam.de> <tomas@tuxteam.de> wrote:
>
>
> On Thu, Mar 21, 2019 at 10:32:06AM -0400, Kenneth Parker wrote:
>> Have you tried the Command Line:   "sync"?
>
> That won't help in the OP's case, I think: sync is about writing out
> the operating system's buffers to the file system. In the OP's case
> it's about the process's I/O buffers which haven't yet gone to the
> operating system.

I'm reading a pty app won't buffer (script, screen, etc.).

And then there's a program called 'unbuffer'?

> Cheers


-- 
“Let us again pretend that life is a solid substance, shaped like a globe,
which we turn about in our fingers. Let us pretend that we can make out a plain
and logical story, so that when one matter is despatched--love for instance--
we go on, in an orderly manner, to the next.” - Virginia Woolf, The Waves

[toc] | [prev] | [next] | [standalone]

#206435

On Thu, Mar 21, 2019 at 06:01:26PM -0000, Curt wrote:
> I'm reading a pty app won't buffer (script, screen, etc.).

Well, it's a convention, adopted by the C library functions in stdio.

stdio(3) says:

       At  program  startup, three text streams are predefined and need not be
       opened explicitly: standard input  (for  reading  conventional  input),
       standard  output  (for writing conventional output), and standard error
       (for  writing  diagnostic  output).   These  streams  are   abbreviated
       stdin,stdout and stderr.  When opened, the standard error stream is not
       fully buffered;  the  standard  input  and  output  streams  are  fully
       buffered  if  and  only  if  the streams do not refer to an interactive
       device.
       
       Output streams that refer to terminal devices are always line  buffered
       by  default;  pending  output  to such streams is written automatically
       whenever an input stream that refers to a terminal device is read.   In
       cases  where  a large amount of computation is done after printing part
       of a line on an output terminal, it is necessary to fflush(3) the stan‐
       dard  output  before  going  off  and computing so that the output will
       appear.


> And then there's a program called 'unbuffer'?

It's a hack, built around Expect.  It basically tries to fool the
application into thinking that standard output is a terminal, so the
application will operate in line-buffering mode.

I've been told that it works quite often, but you can't ever guarantee
success with it.

[toc] | [prev] | [next] | [standalone]

#206436

[Multipart message — attachments visible in raw view] — view raw

Try stdbuf.

[toc] | [prev] | [next] | [standalone]

#206429

Kenneth Parker <sea7kenp@gmail.com> writes:
> Have you tried the Command Line:   "sync"?

	Excellent question and I did, in fact, try that command
just before killing the running process.

	It had no effect.

<tomas@tuxteam.de> also writes:
> That won't help in the OP's case, I think: sync is about writing out
> the operating system's buffers to the file system. In the OP's case
> it's about the process's I/O buffers which haven't yet gone to the
> operating system.

	Thanks to both of you.  I hadn't thought of that but that
probably explains why nothing happened other than the command
"sync" successfully ran.

	I wrote the application that is creating this output in
perl and there may be a unique solution there that solves this specific
problem.  That is not as good as a general course of action which
works in all cases of output redirection but it beats nothing.
	A suggestion on a posting in stackoverflow was that one
could open the file for appending, append your new output and
then close it.

	I'll give that a try which should solve this one case.
Apparently others have dealt with how to shake the most recent
data out of buffers and commit it to disk and it is highly
dependant on the operating system as to when the write actually
goes to disk.

	Again many thanks.

Martin McCormick

[toc] | [prev] | [next] | [standalone]

#206430

On Thu, Mar 21, 2019 at 11:35:51AM -0500, Martin McCormick wrote:
> 	I wrote the application that is creating this output in
> perl

https://perl.plover.com/FAQs/Buffering.html

[toc] | [prev] | [next] | [standalone]

#206432

[Multipart message — attachments visible in raw view] — view raw

On Thu, Mar 21, 2019 at 12:46:17PM -0400, Greg Wooledge wrote:
> On Thu, Mar 21, 2019 at 11:35:51AM -0500, Martin McCormick wrote:
> > 	I wrote the application that is creating this output in
> > perl
> 
> https://perl.plover.com/FAQs/Buffering.html

This is it, thanks, Greg.

Most run times (C's FILE interface, i.e. fopen() and friends also)
have this interface. An abnormal end (i.e. signal) don't give the
application time to flush the buffers. You might want to catch
the signals and flush, but depending on the signal that might be
iffy (you sure you want to crawl on after having received a
SIGSEGV?) and sometimes impossible (SIGKILL, e.g.).

Cheers
-- t

[toc] | [prev] | [next] | [standalone]

#206437

On Thu 21 Mar 2019 at 11:35:51 (-0500), Martin McCormick wrote:

> 	I wrote the application that is creating this output in
> perl and there may be a unique solution there that solves this specific
> problem.  That is not as good as a general course of action which
> works in all cases of output redirection but it beats nothing.
> 	A suggestion on a posting in stackoverflow was that one
> could open the file for appending, append your new output and
> then close it.

An efficient way of doing this is to trap a signal, like USR1,
in your program, and react by either your close/open-append or
just flushing the buffers. That way, the program will run
normally most of the time, without wasting all that time
opening/closing files.

If there's not too much output compared with the computation necessary
to generate it, just setting line-buffering on the output stream
can be sufficient.

I've read that when the program is already running, some languages
(like Python, so probably Perl too) offer a debugger that can
allow you to flush the buffers from "within", but I've not tried
it.

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#206444

David Wright <deblis@lionunicorn.co.uk> writes:
> An efficient way of doing this is to trap a signal, like USR1,
> in your program, and react by either your close/open-append or
> just flushing the buffers. That way, the program will run
> normally most of the time, without wasting all that time
> opening/closing files.
> 
> If there's not too much output compared with the computation necessary
> to generate it, just setting line-buffering on the output stream
> can be sufficient.
> 
> I've read that when the program is already running, some languages
> (like Python, so probably Perl too) offer a debugger that can
> allow you to flush the buffers from "within", but I've not tried
> it.
> 
> Cheers,
> David.

	Before reading this posting, I added code in my perl
script to open, append and close the file but the suggestion to
add a signal handler is a much better idea so thanks for the
suggestion.

	Opening, appending and closing for each new line of
output made me a bit squeamish.  The program is monitoring a
stream of data from a radio scanner.  The data spew in at about
20 or 30 lines per second.  When nothing is happening, there are
3 possible strings that indicate nothing is happening right now.
When something changes, the strings stop matching 3 comparison
strings i put in which match each of the 3 "nothing is happening
right now" strings and the different strings get printed to the
screen and to the disk.  In reality, these strings don't exactly
mean that nothing is happening but that the same non events are
happening.

	When things change and there is output of interest, that
output also spews in at 20 or 30 lines per second so I need to do
as little as possible to handle that so the system doesn't get
swamped.  The signal handler is most likely far more efficient a
method to capture the data of interest as it will essentially not
have to make any decisions until time to shut down the program
and look at the data.

	Even with the open, append and close routine, the strings
it is capturing appear to be good but it could be capturing for
minutes on end at times and it needs to just be able to run like
the wind and store lines as quickly as it can.

	Thank you.

Martin McCormick 
amateur radio WB5AGZ

[toc] | [prev] | [next] | [standalone]

#206445

Martin McCormick wrote:

> Before reading this posting, I added code in my perl
> script to open, append and close the file but the suggestion to
> add a signal handler is a much better idea so thanks for the
> suggestion.

I always use

# Execute anytime before the <STDIN>.
# Causes the currently selected handle to be flushed after every print.
$| = 1;

when needed - but not sure if applies to your case

[toc] | [prev] | [next] | [standalone]

#206460

On Fri, 22 Mar 2019, at 00:53, Martin McCormick wrote:

> 	Opening, appending and closing for each new line of
> output made me a bit squeamish. ...

You could always count lines written and do a close & reopen 
every (say) 1000 lines.  That way there's less overhead but
the amount of data you might lose is reduced.

-- 
Jeremy Nicoll - my opinions are my own.

[toc] | [prev] | [next] | [standalone]

#206462

[Multipart message — attachments visible in raw view] — view raw

On Thu, Mar 21, 2019 at 07:52:33PM -0500, Martin McCormick wrote:

[...]

> 	Opening, appending and closing for each new line of
> output made me a bit squeamish.  The program is monitoring a
> stream of data from a radio scanner.  The data spew in at about
> 20 or 30 lines per second.

Don't fear. Measure :-)

Doesn't sound outrageous to line-buffer your output to file.

The output to screen is already line-buffered (by default,
at least) and isn't killing you, so if I were you, I'd set
up a benchmark run and torture things a bit. Then, *if* you
notice any whiff of a problem, you could try a more clever
scheme like timeout based flush to better get hold of bursts
(if I understood your description, things go out in bursts).

Cheers
-- tomás

[toc] | [prev] | [next] | [standalone]

#206522

On Fri 22 Mar 2019 at 17:45:50 (+0100), tomas@tuxteam.de wrote:
> On Thu, Mar 21, 2019 at 07:52:33PM -0500, Martin McCormick wrote:
> 
> [...]
> 
> > 	Opening, appending and closing for each new line of
> > output made me a bit squeamish.  The program is monitoring a
> > stream of data from a radio scanner.  The data spew in at about
> > 20 or 30 lines per second.
> 
> Don't fear. Measure :-)
> 
> Doesn't sound outrageous to line-buffer your output to file.
> 
> The output to screen is already line-buffered (by default,
> at least) and isn't killing you, so if I were you, I'd set
> up a benchmark run and torture things a bit. Then, *if* you
> notice any whiff of a problem, you could try a more clever
> scheme like timeout based flush to better get hold of bursts
> (if I understood your description, things go out in bursts).

Reading the OP's problem, I wonder how you're meant to detect
"any whiff of a problem". All we know is that the maximum rate
*might* be 30 lines per second, but is that a guess? Are we
already losing the odd line? How would we replicate test runs?

Probably not, at 30 lps, but in principal I would say that
this is a sticking plaster while you write your better method.

The main concern raised in the OP was flushing before termination,
for which a signal is ideal. And for best performance, I'd forget
tee and just look at the output file occasionally, with tail.

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#206526

[Multipart message — attachments visible in raw view] — view raw

On Sat, Mar 23, 2019 at 10:27:01AM -0500, David Wright wrote:
> On Fri 22 Mar 2019 at 17:45:50 (+0100), tomas@tuxteam.de wrote:

> Reading the OP's problem, I wonder how you're meant to detect
> "any whiff of a problem" [...]

Torture tests.

> The main concern raised in the OP was flushing before termination,
> for which a signal is ideal. And for best performance, I'd forget
> tee and just look at the output file occasionally, with tail.

A signal handler is definitely an option, but it can be pretty tricky.
Especially if you are catching things like SEGV (you dare a write()
after that?)

Cheers
-- t

[toc] | [prev] | [next] | [standalone]

#206545

On Sat 23 Mar 2019 at 18:23:47 (+0100), tomas@tuxteam.de wrote:
> On Sat, Mar 23, 2019 at 10:27:01AM -0500, David Wright wrote:
> > On Fri 22 Mar 2019 at 17:45:50 (+0100), tomas@tuxteam.de wrote:
> 
> > Reading the OP's problem, I wonder how you're meant to detect
> > "any whiff of a problem" [...]
> 
> Torture tests.

Like, multiply the number of sources by stealing a few more radio
scanners to connect up, which then all burst into life as the
police scour the neighbourhood for thieves?

When dealing with realtime real information coming in, over which you
have no control, it can be non-trivial to set up such scenarios.
That's why I thought it best to devise a method that's more
efficient than line buffering. After all, that's why buffering
was invented, wasn't it.

> > The main concern raised in the OP was flushing before termination,
> > for which a signal is ideal. And for best performance, I'd forget
> > tee and just look at the output file occasionally, with tail.
> 
> A signal handler is definitely an option, but it can be pretty tricky.
> Especially if you are catching things like SEGV (you dare a write()
> after that?)

SIGUSR1 to flush the buffers; that's all (see Subject line). If you
find progamming it tricky, that's a good reason for line buffering as
a stopgap. What would I do with SEGV, having trapped it? (Or most of
the other signals…)

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#206548

David Wright <deblis@lionunicorn.co.uk> writes:
> On Sat 23 Mar 2019 at 18:23:47 (+0100), tomas@tuxteam.de wrote:
> > On Sat, Mar 23, 2019 at 10:27:01AM -0500, David Wright wrote:
> > > On Fri 22 Mar 2019 at 17:45:50 (+0100), tomas@tuxteam.de wrote:
> >
> > > Reading the OP's problem, I wonder how you're meant to detect
> > > "any whiff of a problem" [...]
> >
> > Torture tests.
> 
> Like, multiply the number of sources by stealing a few more radio
> scanners to connect up, which then all burst into life as the
> police scour the neighbourhood for thieves?
> 
> When dealing with realtime real information coming in, over which you
> have no control, it can be non-trivial to set up such scenarios.
> That's why I thought it best to devise a method that's more
> efficient than line buffering. After all, that's why buffering
> was invented, wasn't it.

	Apparently, the flush after each new cycle of data isn't
taxing the system too much as the output looks correct.  This is
a 600 MHZ Pentium which would have gone in to the recycle bin
years ago if not for Linux.  Older systems like this tend to
accentuate the effects of not being able to keep up much more
obviously than if this was a quad-core 64-bit modern design.

	The best test I can do is to look at the output which is
quite repetitive as it is designed to allow radios to almost
immediately figure out what frequency and "talk group" they
should be on even if their owner turns on the radio in the middle
of a conversation.  Subsequent lines all look the same so if one
is missing part of the data, it looks wrong especially if you
have watched enough of this gibberish to damage one's brain to
the point where it starts making sense.

	Martin

[toc] | [prev] | [next] | [standalone]

#206549

On Sunday 24 March 2019 14:25:47 Martin McCormick wrote:

> David Wright <deblis@lionunicorn.co.uk> writes:
> > On Sat 23 Mar 2019 at 18:23:47 (+0100), tomas@tuxteam.de wrote:
> > > On Sat, Mar 23, 2019 at 10:27:01AM -0500, David Wright wrote:
> > > > On Fri 22 Mar 2019 at 17:45:50 (+0100), tomas@tuxteam.de wrote:
> > > >
> > > > Reading the OP's problem, I wonder how you're meant to detect
> > > > "any whiff of a problem" [...]
> > >
> > > Torture tests.
> >
> > Like, multiply the number of sources by stealing a few more radio
> > scanners to connect up, which then all burst into life as the
> > police scour the neighbourhood for thieves?
> >
> > When dealing with realtime real information coming in, over which
> > you have no control, it can be non-trivial to set up such scenarios.
> > That's why I thought it best to devise a method that's more
> > efficient than line buffering. After all, that's why buffering was
> > invented, wasn't it.
>
> 	Apparently, the flush after each new cycle of data isn't
> taxing the system too much as the output looks correct.  This is
> a 600 MHZ Pentium which would have gone in to the recycle bin
> years ago if not for Linux.  Older systems like this tend to
> accentuate the effects of not being able to keep up much more
> obviously than if this was a quad-core 64-bit modern design.
>
> 	The best test I can do is to look at the output which is
> quite repetitive as it is designed to allow radios to almost
> immediately figure out what frequency and "talk group" they
> should be on even if their owner turns on the radio in the middle
> of a conversation.  Subsequent lines all look the same so if one
> is missing part of the data, it looks wrong especially if you
> have watched enough of this gibberish to damage one's brain to
> the point where it starts making sense.
>
> 	Martin

Martin; Its one of the famous Murphy's Laws, the instant it makes sense, 
thats a security breech so it gets changed again just as soon as they 
can come up with a confusing enough to the frogs excuse. :(


Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

[toc] | [prev] | [next] | [standalone]

#206553

Martin McCormick wrote:

> Apparently, the flush after each new cycle of data isn't
> taxing the system too much as the output looks correct.  This is
> a 600 MHZ Pentium which would have gone in to the recycle bin
> years ago if not for Linux.  Older systems like this tend to
> accentuate the effects of not being able to keep up much more
> obviously than if this was a quad-core 64-bit modern design.
> 

I am not sure if the cost compared to productivity can keep up for this cpu,
but it is your business.

> The best test I can do is to look at the output which is
> quite repetitive as it is designed to allow radios to almost
> immediately figure out what frequency and "talk group" they
> should be on even if their owner turns on the radio in the middle
> of a conversation.  Subsequent lines all look the same so if one
> is missing part of the data, it looks wrong especially if you
> have watched enough of this gibberish to damage one's brain to
> the point where it starts making sense.

I just wonder why one would do that, but it is again your business. Nowdays
all of this communication is encrypted and you have virtually no chance
listening to this. In most of the countries it is even illegal, but ok, one
can do things for fun anyway - braking the encryption though is close to
impossible.
The communication follows well defined protocol, so knowing it, you might be
able to read the frames, but the content will remain hidden.
[https://en.wikipedia.org/wiki/Terrestrial_Trunked_Radio]

regards

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.debian.user

csiph-web