Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #206423 > unrolled thread

Flushing all Buffers Before Exiting

Back to article view | Back to linux.debian.user

Contents

  Flushing all Buffers Before Exiting "Martin McCormick" <martin.m@suddenlink.net> - 2019-03-21 15:10 +0100
    Re: Flushing all Buffers Before Exiting Kenneth Parker <sea7kenp@gmail.com> - 2019-03-21 15:40 +0100
      Re: Flushing all Buffers Before Exiting <tomas@tuxteam.de> - 2019-03-21 16:20 +0100
        Re: Flushing all Buffers Before Exiting Curt <curty@free.fr> - 2019-03-21 19:10 +0100
          Re: Flushing all Buffers Before Exiting Greg Wooledge <wooledg@eeg.ccf.org> - 2019-03-21 19:20 +0100
          Re: Flushing all Buffers Before Exiting Luís Gomes <luismsgomes@gmail.com> - 2019-03-21 19:30 +0100
      Re: Flushing all Buffers Before Exiting "Martin McCormick" <martin.m@suddenlink.net> - 2019-03-21 17:40 +0100
        Re: Flushing all Buffers Before Exiting Greg Wooledge <wooledg@eeg.ccf.org> - 2019-03-21 17:50 +0100
          Re: Flushing all Buffers Before Exiting <tomas@tuxteam.de> - 2019-03-21 18:30 +0100
        Re: Flushing all Buffers Before Exiting David Wright <deblis@lionunicorn.co.uk> - 2019-03-21 19:40 +0100
          Re: Flushing all Buffers Before Exiting "Martin McCormick" <martin.m@suddenlink.net> - 2019-03-22 02:00 +0100
            Re: Flushing all Buffers Before Exiting deloptes <deloptes@gmail.com> - 2019-03-22 06:20 +0100
            Re: Flushing all Buffers Before Exiting "Jeremy Nicoll" <jn.ml.dbn.25@letterboxes.org> - 2019-03-22 17:40 +0100
            Re: Flushing all Buffers Before Exiting <tomas@tuxteam.de> - 2019-03-22 17:50 +0100
              Re: Flushing all Buffers Before Exiting David Wright <deblis@lionunicorn.co.uk> - 2019-03-23 16:30 +0100
                Re: Flushing all Buffers Before Exiting <tomas@tuxteam.de> - 2019-03-23 18:30 +0100
                  Re: Flushing all Buffers Before Exiting David Wright <deblis@lionunicorn.co.uk> - 2019-03-24 17:20 +0100
                    Re: Flushing all Buffers Before Exiting "Martin McCormick" <martin.m@suddenlink.net> - 2019-03-24 19:30 +0100
                      Re: Flushing all Buffers Before Exiting Gene Heskett <gheskett@shentel.net> - 2019-03-24 20:00 +0100
                      Re: Flushing all Buffers Before Exiting deloptes <deloptes@gmail.com> - 2019-03-24 21:40 +0100
                        Re: Flushing all Buffers Before Exiting "Martin McCormick" <martin.m@suddenlink.net> - 2019-03-25 03:00 +0100
                          Re: Flushing all Buffers Before Exiting deloptes <deloptes@gmail.com> - 2019-03-25 07:20 +0100
                    Re: Flushing all Buffers Before Exiting <tomas@tuxteam.de> - 2019-03-25 10:50 +0100
                      Re: Flushing all Buffers Before Exiting David Wright <deblis@lionunicorn.co.uk> - 2019-03-26 17:50 +0100
                        Re: Flushing all Buffers Before Exiting <tomas@tuxteam.de> - 2019-03-26 18:00 +0100
                          Re: Flushing all Buffers Before Exiting Curt <curty@free.fr> - 2019-03-26 18:40 +0100
                            Re: Flushing all Buffers Before Exiting <tomas@tuxteam.de> - 2019-03-26 19:00 +0100
                          Re: Flushing all Buffers Before Exiting David Wright <deblis@lionunicorn.co.uk> - 2019-03-27 16:00 +0100
                            Re: Flushing all Buffers Before Exiting <tomas@tuxteam.de> - 2019-03-27 16:20 +0100
    Re: Flushing all Buffers Before Exiting Lee <ler762@gmail.com> - 2019-03-21 19:50 +0100
      Re: Flushing all Buffers Before Exiting "Martin McCormick" <martin.m@suddenlink.net> - 2019-03-23 22:40 +0100

Page 2 of 2 — ← Prev page 1 [2]

#206558

deloptes <deloptes@gmail.com> writes:
> I just wonder why one would do that, but it is again your business. 

	In all but a very small handful of countries around the
world, the hobby of amateur radio exists and it's justification
for existence is to allow people to self-train as to how
electronic communication, especially radio, works.  The vast
majority of radio amateurs do not do destructive things with the
knowledge they gain but listening to non-amateur communications
systems and understanding how they work is part of the hobby.
So, we do things that may seem really strange to those who don't
look at technology that way.  

	Computers fit right in to this hobby also as they are
part of modern life.

> Nowdays
> all of this communication is encrypted and you have virtually no chance
> listening to this.

	When the day comes in which all radio communications are
encrypted except for amateur radio where encryption is illegal,
we will probably stop listening to signals other than amateur
radio and broadcasting.

	Right now, much is still in the clear.  It may be
digitally encoded but the coding standards are either to
improve reception, compress bandwidth or both.  If they are to
obscure the conversation from eaves-droppers, then the landscape
gets more complicated regarding the law.

> In most of the countries it is even illegal, but ok, 
> one
> can do things for fun anyway - braking the encryption though is close to
> impossible.

	That is quite true.  some of the encoding schemes involve
more than one layer of encryption and use 1024-bit keys or
something similar so a person who doesn't know the key or keys
involved probably doesn't have enough seconds in his or her
natural life to break even 1 set of keys much less deal with the
key-holders changing the keys every hour or so.  There are far
better ways to spend one's life.

> The communication follows well defined protocol, so knowing it, you might 
> be
> able to read the frames, but the content will remain hidden.

Quite true.

	In the case of what I am doing, a web site for scanner
radio enthusiasts published the frequencies and the logical order in
which they should be entered in to a receiver but the index
numbers turned out to be wrong due to changes made to the site
after the information was published.  The control data includes
the index number for the channel to which a conversation or part
of one is assigned so one can learn the list by reading the index
numbers and observing which channels come to life.  This allows
one to fix the list correctly.  It's like solving a partly
assembled puzzle.

	Sorry for getting far afield of the original topic.

Martin  WB5AGZ

[toc] | [prev] | [next] | [standalone]

#206566

Martin McCormick wrote:

> deloptes <deloptes@gmail.com> writes:
>> I just wonder why one would do that, but it is again your business.
> 
> In all but a very small handful of countries around the
> world, the hobby of amateur radio exists and it's justification
> for existence is to allow people to self-train as to how
> electronic communication, especially radio, works.  The vast
> majority of radio amateurs do not do destructive things with the
> knowledge they gain but listening to non-amateur communications
> systems and understanding how they work is part of the hobby.
> So, we do things that may seem really strange to those who don't
> look at technology that way.
> 
> Computers fit right in to this hobby also as they are
> part of modern life.
> 
>> Nowdays
>> all of this communication is encrypted and you have virtually no chance
>> listening to this.
> 
> When the day comes in which all radio communications are
> encrypted except for amateur radio where encryption is illegal,
> we will probably stop listening to signals other than amateur
> radio and broadcasting.
> 
> Right now, much is still in the clear.  It may be
> digitally encoded but the coding standards are either to
> improve reception, compress bandwidth or both.  If they are to
> obscure the conversation from eaves-droppers, then the landscape
> gets more complicated regarding the law.
> 

I was thinking it is forbidden for amateur radio, because theywant to listen
to you.

>> In most of the countries it is even illegal, but ok,
>> one
>> can do things for fun anyway - braking the encryption though is close to
>> impossible.
> 
> That is quite true.  some of the encoding schemes involve
> more than one layer of encryption and use 1024-bit keys or
> something similar so a person who doesn't know the key or keys
> involved probably doesn't have enough seconds in his or her
> natural life to break even 1 set of keys much less deal with the
> key-holders changing the keys every hour or so.  There are far
> better ways to spend one's life.
> 

Look at the wikipedia link I shared - it talks about the keys

>> The communication follows well defined protocol, so knowing it, you might
>> be
>> able to read the frames, but the content will remain hidden.
> 
> Quite true.
> 
> In the case of what I am doing, a web site for scanner
> radio enthusiasts published the frequencies and the logical order in
> which they should be entered in to a receiver but the index
> numbers turned out to be wrong due to changes made to the site
> after the information was published.  The control data includes
> the index number for the channel to which a conversation or part
> of one is assigned so one can learn the list by reading the index
> numbers and observing which channels come to life.  This allows
> one to fix the list correctly.  It's like solving a partly
> assembled puzzle.
> 
> Sorry for getting far afield of the original topic.
> 
> Martin  WB5AGZ

Why don't you get the documentation or at least what is publicly available
and solve it?

regards

[toc] | [prev] | [next] | [standalone]

#206571

[Multipart message — attachments visible in raw view] — view raw

On Sun, Mar 24, 2019 at 11:17:54AM -0500, David Wright wrote:
> On Sat 23 Mar 2019 at 18:23:47 (+0100), tomas@tuxteam.de wrote:
> > On Sat, Mar 23, 2019 at 10:27:01AM -0500, David Wright wrote:
> > > On Fri 22 Mar 2019 at 17:45:50 (+0100), tomas@tuxteam.de wrote:
> > 
> > > Reading the OP's problem, I wonder how you're meant to detect
> > > "any whiff of a problem" [...]
> > 
> > Torture tests.
> 
> Like, multiply the number of sources by stealing a few more radio
> scanners to connect up, which then all burst into life as the
> police scour the neighbourhood for thieves?

Tests require some modeling and phantasy. And some knowledge of
the environment. Otherwise I'd call it "futzing around" -- still
important, but at an earlier stage.

We don't know OP's data sources; for modeling, one should have
a couple of parameters, like expected average bandwidth, expected
peak bandwith wrt interval and so on. For example, if data is
coming in via serial 115kb/s or, say USB (which one?), we'd have
the input bandwith. Knowing something about the process's digestion
would tell us how much output it will produce for a given input.
And so on.

I've been doing some industrial data acquisition stuff. You end
up doing this kind of estimations all the time: you don't want
exceptional conditions to throw your systems into confusion --
you want to have an idea on "how they fail". And then test, test
and test, to know where you made bad assumptions.

"Stealing radio scanners" would be definitely one possibility.
I'd rather try to simulate them in software -- better for the
nerves :-)

For that, you'd have to have an idea about the interface the
radio scanner has to your computer. Serial? (I strongly guess,
but can't know ;-)

We know a bit about the output: "20 to 30 lines per second".
Assuming roughly 60 - 80 chars per line, that'd make 1200 to
2400 char/s, so roughly 12 - 14 kb/s. That's way below of
what a serial interface handles. Pentium 90s did that under
Linux back then, even a bunch of interfaces at a time.

You had to put a bit of care into it, though.

> When dealing with realtime real information coming in, over which you
> have no control, it can be non-trivial to set up such scenarios.
> That's why I thought it best to devise a method that's more
> efficient than line buffering. After all, that's why buffering
> was invented, wasn't it.

Buffering is no magic: it's the tradeoff you have to make between
bandwith, latency and loss granularity (i.e. how much you're willing
to lose, should your "buffering process" fail to work properly).

Also, just saying "buffering" here is confusing, since there are
usually several levels of buffering at work: in the present case
there is

 - in-process buffering (provided by the libraries (here Perl,
   the one you control with $| resp. ->autoflush()) which goes
   the way of the do-do if your process loses control: signal,
   memory corruption, whatnot)

 - OS level buffering (the one controlled with sync and friends),
   which offers somewhat stronger guarantees (because your OS
   kernel --ahem!-- never loses control [1]).

It comes down to understanding what's going on, more or less.
Given the few data points we have, I'm pretty confident in
saying "write away, trust your hardware/OS, but watch out for
possible problems. Make your back-of-the-envelope calculations
and run a few tests to validate them".

Your proposed "catching of signals" sounds elegant, but know that
it'll only protect you during fair weather. For a SIGHUP or
SIGUSR1 it'll be fine, for a SIGSEGV or a SIGILL (which is telling
you that your process might be highly confused: perhaps the
pointer to the buffer you're trying to flush points to something
completely different!) it might be... not that fine. It's a bit
like doing backups: you'd better done that earlier :-D

Cheers

[1] Database folks are a bit more restrictive than that and
   want to perform well even when the OS loses control. One
   can learn quite a bit from them.

-- tomás

[toc] | [prev] | [next] | [standalone]

#206630

On Mon 25 Mar 2019 at 10:47:08 (+0100), tomas@tuxteam.de wrote:
> On Sun, Mar 24, 2019 at 11:17:54AM -0500, David Wright wrote:
> > On Sat 23 Mar 2019 at 18:23:47 (+0100), tomas@tuxteam.de wrote:
> > > On Sat, Mar 23, 2019 at 10:27:01AM -0500, David Wright wrote:
> > > > On Fri 22 Mar 2019 at 17:45:50 (+0100), tomas@tuxteam.de wrote:
> > > 
> > > > Reading the OP's problem, I wonder how you're meant to detect
> > > > "any whiff of a problem" [...]
> > > 
> > > Torture tests.
> > 
> > Like, multiply the number of sources by stealing a few more radio
> > scanners to connect up, which then all burst into life as the
> > police scour the neighbourhood for thieves?
> 
> Tests require some modeling and phantasy. And some knowledge of
> the environment. Otherwise I'd call it "futzing around" -- still
> important, but at an earlier stage.
> 
> We don't know OP's data sources; for modeling, one should have
> a couple of parameters, like expected average bandwidth, expected
> peak bandwith wrt interval and so on. For example, if data is
> coming in via serial 115kb/s or, say USB (which one?), we'd have
> the input bandwith. Knowing something about the process's digestion
> would tell us how much output it will produce for a given input.
> And so on.
> 
> I've been doing some industrial data acquisition stuff. You end
> up doing this kind of estimations all the time: you don't want
> exceptional conditions to throw your systems into confusion --
> you want to have an idea on "how they fail". And then test, test
> and test, to know where you made bad assumptions.
> 
> "Stealing radio scanners" would be definitely one possibility.
> I'd rather try to simulate them in software -- better for the
> nerves :-)
> 
> For that, you'd have to have an idea about the interface the
> radio scanner has to your computer. Serial? (I strongly guess,
> but can't know ;-)
> 
> We know a bit about the output: "20 to 30 lines per second".
> Assuming roughly 60 - 80 chars per line, that'd make 1200 to
> 2400 char/s, so roughly 12 - 14 kb/s. That's way below of
> what a serial interface handles. Pentium 90s did that under
> Linux back then, even a bunch of interfaces at a time.
> 
> You had to put a bit of care into it, though.

Sure, but I didn't want to presume upon the OP. That can be a lot of
effort. (I too have worked on data acquisition with simultaneous
machine control. running on far less flexible OSes.)

> > When dealing with realtime real information coming in, over which you
> > have no control, it can be non-trivial to set up such scenarios.
> > That's why I thought it best to devise a method that's more
> > efficient than line buffering. After all, that's why buffering
> > was invented, wasn't it.
> 
> Buffering is no magic: it's the tradeoff you have to make between
> bandwith, latency and loss granularity (i.e. how much you're willing
> to lose, should your "buffering process" fail to work properly).
> 
> Also, just saying "buffering" here is confusing, since there are
> usually several levels of buffering at work: in the present case
> there is
> 
>  - in-process buffering (provided by the libraries (here Perl,
>    the one you control with $| resp. ->autoflush()) which goes
>    the way of the do-do if your process loses control: signal,
>    memory corruption, whatnot)
> 
>  - OS level buffering (the one controlled with sync and friends),
>    which offers somewhat stronger guarantees (because your OS
>    kernel --ahem!-- never loses control [1]).
> 
> It comes down to understanding what's going on, more or less.
> Given the few data points we have, I'm pretty confident in
> saying "write away, trust your hardware/OS, but watch out for
> possible problems. Make your back-of-the-envelope calculations
> and run a few tests to validate them".
> 
> Your proposed "catching of signals" sounds elegant, but know that
> it'll only protect you during fair weather. For a SIGHUP or
> SIGUSR1 it'll be fine, for a SIGSEGV or a SIGILL (which is telling
> you that your process might be highly confused: perhaps the
> pointer to the buffer you're trying to flush points to something
> completely different!) it might be... not that fine. It's a bit
> like doing backups: you'd better done that earlier :-D
> 
> Cheers
> 
> [1] Database folks are a bit more restrictive than that and
>    want to perform well even when the OS loses control. One
>    can learn quite a bit from them.

I can see why the most important criterion with a database might be
protecting it from any sort of corruption, but I don't think we're
dealing with that here.

In my experience of these sorts of problems, the most important
criterion was not to waste the precious samples being analysed,
which meant preventing the OS from ever losing control. Again,
I don't think we're dealing with that here.

So these are issues beyond the scope of the OP's problem domain.
OTOH, IMO, trapping a USR1 signal before terminating the program
and, say, turning off the scanner, *is* within the scope of the
OP. I just don't understand why one needs to escalate it into
trapping *all* signals and the associated complications with
managing that.

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#206634

[Multipart message — attachments visible in raw view] — view raw

On Tue, Mar 26, 2019 at 11:43:24AM -0500, David Wright wrote:

[...]

> So these are issues beyond the scope of the OP's problem domain.
> OTOH, IMO, trapping a USR1 signal [...]

In this case, an atexit handler seems the right tool. Since the
OP is using Perl, the END {...} block is our friend.

Cheers
-- t

[toc] | [prev] | [next] | [standalone]

#206637

On 2019-03-26, <tomas@tuxteam.de> <tomas@tuxteam.de> wrote:
>
>
> On Tue, Mar 26, 2019 at 11:43:24AM -0500, David Wright wrote:
>
> [...]
>
>> So these are issues beyond the scope of the OP's problem domain.
>> OTOH, IMO, trapping a USR1 signal [...]
>
> In this case, an atexit handler seems the right tool. Since the
> OP is using Perl, the END {...} block is our friend.

Is this in the ballpark?

 $SIG{INT} = \&tsktsk;

 sub tsktsk {
    $SIG{INT} = \&tsktsk;           # See ``Writing A Signal Handler''
    warn "\aThe long habit of living indisposeth us for dying.\n";
}

[toc] | [prev] | [next] | [standalone]

#206639

[Multipart message — attachments visible in raw view] — view raw

On Tue, Mar 26, 2019 at 05:37:49PM -0000, Curt wrote:
> On 2019-03-26, <tomas@tuxteam.de> <tomas@tuxteam.de> wrote:
> >
> >
> > On Tue, Mar 26, 2019 at 11:43:24AM -0500, David Wright wrote:
> >
> > [...]
> >
> >> So these are issues beyond the scope of the OP's problem domain.
> >> OTOH, IMO, trapping a USR1 signal [...]
> >
> > In this case, an atexit handler seems the right tool. Since the
> > OP is using Perl, the END {...} block is our friend.
> 
> Is this in the ballpark?
> 
>  $SIG{INT} = \&tsktsk;
> 
>  sub tsktsk {
>     $SIG{INT} = \&tsktsk;           # See ``Writing A Signal Handler''
>     warn "\aThe long habit of living indisposeth us for dying.\n";
> }

It is -- but as I wrote, rather use

  END {
   warn "blah blah..."
  }

would be (a) simpler and (b) work also for $SIG{HUP}, and for some
uncaught die() and so on.

Cheers
-- t

[toc] | [prev] | [next] | [standalone]

#206672

On Tue 26 Mar 2019 at 17:55:28 (+0100), tomas@tuxteam.de wrote:
> On Tue, Mar 26, 2019 at 11:43:24AM -0500, David Wright wrote:
> 
> [...]
> 
> > So these are issues beyond the scope of the OP's problem domain.
> > OTOH, IMO, trapping a USR1 signal [...]
> 
> In this case, an atexit handler seems the right tool. Since the
> OP is using Perl, the END {...} block is our friend.

Sure. Nothing wrong with that. The advantage of trapping a signal
other than those that terminate the program is that you get more
opportunities to use it. IOW, you can send USR1 and examine whatever
properties of the program you like without actually stopping it.
It sounded as if the OP might be interested in this because they
mentioned tee earlier.

But my point was that prodding the program with signals seemed a
better fit than just forcing line buffering.

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#206674

[Multipart message — attachments visible in raw view] — view raw

On Wed, Mar 27, 2019 at 09:52:21AM -0500, David Wright wrote:

[...]

> But my point was that prodding the program with signals seemed a
> better fit than just forcing line buffering.

...and I still disagree with that (under most conditions, that is).

Cheers
-- t

[toc] | [prev] | [next] | [standalone]

#206438

On 3/21/19, Martin McCormick <martin.m@suddenlink.net> wrote:
> 	I have been using unix of various flavors for 30 years so
> this is a bit of a bone-head question except that different
> styles of unix handle this situation somewhat differently.
>
> 	Imagine that you run a process whose output you want to
> catch so you run it as someproc >catchfile.  The process has an
> end point so anything it produced gets saved in catchfile and all
> is well.
>
> 	Now imagine you run someproc and it either has no end
> condition or you haven't reached it yet so you kill it with
> Control-C.  Some unixen like FreeBSD seem to flush all the
> buffers  and you still get your output but Debian appears to not
> flush the buffers and you get nothing or maybe a partial capture
> with the most recent data lost.
>
> 	Is there a way to make sure we got everything that was
> produced?

https://unix.stackexchange.com/questions/25372/turn-off-buffering-in-pipe/

Regards,
Lee

[toc] | [prev] | [next] | [standalone]

#206531

Lee <ler762@gmail.com> writes:
> https://unix.stackexchange.com/questions/25372/turn-off-buffering-in-pipe/
> 
> Regards,
> Lee

	Thank you and all others.  It turns out that getting the
autoflush to work in perl is on a par with falling off of a log
for ease of execution.

	There is a perl variable called $| which, when set to a
non-0 value can cause an immediate flush of a buffer but perl
experts recommend you not do it that way.  It's even clearer.

	At the beginning of your perl program you add a line

use IO::Handle;

That is like an include in C so the perl language handler will
know what you mean when you add the next line right after you
write to your file:

FH->autoflush(1);

FH is just my example name for the open file handle.  Perl custom
is to use all caps for the name of the file descriptor so FH or
LOGDATA refers to that file and flushes that buffer.

	In this case, it is an endless loop so the buffer gets
flushed after every new write, but that's how you make it happen.

Martin

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.debian.user

csiph-web