Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #262589 > unrolled thread

Re: 12.2: fork() causing getline() to repeat stdin endlessly

Back to article view | Back to linux.debian.user

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: 12.2: fork() causing getline() to repeat stdin endlessly "Thomas Schmitt" <scdbackup@gmx.net> - 2023-10-23 10:50 +0200
    Re: 12.2: fork() causing getline() to repeat stdin endlessly "Thomas Schmitt" <scdbackup@gmx.net> - 2023-10-23 11:20 +0200
      Re: 12.2: fork() causing getline() to repeat stdin endlessly Greg Wooledge <greg@wooledge.org> - 2023-10-23 15:40 +0200
        Re: 12.2: fork() causing getline() to repeat stdin endlessly jleonard@slimy.com (Jon Leonard) - 2023-10-23 16:20 +0200
          Re: 12.2: fork() causing getline() to repeat stdin endlessly tom kronmiller <twk.personal@gmail.com> - 2023-10-23 16:40 +0200
            Re: 12.2: fork() causing getline() to repeat stdin endlessly Greg Wooledge <greg@wooledge.org> - 2023-10-23 17:20 +0200
          Re: 12.2: fork() causing getline() to repeat stdin endlessly gene heskett <gheskett@shentel.net> - 2023-10-23 18:40 +0200
            Re: 12.2: fork() causing getline() to repeat stdin endlessly <tomas@tuxteam.de> - 2023-10-23 20:10 +0200
      Re: 12.2: fork() causing getline() to repeat stdin endlessly tom kronmiller <twk.personal@gmail.com> - 2023-10-23 21:50 +0200
        Re: 12.2: fork() causing getline() to repeat stdin endlessly "Thomas Schmitt" <scdbackup@gmx.net> - 2023-10-23 22:20 +0200
          Re: 12.2: fork() causing getline() to repeat stdin endlessly Greg Wooledge <greg@wooledge.org> - 2023-10-23 22:30 +0200
            Re: 12.2: fork() causing getline() to repeat stdin endlessly "Thomas Schmitt" <scdbackup@gmx.net> - 2023-10-23 22:50 +0200
              Re: 12.2: fork() causing getline() to repeat stdin endlessly Greg Wooledge <greg@wooledge.org> - 2023-10-23 23:00 +0200
          Re: 12.2: fork() causing getline() to repeat stdin endlessly tom kronmiller <twk.personal@gmail.com> - 2023-10-24 07:20 +0200
            Re: 12.2: fork() causing getline() to repeat stdin endlessly Max Nikulin <manikulin@gmail.com> - 2023-10-24 18:10 +0200
              Re: 12.2: fork() causing getline() to repeat stdin endlessly Greg Wooledge <greg@wooledge.org> - 2023-10-24 21:30 +0200
                Re: 12.2: fork() causing getline() to repeat stdin endlessly conover@panix.com (John Conover) - 2023-10-25 01:00 +0200
                Re: 12.2: fork() causing getline() to repeat stdin endlessly jleonard@slimy.com (Jon Leonard) - 2023-10-25 04:20 +0200
                Re: 12.2: fork() causing getline() to repeat stdin endlessly Max Nikulin <manikulin@gmail.com> - 2023-10-25 04:40 +0200
                Re: 12.2: fork() causing getline() to repeat stdin endlessly <tomas@tuxteam.de> - 2023-10-25 06:30 +0200
              Re: 12.2: fork() causing getline() to repeat stdin endlessly tom kronmiller <twk.personal@gmail.com> - 2023-10-25 07:40 +0200
                Re: 12.2: fork() causing getline() to repeat stdin endlessly tom kronmiller <twk.personal@gmail.com> - 2023-10-25 08:00 +0200
    Re: 12.2: fork() causing getline() to repeat stdin endlessly Max Nikulin <manikulin@gmail.com> - 2023-10-23 18:30 +0200
      Re: 12.2: fork() causing getline() to repeat stdin endlessly Stefan Monnier <monnier@iro.umontreal.ca> - 2023-10-24 05:00 +0200

Page 1 of 2  [1] 2  Next page →

#262589 — Re: 12.2: fork() causing getline() to repeat stdin endlessly

Hi,

i can reproduce the problem with the given example after changing
  int main(int, char **) {
to
  int main(int argc, char **argv) {
in order to get it through the compiler.
(There is also a memory leak about line_buf which does not matter now.)

Not only the read offset of stdin seems to get reset by fork() but also
the line_num conter begins again at 0. (Can be an illusion. See surprise
remedy below.)

This does not happen if i watch the program by gdb. Only if i let it
run freely and printf the line_num counter and the lines, i get after
this change

     while (-1 < (num_read = getline(&line_buf, &stg_size, stdin))) {
         ++line_num;
-        printf("%s", line_buf);
-        if (num_read != 53) {
-            printf("num_read: %ld at line %lu\n", num_read, line_num);
-        }
+        printf("pid %d , line %d : %s", (int) getpid(), line_num, line_buf);

         pid_t pid = fork();
         if (0 == pid) {
             exit(0);
         }
-        else if (pid > 0) {
+        printf("fork() = %d\n", (int) pid);
+        if (pid > 0) {
             (void) waitpid(pid, NULL, 0);
         }

this output

  pid 25511 , line 1
  pid 25511 , line 1
  fork() = 25513
  pid 25511 , line 2
  pid 25511 , line 1
  fork() = 25513
  pid 25511 , line 2
  fork() = 25514
  pid 25511 , line 3
  pid 25511 , line 1
  fork() = 25513
  pid 25511 , line 2
  fork() = 25514
  pid 25511 , line 3
  fork() = 25515
  pid 25511 , line 4
  ...

Riddles:
- Why no "fork() = " after the lines which show their number for the first
  time  ?
- Why is it always one line more ?


The phenomenon vanishes if i replace printf() by fprintf(stderr).

Total diff of the changes which did this trick:
------------------------------------------------------------------------
--- getline_fork_orig.c    2023-10-23 08:51:58.788992417 +0200
+++ getline_fork_2.c       2023-10-23 10:37:28.025016315 +0200
@@ -7,7 +7,7 @@
 #include <sys/wait.h>
 #include <unistd.h>

-int main(int, char **) {
+int main(int argc, char **argv) {
     char *  line_buf = NULL;
     size_t  stg_size = 0u;
     ssize_t num_read = 0;
@@ -15,9 +15,9 @@ int main(int, char **) {

     while (-1 < (num_read = getline(&line_buf, &stg_size, stdin))) {
         ++line_num;
-        printf("%s", line_buf);
+        fprintf(stderr,"%s", line_buf);
         if (num_read != 53) {
-            printf("num_read: %ld at line %lu\n", num_read, line_num);
+            fprintf(stderr,"num_read: %ld at line %lu\n", num_read, line_num);
         }

         pid_t pid = fork();
------------------------------------------------------------------------

I am still puzzled ...


Have a nice day :)

Thomas

[toc] | [next] | [standalone]

#262590

Hi,

it helps to do
  fflush((stdout);
after each printf(), or to run before the loop:
  setvbuf(stdout, NULL, _IONBF, 0);

So it is obvious that the usual output buffering of printf() causes the
repetitions of text.

The loop does not do any extra cycles, as i could confirm by inserting
a stderr message after ++line_num:

          fprintf(stderr, "line_num= %d\n", line_num);

While the buffered printf() repeats its lines, the stderr message shows
no repetitions but counts nicely up to the end.

(I assume gdb disables buffering and thus suppresses the stdout
repetitions.)


Have a nice day :)

Thomas

[toc] | [prev] | [next] | [standalone]

#262598

On Mon, Oct 23, 2023 at 11:15:22AM +0200, Thomas Schmitt wrote:
> it helps to do
>   fflush((stdout);
> after each printf(), or to run before the loop:
>   setvbuf(stdout, NULL, _IONBF, 0);
> 
> So it is obvious that the usual output buffering of printf() causes the
> repetitions of text.

Yes, it looks like a buffering issue to me as well.  Or, more generally,
some kind of weird interaction between buffered stdio FILEs and the
underlying Unix file descriptors, when new processes are being fork()ed.

> The loop does not do any extra cycles, as i could confirm by inserting

Here's another interesting observation: using the original program with
only the argc/argv change, there is different behavior depending on
whether stdin points to a file, or a pipe.

unicorn:~$ seq 1 2 | ./foo
1
num_read: 2 at line 1
2
num_read: 2 at line 2
unicorn:~$ seq 1 2 > bar
unicorn:~$ ./foo < bar
1
num_read: 2 at line 1
2
num_read: 2 at line 2
2
num_read: 2 at line 3
unicorn:~$ 

Clearly there are different mechanisms in play based on whether stdin is
a regular file or a pipe.  I can't fully explain this result, though.


On Mon, Oct 23, 2023 at 12:59:38AM -0400, tom kronmiller wrote:
> Without the fork/exit/waitpid block of code, it prints the input
> successfully.  With that code included, it goes into an infinite loop
> printing the first 78 lines of input,  Every second pass over the input,the
> 78th line is garbled.
> 77                77        77        77        77xx
> 78              1         1xx

The 78th line looks like it's truncated after 16 characters.

> num_read: 30 at line 6433
> 2                  2         2         2         2xx
> 
> The input file "lines" is (all lines 52 charactures + \n == 53 characters):

53 * 77 = 4081

Add the 16 characters of the truncated line and you're right around
the magic number 4096, which is very likely the buffer size.  So, even
if I can't explain every detail, this is another strong clue that stdio
buffering is involved somehow.

[toc] | [prev] | [next] | [standalone]

#262601

On Mon, Oct 23, 2023 at 09:31:11AM -0400, Greg Wooledge wrote:
> On Mon, Oct 23, 2023 at 11:15:22AM +0200, Thomas Schmitt wrote:
> > it helps to do
> >   fflush((stdout);
> > after each printf(), or to run before the loop:
> >   setvbuf(stdout, NULL, _IONBF, 0);
> > 
> > So it is obvious that the usual output buffering of printf() causes the
> > repetitions of text.
> 
> Yes, it looks like a buffering issue to me as well.  Or, more generally,
> some kind of weird interaction between buffered stdio FILEs and the
> underlying Unix file descriptors, when new processes are being fork()ed.

More specifically, fork() does not play nicely with stdio buffering.

For performance reasons, stdio tends to read and write in chunks, reducing
the number of read() and write() calls.  Some data is stored in the stdio
buffers, and that data is getting copied to both processes in the fork()
call.

If you want to mix fork() and stdio, be sure to flush buffers before the
call to fork.  Depending on the task, it may be easier to use the underlying
read() and write() calls.

Jon Leonard 

[toc] | [prev] | [next] | [standalone]

#262604

[Multipart message — attachments visible in raw view] — view raw

On Mon, Oct 23, 2023 at 10:16 AM Jon Leonard <jleonard@slimy.com> wrote:

> More specifically, fork() does not play nicely with stdio buffering.
>

But the fork() should not be changing the address space of the calling
process.  The duplicated buffers in the child process might be an issue in
general (they aren't in this case), but the buffers in the parent process
should not be disturbed.  That seems like a bug in fork() to me.
Meanwhile, I will try the various suggestions to see if one works in the
real program.

[toc] | [prev] | [next] | [standalone]

#262607

On Mon, Oct 23, 2023 at 10:29:45AM -0400, tom kronmiller wrote:
> On Mon, Oct 23, 2023 at 10:16 AM Jon Leonard <jleonard@slimy.com> wrote:
> 
> > More specifically, fork() does not play nicely with stdio buffering.
> >
> 
> But the fork() should not be changing the address space of the calling
> process.  The duplicated buffers in the child process might be an issue in
> general (they aren't in this case), but the buffers in the parent process
> should not be disturbed.  That seems like a bug in fork() to me.

There are multiple moving parts here.  You've got the memory buffers
allocated by stdio, which are indeed private to each process.  But
you've also got the underlying file descriptors, which point to
structures in the kernel corresponding to each open file/stream.

unicorn:~$ seq 1 2 > 12
unicorn:~$ strace -f ./foo < 12
[...]
write(1, "1\n", 21
)                      = 2
write(1, "num_read: 2 at line 1\n", 22num_read: 2 at line 1
) = 22
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLDstrace: Process 471118 attached
, child_tidptr=0x7f861ca2aa10) = 471118
[pid 471118] set_robust_list(0x7f861ca2aa20, 24 <unfinished ...>
[pid 471117] wait4(471118,  <unfinished ...>
[pid 471118] <... set_robust_list resumed>) = 0
[pid 471118] lseek(0, -2, SEEK_CUR)     = 2
[pid 471118] exit_group(0)              = ?
[pid 471118] +++ exited with 0 +++
[...]

Here you can see that stdin (which points to a regular file) is rewound
by 2 bytes in the child process.  But since the child and the parent are
both sharing the same open file descriptor (per fork(2)), this means
stdin is rewound for the parent as well.

Now, I have a couple questions at this point:

1) Why did the child process rewind stdin?
2) Why did only *one* of the child processes do this, and not the rest?

But since this is not my project, I don't require answers to these
questions.  I can just live in my confused state, and accept Jon's maxim
that "fork() does not play nicely with stdio buffering".  We're not quite
up to the "nasal demons" level yet, but I feel like we're approaching it.

http://www.catb.org/jargon/html/N/nasal-demons.html

[toc] | [prev] | [next] | [standalone]

#262616

On 10/23/23 10:16, Jon Leonard wrote:
> On Mon, Oct 23, 2023 at 09:31:11AM -0400, Greg Wooledge wrote:
>> On Mon, Oct 23, 2023 at 11:15:22AM +0200, Thomas Schmitt wrote:
>>> it helps to do
>>>    fflush((stdout);
>>> after each printf(), or to run before the loop:
>>>    setvbuf(stdout, NULL, _IONBF, 0);
>>>
>>> So it is obvious that the usual output buffering of printf() causes the
>>> repetitions of text.
>>
>> Yes, it looks like a buffering issue to me as well.  Or, more generally,
>> some kind of weird interaction between buffered stdio FILEs and the
>> underlying Unix file descriptors, when new processes are being fork()ed.
> 
> More specifically, fork() does not play nicely with stdio buffering.
> 
> For performance reasons, stdio tends to read and write in chunks, reducing
> the number of read() and write() calls.  Some data is stored in the stdio
> buffers, and that data is getting copied to both processes in the fork()
> call.
> 
> If you want to mix fork() and stdio, be sure to flush buffers before the
> call to fork.  Depending on the task, it may be easier to use the underlying
> read() and write() calls.
> 
> Jon Leonard

This thread seems related to a problem we've encountered when reloading 
an edited file, forcing the user to use the file->open menu to reload a 
program just edited, Using the file->reload doesn't always work, nor 
does a mouse button marked reload, they both flip a quarter to decide if 
you get the new code, or the old code that still exists in a cache 
somewhere. Probably in /tmp under a hashed name.  Is there a guaranteed 
fix afoot that can be shared?
> 
> .

Cheers, Gene Heskett.
-- 
"There are four boxes to be used in defense of liberty:
  soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
  - Louis D. Brandeis

[toc] | [prev] | [next] | [standalone]

#262623

[Multipart message — attachments visible in raw view] — view raw

On Mon, Oct 23, 2023 at 12:34:17PM -0400, gene heskett wrote:

[...]

> This thread seems related to a problem we've encountered when reloading an
> edited file, forcing the user to use the file->open menu to reload a program
> just edited [...]

Throw away your editor. Mine doesn't do that.

Cheers
-- 
t

[toc] | [prev] | [next] | [standalone]

#262626

[Multipart message — attachments visible in raw view] — view raw

I ended up using setvbuf(stdin, NULL, _IONBF, 0) in the parent process and
that seems to have fixed the actual program I was having trouble with.

On Mon, Oct 23, 2023 at 10:19 AM Thomas Schmitt <scdbackup@gmx.net> wrote:

> Hi,
>
> it helps to do
>   fflush((stdout);
> after each printf(), or to run before the loop:
>   setvbuf(stdout, NULL, _IONBF, 0);
>
> So it is obvious that the usual output buffering of printf() causes the
> repetitions of text.
>
> The loop does not do any extra cycles, as i could confirm by inserting
> a stderr message after ++line_num:
>
>           fprintf(stderr, "line_num= %d\n", line_num);
>
> While the buffered printf() repeats its lines, the stderr message shows
> no repetitions but counts nicely up to the end.
>
> (I assume gdb disables buffering and thus suppresses the stdout
> repetitions.)
>
>
> Have a nice day :)
>
> Thomas
>
>

[toc] | [prev] | [next] | [standalone]

#262628

Hi,

tom kronmiller wrote:
> I ended up using setvbuf(stdin, NULL, _IONBF, 0) in the parent process and
> that seems to have fixed the actual program I was having trouble with.

stdin ? Not setvbuf(stdout, NULL, _IONBF, 0) ?

That would be one of the weirder remedies and explanations which can be
found in the web.

The one pointed to by Max Nikulin:

> https://stackoverflow.com/questions/2530663/printf-anomaly-after-fork
> fork clones stdout buffer and child exit flushes its content.

would halfways explain what i see with unwritten data in the stdout
buffer. But i am not sure that this is what man 2 fork means by:

  *  The child inherits copies of the parent's set of open file  descrip‐
     tors.   Each  file  descriptor  in the child refers to the same open
     file description (see open(2)) as the corresponding file  descriptor
     in  the parent.  This means that the two descriptors share open file
     status flags, current file offset, and signal-driven I/O  attributes
     (see the description of F_SETOWN and F_SETSIG in fcntl(2)).

Further i wonder why i never had such problems with fork() during the
last decades. Will have to inspect my own programs what exactly they do
with stdout around forking.


Jon Leonard wrote:
> More specifically, fork() does not play nicely with stdio buffering.

That's a good summary of the internet's consensus on that topic.


> Depending on the task, it may be easier to use the underlying
> read() and write() calls.

I am curious whether write() to file descriptor 1 would show different
behavior than printf(). Maybe tomorrow ...


Have a nice day :)

Thomas

[toc] | [prev] | [next] | [standalone]

#262630

On Mon, Oct 23, 2023 at 10:12:44PM +0200, Thomas Schmitt wrote:
> The one pointed to by Max Nikulin:
> 
> > https://stackoverflow.com/questions/2530663/printf-anomaly-after-fork
> > fork clones stdout buffer and child exit flushes its content.
> 
> would halfways explain what i see with unwritten data in the stdout
> buffer. But i am not sure that this is what man 2 fork means by:
> 
>   *  The child inherits copies of the parent's set of open file  descrip‐
>      tors.   Each  file  descriptor  in the child refers to the same open
>      file description (see open(2)) as the corresponding file  descriptor
>      in  the parent.  This means that the two descriptors share open file
>      status flags, current file offset, and signal-driven I/O  attributes
>      (see the description of F_SETOWN and F_SETSIG in fcntl(2)).
> 
> Further i wonder why i never had such problems with fork() during the
> last decades. Will have to inspect my own programs what exactly they do
> with stdout around forking.

I feel I only have a partial understanding as well.

The fork(2) man page, at least this section, is talking only about the
kernel's behavior.  The data structure which the kernel maintains for
each open file, which is referred to by numeric file descriptor, acts
as the man page specifies.

This is independent of the memory buffers that the stdio library uses.

If a program which is using buffered stdout (printf and friends) calls
fork(), with unflushed data still in the output buffer, the buffer
gets copied to the child.  When the child exits, that buffer gets
flushed, which can cause duplication of output once the parent also
flushes its copy.

What I still don't understand, however, is why sometimes stdin gets
rewound (lseek backwards) by the child process.

Since the parent and child DO share the kernel's open file structure (as
cited in the man page above), when the child rewinds stdin, this affects
the parent's file descriptor as well.  Thus, the parent re-reads input.

This is what's causing the loop to iterate more times than it should,
and to re-process input.  I believe that was the original question that
prompted this thread (note the words "repeat" and "endlessly" in the
Subject:), and unfortunately I don't have an answer for it yet.

[toc] | [prev] | [next] | [standalone]

#262632

Hi,

Greg Wooledge wrote:
> This is what's causing the loop to iterate more times than it should,
> and to re-process input.

That's not what i see in my experiments.
I see stuttering output which first repeats the lines put out so far
before it adds a new line.
The getline() loop iterates as often as there are input lines.

The fact that you report input woes matches tom kronmiller's reported
remedy of setvbuf(stdin, NULL, _IONBF, 0).
So maybe i see something completely different than both of you.
Will make more experiments.


Have a nice day :)

Thomas

[toc] | [prev] | [next] | [standalone]

#262633

On Mon, Oct 23, 2023 at 10:45:44PM +0200, Thomas Schmitt wrote:
> Greg Wooledge wrote:
> > This is what's causing the loop to iterate more times than it should,
> > and to re-process input.
> 
> That's not what i see in my experiments.
> I see stuttering output which first repeats the lines put out so far
> before it adds a new line.
> The getline() loop iterates as often as there are input lines.

Here's what I'm seeing.  First, with a tiny two-line input file, I
see three loop iterations:

unicorn:~$ cat 12
1
2
unicorn:~$ ./foo < 12
1
num_read: 2 at line 1
2
num_read: 2 at line 2
2
num_read: 2 at line 3
unicorn:~$ 

That's the case where I used strace and saw the lseek() in the first
child process (but not in any later child processes, which was even
*more* confusing).

Next, with a larger three-line input file:

unicorn:~$ cat bar
This is one line of a text file containing many lines of text.
This is one line of a text file containing many lines of text.
This is one line of a text file containing many lines of text.
unicorn:~$ ./foo < bar
This is one line of a text file containing many lines of text.
num_read: 63 at line 1
This is one line of a text file containing many lines of text.
num_read: 63 at line 2
This is one line of a text file containing many lines of text.
num_read: 63 at line 3
This is one line of a text file containing many lines of text.
num_read: 63 at line 4
This is one line of a text file containing many lines of text.
num_read: 63 at line 5
This is one line of a text file containing many lines of text.
num_read: 63 at line 6
This is one line of a text file containing many lines of text.
num_read: 63 at line 7
This is one line of a text file containing many lines of text.
num_read: 63 at line 8
[...]
This is one line of a text file containing many lines of text.
num_read: 63 at line 413
This is one line of a text file containing many lines of text.
num_read: 63 at line 414
^C
unicorn:~$ 

I am fairly sure this would repeat indefinitely.  I didn't let it go
more than a few seconds, though.

My system: Debian 12, Linux 6.1.0-13-amd64, gcc (Debian 12.2.0-14) 12.2.0,
libc6:amd64 2.36-9+deb12u3.

[toc] | [prev] | [next] | [standalone]

#262645

[Multipart message — attachments visible in raw view] — view raw

tom kronmiller wrote:
> I ended up using setvbuf(stdin, NULL, _IONBF, 0) in the parent process and
> that seems to have fixed the actual program I was having trouble with.

thomas schmitt asked:

> stdin ? Not setvbuf(stdout, NULL, _IONBF, 0) ?


Yes, stdin.  The problem I was having was stdin getting repeated endlessly,
so I unbuffered stdin and that seemed to make it happy.  I'm guessing that
the exit of the child process is doing a fflush(stdin) which is causing the
file descriptor to get rewound so that the parent process gets that stdin
data again.  But maybe this is nonsensical, I just observed that
unbuffering stdin made my toy program and the original program it's
extracted from work as expected.

[toc] | [prev] | [next] | [standalone]

#262674

On 24/10/2023 12:18, tom kronmiller wrote:
> so I unbuffered stdin and that seemed to make it happy.

It might be performance killer. Even fflush(NULL) before fork() may be 
better.

https://stackoverflow.com/questions/50110992/why-does-forking-my-process-cause-the-file-to-be-read-infinitely
"Why does forking my process cause the file to be read infinitely"

extensively explains that seek offset is shared by inherited file 
description and flushing stdin in child affects its parent. However 
buffers are independent. Accordingly to POSIX "The application shall 
prepare for a fork() exactly as if it were a change of active handle."

glibc bug report was closed as invalid
https://sourceware.org/bugzilla/show_bug.cgi?id=23151

I am curious why macOS behaves differently.

[toc] | [prev] | [next] | [standalone]

#262682

On Tue, Oct 24, 2023 at 11:01:55PM +0700, Max Nikulin wrote:
> On 24/10/2023 12:18, tom kronmiller wrote:
> > so I unbuffered stdin and that seemed to make it happy.
> 
> It might be performance killer. Even fflush(NULL) before fork() may be
> better.
> 
> https://stackoverflow.com/questions/50110992/why-does-forking-my-process-cause-the-file-to-be-read-infinitely
> "Why does forking my process cause the file to be read infinitely"

Ooh, that's got an *excellent* answer.

> glibc bug report was closed as invalid
> https://sourceware.org/bugzilla/show_bug.cgi?id=23151
> 
> I am curious why macOS behaves differently.

It would have been nice if the glibc developer had explained a bit, but
of course we aren't owed any explanations.

At this point, we can conclude that the bug is in fact in the OP's C
program.  The underlying C compiler, C library, and Linux kernel are
all behaving within specs.

At this point I still don't know *why* glibc rewinds stdin intermittently
on exit().  Apparently Mac OS X doesn't, or at least didn't at the time
the answer was written (2018).  I guess there must be some reason for it,
and it's not just randomly pranking people, even if I don't understand
the reason.

This has been a most educational thread.

[toc] | [prev] | [next] | [standalone]

#262685

Greg Wooledge writes:
> On Tue, Oct 24, 2023 at 11:01:55PM +0700, Max Nikulin wrote:
> 
> At this point, we can conclude that the bug is in fact in the OP's C
> program.  The underlying C compiler, C library, and Linux kernel are
> all behaving within specs.
> 
> At this point I still don't know *why* glibc rewinds stdin intermittently
> on exit().  Apparently Mac OS X doesn't, or at least didn't at the time
> the answer was written (2018).  I guess there must be some reason for it,
> and it's not just randomly pranking people, even if I don't understand
> the reason.
> 
> This has been a most educational thread.

Hi Greg.

Yes, Greg, indeed!

I have been fighting a similar problem in fork(2). The program was
written in the 1990's, and recompiled in 2006 for amd64. The program
ran 24/7 for 17 years, until several months ago, when the file was
erratically truncated to zero bytes, every few days.

Does anyone know if there are any recent changes to gcc/libraries for
fork(2) and/or system(3)?

    Be safe,

    John

-- 

John Conover, conover@panix.com, http://www.johncon.com/

[toc] | [prev] | [next] | [standalone]

#262692

On Tue, Oct 24, 2023 at 03:19:43PM -0400, Greg Wooledge wrote:
> On Tue, Oct 24, 2023 at 11:01:55PM +0700, Max Nikulin wrote:
> > On 24/10/2023 12:18, tom kronmiller wrote:
> > > so I unbuffered stdin and that seemed to make it happy.
> > 
> > It might be performance killer. Even fflush(NULL) before fork() may be
> > better.

fflush(NULL) is almost certainly cheaper, and something like it is necessary.
The call to fork() does undefined things to stdio state if the relevant
files aren't flushed.  Correctness matters more than performance, though,
so "correct but might be slow" could be a fine answer.

The stdio functions and fork() come from different standards, so it's not
all that surprising that it's tricky to use both.

> > https://stackoverflow.com/questions/50110992/why-does-forking-my-process-cause-the-file-to-be-read-infinitely
> > "Why does forking my process cause the file to be read infinitely"
> 
> Ooh, that's got an *excellent* answer.

That brings up a subtlety:  I've been assuming a Linux-like system, where
stdio stuff has an in-process cache, backed by a Unix-style file descriptor.
As that page points out, the relevant standards permit quite a bit more
than that, much of which is "undefined behavior".  That's usually bad news
as compiler-writer code for "you're not allowed to do that, and we make
no promises whatsoever as to what happens if you try."

As this example shows, getting "undefined behavior" can be really quite
surprising.

> > glibc bug report was closed as invalid
> > https://sourceware.org/bugzilla/show_bug.cgi?id=23151
> > 
> > I am curious why macOS behaves differently.
> 
> It would have been nice if the glibc developer had explained a bit, but
> of course we aren't owed any explanations.
> 
> At this point, we can conclude that the bug is in fact in the OP's C
> program.  The underlying C compiler, C library, and Linux kernel are
> all behaving within specs.
> 
> At this point I still don't know *why* glibc rewinds stdin intermittently
> on exit().  Apparently Mac OS X doesn't, or at least didn't at the time
> the answer was written (2018).  I guess there must be some reason for it,
> and it's not just randomly pranking people, even if I don't understand
> the reason.

The standard says that exit() does cleanup like things queued for atexit()
and flushes any open files.  There's a good reason for that:  If you produce
some output from a program, you'd be surprised if it just got discarded
at the end of the program because it was sitting in an output buffer
that hadn't been flushed.  If you want the other behavior, there is the
similarly named _exit() that doesn't flush output, presumably for cases
where there was a fork() for some small task and flushing buffers first
wasn't viable.  I'd read the documentation carefully, or maybe design
to avoid having to read that documnetation.

What's probably going on is that under different circumstances the cache
of the input is used differently.  Maybe the MacOS libc is less prone
to read-ahead.  Without digging through the code or nominally-opaque
FILE* structures, it's hard to say.  But I'm pretty sure it's
"intermittently the FILE was in a state that didn't cause problems", not
"exit() intermittently flushes".

Jon Leonard

[toc] | [prev] | [next] | [standalone]

#262693

On 25/10/2023 02:19, Greg Wooledge wrote:
> At this point I still don't know *why* glibc rewinds stdin 
> intermittently on exit().

Consider a parent process that reads some file. When a specific keyword 
appears there, it should start a child that parses the same file till 
another keyword. When the child exits the parent continues reading the 
the file from the point where the child stopped.

Buffered input in child may cause offset beyond the point where the 
child faced terminating keyword. It is reasonable to rewind the input 
file to the position just after last string actually processed by the child.

So flush on exit looks like a way to make applications a bit more 
reliable. However the price is that developers must be careful with fork.

[toc] | [prev] | [next] | [standalone]

#262697

[Multipart message — attachments visible in raw view] — view raw

On Tue, Oct 24, 2023 at 03:19:43PM -0400, Greg Wooledge wrote:

[...]

> This has been a most educational thread.

Absolutely. Thanks to all :)

Cheers
-- 
t

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.debian.user

csiph-web