Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #247570 > unrolled thread

file born 30 seconds after its creation on ext4 - bug?

Back to article view | Back to linux.debian.user

Contents

  file born 30 seconds after its creation on ext4 - bug? Vincent Lefevre <vincent@vinc17.net> - 2022-04-26 15:50 +0200
    Re: file born 30 seconds after its creation on ext4 - bug? "Thomas Schmitt" <scdbackup@gmx.net> - 2022-04-26 19:10 +0200
      Re: file born 30 seconds after its creation on ext4 - bug? Vincent Lefevre <vincent@vinc17.net> - 2022-04-27 04:40 +0200
        Re: file born 30 seconds after its creation on ext4 - bug? "Thomas Schmitt" <scdbackup@gmx.net> - 2022-04-27 09:10 +0200
          Re: file born 30 seconds after its creation on ext4 - bug? Vincent Lefevre <vincent@vinc17.net> - 2022-04-27 10:30 +0200
            Re: file born 30 seconds after its creation on ext4 - bug? "Thomas Schmitt" <scdbackup@gmx.net> - 2022-04-27 11:40 +0200
              Re: file born 30 seconds after its creation on ext4 - bug? Vincent Lefevre <vincent@vinc17.net> - 2022-04-27 15:20 +0200
                Re: file born 30 seconds after its creation on ext4 - bug? Greg Wooledge <greg@wooledge.org> - 2022-04-27 15:30 +0200
                  Re: file born 30 seconds after its creation on ext4 - bug? Vincent Lefevre <vincent@vinc17.net> - 2022-04-27 16:10 +0200
                    Re: file born 30 seconds after its creation on ext4 - bug? Stefan Monnier <monnier@iro.umontreal.ca> - 2022-04-28 04:50 +0200
                      Re: file born 30 seconds after its creation on ext4 - bug? Greg Wooledge <greg@wooledge.org> - 2022-04-28 05:10 +0200
                        Re: file born 30 seconds after its creation on ext4 - bug? duh <fill_in_the_blanks@email.com> - 2022-04-29 16:20 +0200
                          Re: file born 30 seconds after its creation on ext4 - bug? Marc Auslander <marcausl@gmail.com> - 2022-04-29 20:10 +0200
                            Re: file born 30 seconds after its creation on ext4 - bug? "sp007@caiway.net" <sp007@caiway.net> - 2022-04-29 22:10 +0200
                              Re: file born 30 seconds after its creation on ext4 - bug? <tomas@tuxteam.de> - 2022-04-30 07:40 +0200
                                Re: file born 30 seconds after its creation on ext4 - bug? "Thomas Schmitt" <scdbackup@gmx.net> - 2022-04-30 10:40 +0200
                                  Re: file born 30 seconds after its creation on ext4 - bug? <tomas@tuxteam.de> - 2022-04-30 14:10 +0200
                                    Re: file born 30 seconds after its creation on ext4 - bug? Vincent Lefevre <vincent@vinc17.net> - 2022-05-02 01:20 +0200
                                      Re: file born 30 seconds after its creation on ext4 - bug? <tomas@tuxteam.de> - 2022-05-02 07:30 +0200
                                  Re: file born 30 seconds after its creation on ext4 - bug? Curt <curty@free.fr> - 2022-04-30 14:40 +0200
                                    Re: file born 30 seconds after its creation on ext4 - bug? <tomas@tuxteam.de> - 2022-04-30 15:10 +0200
                                    Re: file born 30 seconds after its creation on ext4 - bug? "Thomas Schmitt" <scdbackup@gmx.net> - 2022-04-30 15:20 +0200
                                      Re: file born 30 seconds after its creation on ext4 - bug? Vincent Lefevre <vincent@vinc17.net> - 2022-05-02 01:40 +0200
                                        Re: file born 30 seconds after its creation on ext4 - bug? Greg Wooledge <greg@wooledge.org> - 2022-05-02 01:40 +0200
                                        Re: file born 30 seconds after its creation on ext4 - bug? David Wright <deblis@lionunicorn.co.uk> - 2022-05-02 17:10 +0200
                                          Re: file born 30 seconds after its creation on ext4 - bug? <tomas@tuxteam.de> - 2022-05-02 18:20 +0200
                      Re: file born 30 seconds after its creation on ext4 - bug? Vincent Lefevre <vincent@vinc17.net> - 2022-04-28 10:40 +0200
              Re: file born 30 seconds after its creation on ext4 - bug? Vincent Lefevre <vincent@vinc17.net> - 2022-04-28 11:40 +0200
    Re: file born 30 seconds after its creation on ext4 - bug? Nicholas Geovanis <nickgeovanis@gmail.com> - 2022-04-26 19:40 +0200
      Re: file born 30 seconds after its creation on ext4 - bug? Nicholas Geovanis <nickgeovanis@gmail.com> - 2022-04-26 19:50 +0200
        Re: file born 30 seconds after its creation on ext4 - bug? Vincent Lefevre <vincent@vinc17.net> - 2022-04-27 04:50 +0200
    Re: file born 30 seconds after its creation on ext4 - bug? Stefan Monnier <monnier@iro.umontreal.ca> - 2022-04-26 20:20 +0200
      Re: file born 30 seconds after its creation on ext4 - bug? Vincent Lefevre <vincent@vinc17.net> - 2022-04-27 05:10 +0200
    Re: file born 30 seconds after its creation on ext4 - bug? "Thomas Schmitt" <scdbackup@gmx.net> - 2022-04-28 11:30 +0200
      Re: file born 30 seconds after its creation on ext4 - bug? Vincent Lefevre <vincent@vinc17.net> - 2022-04-28 12:20 +0200

Page 1 of 2  [1] 2  Next page →

#247570 — file born 30 seconds after its creation on ext4 - bug?

On an ext4 filesystem, I got a file born 30 seconds after its
actual creation. Is this a bug?

I know that such issues can be observed with NFS, but here this
is just a local ext4 filesystem.

Here are the details.

I started a shell script:

cventin:~> ps -p 667828 -o lstart,cmd
                 STARTED CMD
Tue Apr 26 14:43:15 2022 /bin/sh /home/vlefevre/wd/mpfr/tests/mpfrtests.sh

This script creates a file mpfrtests.cventin.lip.ens-lyon.fr.out
very early. But the first attempts to look at this file failed:

cventin:~/software/mpfr> tail -n 30 mpfrtests.*.out; ll mpfrtests.*.out
zsh: no match
zsh: no match
cventin:~/software/mpfr[1]> tail -n 30 mpfrtests.*.out; ll mpfrtests.*.out
zsh: no match
zsh: no match
cventin:~/software/mpfr[1]> lt|head                                   <14:43:42
total 7016
-rw-r--r--  1  188644 2022-04-26 14:43:42 config.log
-rw-r--r--  1    2861 2022-04-26 14:43:42 conftest.c
-rw-r--r--  1       0 2022-04-26 14:43:42 conftest.err
-rw-r--r--  1    1907 2022-04-26 14:43:42 confdefs.h
-rwxr-xr-x  1  632161 2022-04-26 14:43:16 configure.lineno*
drwxr-xr-x  2    4096 2022-04-26 14:43:11 doc/
drwxr-xr-x  3    4096 2022-04-26 14:43:11 tune/
-rwxr-xr-x  1   23568 2022-04-26 14:43:11 depcomp*
drwxr-xr-x  5   36864 2022-04-26 14:43:11 tests/
cventin:~/software/mpfr> lt|head                                      <14:43:47
total 6416
-rw-r--r--  1   19436 2022-04-26 14:43:47 config.log
-rw-r--r--  1     561 2022-04-26 14:43:47 conftest.c
-rw-r--r--  1       0 2022-04-26 14:43:47 conftest.err
-rw-r--r--  1    4138 2022-04-26 14:43:47 mpfrtests.cfgout
-rw-r--r--  1     500 2022-04-26 14:43:47 confdefs.h
-rwxr-xr-x  1  632161 2022-04-26 14:43:45 configure.lineno*
-rw-r--r--  1     878 2022-04-26 14:43:45 mpfrtests.cventin.lip.ens-lyon.fr.out
drwxr-xr-x  3    4096 2022-04-26 14:43:44 tune/
drwxr-xr-x  4   36864 2022-04-26 14:43:44 tests/

According to /usr/bin/stat, the file birth is

 Birth: 2022-04-26 14:43:45.537241731 +0200

thus 30 seconds after the script started!

Note that the configure.lineno file is created *after*
mpfrtests.cventin.lip.ens-lyon.fr.out, and one can see that
at 14:43:16, configure.lineno was already created.

This is a 12-core Debian/unstable machine with

Linux cventin 5.17.0-1-amd64 #1 SMP PREEMPT Debian 5.17.3-1 (2022-04-18) x86_64 GNU/Linux

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

[toc] | [next] | [standalone]

#247580

Hi,

Vincent Lefevre wrote:
> This script creates a file mpfrtests.cventin.lip.ens-lyon.fr.out
> very early. But the first attempts to look at this file failed:

It looks as if the file indeed does not exist when you inquire it.
So if it got created by the script, maybe it was deleted or renamed
shortly afterwards and created again 30 seconds later ?

Consider to inquire the file in the script for birth time and inode number
immediately after it was created. Compare these properties with the
properties of the file which you see 30 seconds later.


Have a nice day :)

Thomas

[toc] | [prev] | [next] | [standalone]

#247591

On 2022-04-26 19:01:26 +0200, Thomas Schmitt wrote:
> It looks as if the file indeed does not exist when you inquire it.
> So if it got created by the script, maybe it was deleted or renamed
> shortly afterwards and created again 30 seconds later ?

The script doesn't do that. The file is created with

  echo "* $fqdn ($(${1:-.}/config.guess) / ${line#PROC:})" > "$out"

and every other line that writes to the file use >> "$out".

Note that if the file were created again, the first line with the FQDN
would no longer be in the file, but the line is still there.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

[toc] | [prev] | [next] | [standalone]

#247594

Hi,

i wrote:
> > So if it got created by the script, maybe it was deleted or renamed
> > shortly afterwards and created again 30 seconds later ?

Vincent Lefevre wrote:
> The script doesn't do that. The file is created with
>   echo "* $fqdn ($(${1:-.}/config.guess) / ${line#PROC:})" > "$out"
> [...]
> Note that if the file were created again, the first line with the FQDN
> would no longer be in the file, but the line is still there.

This is indeed a riddle. But is there a hard reason not to insert

  stat "$out" >/tmp/stat_out_after_creation 2>&1

after that line and to compare /tmp/stat_out_after_creation with the
stat(1) result when the expected file appears 30 seconds later ?


> https://gitlab.inria.fr/mpfr/misc/-/blob/fed7770cf5f712871bd116ef80d93ea5885fc3f7/vl-tests/mpfrtests.sh
> # Written in 2011-2021 by Vincent Lefevre <vincent@vinc17.net>.

So you are supposed to really know what it does. :))

Whatever, it is fewly plausible that the file would not show up in stat(1)
or ls(1) but the script can happily do
  ... >> "$out"
Each of these lines is supposed to do a name lookup of the file.

Maybe another stat(1) should be placed between
  printf "OK, output in %s\n" "$out"
and
  exit 0


Have a nice day :)

Thomas

[toc] | [prev] | [next] | [standalone]

#247595

Hi,

On 2022-04-27 09:07:37 +0200, Thomas Schmitt wrote:
> i wrote:
> > > So if it got created by the script, maybe it was deleted or renamed
> > > shortly afterwards and created again 30 seconds later ?
> 
> Vincent Lefevre wrote:
> > The script doesn't do that. The file is created with
> >   echo "* $fqdn ($(${1:-.}/config.guess) / ${line#PROC:})" > "$out"
> > [...]
> > Note that if the file were created again, the first line with the FQDN
> > would no longer be in the file, but the line is still there.
> 
> This is indeed a riddle. But is there a hard reason not to insert
> 
>   stat "$out" >/tmp/stat_out_after_creation 2>&1
> 
> after that line and to compare /tmp/stat_out_after_creation with the
> stat(1) result when the expected file appears 30 seconds later ?

Because the issue was unexpected (this was the first time it occurred
after 11 years of testing: I wrote the first version of the script in
2011, and I typically look at the output file just after starting the
script) and is probably unlikely to occur again.

Moreover I don't see why this behavior would be allowed with a local
filesystem. If one has in a script with "set -e" (to detect errors,
like in my script)

  something > out
  something else

and if in an interactive shell, one can see the side effect of
"something else", then one should expect file "out" to be visible
in the directory. (This is/was not necessarily the case with NFS
due to caching, even on the same machine.)

> > https://gitlab.inria.fr/mpfr/misc/-/blob/fed7770cf5f712871bd116ef80d93ea5885fc3f7/vl-tests/mpfrtests.sh
> > # Written in 2011-2021 by Vincent Lefevre <vincent@vinc17.net>.
> 
> So you are supposed to really know what it does. :))

Yes, I'm the only author of the script, and I've been working on
it for 11 years. It was much simpler 11 years ago, but there was
already the same kind of code concerning the output file.

> Whatever, it is fewly plausible that the file would not show up in stat(1)
> or ls(1) but the script can happily do
>   ... >> "$out"
> Each of these lines is supposed to do a name lookup of the file.

unless there is some caching bug. The script is likely to run on
the same CPU core, so that the file would still be visible along
the script, possibly via a cache. But some process running on a
different core might not see it for some time.

However, there's also the fact that the birth time was 30 seconds
ahead of the actual file creation, while there was no lockup.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

[toc] | [prev] | [next] | [standalone]

#247597

Hi,

Vincent Lefevre wrote:
> the issue [...] is probably unlikely to occur again.

In this case we will hardly be able to find an explanation.


> However, there's also the fact that the birth time was 30 seconds
> ahead of the actual file creation, while there was no lockup.

Yes, your observations are not yet consistently explainable.
So some of the normal assumptions about your situation must be wrong.
Question is which ones.

The file times which you showed are consistent with a file that was indeed
created and written 30 seconds after your script was supposed to have
created it and to have written to it with repeated name lookups.

> > -rw-r--r--  1     878 2022-04-26 14:43:45 mpfrtests.cventin.lip.ens-lyon.fr.out
> >  Birth: 2022-04-26 14:43:45.537241731 +0200

On the other hand the content looks like the normal work result of your
script which already had ended half a minute ago.

It is normal that data get onto the physical storage medium only quite a
long time after a program wrote them. But this is supposed to be kept
consistent by the VFS and virtual memory of the Linux kernel.


> The script is likely to run on
> the same CPU core, so that the file would still be visible along
> the script, possibly via a cache.

A connection to the CPU cache would be a strange low level problem of
kernel or hardware.
I understand that the filesystem driver writes to memory pages which
are associated to storage device memory. The pages and their association
are managed by the virtual memory facility of the kernel.
  https://www.kernel.org/doc/html/latest/filesystems/vfs.html#the-address-space-object
Any attempt to access the associated to storage device memory of a not
yet written page is supposed to be directed to the cached page in RAM.

If it has indeed to do with the CPU cache then a particular cache would have
delayed its writing to RAM for 30 seconds but would have served its own CPU
with the full results of file system driver and virtual memory activities
around the new file. No inconsistent partial results would have been written
to page cache in RAM which would have caused protests during your attempts
to see the file.

But i deem it unlikely that the kernel threads which operated filesystem and
virtual memory are (nearly) always running on the same CPU cache which is
not shared with all other CPU cores.
Further, if the memory operations were just pending in some secluded cache
why does the inode then bear the time when that cache would finally have
released its content to the wider accessible RAM.


If i was in your situation, i'd add diagnostic messages to the script in the
hope (or fear) that the glitch happens again.
Especially the inode numbers during and after the script run would be
interesting.


Have a nice day :)

Thomas

[toc] | [prev] | [next] | [standalone]

#247613

On 2022-04-27 11:39:17 +0200, Thomas Schmitt wrote:
> If it has indeed to do with the CPU cache then a particular cache would have
> delayed its writing to RAM for 30 seconds but would have served its own CPU
> with the full results of file system driver and virtual memory activities
> around the new file. No inconsistent partial results would have been written
> to page cache in RAM which would have caused protests during your attempts
> to see the file.
> 
> But i deem it unlikely that the kernel threads which operated filesystem and
> virtual memory are (nearly) always running on the same CPU cache which is
> not shared with all other CPU cores.
> Further, if the memory operations were just pending in some secluded cache
> why does the inode then bear the time when that cache would finally have
> released its content to the wider accessible RAM.

I was thinking of a potentially specific FS cache. But perhaps this
is not related to the CPU at all. It could be that a potential bug
(either in the kernel code or in the compiler) would affect some
system calls but not others. So, the ">> $out" done in the script
would work, but not other kinds of operations on $out.

In the strace output of the script, I get

  openat(AT_FDCWD, "….out", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3

when the file is created, then

  openat(AT_FDCWD, "….out", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3

for the commands that append to this file, and one with

  openat(AT_FDCWD, "….out", O_WRONLY|O_CREAT|O_APPEND, 0666 <unfinished ...>
  <... openat resumed>)  = 3

about 30 seconds later.

So, perhaps there were no issues with openat, but when reading
the directory, the file could not be found because some internal
structures might have been incomplete.

This doesn't explain why the birth time of the file was 30 seconds
late. But according to fs/ext4/inode.c, inode handling seems a bit
complex.

To the question why this doesn't occur every time or at least more
often: there might have been an interrupt at the wrong place, or
something like that.

> If i was in your situation, i'd add diagnostic messages to the script in the
> hope (or fear) that the glitch happens again.

I think that a better thing would be to write a specific test script
doing file creations and writes in append mode, that on many files,
and see at the end whether some birth times were unexpectedly late.

> Especially the inode numbers during and after the script run would be
> interesting.

But note that getting other information about the file during the
script could void the effects of a potential bug.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

[toc] | [prev] | [next] | [standalone]

#247615

On Wed, Apr 27, 2022 at 03:11:22PM +0200, Vincent Lefevre wrote:
> So, perhaps there were no issues with openat, but when reading
> the directory, the file could not be found because some internal
> structures might have been incomplete.

If so, this is a bug at the kernel level, perhaps in the VFS code
or a file system driver.  I'm no expert on such things.

> To the question why this doesn't occur every time or at least more
> often: there might have been an interrupt at the wrong place, or
> something like that.

It's going to be extremely hard to make any progress on your issue if
you can't reliably reproduce the outcome.  Frustrating as that may be.

Are you able to reproduce the issue at all, even like 1% of the time?
If so, you could set up a script that sometimes reproduces, and run it
in a loop until you get the erroneous behavior.  Of course, the script
would have to generate enough debugging output to be useful for
diagnosing the underlying problem, and you'd probably have a second
script controlling the loop which runs the first script, and then looks
for the anomalous output in the logs, to stop the loop.

[toc] | [prev] | [next] | [standalone]

#247624

On 2022-04-27 09:26:50 -0400, Greg Wooledge wrote:
> Are you able to reproduce the issue at all, even like 1% of the time?

In the past 11 years, I may have ran the script (and looked at the
output just after that) hundreds or thousands of times; the commands
to look at the output have been in my shell history for 6 years.

This was the first time I noticed such an issue. But if this is a
kernel bug, it may be a new bug.

> If so, you could set up a script that sometimes reproduces, and run it
> in a loop until you get the erroneous behavior.

I'll try to do that.

> Of course, the script would have to generate enough debugging output
> to be useful for diagnosing the underlying problem, and you'd
> probably have a second script controlling the loop which runs the
> first script, and then looks for the anomalous output in the logs,
> to stop the loop.

The first thing would be to try to reproduce the issue without
much debug info (in order to get the same system calls as the ones
generated by my normal script).

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

[toc] | [prev] | [next] | [standalone]

#247684

Vincent Lefevre [2022-04-27 15:59:57] wrote:
> On 2022-04-27 09:26:50 -0400, Greg Wooledge wrote:
>> Are you able to reproduce the issue at all, even like 1% of the time?
> In the past 11 years, I may have ran the script (and looked at the
> output just after that) hundreds or thousands of times; the commands
> to look at the output have been in my shell history for 6 years.
>
> This was the first time I noticed such an issue. But if this is a
> kernel bug, it may be a new bug.

Another option might be that your system's time was "reset".
This shouldn't happen, but it can happen if your NTP was down, the
machine got out-of-sync over time and you restart the NTP server at
which point it may(!) decide to jump the clock if the difference is
large enough (i.e. too large to catch up gradually).
Can't remember how large is "large enough".

Anyway, just throwing this out there in case it rings a bell,


        Stefan

[toc] | [prev] | [next] | [standalone]

#247685

On Wed, Apr 27, 2022 at 10:45:09PM -0400, Stefan Monnier wrote:
> Another option might be that your system's time was "reset".
> This shouldn't happen, but it can happen if your NTP was down, the
> machine got out-of-sync over time and you restart the NTP server at
> which point it may(!) decide to jump the clock if the difference is
> large enough (i.e. too large to catch up gradually).
> Can't remember how large is "large enough".

It would depend on which NTP implementation is in use.  The traditonal
"ntp" package should not do a large jump like that, except at boot.  It
should just make the clock drift toward the correct time.

Other implementations may be more aggressive about it.  Debian 11 uses
systemd-timesyncd by default, but I don't know how it behaves.  I've
not used chrony either.  I did experiment with openntpd for a while, but
it was many years ago when it was pretty new, so it might have changed
a lot.

In any case, that's a clever theory.  The OP could look for evidence of
an NTP time jump in whatever logs survive from the offending time
period.

[toc] | [prev] | [next] | [standalone]

#247733

On 4/27/22 11:05 PM, Greg Wooledge wrote:
> On Wed, Apr 27, 2022 at 10:45:09PM -0400, Stefan Monnier wrote:
>> Another option might be that your system's time was "reset".
>> This shouldn't happen, but it can happen if your NTP was down, the
>> machine got out-of-sync over time and you restart the NTP server at
>> which point it may(!) decide to jump the clock if the difference is
>> large enough (i.e. too large to catch up gradually).
>> Can't remember how large is "large enough".
> It would depend on which NTP implementation is in use.  The traditonal
> "ntp" package should not do a large jump like that, except at boot.  It
> should just make the clock drift toward the correct time.
>
> Other implementations may be more aggressive about it.  Debian 11 uses
> systemd-timesyncd by default, but I don't know how it behaves.  I've
> not used chrony either.  I did experiment with openntpd for a while, but
> it was many years ago when it was pretty new, so it might have changed
> a lot.
>
> In any case, that's a clever theory.  The OP could look for evidence of
> an NTP time jump in whatever logs survive from the offending time
> period.
>
Having skimmed over a number of the replies, and really not being
qualified, may I just

toss out a probably useless ideas to use the "sync" command. Looking at
the 'man sync'

shows at the bottom several variants or whatever to sync. Just a thought
since when does

the data get transferred to the disk versus just being held in memory or
whatever?

This is probably just a useless tangent based on my ignorance, but once
in awhile it is possible

to discover something when falls into a hole.

[toc] | [prev] | [next] | [standalone]

#247738

On 4/29/2022 10:20 AM, duh wrote:
> 
> On 4/27/22 11:05 PM, Greg Wooledge wrote:

....
> Having skimmed over a number of the replies, and really not being
> qualified, may I just
> 
> toss out a probably useless ideas to use the "sync" command. Looking at
> the 'man sync'
> 
> shows at the bottom several variants or whatever to sync. Just a thought
> since when does
> 
> the data get transferred to the disk versus just being held in memory or
> whatever?
> 
> This is probably just a useless tangent based on my ignorance, but once
> in awhile it is possible
> 
> to discover something when falls into a hole.

sync isn't about this.  linux caches file system pages in memory - both 
content and metadata.  sync is about forcing the changed pages back to 
disk, for example before shutting down.  It's done automatically - maybe 
every 30 seconds (I'm not sure about linux on this).  But sync does not 
change what programs see unless they use a direct to disk read, which is 
certainly not what's going on here.

[toc] | [prev] | [next] | [standalone]

#247741

On Fri, 29 Apr 2022 13:45:15 -0400

> linux caches file system pages in memory

how does one force to refresh this memory with a command?

Thanks!

 

[toc] | [prev] | [next] | [standalone]

#247754

[Multipart message — attachments visible in raw view] — view raw

On Fri, Apr 29, 2022 at 09:41:23PM +0200, sp007@caiway.net wrote:
> On Fri, 29 Apr 2022 13:45:15 -0400
> 
> > linux caches file system pages in memory
> 
> how does one force to refresh this memory with a command?

What do you mean by "refresh"? What's in the cache is always
the "freshest" version: nobody's writing to the disk behind
the operating system's back (unless they insist into getting
into trouble).

Cheers
-- 
t

[toc] | [prev] | [next] | [standalone]

#247756

Hi,

sp007@caiway.net wrote:
> how does one force to refresh this memory with a command?

tomas@tuxteam.de wrote:
> What do you mean by "refresh"? What's in the cache is always
> the "freshest" version:

Indeed. With normal filesystem operations there should be no need to call
something like sync(2) in order to get a consistent representation of the
current filesystem state.

That's actually the riddle of this thread:
The filesystem behavior outside the script and the time stamps look like
the file was created 30 seconds after the script began to open, write, and
close the file.
The file content indicates that it was written shortly after the script
began.

I understand that Vincent Lefevre suspects these discrepancies to be a bug
in the ext4 driver. I rather suspect that ext4 is ok and that we observe
the effects of some other glitch which caused the ext4 driver to create
the finally observed file again, 30 seconds later.


> nobody's writing to the disk behind the operating system's back
> (unless they insist into getting into trouble).

Burn programs do, because the kernel insists in being incapable of burning
CD-R and other sequential media types or states.
ioctl(SG_IO) performs SCSI transactions on the storage device by the kernel
but without the higher level drivers being aware. E.g. you don't see the
traffic counted in files of /proc and the block device driver will continue
to show the medium state before the burn run.
Remedy is to eject and reload the optical medium. On hard disks there
would be ioctl(BLKRRPART). But for CDROM there is nothing comparable in
Linux.


Have a nice day :)

Thomas

[toc] | [prev] | [next] | [standalone]

#247760

[Multipart message — attachments visible in raw view] — view raw

On Sat, Apr 30, 2022 at 10:33:34AM +0200, Thomas Schmitt wrote:
> Hi,
> 
> sp007@caiway.net wrote:
> > how does one force to refresh this memory with a command?
> 
> tomas@tuxteam.de wrote:
> > What do you mean by "refresh"? What's in the cache is always
> > the "freshest" version:
> 
> Indeed. With normal filesystem operations there should be no need to call
> something like sync(2) in order to get a consistent representation of the
> current filesystem state.

Exactly: what we "see", i.e. what we interact with via the operating
system facilities is the cache's view. The disk's view will become
consistent with that at some time in the future (at least we hope
that).

[...]

> I understand that Vincent Lefevre suspects these discrepancies to be a bug
> in the ext4 driver. I rather suspect that ext4 is ok and that we observe
> the effects of some other glitch which caused the ext4 driver to create
> the finally observed file again, 30 seconds later.

It sure looks like a bug. But it would be a bug at a spot where one would
expect that it should have bitten oodles of other people by now, so that
feeling that we must be missing something is as understandable.

Cheers
-- 
t

[toc] | [prev] | [next] | [standalone]

#247795

On 2022-04-30 14:06:53 +0200, tomas@tuxteam.de wrote:
> On Sat, Apr 30, 2022 at 10:33:34AM +0200, Thomas Schmitt wrote:
> > I understand that Vincent Lefevre suspects these discrepancies to be a bug
> > in the ext4 driver. I rather suspect that ext4 is ok and that we observe
> > the effects of some other glitch which caused the ext4 driver to create
> > the finally observed file again, 30 seconds later.
> 
> It sure looks like a bug. But it would be a bug at a spot where one would
> expect that it should have bitten oodles of other people by now, so that
> feeling that we must be missing something is as understandable.

Not necessarily. The bug might be new. It might be difficult to
reproduce, in particular if there is a race condition behind. It
might have occurred on other machines, but remained unnoticed,
because to notice it, one needs a specific sequence of operations
performed in a timely manner.

Bugs may exist, but may remain unnoticed for years. See for instance
this one I had found in 2006:

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=356083#15

  "Good catch.  Wow, that bug has been in GNU cmp ever since it was
  entered into CVS in 1991."

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

[toc] | [prev] | [next] | [standalone]

#247798

[Multipart message — attachments visible in raw view] — view raw

On Mon, May 02, 2022 at 01:10:04AM +0200, Vincent Lefevre wrote:
> On 2022-04-30 14:06:53 +0200, tomas@tuxteam.de wrote:

[...]

> > It sure looks like a bug. But it would be a bug at a spot where one would
> > expect that it should have bitten oodles of other people by now, so that
> > feeling that we must be missing something is as understandable.
> 
> Not necessarily. The bug might be new. It might be difficult to
> reproduce, in particular if there is a race condition behind. It
> might have occurred on other machines, but remained unnoticed,
> because to notice it, one needs a specific sequence of operations
> performed in a timely manner.

[...]

Yes, of course. I didn't say it can't be a bug. Just it may be
hard to convince people. As it is sporadic, it may be hard to
come up with a somewhat reliable reproducer.

Cheers
-- 
t

[toc] | [prev] | [next] | [standalone]

#247762

On 2022-04-30, Thomas Schmitt <scdbackup@gmx.net> wrote:
>
> Indeed. With normal filesystem operations there should be no need to call
> something like sync(2) in order to get a consistent representation of the
> current filesystem state.
>

What does the following mean, then, in that light:

 Because of delayed allocation and other performance optimizations, ext4's
 behavior of writing files to disk is different from ext3. In ext4, when a
 program writes to the file system, it is not guaranteed to be on-disk unless
 the program issues an fsync() call afterwards. 

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/storage_administration_guide/ch-ext4

Or this:

 So on ext4 ‘data=ordered’ and ‘data=writeback’ are somehow similar when a file
 is enlarged, which is quite confusing and not clearly stated in the man pages.
 And the delayed allocation will actually commit the data to disk only after
 30-150 seconds (it is not very clear on this exact window of data loss) even
 when ‘commit=5′ is supposed (cf. ‘man mount’) to do it after 5 seconds.  In
 conclusion, ext4 with default options guarantees after a crash only the
 atomicity and consistency of the filesystem changes (all metadata) with a
 maximum loss of 5 seconds on metadata changes. The data changes may suffer a
 loss of 30-150 seconds and in the majority of cases all changed files in this
 window will be completely wiped with zero bytes! The atomicity of file changes
 is not working anymore with the rename-idiom.

https://www.pointsoftware.ch/2014/02/05/linux-filesystems-part-4-ext4-vs-ext3-and-why-delayed-allocation-is-bad/

Or maybe none of this is  pertinent.

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.debian.user

csiph-web