Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1614272 > unrolled thread

[RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it

Back to article view | Back to linux.kernel

Contents

  [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-03-31 21:30 +0200
    [RFC PATCH 2/4] dax: set errors in mapping when writeback fails Jeff Layton <jlayton@redhat.com> - 2017-03-31 21:30 +0200
    [RFC PATCH 1/4] fs: new infrastructure for writeback error handling and reporting Jeff Layton <jlayton@redhat.com> - 2017-03-31 21:30 +0200
      Re: [RFC PATCH 1/4] fs: new infrastructure for writeback error  handling and reporting Nikolay Borisov <nborisov@suse.com> - 2017-04-03 09:20 +0200
        Re: [RFC PATCH 1/4] fs: new infrastructure for writeback error  handling and reporting Jeff Layton <jlayton@redhat.com> - 2017-04-03 12:30 +0200
      Re: [RFC PATCH 1/4] fs: new infrastructure for writeback error  handling and reporting Matthew Wilcox <willy@infradead.org> - 2017-04-03 16:50 +0200
        Re: [RFC PATCH 1/4] fs: new infrastructure for writeback error  handling and reporting Jeff Layton <jlayton@redhat.com> - 2017-04-03 17:30 +0200
          Re: [RFC PATCH 1/4] fs: new infrastructure for writeback error  handling and reporting Matthew Wilcox <willy@infradead.org> - 2017-04-03 18:20 +0200
            Re: [RFC PATCH 1/4] fs: new infrastructure for writeback error  handling and reporting Jeff Layton <jlayton@redhat.com> - 2017-04-03 18:40 +0200
    [RFC PATCH 4/4] ext4: wire it up to the new writeback error reporting infrastructure Jeff Layton <jlayton@redhat.com> - 2017-03-31 21:30 +0200
    Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it NeilBrown <neilb@suse.com> - 2017-04-03 06:30 +0200
      Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-04-03 12:30 +0200
        Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Matthew Wilcox <willy@infradead.org> - 2017-04-03 16:40 +0200
          Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-04-03 19:50 +0200
            Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeremy Allison <jra@samba.org> - 2017-04-03 20:20 +0200
              Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-04-03 20:20 +0200
                Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeremy Allison <jra@samba.org> - 2017-04-03 20:40 +0200
                  Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeremy Allison <jra@samba.org> - 2017-04-03 20:50 +0200
                    Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-04-03 20:50 +0200
            Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Matthew Wilcox <willy@infradead.org> - 2017-04-03 21:20 +0200
              Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-04-03 22:20 +0200
                Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Matthew Wilcox <willy@infradead.org> - 2017-04-04 04:50 +0200
                Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it NeilBrown <neilb@suse.com> - 2017-04-04 05:10 +0200
                  Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-04-04 13:50 +0200
                    Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it NeilBrown <neilb@suse.com> - 2017-04-05 00:50 +0200
                  Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Matthew Wilcox <willy@infradead.org> - 2017-04-04 14:00 +0200
                    Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-04-04 14:20 +0200
                      Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Matthew Wilcox <willy@infradead.org> - 2017-04-04 18:20 +0200
                        Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-04-04 18:30 +0200
                          Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Matthew Wilcox <willy@infradead.org> - 2017-04-04 19:10 +0200
                            Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-04-04 20:10 +0200
                            Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it NeilBrown <neilb@suse.com> - 2017-04-05 01:00 +0200
                            Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-04-05 22:00 +0200
                              Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Matthew Wilcox <willy@infradead.org> - 2017-04-05 23:10 +0200
                                Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it NeilBrown <neilb@suse.com> - 2017-04-06 02:30 +0200
                              Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it NeilBrown <neilb@suse.com> - 2017-04-06 02:10 +0200
                                Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Matthew Wilcox <willy@infradead.org> - 2017-04-06 05:00 +0200
                                  Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it NeilBrown <neilb@suse.com> - 2017-04-06 07:20 +0200
                                    Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Matthew Wilcox <willy@infradead.org> - 2017-04-06 15:40 +0200
                                      Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it NeilBrown <neilb@suse.com> - 2017-04-07 00:00 +0200
                                Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-04-06 16:10 +0200
                                Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-04-06 21:20 +0200
                                  Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Matthew Wilcox <willy@infradead.org> - 2017-04-06 22:20 +0200
                                    Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-04-07 15:20 +0200
                                      Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it NeilBrown <neilb@suse.com> - 2017-04-10 01:20 +0200
                                        Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-04-10 15:30 +0200
                                  Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it NeilBrown <neilb@suse.com> - 2017-04-07 00:20 +0200
                          Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it NeilBrown <neilb@suse.com> - 2017-04-05 01:20 +0200
                            Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-04-05 13:20 +0200
                              Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it NeilBrown <neilb@suse.com> - 2017-04-06 02:50 +0200
                    Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Theodore Ts'o <tytso@mit.edu> - 2017-04-04 15:40 +0200
                    Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it NeilBrown <neilb@suse.com> - 2017-04-05 00:30 +0200
      Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Matthew Wilcox <willy@infradead.org> - 2017-04-03 17:00 +0200

Page 3 of 3 — ← Prev page 1 2 [3]

#1618055 — Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it

On Thu, 2017-04-06 at 10:02 +1000, NeilBrown wrote:
> On Thu, Apr 06 2017, Jeff Layton wrote:
> 
> > On Tue, 2017-04-04 at 10:09 -0700, Matthew Wilcox wrote:
> > > On Tue, Apr 04, 2017 at 12:25:46PM -0400, Jeff Layton wrote:
> > > > That said, I think giving more specific errors where we can is useful.
> > > > When your program is erroring out and writing 'I/O error' to the logs,
> > > > then how much time will your admins burn before they figure out that it
> > > > really failed because the filesystem was full?
> > > 
> > > df is one of the first things I check ... a few years ago, I also learned
> > > to check df -i ... ;-)
> > > 
> > > Anyway, given the decision to simply report the last error lets us do this
> > > implementation:
> > > 
> > > void filemap_set_wb_error(struct address_space *mapping, int err)
> > > {
> > > 	struct inode *inode = mapping->host;
> > > 	unsigned int wb_err;
> > > 
> > > 	if (!err)
> > > 		return;
> > > 	/*
> > > 	 * This should be called with the error code that we want to return
> > > 	 * on fsync. Thus, it should always be <= 0.
> > > 	 */
> > > 	WARN_ON(err > 0 || err < -MAX_ERRNO);
> > > 
> > > 	spin_lock(&inode->i_lock);
> > > 	wb_err = ((mapping->wb_err & ~MAX_ERRNO) + (1 << 12)) | -err;
> > > 	WRITE_ONCE(mapping->wb_err, wb_err);
> > > 	spin_unlock(&inode->i_lock);
> > > }
> > > 
> > 
> > I like this idea of being able to store arbitrary error codes there.
> > That should be used judiciously of course, but we already allow
> > returning arbitrary errors via the ->fsync op anyway.
> > 
> > I'll plan to incorporate something like that into the next set (with
> > judicious comments and constants).
> > 
> > One question...is the i_lock the right way to protect this? I think we
> > could do this locklessly too (cmpxchg in a loop, for instance). I'm not
> > worried about performance here -- it's just nice to be able to call
> > simple stuff like this without worrying about locking.
> 
> I like the idea of using cmpxchg.
> 
> 
> > 
> > > int filemap_report_wb_error(struct file *file)
> > > {
> > > 	struct inode *inode = file_inode(file);
> > > 	unsigned int wb_err = READ_ONCE(mapping->wb_err);
> > > 
> > > 	if (file->f_wb_err == wb_err)
> > > 		return 0;
> > > 	return -(wb_err & 4095);
> > > }
> > > 
> > > That only gives us 20 bits of counter, but I think that's enough.
> > 
> > 2^20 is 1048576, which seems a little small to me.
> > 
> > We may end up bumping the counter on every failed I/O. How fast can we
> > generate 1M failed I/Os? :)
> 
> Do we need to count all of those if no-one sees them?
> i.e. use one bit to say "this error hasn't been seen".
> If an error occurs with has the name error code as is currently stored,
> and the bit is set, don't make a change.  Otherwise make the change,
> inc the counter, set the bit.
> When checking for an error, if the bit is set, clear it first.
> Then you can count 500,000 errors-returned-to-some-thread, which is
> probably enough.
> 

Yeah, that seems like it might be a good idea if we want to stick to a
small value here.

> > 
> > 2^52 however is 4503599627370496 (4Tios or so) ... that might take a
> > little longer to overflow. Is it worth the cost here to ensure that
> > this won't occur?
> > 
> > Actually...we could put this field in the inode instead of the mapping.
> > I know we've traditionally tracked this in the mapping, but is that
> > required here?
> 
> What if the address_space is shared by two inodes?  That is the whole
> point of the i_mapping pointer.  This would make it harder for the
> "other" inode to get the error.
> (Does anyone actually use fs/coda ??
>  Actually, block devices use i_mapping too.
>  If two block device inodes have the same major/minor number, they
>  end up having i_mapping point to the same place)
> 

Ahh ok, that makes sense. I'll plan to keep it part of the mapping.

> If you are concerned about space in 'struct address_space', just prune
> some wastage.
> The "host" field brings no value.  It is only ever assigned in
> inode_init_always():
> 
> 	struct address_space *const mapping = &inode->i_data;
> ......
> 	mapping->host = inode;
> 
> So you could change all references to use
>    container_of(mapping, struct inode, i_data)
> 

That might be a nice cleanup, but I think I'll leave that to be done
separately.

-- 
Jeff Layton <jlayton@redhat.com>

[toc] | [prev] | [next] | [standalone]

#1618293 — Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it

On Thu, 2017-04-06 at 10:02 +1000, NeilBrown wrote:
> > 

> On Thu, Apr 06 2017, Jeff Layton wrote:
> 
> > On Tue, 2017-04-04 at 10:09 -0700, Matthew Wilcox wrote:
> > > On Tue, Apr 04, 2017 at 12:25:46PM -0400, Jeff Layton wrote:
> > > > That said, I think giving more specific errors where we can is useful.
> > > > When your program is erroring out and writing 'I/O error' to the logs,
> > > > then how much time will your admins burn before they figure out that it
> > > > really failed because the filesystem was full?
> > > 
> > > df is one of the first things I check ... a few years ago, I also learned
> > > to check df -i ... ;-)
> > > 
> > > Anyway, given the decision to simply report the last error lets us do this
> > > implementation:
> > > 
> > > void filemap_set_wb_error(struct address_space *mapping, int err)
> > > {
> > > 	struct inode *inode = mapping->host;
> > > 	unsigned int wb_err;
> > > 
> > > 	if (!err)
> > > 		return;
> > > 	/*
> > > 	 * This should be called with the error code that we want to return
> > > 	 * on fsync. Thus, it should always be <= 0.
> > > 	 */
> > > 	WARN_ON(err > 0 || err < -MAX_ERRNO);
> > > 
> > > 	spin_lock(&inode->i_lock);
> > > 	wb_err = ((mapping->wb_err & ~MAX_ERRNO) + (1 << 12)) | -err;
> > > 	WRITE_ONCE(mapping->wb_err, wb_err);
> > > 	spin_unlock(&inode->i_lock);
> > > }
> > > 
> > 
> > I like this idea of being able to store arbitrary error codes there.
> > That should be used judiciously of course, but we already allow
> > returning arbitrary errors via the ->fsync op anyway.
> > 
> > I'll plan to incorporate something like that into the next set (with
> > judicious comments and constants).
> > 
> > One question...is the i_lock the right way to protect this? I think we
> > could do this locklessly too (cmpxchg in a loop, for instance). I'm not
> > worried about performance here -- it's just nice to be able to call
> > simple stuff like this without worrying about locking.
> 
> I like the idea of using cmpxchg.
> 
> 
> > 
> > > int filemap_report_wb_error(struct file *file)
> > > {
> > > 	struct inode *inode = file_inode(file);
> > > 	unsigned int wb_err = READ_ONCE(mapping->wb_err);
> > > 
> > > 	if (file->f_wb_err == wb_err)
> > > 		return 0;
> > > 	return -(wb_err & 4095);
> > > }
> > > 
> > > That only gives us 20 bits of counter, but I think that's enough.
> > 
> > 2^20 is 1048576, which seems a little small to me.
> > 
> > We may end up bumping the counter on every failed I/O. How fast can we
> > generate 1M failed I/Os? :)
> 
> Do we need to count all of those if no-one sees them?
> i.e. use one bit to say "this error hasn't been seen".
> If an error occurs with has the name error code as is currently stored,
> and the bit is set, don't make a change.  Otherwise make the change,
> inc the counter, set the bit.
> When checking for an error, if the bit is set, clear it first.
> Then you can count 500,000 errors-returned-to-some-thread, which is
> probably enough.
> 

Ok, so here's a replacement for patch #1. The other 3 are pretty much
the same. The main changes are:

- 32 bit value:
  - 12 bits for error code
  - 1 bit for "seen" flag
  - 19 bits for the counter
- mapping->wb_err is managed with cmpxchg
- file->f_wb_err is protected with file->f_lock

I tried to avoid updating things unnecesssarily. I could use some
guidance on how to specify the constants in terms of MAX_ERRNO as well.

It seems to work, in very basic by-hand testing.

If this looks reasonable, I may try again to plug this in at a higher
level, so we don't need to change so much filesystem code. IOW:

- make filemap_set_wb_error the new implementation of mapping_set_error
- have vfs_fsync_range call filemap_report_wb_error, and return what it
  returns if it's non-zero
- have filemap_check_error grab the current error code without updating
  the counter or the seen flag

That approach may not work, but I'll see. Anyway, here's the updated
patch. I may need to revise the changelog too.

--------------------------8<---------------------

[PATCH] fs: new infrastructure for writeback error handling and reporting

Most filesystems currently use mapping_set_error and
filemap_check_errors for setting and reporting/clearing writeback errors
at the mapping level. filemap_check_errors is indirectly called from
most of the filemap_fdatawait_* functions and from
filemap_write_and_wait*. These functions are called from all sorts of
contexts to wait on writeback to finish -- e.g. mostly in fsync, but
also in truncate calls, getattr, etc.

It's those non-fsync callers that are problematic. We should be
reporting writeback errors during fsync, but many places in the code
clear out errors before they can be properly reported, or report errors
at nonsensical times. If I get -EIO on a stat() call, how do I know that
was because writeback failed?

This patch adds a small bit of new infrastructure for setting and
reporting errors during pagecache writeback. While the above was my
original impetus for adding this, I think it's also the case that
current fsync semantics are just problematic for userland. Most
applications that call fsync do so to ensure that the data they wrote
has hit the backing store.

In the case where there are multiple writers to the file at the same
time, this is really hard to determine. The first one to call fsync will
see any stored error, and the rest get back 0. The processes with open
fd may not be associated with one another in any way. They could even be
in different containers, so ensuring coordination between all fsync
callers is not really an option.

One way to remedy this would be to track what file descriptor was used
to dirty the file, but that's rather cumbersome and would likely be
slow. However, there is a simpler way to improve the semantics here
without incurring too much overhead.

This set adds a wb_error field and a sequence counter to the
address_space, and a corresponding sequence counter in the struct file.
When errors are reported during writeback, we set the error field in the
mapping and increment the sequence counter.

When fsync or flush is called, we check the sequence in the file vs. the
one in the mapping. If the file's counter is behind the one in the
mapping, then we update the sequence counter in the file to the value of
the one in the mapping and report the error. If the file is "caught up"
then we just report 0.

This changes the semantics of fsync such that applications can now use
it to determine whether there were any writeback errors since fsync(fd)
was last called (or since the file was opened in the case of fsync
having never been called).

Note that those writeback errors may have occurred when writing data
that was dirtied via an entirely different fd, but that's the case now
with the current mapping_set_error/filemap_check_error infrastructure.
This will at least prevent you from getting a false report of success.

The basic idea here is for filesystems to use filemap_set_wb_error to
set the error in the mapping when there are writeback errors, and then
have the fsync and flush operations call filemap_report_wb_error just
before returning to ensure that those errors get reported properly.

Eventually, it may make sense to move the reporting into the generic
vfs_fsync_range helper, but doing it this way for now makes it simpler
to convert filesystems to the new API individually.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
---
 Documentation/filesystems/vfs.txt |  14 +++-
 fs/open.c                         |   3 +
 include/linux/fs.h                |   4 +
 mm/filemap.c                      | 162 ++++++++++++++++++++++++++++++++++++++
 4 files changed, 181 insertions(+), 2 deletions(-)

diff --git a/Documentation/filesystems/vfs.txt b/Documentation/filesystems/vfs.txt
index 569211703721..b2b5e411b340 100644
--- a/Documentation/filesystems/vfs.txt
+++ b/Documentation/filesystems/vfs.txt
@@ -577,6 +577,11 @@ should clear PG_Dirty and set PG_Writeback.  It can be actually
 written at any point after PG_Dirty is clear.  Once it is known to be
 safe, PG_Writeback is cleared.
 
+If there is an error during writeback, then the address_space should be
+marked with an error (typically using filemap_set_wb_error), in order to
+ensure that the error can later be reported to the application at fsync
+or close.
+
 Writeback makes use of a writeback_control structure...
 
 struct address_space_operations
@@ -885,11 +890,16 @@ otherwise noted.
 	"private_data" member in the file structure if you want to point
 	to a device structure
 
-  flush: called by the close(2) system call to flush a file
+  flush: called by the close(2) system call to flush a file. Writeback
+	errors not previously reported via fsync should be reported
+	here as you would for fsync.
 
   release: called when the last reference to an open file is closed
 
-  fsync: called by the fsync(2) system call
+  fsync: called by the fsync(2) system call. Filesystems that use the
+	pagecache should call filemap_report_wb_error before returning
+	to ensure that any errors that occurred during writeback are
+	reported and the file's error sequence advanced.
 
   fasync: called by the fcntl(2) system call when asynchronous
 	(non-blocking) mode is enabled for a file
diff --git a/fs/open.c b/fs/open.c
index 949cef29c3bb..baf82f2c642e 100644
--- a/fs/open.c
+++ b/fs/open.c
@@ -709,6 +709,9 @@ static int do_dentry_open(struct file *f,
 	f->f_inode = inode;
 	f->f_mapping = inode->i_mapping;
 
+	/* Ensure that we skip any errors that predate opening of the file */
+	f->f_wb_err = READ_ONCE(inode->i_mapping->wb_err);
+
 	if (unlikely(f->f_flags & O_PATH)) {
 		f->f_mode = FMODE_PATH;
 		f->f_op = &empty_fops;
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 7251f7bb45e8..f33857113ff4 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -394,6 +394,7 @@ struct address_space {
 	gfp_t			gfp_mask;	/* implicit gfp mask for allocations */
 	struct list_head	private_list;	/* ditto */
 	void			*private_data;	/* ditto */
+	u32			wb_err;
 } __attribute__((aligned(sizeof(long))));
 	/*
 	 * On most architectures that alignment is already the case; but
@@ -868,6 +869,7 @@ struct file {
 	struct list_head	f_tfile_llink;
 #endif /* #ifdef CONFIG_EPOLL */
 	struct address_space	*f_mapping;
+	u32			f_wb_err;
 } __attribute__((aligned(4)));	/* lest something weird decides that 2 is OK */
 
 struct file_handle {
@@ -2521,6 +2523,8 @@ extern int __filemap_fdatawrite_range(struct address_space *mapping,
 extern int filemap_fdatawrite_range(struct address_space *mapping,
 				loff_t start, loff_t end);
 extern int filemap_check_errors(struct address_space *mapping);
+extern void filemap_set_wb_error(struct address_space *mapping, int err);
+extern int filemap_report_wb_error(struct file *file);
 
 extern int vfs_fsync_range(struct file *file, loff_t start, loff_t end,
 			   int datasync);
diff --git a/mm/filemap.c b/mm/filemap.c
index 1694623a6289..60b6fa417b98 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -545,6 +545,168 @@ int filemap_write_and_wait_range(struct address_space *mapping,
 }
 EXPORT_SYMBOL(filemap_write_and_wait_range);
 
+/*
+ * The wb_err field in the address_space provides a place to store writeback
+ * errors. We endeavor to deliver writeback errors to fsync on all open file
+ * descriptors that were open at the time that the error was caught. We do
+ * this using a 32-bit value to store the error, with the upper bits as a
+ * sequence counter. We can store any error up to MAX_ERROR.
+ *
+ * Additionally, we reserve one bit to indicate whether any fd has grabbed the
+ * value to record in its struct file. If nothing has, then we don't really
+ * need to increment the counter.
+ */
+
+/* This bit is used as a flag to indicate whether the value has been seen */
+#define WB_ERR_SEEN		(1 << 12)
+
+/* Increment the counter by this much to ensure that we don't touch earlier
+ * values */
+#define WB_ERR_CTR_INC		(1 << 13)
+
+/**
+ * filemap_set_wb_error - set the wb error in the mapping for later reporting
+ * @mapping: mapping in which the error should be set
+ * @err: error to set. must be negative value but not less than -MAX_ERRNO
+ *
+ * When an error occurs during writeback of inode data, we must report that
+ * error during fsync. This function sets the writeback error field in the
+ * mapping, and increments the sequence counter. When fsync or close is later
+ * performed, the caller can then check the sequence in the mapping against
+ * the one in the file to determine whether the error should be reported.
+ *
+ * Because there are so few bits for the counter, we try to avoid incrementing
+ * it unless someone is going to record the value for later comparison. This
+ * is tracked by a bit in the 32 bit word that we use as a "seen" flag.
+ *
+ * Note that we always use the latest writeback error, since POSIX states
+ * that when there are multiple errors (e.g. -EIO followed by -ENOSPC),
+ * that any possible error may be returned.
+ */
+void filemap_set_wb_error(struct address_space *mapping, int err)
+{
+	u32 old;
+
+	/*
+	 * The above constants rely indirectly on MAX_ERRNO not changing
+	 * since I'm not sure how to take a log at build time. Suggestions
+	 * of better ways to phrase the flag values would be welcome.
+	 */
+	BUILD_BUG_ON(MAX_ERRNO + 1 != WB_ERR_SEEN);
+
+	/* Optimize for common case of no error */
+	if (likely(!err))
+		return;
+
+	/*
+	 * Ensure the error code actually fits where we want it to go. If it
+	 * doesn't then just throw a warning and don't record anything.
+	 */
+	if (unlikely(err > 0 || err < -MAX_ERRNO)) {
+		WARN(1, "err=%d\n", err);
+		return;
+	}
+
+	old = READ_ONCE(mapping->wb_err);
+	for (;;) {
+		u32 new, cur;
+
+		/* Clear out error bits and set new error */
+		new = (old & ~MAX_ERRNO) | -err;
+
+		/* Only increment if someone has looked at it */
+		if (old & WB_ERR_SEEN) {
+			new += WB_ERR_CTR_INC;
+			new &= ~WB_ERR_SEEN;
+		}
+
+		/* Try to swap the new value into place */
+		cur = cmpxchg(&mapping->wb_err, old, new);
+
+		/*
+		 * Call it success if we did the swap or someone else beat us
+		 * to it for the same value.
+		 */
+		if (likely(cur == old || cur == new))
+			break;
+
+		/* Raced with an update, try again */
+		old = cur;
+	}
+}
+EXPORT_SYMBOL(filemap_set_wb_error);
+
+/**
+ * filemap_report_wb_error - report wb error (if any) that was previously set
+ * @file: struct file on which the error is being reported
+ *
+ * When userland calls fsync or close (or something like nfsd does the
+ * equivalent), we want to report any writeback errors that occurred since
+ * the last fsync (or since the file was opened if there haven't been any).
+ *
+ * Grab the wb_err from the mapping. If it matches what we have in the file,
+ * then just quickly return 0. The file is all caught up.
+ *
+ * If it doesn't match, then take the mapping value, set the "seen" flag in
+ * it and try to swap it into place. If it works, or another task beat us
+ * to it with the new value, then update the f_wb_err and return the error
+ * portion. The error at this point _should_ be reported to userland.
+ *
+ * While we handle mapping->wb_err with atomic operations, the f_wb_err
+ * value is protected by the f_lock since we must ensure that it reflects
+ * the latest value swapped in for this file descriptor.
+ */
+int filemap_report_wb_error(struct file *file)
+{
+	int err = 0;
+	struct address_space *mapping = file->f_mapping;
+	u32 old;
+
+	old = READ_ONCE(mapping->wb_err);
+
+	/*
+	 * This catches the common case of no errors, and the case where
+	 * nothing has changed since we last checked.
+	 */
+	if (old == READ_ONCE(file->f_wb_err))
+		goto out;
+
+	spin_lock(&file->f_lock);
+	for (;;) {
+		u32 cur, new;
+
+		/*
+		 * We always store values with the "seen" bit set, so if this
+		 * matches what we already have, then we can call it done.
+		 * There is nothing to update so just return 0.
+		 */
+		if (old == file->f_wb_err)
+			break;
+
+		/* set flag and try to swap it into place */
+		new = old | WB_ERR_SEEN;
+		cur = cmpxchg(&mapping->wb_err, old, new);
+
+		/*
+		 * We can quit now if we successfully swapped in the new value
+		 * or someone else beat us to it with the same value that we
+		 * were planning to store.
+		 */
+		if (likely(cur == old || cur == new)) {
+			file->f_wb_err = new;
+			err = -(new & MAX_ERRNO);
+			break;
+		}
+
+		/* Raced with an update, try again */
+		old = cur;
+	}
+	spin_unlock(&file->f_lock);
+out:
+	return err;
+}
+EXPORT_SYMBOL(filemap_report_wb_error);
+
 /**
  * replace_page_cache_page - replace a pagecache page with a new one
  * @old:	page to be replaced
-- 
2.9.3

[toc] | [prev] | [next] | [standalone]

#1618322 — Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it

On Thu, Apr 06, 2017 at 03:14:52PM -0400, Jeff Layton wrote:
> @@ -868,6 +869,7 @@ struct file {
>  	struct list_head	f_tfile_llink;
>  #endif /* #ifdef CONFIG_EPOLL */
>  	struct address_space	*f_mapping;
> +	u32			f_wb_err;
>  } __attribute__((aligned(4)));	/* lest something weird decides that 2 is OK */
>  

I think we can squeeze that in next to f_flags?

> +/**
> + * filemap_set_wb_error - set the wb error in the mapping for later reporting
> + * @mapping: mapping in which the error should be set
> + * @err: error to set. must be negative value but not less than -MAX_ERRNO

Do we want to have users call filemap_set_wb_error(mapping, EIO)
or filemap_set_wb_error(mapping, -EIO)?  Either way, we can assert
that it's in the correct range (oh look, we have at least one user of
mapping_set_error calling it with a positive errno ...)

I've been playing with positive or negative errnos for the xarray, and
positive looks better to me, although there's a definite advantage to
being able to just call filemap_set_wb_error(mapping, result).

#define XAS_ERROR(errno)        ((struct xa_node *)((errno << 1) | 1))

static inline int xas_error(const struct xa_state *xas)
{
        unsigned long v = (unsigned long)xas->xa_node;
        return (v & 1) ? -(v >> 1) : 0;
}

static inline void xas_set_err(struct xa_state *xas, unsigned long err)
{
        XA_BUG_ON(err > MAX_ERRNO);
        xas->xa_node = XAS_ERROR(err);
}

> +	/*
> +	 * Ensure the error code actually fits where we want it to go. If it
> +	 * doesn't then just throw a warning and don't record anything.
> +	 */
> +	if (unlikely(err > 0 || err < -MAX_ERRNO)) {
> +		WARN(1, "err=%d\n", err);
> +		return;
> +	}

Cute trick to make this more succinct:

	if (WARN(err > 0 || err < -MAX_ERRNO), "err = %d\n", err)
		return;
or even ...

	if (WARN((unsigned int)-err > MAX_ERRNO), "err = %d\n", err)
		return;

> +		/* Clear out error bits and set new error */
> +		new = (old & ~MAX_ERRNO) | -err;
> +
> +		/* Only increment if someone has looked at it */
> +		if (old & WB_ERR_SEEN) {
> +			new += WB_ERR_CTR_INC;
> +			new &= ~WB_ERR_SEEN;
> +		}

Although we always want to clear out the SEEN bit if we're updating ... so

		new = (old & ~(MAX_ERRNO | WB_ERR_SEEN) | -err;

		/* Only increment if someone has looked at it */
		if (old & WB_ERR_SEEN)
			new += WB_ERR_CTR_INC;

... and then there's no need to update if it's the same errno and nobody's
seen it:

		if (old == new)
			break;

[...]

> +		/*
> +		 * We always store values with the "seen" bit set, so if this
> +		 * matches what we already have, then we can call it done.
> +		 * There is nothing to update so just return 0.
> +		 */
> +		if (old == file->f_wb_err)
> +			break;
> +
> +		/* set flag and try to swap it into place */
> +		new = old | WB_ERR_SEEN;

Again, I think we should avoid the cmpxchg with:

		if (old == new)
			break;

> +		cur = cmpxchg(&mapping->wb_err, old, new);
> +
> +		/*
> +		 * We can quit now if we successfully swapped in the new value
> +		 * or someone else beat us to it with the same value that we
> +		 * were planning to store.
> +		 */
> +		if (likely(cur == old || cur == new)) {
> +			file->f_wb_err = new;
> +			err = -(new & MAX_ERRNO);
> +			break;
> +		}
> +
> +		/* Raced with an update, try again */
> +		old = cur;

Well ... should we?  We're returning an error which is new to this fd anyway.
Do we want to return the most recent error by a nanosecond, or should we
return the previous one and then see this one next time we call fsync()?

I'd lean towards not looping here; not even looking at 'cur'.

[toc] | [prev] | [next] | [standalone]

#1618785 — Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it

On Thu, 2017-04-06 at 13:05 -0700, Matthew Wilcox wrote:
> On Thu, Apr 06, 2017 at 03:14:52PM -0400, Jeff Layton wrote:
> > @@ -868,6 +869,7 @@ struct file {
> >  	struct list_head	f_tfile_llink;
> >  #endif /* #ifdef CONFIG_EPOLL */
> >  	struct address_space	*f_mapping;
> > +	u32			f_wb_err;
> >  } __attribute__((aligned(4)));	/* lest something weird decides that 2 is OK */
> >  
> 
> I think we can squeeze that in next to f_flags?
> 

Sure, will do. I meant to look at pahole output and see if there are
existing holes.

> > +/**
> > + * filemap_set_wb_error - set the wb error in the mapping for later reporting
> > + * @mapping: mapping in which the error should be set
> > + * @err: error to set. must be negative value but not less than -MAX_ERRNO
> 
> Do we want to have users call filemap_set_wb_error(mapping, EIO)
> or filemap_set_wb_error(mapping, -EIO)?  Either way, we can assert
> that it's in the correct range (oh look, we have at least one user of
> mapping_set_error calling it with a positive errno ...)
> 

Yeah, I sent a patch for that a while back but I don't think anyone
picked it up. Luckily that caller is harmless since EIO just ends up in
the default case and gets turned into -EIO.

> I've been playing with positive or negative errnos for the xarray, and
> positive looks better to me, although there's a definite advantage to
> being able to just call filemap_set_wb_error(mapping, result).
> 

That's my main rationale. We generally use negative error codes in the
kernel, so let's do what's easiest for most callsites. I say negative
error codes here.


> #define XAS_ERROR(errno)        ((struct xa_node *)((errno << 1) | 1))
> 
> static inline int xas_error(const struct xa_state *xas)
> {
>         unsigned long v = (unsigned long)xas->xa_node;
>         return (v & 1) ? -(v >> 1) : 0;
> }
> 
> static inline void xas_set_err(struct xa_state *xas, unsigned long err)
> {
>         XA_BUG_ON(err > MAX_ERRNO);
>         xas->xa_node = XAS_ERROR(err);
> }
> 
> > +	/*
> > +	 * Ensure the error code actually fits where we want it to go. If it
> > +	 * doesn't then just throw a warning and don't record anything.
> > +	 */
> > +	if (unlikely(err > 0 || err < -MAX_ERRNO)) {
> > +		WARN(1, "err=%d\n", err);
> > +		return;
> > +	}
> 
> Cute trick to make this more succinct:
> 
> 	if (WARN(err > 0 || err < -MAX_ERRNO), "err = %d\n", err)
> 		return;
> or even ...
> 
> 	if (WARN((unsigned int)-err > MAX_ERRNO), "err = %d\n", err)
> 		return;
> 

Nice. I always forget that WARN has a return. Will fix.

> > +		/* Clear out error bits and set new error */
> > +		new = (old & ~MAX_ERRNO) | -err;
> > +
> > +		/* Only increment if someone has looked at it */
> > +		if (old & WB_ERR_SEEN) {
> > +			new += WB_ERR_CTR_INC;
> > +			new &= ~WB_ERR_SEEN;
> > +		}
> 
> Although we always want to clear out the SEEN bit if we're updating ... so
> 
> 		new = (old & ~(MAX_ERRNO | WB_ERR_SEEN) | -err;
> 
> 		/* Only increment if someone has looked at it */
> 		if (old & WB_ERR_SEEN)
> 			new += WB_ERR_CTR_INC;
> 

Sure, that is more succinct.

> ... and then there's no need to update if it's the same errno and nobody's
> seen it:
> 
> 		if (old == new)
> 			break;
> 

No, we can't do this. The thing could have just been updated by a task
that is setting the "seen" bit. We don't want to lose the error here. We
always have to do the cmpxchg on the set_wb_error side, I think.

> [...]
> 
> > +		/*
> > +		 * We always store values with the "seen" bit set, so if this
> > +		 * matches what we already have, then we can call it done.
> > +		 * There is nothing to update so just return 0.
> > +		 */
> > +		if (old == file->f_wb_err)
> > +			break;
> > +
> > +		/* set flag and try to swap it into place */
> > +		new = old | WB_ERR_SEEN;
> 
> Again, I think we should avoid the cmpxchg with:
> 
> 		if (old == new)
> 			break;
> 

Yeah, we may be able to do this one. I had myself convinced otherwise
yesterday, but I think you may be right.

> > +		cur = cmpxchg(&mapping->wb_err, old, new);
> > +
> > +		/*
> > +		 * We can quit now if we successfully swapped in the new value
> > +		 * or someone else beat us to it with the same value that we
> > +		 * were planning to store.
> > +		 */
> > +		if (likely(cur == old || cur == new)) {
> > +			file->f_wb_err = new;
> > +			err = -(new & MAX_ERRNO);
> > +			break;
> > +		}
> > +
> > +		/* Raced with an update, try again */
> > +		old = cur;
> 
> Well ... should we?  We're returning an error which is new to this fd anyway.
> Do we want to return the most recent error by a nanosecond, or should we
> return the previous one and then see this one next time we call fsync()?
> 
> I'd lean towards not looping here; not even looking at 'cur'.
> 

Yeah, that might be fine here. Let me think about it a bit more.

-- 
Jeff Layton <jlayton@redhat.com>

[toc] | [prev] | [next] | [standalone]

#1619571

[Multipart message — attachments visible in raw view] — view raw

On Fri, Apr 07 2017, Jeff Layton wrote:

>
>> ... and then there's no need to update if it's the same errno and nobody's
>> seen it:
>> 
>> 		if (old == new)
>> 			break;
>> 
>
> No, we can't do this. The thing could have just been updated by a task
> that is setting the "seen" bit. We don't want to lose the error here. We
> always have to do the cmpxchg on the set_wb_error side, I think.

I don't follow your logic.
If (old == new) then there was a moment since this function started when
performing the cmpxchg() would not have changed the contents of memory.
So let's pretend it did actually happen at that moment, and not change
memory.

If we race with a task setting the "seen" bit, then it will have seen
the error *after* the new error, that this thread is reporting, actually
happened.  So the result is still correct.

Thanks,
NeilBrown

[toc] | [prev] | [next] | [standalone]

#1619927 — Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it

On Mon, 2017-04-10 at 09:15 +1000, NeilBrown wrote:
> On Fri, Apr 07 2017, Jeff Layton wrote:
> 
> > 
> > > ... and then there's no need to update if it's the same errno and nobody's
> > > seen it:
> > > 
> > > 		if (old == new)
> > > 			break;
> > > 
> > 
> > No, we can't do this. The thing could have just been updated by a task
> > that is setting the "seen" bit. We don't want to lose the error here. We
> > always have to do the cmpxchg on the set_wb_error side, I think.
> 
> I don't follow your logic.
> If (old == new) then there was a moment since this function started when
> performing the cmpxchg() would not have changed the contents of memory.
> So let's pretend it did actually happen at that moment, and not change
> memory.
> 
> If we race with a task setting the "seen" bit, then it will have seen
> the error *after* the new error, that this thread is reporting, actually
> happened.  So the result is still correct.
> 

Ok, that does make sense. I'll plan to do that.

There's also a bug in the last patch that I sent. We need to mark the
SEEN bit when we sample the value at open time, so we need a
filemap_sample_wb_error function to grab the current wb_err_t and mark
it SEEN if necessary.

That also gives us a way to handle something like filemap_write_and_wait
(which doesn't take a struct file). We can sample the wb_err_t prior to
starting writeback, and then return an error if anything failed after
that point.

I think that's probably close enough to how the current code works that
we can use it to make drop-in replacements for filemap_write_and_wait*
which should keep us from having to change so much existing code here.

filemap_check_errors would need to take a previously-sampled wb_err_t
argument, but only the lowest-level callers of that and
filemap_fdatawait* would need to deal with them directly.
-- 
Jeff Layton <jlayton@redhat.com>

[toc] | [prev] | [next] | [standalone]

#1618377

[Multipart message — attachments visible in raw view] — view raw

On Thu, Apr 06 2017, Jeff Layton wrote:

>
> I tried to avoid updating things unnecesssarily. I could use some
> guidance on how to specify the constants in terms of MAX_ERRNO as well.

ilog2() defined in include/linux/log2.h

And you have MAX_ERROR in one comment, instead of MAX_ERRNO :-)

>  
> -  flush: called by the close(2) system call to flush a file
> +  flush: called by the close(2) system call to flush a file. Writeback
> +	errors not previously reported via fsync should be reported
> +	here as you would for fsync.

"could", not "should".  I think it is agreed that this is a good
idea, is it?

> diff --git a/include/linux/fs.h b/include/linux/fs.h
> index 7251f7bb45e8..f33857113ff4 100644
> --- a/include/linux/fs.h
> +++ b/include/linux/fs.h
> @@ -394,6 +394,7 @@ struct address_space {
>  	gfp_t			gfp_mask;	/* implicit gfp mask for allocations */
>  	struct list_head	private_list;	/* ditto */
>  	void			*private_data;	/* ditto */
> +	u32			wb_err;

I would rather this was a wb_err_t or similar, and that the functions
which implement it take a pointer to a wb_err_t.
Then the 'error' in 'struct nfs_open_context' could become a 'wb_err_t',
and nfs could use these functions to do error tracking the way it wants
to.

Thanks - looking good.

NeilBrown

[toc] | [prev] | [next] | [standalone]

#1616443

[Multipart message — attachments visible in raw view] — view raw

On Tue, Apr 04 2017, Jeff Layton wrote:

> On Tue, 2017-04-04 at 09:12 -0700, Matthew Wilcox wrote:
>> On Tue, Apr 04, 2017 at 08:17:48AM -0400, Jeff Layton wrote:
>> > Agreed that we should focus on POSIX compliance. I'll also note that
>> > POSIX states:
>> > 
>> > "If more than one error occurs in processing a function call, any one
>> > of the possible errors may be returned, as the order of
>> > detection is undefined."
>> > 
>> >     http://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html#tag_15_03
>> > 
>> > So, I'd like to push back on this idea that we need to prefer reporting
>> > -EIO over other errors. POSIX certainly doesn't mandate that. 
>> 
>> I honestly wonder if we need to support ENOSPC from writeback at all.
>> Looking at our history, the AS_EIO / AS_ENOSPC came from this patch
>> in 2003:
>> 
>> https://git.kernel.org/pub/scm/linux/kernel/git/tglx/history.git/commit/?id=fcad2b42fc2e15a94ba1a1ba8535681a735bfd16
>> 
>> That seems to come from here:
>> http://lkml.iu.edu/hypermail/linux/kernel/0308.0/0205.html
>> which is marked as a resend, but I can't find the original.
>> 
>> It's a little misleading because the immediately preceding patch
>> introduced mapping->error, so there's no precedent here to speak of.
>> It looks like we used to just silently lose writeback errors (*cough*).
>> 
>> I'd like to suggest that maybe we don't need to support multiple errors
>> at all.  That all errors, including ENOSPC, get collapsed into EIO.
>> POSIX already tells us to do that for close() and permits us to do that
>> for fsync().
>> 
>
> That is certainly allowed under POSIX as I interpret the spec. At a
> minimum we just need a single flag and can collapse all errors under
> that.
>
> That said, I think giving more specific errors where we can is useful.
> When your program is erroring out and writing 'I/O error' to the logs,
> then how much time will your admins burn before they figure out that it
> really failed because the filesystem was full?

What if you don't have an admin?  What if it was an over-quota error?
I think precise error messages are valuable.
I am leaning towards "last error wins" though.  The complexity of any
scheme that reports "worst recent error" seems to out weigh the value.

I think we should present this as a service to filesystems. e.g. create
a "recent_wb_error" structure which the filesystem can record errors in
when they occur, and syscalls can read errors from.
One of these would be provided in 'struct address_space', but
filesystems can easily embed one in their own data structure
(e.g. nfs_open_context) if they want to.

I don't think we should return a recent_wb_error on close by default,
but individual filesystems can ("man 2 close" implies NFS does this for
EDQUOT at it should continue to do so).

fsync() (and file_sync_range()) should return a recent_wb_error, but
what about write()?  It would be a suitable way to stop an application
early, but it isn't exactly the requested write that failed...
Posix says of EIO from write:

    A physical I/O error has occurred.

which is rather vague.  Where and when did this error in physics (:-)
occur?

O_DIRECT write() can get an EIO from a previous write-back write to the
same file.  Maybe non-O_DIRECT writes should too?

Thanks,
NeilBrown

[toc] | [prev] | [next] | [standalone]

#1616848 — Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it

On Wed, 2017-04-05 at 09:13 +1000, NeilBrown wrote:
> On Tue, Apr 04 2017, Jeff Layton wrote:
> 
> > On Tue, 2017-04-04 at 09:12 -0700, Matthew Wilcox wrote:
> > > On Tue, Apr 04, 2017 at 08:17:48AM -0400, Jeff Layton wrote:
> > > > Agreed that we should focus on POSIX compliance. I'll also note that
> > > > POSIX states:
> > > > 
> > > > "If more than one error occurs in processing a function call, any one
> > > > of the possible errors may be returned, as the order of
> > > > detection is undefined."
> > > > 
> > > >     http://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html#tag_15_03
> > > > 
> > > > So, I'd like to push back on this idea that we need to prefer reporting
> > > > -EIO over other errors. POSIX certainly doesn't mandate that. 
> > > 
> > > I honestly wonder if we need to support ENOSPC from writeback at all.
> > > Looking at our history, the AS_EIO / AS_ENOSPC came from this patch
> > > in 2003:
> > > 
> > > https://git.kernel.org/pub/scm/linux/kernel/git/tglx/history.git/commit/?id=fcad2b42fc2e15a94ba1a1ba8535681a735bfd16
> > > 
> > > That seems to come from here:
> > > http://lkml.iu.edu/hypermail/linux/kernel/0308.0/0205.html
> > > which is marked as a resend, but I can't find the original.
> > > 
> > > It's a little misleading because the immediately preceding patch
> > > introduced mapping->error, so there's no precedent here to speak of.
> > > It looks like we used to just silently lose writeback errors (*cough*).
> > > 
> > > I'd like to suggest that maybe we don't need to support multiple errors
> > > at all.  That all errors, including ENOSPC, get collapsed into EIO.
> > > POSIX already tells us to do that for close() and permits us to do that
> > > for fsync().
> > > 
> > 
> > That is certainly allowed under POSIX as I interpret the spec. At a
> > minimum we just need a single flag and can collapse all errors under
> > that.
> > 
> > That said, I think giving more specific errors where we can is useful.
> > When your program is erroring out and writing 'I/O error' to the logs,
> > then how much time will your admins burn before they figure out that it
> > really failed because the filesystem was full?
> 
> What if you don't have an admin?  What if it was an over-quota error?
> I think precise error messages are valuable.
> I am leaning towards "last error wins" though.  The complexity of any
> scheme that reports "worst recent error" seems to out weigh the value.
> 
> I think we should present this as a service to filesystems. e.g. create
> a "recent_wb_error" structure which the filesystem can record errors in
> when they occur, and syscalls can read errors from.
> One of these would be provided in 'struct address_space', but
> filesystems can easily embed one in their own data structure
> (e.g. nfs_open_context) if they want to.
> 
> I don't think we should return a recent_wb_error on close by default,
> but individual filesystems can ("man 2 close" implies NFS does this for
> EDQUOT at it should continue to do so).
> 
> fsync() (and file_sync_range()) should return a recent_wb_error, but
> what about write()?  It would be a suitable way to stop an application
> early, but it isn't exactly the requested write that failed...
> Posix says of EIO from write:
> 
>     A physical I/O error has occurred.
> 
> which is rather vague.  Where and when did this error in physics (:-)
> occur?
> 
> O_DIRECT write() can get an EIO from a previous write-back write to the
> same file.  Maybe non-O_DIRECT writes should too?
> 

Some already do this for buffered writes.

This is really a philosophical question, IMO...is it correct to return
an error on a write call, due to writeback failing previously or during
the write call, quite possibly to a range that the write call does not
touch? I can see an argument either way for this.

Also, if we do think that returning an error on the write is the right
thing to do, should that error advance the sequence counter in the
struct file, such that an fsync afterward gets back 0? My feeling here
is that fsync should still report an error after a failed write, but
maybe that's wrong?

This is certainly one area where switching to synchronous writes on
error would make things a little simpler.
-- 
Jeff Layton <jlayton@redhat.com>

[toc] | [prev] | [next] | [standalone]

#1617447

[Multipart message — attachments visible in raw view] — view raw

On Wed, Apr 05 2017, Jeff Layton wrote:

>> 
>> O_DIRECT write() can get an EIO from a previous write-back write to the
>> same file.  Maybe non-O_DIRECT writes should too?
>> 
>
> Some already do this for buffered writes.
>
> This is really a philosophical question, IMO...is it correct to return
> an error on a write call, due to writeback failing previously or during
> the write call, quite possibly to a range that the write call does not
> touch? I can see an argument either way for this.

I like the "we already do" argument.

>
> Also, if we do think that returning an error on the write is the right
> thing to do, should that error advance the sequence counter in the
> struct file, such that an fsync afterward gets back 0? My feeling here
> is that fsync should still report an error after a failed write, but
> maybe that's wrong?

My first thought was that one the error has been returned to any
syscall on a given fd, it has been returned.  Once is enough.
My second thought was that maybe your feeling is right.  Having a well
defined error-return point in fsync feels like a nice design.
My third thought was that this would mean either
 - write continues to fail until fsync is called (probably bad), or
 - we need two counters per "struct file", one for fsync, one for write.
   I don't like that much.

So I'm going back to my first thought.

Thanks,
NeilBrown


>
> This is certainly one area where switching to synchronous writes on
> error would make things a little simpler.
> -- 
> Jeff Layton <jlayton@redhat.com>

[toc] | [prev] | [next] | [standalone]

#1616030 — Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it

On Tue, Apr 04, 2017 at 04:53:58AM -0700, Matthew Wilcox wrote:
> I'd like us to focus on our POSIX compliance here and not return
> arbitrary errors.  The relevant pages are here:
> 
> http://pubs.opengroup.org/onlinepubs/9699919799/functions/fsync.html
> http://pubs.opengroup.org/onlinepubs/9699919799/functions/write.html
> http://pubs.opengroup.org/onlinepubs/9699919799/functions/close.html
> 
> For close(), we have to map every error to EIO.

I don't believe that's true.  POSIX explicitly states that
implementations may return additional errors, and define additional
errors, in addition to the ones that are listeds in the specification.

The specification is pretty clear about saying when a particular
system call "shall" fail (meaning it must fail if it was so listed),
and when it "may" fail.  But no where does it say that these are the
only situations when a system call is allowed to fail.

Which is good, because ext4 and xfs will both return EUCLEAN if the
file system is corrupted.  (Mainly because it's too painful to define
a new errno, EFSCORRUPTED --- not because of trying to get it into
Posix, but because it's painful to get new errno's defined in glibc.)
POSIX says *nothing* about file systems being corrupted, and if your
interpretation were correct, we're already in violation of POSIX....

	       	    	     	   	      - Ted
					      

[toc] | [prev] | [next] | [standalone]

#1616425

[Multipart message — attachments visible in raw view] — view raw

On Tue, Apr 04 2017, Matthew Wilcox wrote:

> On Tue, Apr 04, 2017 at 01:03:22PM +1000, NeilBrown wrote:
>> On Mon, Apr 03 2017, Jeff Layton wrote:
>> 
>> > On Mon, 2017-04-03 at 12:16 -0700, Matthew Wilcox wrote:
>> >> So, OK, that makes sense, we should keep allowing filesystems to report
>> >> ENOSPC as a writeback error.  But I think much of the argument below
>> >> still holds, and we should continue to have a prior EIO to be reported
>> >> over a new ENOSPC (even if the program has already consumed the EIO).
>> >
>> > I'm fine with that (though I'd like Neil's thoughts before we decide
>> > anything) there.
>> 
>> I'd like there be a well defined time when old errors were forgotten.
>> It does make sense for EIO to persist even if ENOSPC or EDQUOT is
>> received, but not forever.
>> Clearing the remembered errors when put_write_access() causes
>> i_writecount to reach zero is one option (as suggested), but I'm not
>> sure I'm happy with it.
>> 
>> Local filesystems, or network filesystems which receive strong write
>> delegations, should only ever return EIO to fsync.  We should
>> concentrate on them first, I think.  As there is only one possible
>> error, the seq counter is sufficient to "clear" it once it has been
>> reported to fsync() (or write()?).
>> 
>> Other network filesystems could return a whole host of errors: ENOSPC
>> EDQUOT ESTALE EPERM EFBIG ...
>> Do we want to limit exactly which errors are allowed in generic code, or
>> do we just support EIO generically and expect the filesystem to sort out
>> the details for anything else?
>
> I'd like us to focus on our POSIX compliance here and not return
> arbitrary errors.  The relevant pages are here:
>
> http://pubs.opengroup.org/onlinepubs/9699919799/functions/fsync.html
> http://pubs.opengroup.org/onlinepubs/9699919799/functions/write.html
> http://pubs.opengroup.org/onlinepubs/9699919799/functions/close.html
>
> For close(), we have to map every error to EIO.
> For fsync(), we can return any error that write() could have.  That limits
> us to:
>
> EFBIG ENOSPC EIO ENOBUFS ENXIO
>
> I think EFBIG really isn't a writeback error; are there any network
> filesystems that don't know the file size limit at the time they accept
> the original write?  ENOBUFS seems like a transient error (*this* call to
> fsync() failed, but the next one may succeed ... it's the equivalent of
> ENOMEM).  ENXIO seems to me like it's a submission error, not a writeback
> error.  So that leaves us with ENOSPC and EIO, as we have support today.

I guess Posix doesn't acknowledge the existence of disk quotas?
I think we need to add EDQUOT to your list.
Other hypothetical errors errors from the server such as EPERM or ESTALE
can reasonably be mapped to EIO.

>
>> One possible approach a filesystem could take is just to allow a single
>> async writeback error.  After that error, all subsequent write()
>> system calls become synchronous. As write() or fsync() is called on each
>> file descriptor (which could possibly have sent the write which caused
>> the error), an error is returned and that fact is counted.  Once we have
>> returned as many errors as there are open file descriptors
>> (i_writecount?), and have seen a successful write, the filesystem
>> forgets all recorded errors and switches back to async writes (for that
>> inode).   NFS does this switch-to-sync-on-error.  See nfs_need_check_write().
>> 
>> The "which could possibly have sent the write which caused the error" is
>> an explicit reference to NFS.  NFS doesn't use the AS_EIO/AS_ENOSPC
>> flags to return async errors.  It allocates an nfs_open_context for each
>> user who opens a given inode, and stores an error in there.  Each dirty
>> pages is associated with one of these, so errors a sure to go to the
>> correct user, though not necessarily the correct fd at present.
>
> ... and you need the nfs_open_context in order to use the correct
> credentials when writing a page to the server, correct?

Correct.

Thanks,
NeilBrown


>
>> When we specify the new behaviour we should be careful to be as vague as
>> possible while still saying what we need.  This allows filesystems some
>> flexibility.
>> 
>>   If an error happens during writeback, the next write() or fsync() (or
>>   ....) on the file descriptor to which data was written will return -1
>>   with errno set to EIO or some other relevant error.  Other file
>>   descriptors open on the same file may receive EIO or some other error
>>   on a subsequent appropriate system call.
>>   It should not be assumed that close() will return an error.  fsync()
>>   must be called before close() if writeback errors are important to the
>>   application.
>
> Thanks for explaining what NFS does today.

[toc] | [prev] | [next] | [standalone]

#1615281 — Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it

On Mon, Apr 03, 2017 at 02:25:11PM +1000, NeilBrown wrote:
> I don't like that you need to add a 'flush' handler to every filesystem,
> most of which just call
>  +	return filemap_report_wb_error(file);
> 
> Could we just have
> 	if (filp->f_op->flush)
> 		retval = filp->f_op->flush(filp, id);
> +	else
> +		retval = filemap_report_wb_error(filp);
> in flip_close() ??

Maybe this is badly named as ext4_flush_file().  Maybe this should be
generic_flush_file(), and then there's no per-filesystem overhead to this?

[toc] | [prev] | [standalone]

Page 3 of 3 — ← Prev page 1 2 [3]

Back to top | Article view | linux.kernel

csiph-web