Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1614272 > unrolled thread

[RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it

Back to article view | Back to linux.kernel

Contents

  [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-03-31 21:30 +0200
    [RFC PATCH 2/4] dax: set errors in mapping when writeback fails Jeff Layton <jlayton@redhat.com> - 2017-03-31 21:30 +0200
    [RFC PATCH 1/4] fs: new infrastructure for writeback error handling and reporting Jeff Layton <jlayton@redhat.com> - 2017-03-31 21:30 +0200
      Re: [RFC PATCH 1/4] fs: new infrastructure for writeback error  handling and reporting Nikolay Borisov <nborisov@suse.com> - 2017-04-03 09:20 +0200
        Re: [RFC PATCH 1/4] fs: new infrastructure for writeback error  handling and reporting Jeff Layton <jlayton@redhat.com> - 2017-04-03 12:30 +0200
      Re: [RFC PATCH 1/4] fs: new infrastructure for writeback error  handling and reporting Matthew Wilcox <willy@infradead.org> - 2017-04-03 16:50 +0200
        Re: [RFC PATCH 1/4] fs: new infrastructure for writeback error  handling and reporting Jeff Layton <jlayton@redhat.com> - 2017-04-03 17:30 +0200
          Re: [RFC PATCH 1/4] fs: new infrastructure for writeback error  handling and reporting Matthew Wilcox <willy@infradead.org> - 2017-04-03 18:20 +0200
            Re: [RFC PATCH 1/4] fs: new infrastructure for writeback error  handling and reporting Jeff Layton <jlayton@redhat.com> - 2017-04-03 18:40 +0200
    [RFC PATCH 4/4] ext4: wire it up to the new writeback error reporting infrastructure Jeff Layton <jlayton@redhat.com> - 2017-03-31 21:30 +0200
    Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it NeilBrown <neilb@suse.com> - 2017-04-03 06:30 +0200
      Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-04-03 12:30 +0200
        Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Matthew Wilcox <willy@infradead.org> - 2017-04-03 16:40 +0200
          Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-04-03 19:50 +0200
            Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeremy Allison <jra@samba.org> - 2017-04-03 20:20 +0200
              Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-04-03 20:20 +0200
                Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeremy Allison <jra@samba.org> - 2017-04-03 20:40 +0200
                  Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeremy Allison <jra@samba.org> - 2017-04-03 20:50 +0200
                    Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-04-03 20:50 +0200
            Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Matthew Wilcox <willy@infradead.org> - 2017-04-03 21:20 +0200
              Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-04-03 22:20 +0200
                Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Matthew Wilcox <willy@infradead.org> - 2017-04-04 04:50 +0200
                Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it NeilBrown <neilb@suse.com> - 2017-04-04 05:10 +0200
                  Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-04-04 13:50 +0200
                    Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it NeilBrown <neilb@suse.com> - 2017-04-05 00:50 +0200
                  Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Matthew Wilcox <willy@infradead.org> - 2017-04-04 14:00 +0200
                    Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-04-04 14:20 +0200
                      Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Matthew Wilcox <willy@infradead.org> - 2017-04-04 18:20 +0200
                        Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-04-04 18:30 +0200
                          Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Matthew Wilcox <willy@infradead.org> - 2017-04-04 19:10 +0200
                            Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-04-04 20:10 +0200
                            Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it NeilBrown <neilb@suse.com> - 2017-04-05 01:00 +0200
                            Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-04-05 22:00 +0200
                              Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Matthew Wilcox <willy@infradead.org> - 2017-04-05 23:10 +0200
                                Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it NeilBrown <neilb@suse.com> - 2017-04-06 02:30 +0200
                              Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it NeilBrown <neilb@suse.com> - 2017-04-06 02:10 +0200
                                Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Matthew Wilcox <willy@infradead.org> - 2017-04-06 05:00 +0200
                                  Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it NeilBrown <neilb@suse.com> - 2017-04-06 07:20 +0200
                                    Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Matthew Wilcox <willy@infradead.org> - 2017-04-06 15:40 +0200
                                      Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it NeilBrown <neilb@suse.com> - 2017-04-07 00:00 +0200
                                Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-04-06 16:10 +0200
                                Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-04-06 21:20 +0200
                                  Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Matthew Wilcox <willy@infradead.org> - 2017-04-06 22:20 +0200
                                    Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-04-07 15:20 +0200
                                      Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it NeilBrown <neilb@suse.com> - 2017-04-10 01:20 +0200
                                        Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-04-10 15:30 +0200
                                  Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it NeilBrown <neilb@suse.com> - 2017-04-07 00:20 +0200
                          Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it NeilBrown <neilb@suse.com> - 2017-04-05 01:20 +0200
                            Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Jeff Layton <jlayton@redhat.com> - 2017-04-05 13:20 +0200
                              Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it NeilBrown <neilb@suse.com> - 2017-04-06 02:50 +0200
                    Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Theodore Ts'o <tytso@mit.edu> - 2017-04-04 15:40 +0200
                    Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it NeilBrown <neilb@suse.com> - 2017-04-05 00:30 +0200
      Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking  infrastructure and convert ext4 to use it Matthew Wilcox <willy@infradead.org> - 2017-04-03 17:00 +0200

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

#1615533 — Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it

On Mon, 2017-04-03 at 12:16 -0700, Matthew Wilcox wrote:
> On Mon, Apr 03, 2017 at 01:47:37PM -0400, Jeff Layton wrote:
> > > I wonder whether it's even worth supporting both EIO and ENOSPC for a
> > > writeback problem.  If I understand correctly, at the time of write(),
> > > filesystems check to see if they have enough blocks to satisfy the
> > > request, so ENOSPC only comes up in the writeback context for thinly
> > > provisioned devices.
> > 
> > No, ENOSPC on writeback can certainly happen with network filesystems.
> > NFS and CIFS have no way to reserve space. You wouldn't want to have to
> > do an extra RPC on every buffered write. :)
> 
> Aaah, yes, network filesystems.  I would indeed not want to do an extra
> RPC on every write to a hole (it's a hole vs non-hole question, rather
> than a buffered/unbuffered question ... unless you're WAFLing and not
> reclaiming quickly enough, I suppose).
> 
> So, OK, that makes sense, we should keep allowing filesystems to report
> ENOSPC as a writeback error.  But I think much of the argument below
> still holds, and we should continue to have a prior EIO to be reported
> over a new ENOSPC (even if the program has already consumed the EIO).
> 

I'm fine with that (though I'd like Neil's thoughts before we decide
anything) there.

> If you find that unconvincing, we could do something like this ...
> 
> void filemap_set_wb_error(struct address_space *mapping, int err)
> {
> 	struct inode *inode = mapping->host;
> 
> 	if (!err)
> 		return;
> 	/*
> 	 * This should be called with the error code that we want to return
> 	 * on fsync. Thus, it should always be <= 0.
> 	 */
> 	WARN_ON(err > 0);
> 
> 	spin_lock(&inode->i_lock);
> 	if (err == -EIO)
> 		mapping->wb_err |= 1;
> 	else if (err == -ENOSPC)
> 		mapping->wb_err |= 2;
> 	mapping->wb_err += 4;
> 	spin_unlock(&inode->i_lock);
> }
> 
> int filemap_report_wb_error(struct file *file)
> {
> 	struct inode *inode = file_inode(file);
> 	struct address_space *mapping = file->f_mapping;
> 	int err;
> 
> 	spin_lock(&inode->i_lock);
> 	if (file->f_wb_err == mapping->wb_err) {
> 		err = 0;
> 	} else if (mapping->wb_err & 1) {
> 		filp->f_wb_err = mapping->wb_err & ~2;
> 		err = -EIO;
> 	} else {
> 		filp->f_wb_err = mapping->wb_err;
> 		err = -ENOSPC;
> 	}
> 	spin_unlock(&inode->i_lock);
> 	return err;
> }
> 
> If I got that right, calling fsync() on an inode which has experienced
> both errors would first get an EIO.  Calling fsync() on it again would
> get an ENOSPC.  Calling fsync() on it a third time would get 0.  When
> either error occurs again, the thread will go back through the cycle
> (EIO -> ENOSPC -> 0).
> 

I don't think so? mapping->wb_err would still have 0x1 set after the
first call so you'd always end up in the first else if branch.

It's getting toward beer 30 here though so I could be misreading it.

In any case, I'd rather not do this any more cleverly than we have to.
Simpler is better here, and letting EIO override ENOSPC would be
preferable to me.

Also, since we're going to do this with 32 bit values, it might be nice
to use atomics and not need the spinlock there, especially if we want
to be able to clear that value out when the i_writecount goes to 0.

-- 
Jeff Layton <jlayton@redhat.com>

[toc] | [prev] | [next] | [standalone]

#1615669 — Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it

On Mon, Apr 03, 2017 at 04:16:17PM -0400, Jeff Layton wrote:
> On Mon, 2017-04-03 at 12:16 -0700, Matthew Wilcox wrote:
> > int filemap_report_wb_error(struct file *file)
> > {
> > 	struct inode *inode = file_inode(file);
> > 	struct address_space *mapping = file->f_mapping;
> > 	int err;
> > 
> > 	spin_lock(&inode->i_lock);
> > 	if (file->f_wb_err == mapping->wb_err) {
> > 		err = 0;
> > 	} else if (mapping->wb_err & 1) {
> > 		filp->f_wb_err = mapping->wb_err & ~2;
> > 		err = -EIO;
> > 	} else {
> > 		filp->f_wb_err = mapping->wb_err;
> > 		err = -ENOSPC;
> > 	}
> > 	spin_unlock(&inode->i_lock);
> > 	return err;
> > }
> > 
> > If I got that right, calling fsync() on an inode which has experienced
> > both errors would first get an EIO.  Calling fsync() on it again would
> > get an ENOSPC.  Calling fsync() on it a third time would get 0.  When
> > either error occurs again, the thread will go back through the cycle
> > (EIO -> ENOSPC -> 0).
> > 
> 
> I don't think so? mapping->wb_err would still have 0x1 set after the
> first call so you'd always end up in the first else if branch.
> 
> It's getting toward beer 30 here though so I could be misreading it.

Well, yes, of course you misread it.  You read what I actually wrote
instead of what I intended to write.  Silly Jeff ...

int filemap_report_wb_error(struct file *file)
{
	struct inode *inode = file_inode(file);
	struct address_space *mapping = file->f_mapping;
	int err;

	spin_lock(&inode->i_lock);
	if (file->f_wb_err == mapping->wb_err) {
		err = 0;
	} else if ((mapping->wb_err ^ file->f_wb_err) == 2) {
		filp->f_wb_err = mapping->wb_err;
		err = -ENOSPC;
	} else {
		filp->f_wb_err = mapping->wb_err & ~2;
		err = -EIO;
	}
	spin_unlock(&inode->i_lock);
	return err;
}

The read side is easier in terms of atomic ...

int filemap_report_wb_error(struct file *file)
{
	unsigned int wb_err = atomic_read(&file->f_mapping->wb_err)

	if (file->f_wb_err == wb_err)
		return 0;
	if ((file->f_wb_err ^ wb_err) == 2) {
		filp->f_wb_err = wb_err;
		return -ENOSPC;
	} else {
		filp->f_wb_err = wb_err & ~2;
		return -EIO;
	}
}

but doing the write side with an atomic looks incredibly painful.  Since
we don't actually need to make the write side scalable, I'd rather see the
write side continue to use a spinlock and do the read side this way:

int filemap_report_wb_error(struct file *file)
{
	unsigned int wb_err = READ_ONCE(file->f_mapping->wb_err)

	if (file->f_wb_err == wb_err)
		return 0;
	if ((file->f_wb_err ^ wb_err) == 2) {
		filp->f_wb_err = wb_err;
		return -ENOSPC;
	} else {
		filp->f_wb_err = wb_err & ~2;
		return -EIO;
	}
}

[toc] | [prev] | [next] | [standalone]

#1615677

[Multipart message — attachments visible in raw view] — view raw

On Mon, Apr 03 2017, Jeff Layton wrote:

> On Mon, 2017-04-03 at 12:16 -0700, Matthew Wilcox wrote:
>> On Mon, Apr 03, 2017 at 01:47:37PM -0400, Jeff Layton wrote:
>> > > I wonder whether it's even worth supporting both EIO and ENOSPC for a
>> > > writeback problem.  If I understand correctly, at the time of write(),
>> > > filesystems check to see if they have enough blocks to satisfy the
>> > > request, so ENOSPC only comes up in the writeback context for thinly
>> > > provisioned devices.
>> > 
>> > No, ENOSPC on writeback can certainly happen with network filesystems.
>> > NFS and CIFS have no way to reserve space. You wouldn't want to have to
>> > do an extra RPC on every buffered write. :)
>> 
>> Aaah, yes, network filesystems.  I would indeed not want to do an extra
>> RPC on every write to a hole (it's a hole vs non-hole question, rather
>> than a buffered/unbuffered question ... unless you're WAFLing and not
>> reclaiming quickly enough, I suppose).
>> 
>> So, OK, that makes sense, we should keep allowing filesystems to report
>> ENOSPC as a writeback error.  But I think much of the argument below
>> still holds, and we should continue to have a prior EIO to be reported
>> over a new ENOSPC (even if the program has already consumed the EIO).
>> 
>
> I'm fine with that (though I'd like Neil's thoughts before we decide
> anything) there.

I'd like there be a well defined time when old errors were forgotten.
It does make sense for EIO to persist even if ENOSPC or EDQUOT is
received, but not forever.
Clearing the remembered errors when put_write_access() causes
i_writecount to reach zero is one option (as suggested), but I'm not
sure I'm happy with it.

Local filesystems, or network filesystems which receive strong write
delegations, should only ever return EIO to fsync.  We should
concentrate on them first, I think.  As there is only one possible
error, the seq counter is sufficient to "clear" it once it has been
reported to fsync() (or write()?).

Other network filesystems could return a whole host of errors: ENOSPC
EDQUOT ESTALE EPERM EFBIG ...
Do we want to limit exactly which errors are allowed in generic code, or
do we just support EIO generically and expect the filesystem to sort out
the details for anything else?

One possible approach a filesystem could take is just to allow a single
async writeback error.  After that error, all subsequent write()
system calls become synchronous. As write() or fsync() is called on each
file descriptor (which could possibly have sent the write which caused
the error), an error is returned and that fact is counted.  Once we have
returned as many errors as there are open file descriptors
(i_writecount?), and have seen a successful write, the filesystem
forgets all recorded errors and switches back to async writes (for that
inode).   NFS does this switch-to-sync-on-error.  See nfs_need_check_write().

The "which could possibly have sent the write which caused the error" is
an explicit reference to NFS.  NFS doesn't use the AS_EIO/AS_ENOSPC
flags to return async errors.  It allocates an nfs_open_context for each
user who opens a given inode, and stores an error in there.  Each dirty
pages is associated with one of these, so errors a sure to go to the
correct user, though not necessarily the correct fd at present.

When we specify the new behaviour we should be careful to be as vague as
possible while still saying what we need.  This allows filesystems some
flexibility.

  If an error happens during writeback, the next write() or fsync() (or
  ....) on the file descriptor to which data was written will return -1
  with errno set to EIO or some other relevant error.  Other file
  descriptors open on the same file may receive EIO or some other error
  on a subsequent appropriate system call.
  It should not be assumed that close() will return an error.  fsync()
  must be called before close() if writeback errors are important to the
  application.


Thanks,
NeilBrown

[toc] | [prev] | [next] | [standalone]

#1615932 — Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it

On Tue, 2017-04-04 at 13:03 +1000, NeilBrown wrote:
> On Mon, Apr 03 2017, Jeff Layton wrote:
> 
> > On Mon, 2017-04-03 at 12:16 -0700, Matthew Wilcox wrote:
> > > On Mon, Apr 03, 2017 at 01:47:37PM -0400, Jeff Layton wrote:
> > > > > I wonder whether it's even worth supporting both EIO and ENOSPC for a
> > > > > writeback problem.  If I understand correctly, at the time of write(),
> > > > > filesystems check to see if they have enough blocks to satisfy the
> > > > > request, so ENOSPC only comes up in the writeback context for thinly
> > > > > provisioned devices.
> > > > 
> > > > No, ENOSPC on writeback can certainly happen with network filesystems.
> > > > NFS and CIFS have no way to reserve space. You wouldn't want to have to
> > > > do an extra RPC on every buffered write. :)
> > > 
> > > Aaah, yes, network filesystems.  I would indeed not want to do an extra
> > > RPC on every write to a hole (it's a hole vs non-hole question, rather
> > > than a buffered/unbuffered question ... unless you're WAFLing and not
> > > reclaiming quickly enough, I suppose).
> > > 
> > > So, OK, that makes sense, we should keep allowing filesystems to report
> > > ENOSPC as a writeback error.  But I think much of the argument below
> > > still holds, and we should continue to have a prior EIO to be reported
> > > over a new ENOSPC (even if the program has already consumed the EIO).
> > > 
> > 
> > I'm fine with that (though I'd like Neil's thoughts before we decide
> > anything) there.
> 
> I'd like there be a well defined time when old errors were forgotten.
> It does make sense for EIO to persist even if ENOSPC or EDQUOT is
> received, but not forever.
> Clearing the remembered errors when put_write_access() causes
> i_writecount to reach zero is one option (as suggested), but I'm not
> sure I'm happy with it.
> 
> Local filesystems, or network filesystems which receive strong write
> delegations, should only ever return EIO to fsync.  We should
> concentrate on them first, I think.  As there is only one possible
> error, the seq counter is sufficient to "clear" it once it has been
> reported to fsync() (or write()?).
> 
> Other network filesystems could return a whole host of errors: ENOSPC
> EDQUOT ESTALE EPERM EFBIG ...
> Do we want to limit exactly which errors are allowed in generic code, or
> do we just support EIO generically and expect the filesystem to sort out
> the details for anything else?
> 
> One possible approach a filesystem could take is just to allow a single
> async writeback error.  After that error, all subsequent write()
> system calls become synchronous. As write() or fsync() is called on each
> file descriptor (which could possibly have sent the write which caused
> the error), an error is returned and that fact is counted.  Once we have
> returned as many errors as there are open file descriptors
> (i_writecount?), and have seen a successful write, the filesystem
> forgets all recorded errors and switches back to async writes (for that
> inode).   NFS does this switch-to-sync-on-error.  See nfs_need_check_write().
> 
> The "which could possibly have sent the write which caused the error" is
> an explicit reference to NFS.  NFS doesn't use the AS_EIO/AS_ENOSPC
> flags to return async errors.  It allocates an nfs_open_context for each
> user who opens a given inode, and stores an error in there.  Each dirty
> pages is associated with one of these, so errors a sure to go to the
> correct user, though not necessarily the correct fd at present.
> 
> When we specify the new behaviour we should be careful to be as vague as
> possible while still saying what we need.  This allows filesystems some
> flexibility.
> 
>   If an error happens during writeback, the next write() or fsync() (or
>   ....) on the file descriptor to which data was written will return -1
>   with errno set to EIO or some other relevant error.  Other file
>   descriptors open on the same file may receive EIO or some other error
>   on a subsequent appropriate system call.
>   It should not be assumed that close() will return an error.  fsync()
>   must be called before close() if writeback errors are important to the
>   application.
> 
> 

A lot in here... :)

While I like the NFS method of switching to sync I/O on error (and
indeed, I'm copying that in the Ceph ENOSPC patches I have), I'm not
sure it would really help anything here. The main reason NFS does that
is to prevent you from dirtying tons of pages that can't be cleaned. 

While that is a laudable goal, it's not really the problem I'm
interested in solving here. My goal is simply to ensure that you see a
writeback error on fsync if one occurred since the last fsync.

I think it just comes down to the fact that I'm not convinced that it
really matters much _what_ error gets reported, as long as you get one.
As you've mentioned in earlier discussions, most programs just treat it
as a fatal error anyway. As long as that error is representative of
some error that occurred during writeback, do we really care what it
was?

Suppose we have a bunch of dirty pages on an inode, get an EIO error
and then ENOSPC on a different write (maybe issued in parallel). We
send the ENOSPC error back to the application on an fsync (since it
came in last). Application then cleans out some junk from the fs and
then reissues the writes. They fail again and then he gets EIO from the
fsync and aborts.

Ok, so we might not have had to clean out the files and reissue the
writes there since we were going to give up anyway. Is it worth going
to extra lengths to avoid that there, given that we're in an error
condition anyway?

I'm just trying to understand why it matters at all what error you get
back when there multiple problems. They all seem equally valid to me in
that situation.

-- 
Jeff Layton <jlayton@redhat.com>

[toc] | [prev] | [next] | [standalone]

#1616434

[Multipart message — attachments visible in raw view] — view raw

On Tue, Apr 04 2017, Jeff Layton wrote:

> On Tue, 2017-04-04 at 13:03 +1000, NeilBrown wrote:
>> On Mon, Apr 03 2017, Jeff Layton wrote:
>> 
>> > On Mon, 2017-04-03 at 12:16 -0700, Matthew Wilcox wrote:
>> > > On Mon, Apr 03, 2017 at 01:47:37PM -0400, Jeff Layton wrote:
>> > > > > I wonder whether it's even worth supporting both EIO and ENOSPC for a
>> > > > > writeback problem.  If I understand correctly, at the time of write(),
>> > > > > filesystems check to see if they have enough blocks to satisfy the
>> > > > > request, so ENOSPC only comes up in the writeback context for thinly
>> > > > > provisioned devices.
>> > > > 
>> > > > No, ENOSPC on writeback can certainly happen with network filesystems.
>> > > > NFS and CIFS have no way to reserve space. You wouldn't want to have to
>> > > > do an extra RPC on every buffered write. :)
>> > > 
>> > > Aaah, yes, network filesystems.  I would indeed not want to do an extra
>> > > RPC on every write to a hole (it's a hole vs non-hole question, rather
>> > > than a buffered/unbuffered question ... unless you're WAFLing and not
>> > > reclaiming quickly enough, I suppose).
>> > > 
>> > > So, OK, that makes sense, we should keep allowing filesystems to report
>> > > ENOSPC as a writeback error.  But I think much of the argument below
>> > > still holds, and we should continue to have a prior EIO to be reported
>> > > over a new ENOSPC (even if the program has already consumed the EIO).
>> > > 
>> > 
>> > I'm fine with that (though I'd like Neil's thoughts before we decide
>> > anything) there.
>> 
>> I'd like there be a well defined time when old errors were forgotten.
>> It does make sense for EIO to persist even if ENOSPC or EDQUOT is
>> received, but not forever.
>> Clearing the remembered errors when put_write_access() causes
>> i_writecount to reach zero is one option (as suggested), but I'm not
>> sure I'm happy with it.
>> 
>> Local filesystems, or network filesystems which receive strong write
>> delegations, should only ever return EIO to fsync.  We should
>> concentrate on them first, I think.  As there is only one possible
>> error, the seq counter is sufficient to "clear" it once it has been
>> reported to fsync() (or write()?).
>> 
>> Other network filesystems could return a whole host of errors: ENOSPC
>> EDQUOT ESTALE EPERM EFBIG ...
>> Do we want to limit exactly which errors are allowed in generic code, or
>> do we just support EIO generically and expect the filesystem to sort out
>> the details for anything else?
>> 
>> One possible approach a filesystem could take is just to allow a single
>> async writeback error.  After that error, all subsequent write()
>> system calls become synchronous. As write() or fsync() is called on each
>> file descriptor (which could possibly have sent the write which caused
>> the error), an error is returned and that fact is counted.  Once we have
>> returned as many errors as there are open file descriptors
>> (i_writecount?), and have seen a successful write, the filesystem
>> forgets all recorded errors and switches back to async writes (for that
>> inode).   NFS does this switch-to-sync-on-error.  See nfs_need_check_write().
>> 
>> The "which could possibly have sent the write which caused the error" is
>> an explicit reference to NFS.  NFS doesn't use the AS_EIO/AS_ENOSPC
>> flags to return async errors.  It allocates an nfs_open_context for each
>> user who opens a given inode, and stores an error in there.  Each dirty
>> pages is associated with one of these, so errors a sure to go to the
>> correct user, though not necessarily the correct fd at present.
>> 
>> When we specify the new behaviour we should be careful to be as vague as
>> possible while still saying what we need.  This allows filesystems some
>> flexibility.
>> 
>>   If an error happens during writeback, the next write() or fsync() (or
>>   ....) on the file descriptor to which data was written will return -1
>>   with errno set to EIO or some other relevant error.  Other file
>>   descriptors open on the same file may receive EIO or some other error
>>   on a subsequent appropriate system call.
>>   It should not be assumed that close() will return an error.  fsync()
>>   must be called before close() if writeback errors are important to the
>>   application.
>> 
>> 
>
> A lot in here... :)
>
> While I like the NFS method of switching to sync I/O on error (and
> indeed, I'm copying that in the Ceph ENOSPC patches I have), I'm not
> sure it would really help anything here. The main reason NFS does that
> is to prevent you from dirtying tons of pages that can't be cleaned. 

It would help because it means there are no longer any async errors, so
there is no need to try to keep track of them.

If we decided that "last error wins", then that becomes irrelevant.  But
if we do care about any precedence of errors, then going sync is an easy
way to make sure the right error gets to the right place.

>
> While that is a laudable goal, it's not really the problem I'm
> interested in solving here. My goal is simply to ensure that you see a
> writeback error on fsync if one occurred since the last fsync.
>
> I think it just comes down to the fact that I'm not convinced that it
> really matters much _what_ error gets reported, as long as you get one.
> As you've mentioned in earlier discussions, most programs just treat it
> as a fatal error anyway. As long as that error is representative of
> some error that occurred during writeback, do we really care what it
> was?

I don't think I personally care at all, but there might be programs out
there...
I think it would be a good design goal to ensure that the behaviour seen
when there is only one open file descriptor on a file, remains
unchanged.
That means that file the fd is help open, multiple different error codes
can be returned in arbitrary order (unlikely, but possible).

Thanks,
NeilBrown


>
> Suppose we have a bunch of dirty pages on an inode, get an EIO error
> and then ENOSPC on a different write (maybe issued in parallel). We
> send the ENOSPC error back to the application on an fsync (since it
> came in last). Application then cleans out some junk from the fs and
> then reissues the writes. They fail again and then he gets EIO from the
> fsync and aborts.
>
> Ok, so we might not have had to clean out the files and reissue the
> writes there since we were going to give up anyway. Is it worth going
> to extra lengths to avoid that there, given that we're in an error
> condition anyway?
>
> I'm just trying to understand why it matters at all what error you get
> back when there multiple problems. They all seem equally valid to me in
> that situation.
>
> -- 
> Jeff Layton <jlayton@redhat.com>

[toc] | [prev] | [next] | [standalone]

#1615940 — Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it

On Tue, Apr 04, 2017 at 01:03:22PM +1000, NeilBrown wrote:
> On Mon, Apr 03 2017, Jeff Layton wrote:
> 
> > On Mon, 2017-04-03 at 12:16 -0700, Matthew Wilcox wrote:
> >> So, OK, that makes sense, we should keep allowing filesystems to report
> >> ENOSPC as a writeback error.  But I think much of the argument below
> >> still holds, and we should continue to have a prior EIO to be reported
> >> over a new ENOSPC (even if the program has already consumed the EIO).
> >
> > I'm fine with that (though I'd like Neil's thoughts before we decide
> > anything) there.
> 
> I'd like there be a well defined time when old errors were forgotten.
> It does make sense for EIO to persist even if ENOSPC or EDQUOT is
> received, but not forever.
> Clearing the remembered errors when put_write_access() causes
> i_writecount to reach zero is one option (as suggested), but I'm not
> sure I'm happy with it.
> 
> Local filesystems, or network filesystems which receive strong write
> delegations, should only ever return EIO to fsync.  We should
> concentrate on them first, I think.  As there is only one possible
> error, the seq counter is sufficient to "clear" it once it has been
> reported to fsync() (or write()?).
> 
> Other network filesystems could return a whole host of errors: ENOSPC
> EDQUOT ESTALE EPERM EFBIG ...
> Do we want to limit exactly which errors are allowed in generic code, or
> do we just support EIO generically and expect the filesystem to sort out
> the details for anything else?

I'd like us to focus on our POSIX compliance here and not return
arbitrary errors.  The relevant pages are here:

http://pubs.opengroup.org/onlinepubs/9699919799/functions/fsync.html
http://pubs.opengroup.org/onlinepubs/9699919799/functions/write.html
http://pubs.opengroup.org/onlinepubs/9699919799/functions/close.html

For close(), we have to map every error to EIO.
For fsync(), we can return any error that write() could have.  That limits
us to:

EFBIG ENOSPC EIO ENOBUFS ENXIO

I think EFBIG really isn't a writeback error; are there any network
filesystems that don't know the file size limit at the time they accept
the original write?  ENOBUFS seems like a transient error (*this* call to
fsync() failed, but the next one may succeed ... it's the equivalent of
ENOMEM).  ENXIO seems to me like it's a submission error, not a writeback
error.  So that leaves us with ENOSPC and EIO, as we have support today.

> One possible approach a filesystem could take is just to allow a single
> async writeback error.  After that error, all subsequent write()
> system calls become synchronous. As write() or fsync() is called on each
> file descriptor (which could possibly have sent the write which caused
> the error), an error is returned and that fact is counted.  Once we have
> returned as many errors as there are open file descriptors
> (i_writecount?), and have seen a successful write, the filesystem
> forgets all recorded errors and switches back to async writes (for that
> inode).   NFS does this switch-to-sync-on-error.  See nfs_need_check_write().
> 
> The "which could possibly have sent the write which caused the error" is
> an explicit reference to NFS.  NFS doesn't use the AS_EIO/AS_ENOSPC
> flags to return async errors.  It allocates an nfs_open_context for each
> user who opens a given inode, and stores an error in there.  Each dirty
> pages is associated with one of these, so errors a sure to go to the
> correct user, though not necessarily the correct fd at present.

... and you need the nfs_open_context in order to use the correct
credentials when writing a page to the server, correct?

> When we specify the new behaviour we should be careful to be as vague as
> possible while still saying what we need.  This allows filesystems some
> flexibility.
> 
>   If an error happens during writeback, the next write() or fsync() (or
>   ....) on the file descriptor to which data was written will return -1
>   with errno set to EIO or some other relevant error.  Other file
>   descriptors open on the same file may receive EIO or some other error
>   on a subsequent appropriate system call.
>   It should not be assumed that close() will return an error.  fsync()
>   must be called before close() if writeback errors are important to the
>   application.

Thanks for explaining what NFS does today.

[toc] | [prev] | [next] | [standalone]

#1615955 — Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it

On Tue, 2017-04-04 at 04:53 -0700, Matthew Wilcox wrote:
> On Tue, Apr 04, 2017 at 01:03:22PM +1000, NeilBrown wrote:
> > On Mon, Apr 03 2017, Jeff Layton wrote:
> > 
> > > On Mon, 2017-04-03 at 12:16 -0700, Matthew Wilcox wrote:
> > > > So, OK, that makes sense, we should keep allowing filesystems to report
> > > > ENOSPC as a writeback error.  But I think much of the argument below
> > > > still holds, and we should continue to have a prior EIO to be reported
> > > > over a new ENOSPC (even if the program has already consumed the EIO).
> > > 
> > > I'm fine with that (though I'd like Neil's thoughts before we decide
> > > anything) there.
> > 
> > I'd like there be a well defined time when old errors were forgotten.
> > It does make sense for EIO to persist even if ENOSPC or EDQUOT is
> > received, but not forever.
> > Clearing the remembered errors when put_write_access() causes
> > i_writecount to reach zero is one option (as suggested), but I'm not
> > sure I'm happy with it.
> > 
> > Local filesystems, or network filesystems which receive strong write
> > delegations, should only ever return EIO to fsync.  We should
> > concentrate on them first, I think.  As there is only one possible
> > error, the seq counter is sufficient to "clear" it once it has been
> > reported to fsync() (or write()?).
> > 
> > Other network filesystems could return a whole host of errors: ENOSPC
> > EDQUOT ESTALE EPERM EFBIG ...
> > Do we want to limit exactly which errors are allowed in generic code, or
> > do we just support EIO generically and expect the filesystem to sort out
> > the details for anything else?
> 
> I'd like us to focus on our POSIX compliance here and not return
> arbitrary errors.  The relevant pages are here:
> 
> http://pubs.opengroup.org/onlinepubs/9699919799/functions/fsync.html
> http://pubs.opengroup.org/onlinepubs/9699919799/functions/write.html
> http://pubs.opengroup.org/onlinepubs/9699919799/functions/close.html
> 
> For close(), we have to map every error to EIO.
> For fsync(), we can return any error that write() could have.  That limits
> us to:
> 
> EFBIG ENOSPC EIO ENOBUFS ENXIO
> 
> I think EFBIG really isn't a writeback error; are there any network
> filesystems that don't know the file size limit at the time they accept
> the original write?  ENOBUFS seems like a transient error (*this* call to
> fsync() failed, but the next one may succeed ... it's the equivalent of
> ENOMEM).  ENXIO seems to me like it's a submission error, not a writeback
> error.  So that leaves us with ENOSPC and EIO, as we have support today.
> 

Agreed that we should focus on POSIX compliance. I'll also note that
POSIX states:

"If more than one error occurs in processing a function call, any one
of the possible errors may be returned, as the order of
detection is undefined."

    http://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html#tag_15_03

So, I'd like to push back on this idea that we need to prefer reporting
-EIO over other errors. POSIX certainly doesn't mandate that. 

If we agree that that is the case, then I think the simplest thing to
do here would be to clear the other error flag(s) when we get a new
error, such that we only preserve the latest one. With that, we also
wouldn't need to clear anything out when i_writecount goes to zero
either. It would "just work" without that.

> > One possible approach a filesystem could take is just to allow a single
> > async writeback error.  After that error, all subsequent write()
> > system calls become synchronous. As write() or fsync() is called on each
> > file descriptor (which could possibly have sent the write which caused
> > the error), an error is returned and that fact is counted.  Once we have
> > returned as many errors as there are open file descriptors
> > (i_writecount?), and have seen a successful write, the filesystem
> > forgets all recorded errors and switches back to async writes (for that
> > inode).   NFS does this switch-to-sync-on-error.  See nfs_need_check_write().
> > 
> > The "which could possibly have sent the write which caused the error" is
> > an explicit reference to NFS.  NFS doesn't use the AS_EIO/AS_ENOSPC
> > flags to return async errors.  It allocates an nfs_open_context for each
> > user who opens a given inode, and stores an error in there.  Each dirty
> > pages is associated with one of these, so errors a sure to go to the
> > correct user, though not necessarily the correct fd at present.
> 
> ... and you need the nfs_open_context in order to use the correct
> credentials when writing a page to the server, correct?
> 

Yes, and it is expensive. I don't think we want to do that at the
generic VFS layer if we can at all help it.

> > When we specify the new behaviour we should be careful to be as vague as
> > possible while still saying what we need.  This allows filesystems some
> > flexibility.
> > 
> >   If an error happens during writeback, the next write() or fsync() (or
> >   ....) on the file descriptor to which data was written will return -1
> >   with errno set to EIO or some other relevant error.  Other file
> >   descriptors open on the same file may receive EIO or some other error
> >   on a subsequent appropriate system call.
> >   It should not be assumed that close() will return an error.  fsync()
> >   must be called before close() if writeback errors are important to the
> >   application.
> 

...and I also agree that we leave as much grey area as possible here to
allow for a wide range of implementations.
-- 
Jeff Layton <jlayton@redhat.com>

[toc] | [prev] | [next] | [standalone]

#1616188 — Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it

On Tue, Apr 04, 2017 at 08:17:48AM -0400, Jeff Layton wrote:
> Agreed that we should focus on POSIX compliance. I'll also note that
> POSIX states:
> 
> "If more than one error occurs in processing a function call, any one
> of the possible errors may be returned, as the order of
> detection is undefined."
> 
>     http://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html#tag_15_03
> 
> So, I'd like to push back on this idea that we need to prefer reporting
> -EIO over other errors. POSIX certainly doesn't mandate that. 

I honestly wonder if we need to support ENOSPC from writeback at all.
Looking at our history, the AS_EIO / AS_ENOSPC came from this patch
in 2003:

https://git.kernel.org/pub/scm/linux/kernel/git/tglx/history.git/commit/?id=fcad2b42fc2e15a94ba1a1ba8535681a735bfd16

That seems to come from here:
http://lkml.iu.edu/hypermail/linux/kernel/0308.0/0205.html
which is marked as a resend, but I can't find the original.

It's a little misleading because the immediately preceding patch
introduced mapping->error, so there's no precedent here to speak of.
It looks like we used to just silently lose writeback errors (*cough*).

I'd like to suggest that maybe we don't need to support multiple errors
at all.  That all errors, including ENOSPC, get collapsed into EIO.
POSIX already tells us to do that for close() and permits us to do that
for fsync().

[toc] | [prev] | [next] | [standalone]

#1616195 — Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it

On Tue, 2017-04-04 at 09:12 -0700, Matthew Wilcox wrote:
> On Tue, Apr 04, 2017 at 08:17:48AM -0400, Jeff Layton wrote:
> > Agreed that we should focus on POSIX compliance. I'll also note that
> > POSIX states:
> > 
> > "If more than one error occurs in processing a function call, any one
> > of the possible errors may be returned, as the order of
> > detection is undefined."
> > 
> >     http://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html#tag_15_03
> > 
> > So, I'd like to push back on this idea that we need to prefer reporting
> > -EIO over other errors. POSIX certainly doesn't mandate that. 
> 
> I honestly wonder if we need to support ENOSPC from writeback at all.
> Looking at our history, the AS_EIO / AS_ENOSPC came from this patch
> in 2003:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/tglx/history.git/commit/?id=fcad2b42fc2e15a94ba1a1ba8535681a735bfd16
> 
> That seems to come from here:
> http://lkml.iu.edu/hypermail/linux/kernel/0308.0/0205.html
> which is marked as a resend, but I can't find the original.
> 
> It's a little misleading because the immediately preceding patch
> introduced mapping->error, so there's no precedent here to speak of.
> It looks like we used to just silently lose writeback errors (*cough*).
> 
> I'd like to suggest that maybe we don't need to support multiple errors
> at all.  That all errors, including ENOSPC, get collapsed into EIO.
> POSIX already tells us to do that for close() and permits us to do that
> for fsync().
> 

That is certainly allowed under POSIX as I interpret the spec. At a
minimum we just need a single flag and can collapse all errors under
that.

That said, I think giving more specific errors where we can is useful.
When your program is erroring out and writing 'I/O error' to the logs,
then how much time will your admins burn before they figure out that it
really failed because the filesystem was full?
-- 
Jeff Layton <jlayton@redhat.com>

[toc] | [prev] | [next] | [standalone]

#1616217 — Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it

On Tue, Apr 04, 2017 at 12:25:46PM -0400, Jeff Layton wrote:
> That said, I think giving more specific errors where we can is useful.
> When your program is erroring out and writing 'I/O error' to the logs,
> then how much time will your admins burn before they figure out that it
> really failed because the filesystem was full?

df is one of the first things I check ... a few years ago, I also learned
to check df -i ... ;-)

Anyway, given the decision to simply report the last error lets us do this
implementation:

void filemap_set_wb_error(struct address_space *mapping, int err)
{
	struct inode *inode = mapping->host;
	unsigned int wb_err;

	if (!err)
		return;
	/*
	 * This should be called with the error code that we want to return
	 * on fsync. Thus, it should always be <= 0.
	 */
	WARN_ON(err > 0 || err < -MAX_ERRNO);

	spin_lock(&inode->i_lock);
	wb_err = ((mapping->wb_err & ~MAX_ERRNO) + (1 << 12)) | -err;
	WRITE_ONCE(mapping->wb_err, wb_err);
	spin_unlock(&inode->i_lock);
}

int filemap_report_wb_error(struct file *file)
{
	struct inode *inode = file_inode(file);
	unsigned int wb_err = READ_ONCE(mapping->wb_err);

	if (file->f_wb_err == wb_err)
		return 0;
	return -(wb_err & 4095);
}

That only gives us 20 bits of counter, but I think that's enough.

[toc] | [prev] | [next] | [standalone]

#1616293 — Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it

On Tue, 2017-04-04 at 10:09 -0700, Matthew Wilcox wrote:
> On Tue, Apr 04, 2017 at 12:25:46PM -0400, Jeff Layton wrote:
> > That said, I think giving more specific errors where we can is useful.
> > When your program is erroring out and writing 'I/O error' to the logs,
> > then how much time will your admins burn before they figure out that it
> > really failed because the filesystem was full?
> 
> df is one of the first things I check ... a few years ago, I also learned
> to check df -i ... ;-)
> 
> Anyway, given the decision to simply report the last error lets us do this
> implementation:
> 
> void filemap_set_wb_error(struct address_space *mapping, int err)
> {
> 	struct inode *inode = mapping->host;
> 	unsigned int wb_err;
> 
> 	if (!err)
> 		return;
> 	/*
> 	 * This should be called with the error code that we want to return
> 	 * on fsync. Thus, it should always be <= 0.
> 	 */
> 	WARN_ON(err > 0 || err < -MAX_ERRNO);
> 
> 	spin_lock(&inode->i_lock);
> 	wb_err = ((mapping->wb_err & ~MAX_ERRNO) + (1 << 12)) | -err;
> 	WRITE_ONCE(mapping->wb_err, wb_err);

Do we need the WRITE_ONCE, given that you're under a spinlock there?

> 	spin_unlock(&inode->i_lock);
> }
> 
> int filemap_report_wb_error(struct file *file)
> {
> 	struct inode *inode = file_inode(file);
> 	unsigned int wb_err = READ_ONCE(mapping->wb_err);
> 
> 	if (file->f_wb_err == wb_err)
> 		return 0;
> 	return -(wb_err & 4095);
> }
> 
> That only gives us 20 bits of counter, but I think that's enough.

That'd be fine with me, but I'm all for allowing filesystems to return
arbitrary writeback errors on fsync.

Others may have different opinions there. We could add a wrapper
function that sanitizes the error codes if some filesystems wanted that
though.

-- 
Jeff Layton <jlayton@redhat.com>

[toc] | [prev] | [next] | [standalone]

#1616436

[Multipart message — attachments visible in raw view] — view raw

On Tue, Apr 04 2017, Matthew Wilcox wrote:

> On Tue, Apr 04, 2017 at 12:25:46PM -0400, Jeff Layton wrote:
>> That said, I think giving more specific errors where we can is useful.
>> When your program is erroring out and writing 'I/O error' to the logs,
>> then how much time will your admins burn before they figure out that it
>> really failed because the filesystem was full?
>
> df is one of the first things I check ... a few years ago, I also learned
> to check df -i ... ;-)
>
> Anyway, given the decision to simply report the last error lets us do this
> implementation:
>
> void filemap_set_wb_error(struct address_space *mapping, int err)
> {
> 	struct inode *inode = mapping->host;
> 	unsigned int wb_err;
>
> 	if (!err)
> 		return;
> 	/*
> 	 * This should be called with the error code that we want to return
> 	 * on fsync. Thus, it should always be <= 0.
> 	 */
> 	WARN_ON(err > 0 || err < -MAX_ERRNO);
>
> 	spin_lock(&inode->i_lock);
> 	wb_err = ((mapping->wb_err & ~MAX_ERRNO) + (1 << 12)) | -err;

Seriously?  You are missing "MAX_ERRNO" (4095) together with "1 << 12"
(4096) in the one expression without a big comment saying why?

Surely:
>       BUILD_BUG_ON_NOT_POWER_OF_2(MAX_ERRNO+1);
> 	wb_err = ((mapping->wb_err & ~MAX_ERRNO) + (MAX_ERRNO+1)) | -err;

NeilBrown

[toc] | [prev] | [next] | [standalone]

#1617319 — Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it

On Tue, 2017-04-04 at 10:09 -0700, Matthew Wilcox wrote:
> On Tue, Apr 04, 2017 at 12:25:46PM -0400, Jeff Layton wrote:
> > That said, I think giving more specific errors where we can is useful.
> > When your program is erroring out and writing 'I/O error' to the logs,
> > then how much time will your admins burn before they figure out that it
> > really failed because the filesystem was full?
> 
> df is one of the first things I check ... a few years ago, I also learned
> to check df -i ... ;-)
> 
> Anyway, given the decision to simply report the last error lets us do this
> implementation:
> 
> void filemap_set_wb_error(struct address_space *mapping, int err)
> {
> 	struct inode *inode = mapping->host;
> 	unsigned int wb_err;
> 
> 	if (!err)
> 		return;
> 	/*
> 	 * This should be called with the error code that we want to return
> 	 * on fsync. Thus, it should always be <= 0.
> 	 */
> 	WARN_ON(err > 0 || err < -MAX_ERRNO);
> 
> 	spin_lock(&inode->i_lock);
> 	wb_err = ((mapping->wb_err & ~MAX_ERRNO) + (1 << 12)) | -err;
> 	WRITE_ONCE(mapping->wb_err, wb_err);
> 	spin_unlock(&inode->i_lock);
> }
> 

I like this idea of being able to store arbitrary error codes there.
That should be used judiciously of course, but we already allow
returning arbitrary errors via the ->fsync op anyway.

I'll plan to incorporate something like that into the next set (with
judicious comments and constants).

One question...is the i_lock the right way to protect this? I think we
could do this locklessly too (cmpxchg in a loop, for instance). I'm not
worried about performance here -- it's just nice to be able to call
simple stuff like this without worrying about locking.

> int filemap_report_wb_error(struct file *file)
> {
> 	struct inode *inode = file_inode(file);
> 	unsigned int wb_err = READ_ONCE(mapping->wb_err);
> 
> 	if (file->f_wb_err == wb_err)
> 		return 0;
> 	return -(wb_err & 4095);
> }
> 
> That only gives us 20 bits of counter, but I think that's enough.

2^20 is 1048576, which seems a little small to me.

We may end up bumping the counter on every failed I/O. How fast can we
generate 1M failed I/Os? :)

2^52 however is 4503599627370496 (4Tios or so) ... that might take a
little longer to overflow. Is it worth the cost here to ensure that
this won't occur?

Actually...we could put this field in the inode instead of the mapping.
I know we've traditionally tracked this in the mapping, but is that
required here?

If we put this field in the inode then perhaps we can union it with
something and mitigate the cost of a larger counter...maybe in the
i_pipe union? I don't think S_ISREG inodes use anything in there, do
they?
-- 
Jeff Layton <jlayton@redhat.com>

[toc] | [prev] | [next] | [standalone]

#1617379 — Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it

On Wed, Apr 05, 2017 at 03:49:52PM -0400, Jeff Layton wrote:
> > That only gives us 20 bits of counter, but I think that's enough.
> 
> 2^20 is 1048576, which seems a little small to me.
> 
> We may end up bumping the counter on every failed I/O. How fast can we
> generate 1M failed I/Os? :)

So there's a one-in-a-million chance of missing a failed I/O ... if
we're generating lots of errors, the next time the app calls fsync(),
it'll notice the other million times we've hit the problem :-)

> Actually...we could put this field in the inode instead of the mapping.
> I know we've traditionally tracked this in the mapping, but is that
> required here?
> 
> If we put this field in the inode then perhaps we can union it with
> something and mitigate the cost of a larger counter...maybe in the
> i_pipe union? I don't think S_ISREG inodes use anything in there, do
> they?

But writeback isn't just done on ISREG inodes, but also on S_ISBLK inodes,
which use i_bdev (right?)

Another possibility is to move this out of the address_space and into
either the super_block or the backing_device_info.  Errors don't tend
to be constrained to a single file but affect the entire filesystem,
or even multiple filesystems if you have a partitioned block device ...

[toc] | [prev] | [next] | [standalone]

#1617441

[Multipart message — attachments visible in raw view] — view raw

On Wed, Apr 05 2017, Matthew Wilcox wrote:

> On Wed, Apr 05, 2017 at 03:49:52PM -0400, Jeff Layton wrote:
>> > That only gives us 20 bits of counter, but I think that's enough.
>> 
>> 2^20 is 1048576, which seems a little small to me.
>> 
>> We may end up bumping the counter on every failed I/O. How fast can we
>> generate 1M failed I/Os? :)
>
> So there's a one-in-a-million chance of missing a failed I/O ... if
> we're generating lots of errors, the next time the app calls fsync(),
> it'll notice the other million times we've hit the problem :-)
>
>> Actually...we could put this field in the inode instead of the mapping.
>> I know we've traditionally tracked this in the mapping, but is that
>> required here?
>> 
>> If we put this field in the inode then perhaps we can union it with
>> something and mitigate the cost of a larger counter...maybe in the
>> i_pipe union? I don't think S_ISREG inodes use anything in there, do
>> they?
>
> But writeback isn't just done on ISREG inodes, but also on S_ISBLK inodes,
> which use i_bdev (right?)
>
> Another possibility is to move this out of the address_space and into
> either the super_block or the backing_device_info.  Errors don't tend
> to be constrained to a single file but affect the entire filesystem,
> or even multiple filesystems if you have a partitioned block device ...

EDQUOT.  Remember EDQUOT.  It certainly don't affect the whole
filesystem.
Even without that, filesystems can easily treat different files
differently.  We shouldn't assume one-failes-all-fail.

NeilBrown

[toc] | [prev] | [next] | [standalone]

#1617434

[Multipart message — attachments visible in raw view] — view raw

On Thu, Apr 06 2017, Jeff Layton wrote:

> On Tue, 2017-04-04 at 10:09 -0700, Matthew Wilcox wrote:
>> On Tue, Apr 04, 2017 at 12:25:46PM -0400, Jeff Layton wrote:
>> > That said, I think giving more specific errors where we can is useful.
>> > When your program is erroring out and writing 'I/O error' to the logs,
>> > then how much time will your admins burn before they figure out that it
>> > really failed because the filesystem was full?
>> 
>> df is one of the first things I check ... a few years ago, I also learned
>> to check df -i ... ;-)
>> 
>> Anyway, given the decision to simply report the last error lets us do this
>> implementation:
>> 
>> void filemap_set_wb_error(struct address_space *mapping, int err)
>> {
>> 	struct inode *inode = mapping->host;
>> 	unsigned int wb_err;
>> 
>> 	if (!err)
>> 		return;
>> 	/*
>> 	 * This should be called with the error code that we want to return
>> 	 * on fsync. Thus, it should always be <= 0.
>> 	 */
>> 	WARN_ON(err > 0 || err < -MAX_ERRNO);
>> 
>> 	spin_lock(&inode->i_lock);
>> 	wb_err = ((mapping->wb_err & ~MAX_ERRNO) + (1 << 12)) | -err;
>> 	WRITE_ONCE(mapping->wb_err, wb_err);
>> 	spin_unlock(&inode->i_lock);
>> }
>> 
>
> I like this idea of being able to store arbitrary error codes there.
> That should be used judiciously of course, but we already allow
> returning arbitrary errors via the ->fsync op anyway.
>
> I'll plan to incorporate something like that into the next set (with
> judicious comments and constants).
>
> One question...is the i_lock the right way to protect this? I think we
> could do this locklessly too (cmpxchg in a loop, for instance). I'm not
> worried about performance here -- it's just nice to be able to call
> simple stuff like this without worrying about locking.

I like the idea of using cmpxchg.


>
>> int filemap_report_wb_error(struct file *file)
>> {
>> 	struct inode *inode = file_inode(file);
>> 	unsigned int wb_err = READ_ONCE(mapping->wb_err);
>> 
>> 	if (file->f_wb_err == wb_err)
>> 		return 0;
>> 	return -(wb_err & 4095);
>> }
>> 
>> That only gives us 20 bits of counter, but I think that's enough.
>
> 2^20 is 1048576, which seems a little small to me.
>
> We may end up bumping the counter on every failed I/O. How fast can we
> generate 1M failed I/Os? :)

Do we need to count all of those if no-one sees them?
i.e. use one bit to say "this error hasn't been seen".
If an error occurs with has the name error code as is currently stored,
and the bit is set, don't make a change.  Otherwise make the change,
inc the counter, set the bit.
When checking for an error, if the bit is set, clear it first.
Then you can count 500,000 errors-returned-to-some-thread, which is
probably enough.

>
> 2^52 however is 4503599627370496 (4Tios or so) ... that might take a
> little longer to overflow. Is it worth the cost here to ensure that
> this won't occur?
>
> Actually...we could put this field in the inode instead of the mapping.
> I know we've traditionally tracked this in the mapping, but is that
> required here?

What if the address_space is shared by two inodes?  That is the whole
point of the i_mapping pointer.  This would make it harder for the
"other" inode to get the error.
(Does anyone actually use fs/coda ??
 Actually, block devices use i_mapping too.
 If two block device inodes have the same major/minor number, they
 end up having i_mapping point to the same place)

If you are concerned about space in 'struct address_space', just prune
some wastage.
The "host" field brings no value.  It is only ever assigned in
inode_init_always():

	struct address_space *const mapping = &inode->i_data;
......
	mapping->host = inode;

So you could change all references to use
   container_of(mapping, struct inode, i_data)

NeilBrown



>
> If we put this field in the inode then perhaps we can union it with
> something and mitigate the cost of a larger counter...maybe in the
> i_pipe union? I don't think S_ISREG inodes use anything in there, do
> they?
> -- 
> Jeff Layton <jlayton@redhat.com>

[toc] | [prev] | [next] | [standalone]

#1617474 — Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it

On Thu, Apr 06, 2017 at 10:02:48AM +1000, NeilBrown wrote:
> If you are concerned about space in 'struct address_space', just prune
> some wastage.

I'm trying to (via wlists).  still buggy though.

> The "host" field brings no value.  It is only ever assigned in
> inode_init_always():
> 
> 	struct address_space *const mapping = &inode->i_data;
> ......
> 	mapping->host = inode;
> 
> So you could change all references to use
>    container_of(mapping, struct inode, i_data)

Alas, no:

drivers/dax/dax.c:      inode->i_mapping->host = dax_dev->inode;
fs/gfs2/glock.c:                mapping->host = s->s_bdev->bd_inode;
fs/gfs2/ops_fstype.c:   mapping->host = sb->s_bdev->bd_inode;
fs/nilfs2/page.c:       mapping->host = inode;

[toc] | [prev] | [next] | [standalone]

#1617505

[Multipart message — attachments visible in raw view] — view raw

On Wed, Apr 05 2017, Matthew Wilcox wrote:

> On Thu, Apr 06, 2017 at 10:02:48AM +1000, NeilBrown wrote:
>> If you are concerned about space in 'struct address_space', just prune
>> some wastage.
>
> I'm trying to (via wlists).  still buggy though.

Cool.
(I wonder what a wlist is.... weighted list?)

>
>> The "host" field brings no value.  It is only ever assigned in
>> inode_init_always():
>> 
>> 	struct address_space *const mapping = &inode->i_data;
>> ......
>> 	mapping->host = inode;
>> 
>> So you could change all references to use
>>    container_of(mapping, struct inode, i_data)
>
> Alas, no:
>
> drivers/dax/dax.c:      inode->i_mapping->host = dax_dev->inode;

	inode->i_mapping = dax_dev->inode->i_mapping;
	inode->i_mapping->host = dax_dev->inode;
so that second line is equivalent to
        dax_dev->inode->i_mapping->host = dax_dev->inode;
so inode->mapping->host leads back to inode.  So this doesn't break the
invariant.
       

> fs/gfs2/glock.c:                mapping->host = s->s_bdev->bd_inode;
> fs/gfs2/ops_fstype.c:   mapping->host = sb->s_bdev->bd_inode;

Hmm.. that's weird.  I cannot quite follow what is happening there.
It creates an address-space for metadata which doesn't have a real
inode, and borrows bits of the bdev inode ... possibly just to be able
to find the blocksize deep in buffer.c or similar.
I suspect that using an 'inode' instead of a 'mapping' would make the
code clearer.

> fs/nilfs2/page.c:       mapping->host = inode;

A nilfs inode is allocated with 2 address spaces,
one for the data and one for btree indexing metadata.
And then there are a couple of extra address spaces for the
global metadata-file (mtd).

I wonder what the ->host pointer is actually used for.
buffer.c uses it:
 - to mark the inode 'dirty' when the page is marked dirty
 - to find the blocksize of the inode, for creating buffer_heads
 - find the size of the mapping (i_size)

I could probably argue that the 'dirty' flag (at least for the data) and
the size really belong in the address_space, not in the inode.
The blocksize, I'm less sure of.

I suspect gfs2 and nilfs2 could be changed to allocate a separate inode
(instead of address_space), or to not make use of the ->host pointer.
It would be more work than I at first thought though.

Thanks,
NeilBrown

[toc] | [prev] | [next] | [standalone]

#1618012 — Re: [RFC PATCH 0/4] fs: introduce new writeback error tracking infrastructure and convert ext4 to use it

On Thu, Apr 06, 2017 at 03:12:42PM +1000, NeilBrown wrote:
> On Wed, Apr 05 2017, Matthew Wilcox wrote:
> 
> > On Thu, Apr 06, 2017 at 10:02:48AM +1000, NeilBrown wrote:
> >> If you are concerned about space in 'struct address_space', just prune
> >> some wastage.
> >
> > I'm trying to (via wlists).  still buggy though.
> 
> Cool.
> (I wonder what a wlist is.... weighted list?)

A homonym ;-)  Waitlists.  Yet Another Variation of a doubly linked list.
This variation has only a single head pointer (like the hlist), allows
O(1) deletion from any point in the list (but requires that you know the
head of the list).  The tail of the list is pointed to by head->prev.

http://git.infradead.org/users/willy/linux-dax.git/shortlog/refs/heads/wlist

It also needs updating to the latest tree; WW locks got revamped.

[toc] | [prev] | [next] | [standalone]

#1618366

[Multipart message — attachments visible in raw view] — view raw

On Thu, Apr 06 2017, Matthew Wilcox wrote:

> On Thu, Apr 06, 2017 at 03:12:42PM +1000, NeilBrown wrote:
>> On Wed, Apr 05 2017, Matthew Wilcox wrote:
>> 
>> > On Thu, Apr 06, 2017 at 10:02:48AM +1000, NeilBrown wrote:
>> >> If you are concerned about space in 'struct address_space', just prune
>> >> some wastage.
>> >
>> > I'm trying to (via wlists).  still buggy though.
>> 
>> Cool.
>> (I wonder what a wlist is.... weighted list?)
>
> A homonym ;-)  Waitlists.  Yet Another Variation of a doubly linked list.
> This variation has only a single head pointer (like the hlist), allows
> O(1) deletion from any point in the list (but requires that you know the
> head of the list).  The tail of the list is pointed to by head->prev.
>
> http://git.infradead.org/users/willy/linux-dax.git/shortlog/refs/heads/wlist
>
> It also needs updating to the latest tree; WW locks got revamped.

So nothing points to the head, meaning you need both the node and the
head to delete something.  But the head is smaller.
I can see how that would be useful.
Thanks,
NeilBrown

[toc] | [prev] | [next] | [standalone]

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

Back to top | Article view | linux.kernel

csiph-web