Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1356098 > unrolled thread

Re: Freezable workqueue blocks non-freezable workqueue during the system resume process

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Freezable workqueue blocks non-freezable workqueue during the  system resume process Tejun Heo <tj@kernel.org> - 2016-03-11 19:00 +0100
    Re: Freezable workqueue blocks non-freezable workqueue during the  system resume process Jan Kara <jack@suse.cz> - 2016-03-14 08:30 +0100
      Re: Freezable workqueue blocks non-freezable workqueue during the  system resume process Alan Stern <stern@rowland.harvard.edu> - 2016-03-14 15:40 +0100
        Re: Freezable workqueue blocks non-freezable workqueue during the  system resume process Jan Kara <jack@suse.cz> - 2016-03-15 10:30 +0100
          Re: Freezable workqueue blocks non-freezable workqueue during the  system resume process Tejun Heo <tj@kernel.org> - 2016-03-16 16:10 +0100

#1356098 — Re: Freezable workqueue blocks non-freezable workqueue during the system resume process

Hello, Jan.

On Thu, Mar 03, 2016 at 10:33:10AM +0100, Jan Kara wrote:
> > Ugh... that's nasty.  I wonder whether the right thing to do is making
> > writeback workers non-freezable.  IOs are supposed to be blocked from
> > lower layer anyway.  Jan, what do you think?
> 
> Well no, at least currently IO is not blocked in lower layers AFAIK - for
> that you'd need to freeze block devices & filesystems and there are issues

At least libata does and I think SCSI does too, but yeah, there
probably are drivers which depend on block layer blocking IOs, which
btw is a pretty fragile way to go about as upper layers might not be
the only source of activities.

> with that (Jiri Kosina was the last one which was trying to make this work
> AFAIR). And I think you need to stop writeback (and generally any IO) to be
> generated so that it doesn't interact in a strange way with device drivers
> being frozen. So IMO until suspend freezes filesystems & devices properly
> you have to freeze writeback workqueue.

I still think the right thing to do is plugging that block layer or
low level drivers.  It's like we're trying to plug multiple sources
when we can plug the point where they come together anyway.

Thanks.

-- 
tejun

[toc] | [next] | [standalone]

#1356993

On Fri 11-03-16 12:56:10, Tejun Heo wrote:
> Hello, Jan.
> 
> On Thu, Mar 03, 2016 at 10:33:10AM +0100, Jan Kara wrote:
> > > Ugh... that's nasty.  I wonder whether the right thing to do is making
> > > writeback workers non-freezable.  IOs are supposed to be blocked from
> > > lower layer anyway.  Jan, what do you think?
> > 
> > Well no, at least currently IO is not blocked in lower layers AFAIK - for
> > that you'd need to freeze block devices & filesystems and there are issues
> 
> At least libata does and I think SCSI does too, but yeah, there
> probably are drivers which depend on block layer blocking IOs, which
> btw is a pretty fragile way to go about as upper layers might not be
> the only source of activities.
> 
> > with that (Jiri Kosina was the last one which was trying to make this work
> > AFAIR). And I think you need to stop writeback (and generally any IO) to be
> > generated so that it doesn't interact in a strange way with device drivers
> > being frozen. So IMO until suspend freezes filesystems & devices properly
> > you have to freeze writeback workqueue.
> 
> I still think the right thing to do is plugging that block layer or
> low level drivers.  It's like we're trying to plug multiple sources
> when we can plug the point where they come together anyway.

I agree that freezing writeback workers is a workaround for real issues at
best and ideally we shouldn't have to do that. But at least for now I had
the impression that it is needed for suspend to work reasonably reliably.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

[toc] | [prev] | [next] | [standalone]

#1357293

On Mon, 14 Mar 2016, Jan Kara wrote:

> On Fri 11-03-16 12:56:10, Tejun Heo wrote:
> > Hello, Jan.
> > 
> > On Thu, Mar 03, 2016 at 10:33:10AM +0100, Jan Kara wrote:
> > > > Ugh... that's nasty.  I wonder whether the right thing to do is making
> > > > writeback workers non-freezable.  IOs are supposed to be blocked from
> > > > lower layer anyway.  Jan, what do you think?
> > > 
> > > Well no, at least currently IO is not blocked in lower layers AFAIK - for
> > > that you'd need to freeze block devices & filesystems and there are issues
> > 
> > At least libata does and I think SCSI does too, but yeah, there
> > probably are drivers which depend on block layer blocking IOs, which
> > btw is a pretty fragile way to go about as upper layers might not be
> > the only source of activities.
> > 
> > > with that (Jiri Kosina was the last one which was trying to make this work
> > > AFAIR). And I think you need to stop writeback (and generally any IO) to be
> > > generated so that it doesn't interact in a strange way with device drivers
> > > being frozen. So IMO until suspend freezes filesystems & devices properly
> > > you have to freeze writeback workqueue.

What do you mean by "freezes ... devices"?  Only a piece of code can be 
frozen -- not a device.

The kernel does suspend device drivers; that is, it invokes their
suspend callbacks.  But it doesn't "freeze" them in any sense.  Once a 
driver has been suspended, it assumes it won't receive any I/O requests 
until it has been resumed.  Therefore the kernel first has to prevent 
all the upper layers from generating such requests and/or sending them 
to the low-level drivers.

> > I still think the right thing to do is plugging that block layer or
> > low level drivers.  It's like we're trying to plug multiple sources
> > when we can plug the point where they come together anyway.
> 
> I agree that freezing writeback workers is a workaround for real issues at
> best and ideally we shouldn't have to do that. But at least for now I had
> the impression that it is needed for suspend to work reasonably reliably.

The design is not to plug low-level drivers, but instead to prevent
them from receiving any requests by plugging or freezing high-level
code.

It's pretty clear that we don't want to have ongoing I/O during a 
system suspend, right?  And that means the I/O has to be prevented (or 
"plugged", if you prefer) somewhere -- either at an upper layer or at a 
lower layer.  There was a choice to be made, and the decision was to do 
it at an upper layer.

Alan Stern

[toc] | [prev] | [next] | [standalone]

#1357941

On Mon 14-03-16 10:37:22, Alan Stern wrote:
> On Mon, 14 Mar 2016, Jan Kara wrote:
> 
> > On Fri 11-03-16 12:56:10, Tejun Heo wrote:
> > > Hello, Jan.
> > > 
> > > On Thu, Mar 03, 2016 at 10:33:10AM +0100, Jan Kara wrote:
> > > > > Ugh... that's nasty.  I wonder whether the right thing to do is making
> > > > > writeback workers non-freezable.  IOs are supposed to be blocked from
> > > > > lower layer anyway.  Jan, what do you think?
> > > > 
> > > > Well no, at least currently IO is not blocked in lower layers AFAIK - for
> > > > that you'd need to freeze block devices & filesystems and there are issues
> > > 
> > > At least libata does and I think SCSI does too, but yeah, there
> > > probably are drivers which depend on block layer blocking IOs, which
> > > btw is a pretty fragile way to go about as upper layers might not be
> > > the only source of activities.
> > > 
> > > > with that (Jiri Kosina was the last one which was trying to make this work
> > > > AFAIR). And I think you need to stop writeback (and generally any IO) to be
> > > > generated so that it doesn't interact in a strange way with device drivers
> > > > being frozen. So IMO until suspend freezes filesystems & devices properly
> > > > you have to freeze writeback workqueue.
> 
> What do you mean by "freezes ... devices"?  Only a piece of code can be 
> frozen -- not a device.

By that I meant block device and filesystem freezing. That way filesystem
is frozen so that it doesn't submit any more IO to the device.

> The kernel does suspend device drivers; that is, it invokes their
> suspend callbacks.  But it doesn't "freeze" them in any sense.  Once a 
> driver has been suspended, it assumes it won't receive any I/O requests 
> until it has been resumed.  Therefore the kernel first has to prevent 
> all the upper layers from generating such requests and/or sending them 
> to the low-level drivers.

OK, so Tejun and you should talk together because you both seem to want
something else... If I understand it right, Tejun wants suspended devices
to just queue requests that have been submitted after these devices were
suspended and complete them once they are resumed...

> > > I still think the right thing to do is plugging that block layer or
> > > low level drivers.  It's like we're trying to plug multiple sources
> > > when we can plug the point where they come together anyway.
> > 
> > I agree that freezing writeback workers is a workaround for real issues at
> > best and ideally we shouldn't have to do that. But at least for now I had
> > the impression that it is needed for suspend to work reasonably reliably.
> 
> The design is not to plug low-level drivers, but instead to prevent
> them from receiving any requests by plugging or freezing high-level
> code.
> 
> It's pretty clear that we don't want to have ongoing I/O during a 
> system suspend, right?  And that means the I/O has to be prevented (or 
> "plugged", if you prefer) somewhere -- either at an upper layer or at a 
> lower layer.  There was a choice to be made, and the decision was to do 
> it at an upper layer.

I agree the IO has to be plugged somewhere. And Tejun seems to want to plug
it at lower layer...

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

[toc] | [prev] | [next] | [standalone]

#1359072

Hello, Jan, Alan.

On Tue, Mar 15, 2016 at 10:25:43AM +0100, Jan Kara wrote:
> > The kernel does suspend device drivers; that is, it invokes their
> > suspend callbacks.  But it doesn't "freeze" them in any sense.  Once a 
> > driver has been suspended, it assumes it won't receive any I/O requests 
> > until it has been resumed.  Therefore the kernel first has to prevent 
> > all the upper layers from generating such requests and/or sending them 
> > to the low-level drivers.
> 
> OK, so Tejun and you should talk together because you both seem to want
> something else... If I understand it right, Tejun wants suspended devices
> to just queue requests that have been submitted after these devices were
> suspended and complete them once they are resumed...

Yeah, I suppose that's why we have the code base we do now.  I don't
think freezing kernel threads is the right mechanism to plug IO
devices during suspend.  It's way too error-prone and causes a
dependency nightmare as it acts essentially as a system-wide lock.

More complex drivers already plug themselves which are necessary no
matter what as upper layers or some kthreads aren't the only sources
of commands to devices.  We can plug at block layer for IOs coming
down from higher layers.  We can even provide a mechanism to plug
certain kthreads if necessary but they should be contained in the
driver - e.g. the suspend callback specifically blocking certain
specific kthreads - instead of the vague "the system is generally
stopped now and it seems to work most of the time" that we're doing
now.

Thanks.

-- 
tejun

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web