Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1736190 > unrolled thread

Re: I/O hangs after resuming from suspend-to-ram

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: I/O hangs after resuming from suspend-to-ram Ming Lei <ming.lei@redhat.com> - 2017-09-21 00:20 +0200
    Re: I/O hangs after resuming from suspend-to-ram Martin Steigerwald <martin@lichtvoll.de> - 2017-09-24 19:40 +0200
      Re: I/O hangs after resuming from suspend-to-ram Ming Lei <ming.lei@redhat.com> - 2017-09-25 05:10 +0200
        Re: I/O hangs after resuming from suspend-to-ram Martin Steigerwald <martin@lichtvoll.de> - 2017-09-25 16:20 +0200

#1736190 — Re: I/O hangs after resuming from suspend-to-ram

On Wed, Sep 20, 2017 at 07:25:02PM +0200, Martin Steigerwald wrote:
> Ming Lei - 28.08.17, 21:32:
> > On Mon, Aug 28, 2017 at 03:10:35PM +0200, Martin Steigerwald wrote:
> > > Ming Lei - 28.08.17, 20:58:
> > > > On Sun, Aug 27, 2017 at 09:43:52AM +0200, Oleksandr Natalenko wrote:
> > > > > Hi.
> > > > > 
> > > > > Here is disk setup for QEMU VM:
> […]
> > > > > In words: 2 virtual disks, RAID10 setup with far-2 layout, LUKS on it,
> > > > > then
> > > > > LVM, then ext4 for boot, swap and btrfs for /.
> > > > > 
> > > > > I couldn't reproduce the issue with single disk without RAID.
> > > > 
> > > > Could you verify if the following patch fixes your issue?
> > > 
> > > Could this also apply to non MD RAID systems? I am using BTRFS RAID
> > > 1 with two SSDs. So far with CFQ it runs stable.
> > 
> > It is for fixing Oleksandr's issue wrt. blk-mq, and looks not for you.
> 
> My findings are different:
> 
> On 4.12.10 with CONFIG_HZ=1000, CONFIG_PREEMPT=y and optimizations for Intel 
> Core/newer Xeon I see this:
> 
> 1) Running with CFQ: No hang after resume
> 
> 2) Running with scsi_mod.use_blk_mq=1 + BFQ: Hang after resume within first 1-2 
> days.

Hi Martin,

Thanks for your report!

Could you test the following patchset to see if it fixes your issue?

	https://marc.info/?l=linux-block&m=150579298505484&w=2


-- 
Ming

[toc] | [next] | [standalone]

#1738234

Ming Lei - 21.09.17, 06:17:
> On Wed, Sep 20, 2017 at 07:25:02PM +0200, Martin Steigerwald wrote:
> > Ming Lei - 28.08.17, 21:32:
> > > On Mon, Aug 28, 2017 at 03:10:35PM +0200, Martin Steigerwald wrote:
> > > > Ming Lei - 28.08.17, 20:58:
> > > > > On Sun, Aug 27, 2017 at 09:43:52AM +0200, Oleksandr Natalenko wrote:
> > > > > > Hi.
> > 
> > > > > > Here is disk setup for QEMU VM:
> > […]
> > 
> > > > > > In words: 2 virtual disks, RAID10 setup with far-2 layout, LUKS on
> > > > > > it,
> > > > > > then
> > > > > > LVM, then ext4 for boot, swap and btrfs for /.
> > > > > > 
> > > > > > I couldn't reproduce the issue with single disk without RAID.
> > > > > 
> > > > > Could you verify if the following patch fixes your issue?
> > > > 
> > > > Could this also apply to non MD RAID systems? I am using BTRFS RAID
> > > > 1 with two SSDs. So far with CFQ it runs stable.
> > > 
> > > It is for fixing Oleksandr's issue wrt. blk-mq, and looks not for you.
> > 
> > My findings are different:
> > 
> > On 4.12.10 with CONFIG_HZ=1000, CONFIG_PREEMPT=y and optimizations for
> > Intel Core/newer Xeon I see this:
> > 
> > 1) Running with CFQ: No hang after resume
> > 
> > 2) Running with scsi_mod.use_blk_mq=1 + BFQ: Hang after resume within
> > first 1-2 days.
> 
> Hi Martin,
> 
> Thanks for your report!
> 
> Could you test the following patchset to see if it fixes your issue?
> 
> 	https://marc.info/?l=linux-block&m=150579298505484&w=2

Testing with https://github.com/ming1/linux.git, my_v4.13-safe-scsi-
quiesce_V5_for_test branch as of 53954fd58fb9fe6894b1635dad1acec96ca51a2f I 
now have a bit more than three days and 8 hours of uptime without a hang on 
resume from memory or disk, despite using blk-mq + BFQ.

So it looks like this patch set fixes the issue for me. To say for sure I´d say 
some more days of testing are needed. But it looks like the hang on resume is 
fixed by this patch set.

Thanks,
-- 
Martin

[toc] | [prev] | [next] | [standalone]

#1738719

On Sun, Sep 24, 2017 at 07:33:00PM +0200, Martin Steigerwald wrote:
> Ming Lei - 21.09.17, 06:17:
> > On Wed, Sep 20, 2017 at 07:25:02PM +0200, Martin Steigerwald wrote:
> > > Ming Lei - 28.08.17, 21:32:
> > > > On Mon, Aug 28, 2017 at 03:10:35PM +0200, Martin Steigerwald wrote:
> > > > > Ming Lei - 28.08.17, 20:58:
> > > > > > On Sun, Aug 27, 2017 at 09:43:52AM +0200, Oleksandr Natalenko wrote:
> > > > > > > Hi.
> > > 
> > > > > > > Here is disk setup for QEMU VM:
> > > […]
> > > 
> > > > > > > In words: 2 virtual disks, RAID10 setup with far-2 layout, LUKS on
> > > > > > > it,
> > > > > > > then
> > > > > > > LVM, then ext4 for boot, swap and btrfs for /.
> > > > > > > 
> > > > > > > I couldn't reproduce the issue with single disk without RAID.
> > > > > > 
> > > > > > Could you verify if the following patch fixes your issue?
> > > > > 
> > > > > Could this also apply to non MD RAID systems? I am using BTRFS RAID
> > > > > 1 with two SSDs. So far with CFQ it runs stable.
> > > > 
> > > > It is for fixing Oleksandr's issue wrt. blk-mq, and looks not for you.
> > > 
> > > My findings are different:
> > > 
> > > On 4.12.10 with CONFIG_HZ=1000, CONFIG_PREEMPT=y and optimizations for
> > > Intel Core/newer Xeon I see this:
> > > 
> > > 1) Running with CFQ: No hang after resume
> > > 
> > > 2) Running with scsi_mod.use_blk_mq=1 + BFQ: Hang after resume within
> > > first 1-2 days.
> > 
> > Hi Martin,
> > 
> > Thanks for your report!
> > 
> > Could you test the following patchset to see if it fixes your issue?
> > 
> > 	https://marc.info/?l=linux-block&m=150579298505484&w=2
> 
> Testing with https://github.com/ming1/linux.git, my_v4.13-safe-scsi-
> quiesce_V5_for_test branch as of 53954fd58fb9fe6894b1635dad1acec96ca51a2f I 
> now have a bit more than three days and 8 hours of uptime without a hang on 
> resume from memory or disk, despite using blk-mq + BFQ.
> 
> So it looks like this patch set fixes the issue for me. To say for sure I´d say 
> some more days of testing are needed. But it looks like the hang on resume is 
> fixed by this patch set.

Martin, thanks for your test, it is great to see your issue is fixed
by this patchset.

Also I remember that your issue wasn't related with MD, and actually
you were using BTRFS(RAID), just want to double check with you,
is that true?

-- 
Ming

[toc] | [prev] | [next] | [standalone]

#1739081

Ming Lei - 25.09.17, 10:59:
> On Sun, Sep 24, 2017 at 07:33:00PM +0200, Martin Steigerwald wrote:
> > Ming Lei - 21.09.17, 06:17:
> > > On Wed, Sep 20, 2017 at 07:25:02PM +0200, Martin Steigerwald wrote:
> > > > Ming Lei - 28.08.17, 21:32:
> > > > > On Mon, Aug 28, 2017 at 03:10:35PM +0200, Martin Steigerwald wrote:
> > > > > > Ming Lei - 28.08.17, 20:58:
> > > > > > > On Sun, Aug 27, 2017 at 09:43:52AM +0200, Oleksandr Natalenko 
wrote:
> > > > > > > > Hi.
> > > > 
> > > > > > > > Here is disk setup for QEMU VM:
> > > > […]
> > > > 
> > > > > > > > In words: 2 virtual disks, RAID10 setup with far-2 layout,
> > > > > > > > LUKS on
> > > > > > > > it,
> > > > > > > > then
> > > > > > > > LVM, then ext4 for boot, swap and btrfs for /.
> > > > > > > > 
> > > > > > > > I couldn't reproduce the issue with single disk without RAID.
> > > > > > > 
> > > > > > > Could you verify if the following patch fixes your issue?
> > > > > > 
> > > > > > Could this also apply to non MD RAID systems? I am using BTRFS
> > > > > > RAID
> > > > > > 1 with two SSDs. So far with CFQ it runs stable.
> > > > > 
> > > > > It is for fixing Oleksandr's issue wrt. blk-mq, and looks not for
> > > > > you.
> > > > 
> > > > My findings are different:
> > > > 
> > > > On 4.12.10 with CONFIG_HZ=1000, CONFIG_PREEMPT=y and optimizations for
> > > > Intel Core/newer Xeon I see this:
> > > > 
> > > > 1) Running with CFQ: No hang after resume
> > > > 
> > > > 2) Running with scsi_mod.use_blk_mq=1 + BFQ: Hang after resume within
> > > > first 1-2 days.
> > > 
> > > Hi Martin,
> > > 
> > > Thanks for your report!
> > > 
> > > Could you test the following patchset to see if it fixes your issue?
> > > 
> > > 	https://marc.info/?l=linux-block&m=150579298505484&w=2
> > 
> > Testing with https://github.com/ming1/linux.git, my_v4.13-safe-scsi-
> > quiesce_V5_for_test branch as of 53954fd58fb9fe6894b1635dad1acec96ca51a2f
> > I
> > now have a bit more than three days and 8 hours of uptime without a hang
> > on
> > resume from memory or disk, despite using blk-mq + BFQ.
> > 
> > So it looks like this patch set fixes the issue for me. To say for sure
> > I´d say some more days of testing are needed. But it looks like the hang
> > on resume is fixed by this patch set.
> 
> Martin, thanks for your test, it is great to see your issue is fixed
> by this patchset.
> 
> Also I remember that your issue wasn't related with MD, and actually
> you were using BTRFS(RAID), just want to double check with you,
> is that true?

Yes. But *additionally*, as I explained, the hangs only occured with 
CONFIG_HZ=1000, Low latency + CPU optimized kernel, not with CONFIG_HZ=250, 
voluntary preemption + generic x86 kernel. This hints a timing issue for me. 
So blk-mq + BFQ + CONFIG_HZ=1000 + Low Latency + CPU optimization = hang.

I did not investigate whether blk-mq without scheduler would work. And I did 
not investigate which of the three changes to the kernel configurations 
triggers the issue. I suspect its either CONFIG_HZ or the preemption option or 
both of them.

My original mail had both working and non working kernel configurations 
attached.

Thanks,
-- 
Martin

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web