Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1531078 > unrolled thread

Re: 4.8.8 kernel trigger OOM killer repeatedly when I have lots of RAM that should be free

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: 4.8.8 kernel trigger OOM killer repeatedly when I have lots of  RAM that should be free Michal Hocko <mhocko@kernel.org> - 2016-11-28 08:30 +0100
    Re: 4.8.8 kernel trigger OOM killer repeatedly when I have lots of RAM that should be free Marc MERLIN <marc@merlins.org> - 2016-11-28 22:00 +0100
    Re: 4.8.8 kernel trigger OOM killer repeatedly when I have lots of  RAM that should be free Marc MERLIN <marc@merlins.org> - 2016-11-29 17:00 +0100
      Re: 4.8.8 kernel trigger OOM killer repeatedly when I have lots of  RAM that should be free Michal Hocko <mhocko@kernel.org> - 2016-11-29 17:10 +0100
        Re: 4.8.8 kernel trigger OOM killer repeatedly when I have lots of  RAM that should be free Marc MERLIN <marc@merlins.org> - 2016-11-29 17:40 +0100
          Re: 4.8.8 kernel trigger OOM killer repeatedly when I have lots of  RAM that should be free Linus Torvalds <torvalds@linux-foundation.org> - 2016-11-29 18:10 +0100
            Re: 4.8.8 kernel trigger OOM killer repeatedly when I have lots of  RAM that should be free Marc MERLIN <marc@merlins.org> - 2016-11-29 18:50 +0100
              Re: 4.8.8 kernel trigger OOM killer repeatedly when I have lots of  RAM that should be free Linus Torvalds <torvalds@linux-foundation.org> - 2016-11-29 19:10 +0100
                Re: 4.8.8 kernel trigger OOM killer repeatedly when I have lots of  RAM that should be free Marc MERLIN <marc@merlins.org> - 2016-11-30 18:50 +0100
                  Re: 4.8.8 kernel trigger OOM killer repeatedly when I have lots of  RAM that should be free Linus Torvalds <torvalds@linux-foundation.org> - 2016-11-30 19:20 +0100
                    Re: 4.8.8 kernel trigger OOM killer repeatedly when I have lots of  RAM that should be free Marc MERLIN <marc@merlins.org> - 2016-11-30 19:30 +0100
                    Re: 4.8.8 kernel trigger OOM killer repeatedly when I have lots of  RAM that should be free Tejun Heo <tj@kernel.org> - 2016-11-30 21:40 +0100
                      Re: 4.8.8 kernel trigger OOM killer repeatedly when I have lots of  RAM that should be free Kent Overstreet <kent.overstreet@gmail.com> - 2016-12-01 15:00 +0100
                        Re: 4.8.8 kernel trigger OOM killer repeatedly when I have lots of  RAM that should be free Linus Torvalds <torvalds@linux-foundation.org> - 2016-12-01 19:20 +0100
                          Re: 4.8.8 kernel trigger OOM killer repeatedly when I have lots of  RAM that should be free Linus Torvalds <torvalds@linux-foundation.org> - 2016-12-01 19:40 +0100
              Re: 4.8.8 kernel trigger OOM killer repeatedly when I have lots of RAM that should be free Marc MERLIN <marc@merlins.org> - 2016-11-30 00:10 +0100
                Re: 4.8.8 kernel trigger OOM killer repeatedly when I have lots of  RAM that should be free Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp> - 2016-11-30 15:00 +0100
    Re: 4.8.8 kernel trigger OOM killer repeatedly when I have lots of  RAM that should be free Marc MERLIN <marc@merlins.org> - 2016-11-29 17:20 +0100

#1531078 — Re: 4.8.8 kernel trigger OOM killer repeatedly when I have lots of RAM that should be free

Marc, could you try this patch please? I think it should be pretty clear
it should help you but running it through your use case would be more
than welcome before I ask Greg to take this to the 4.8 stable tree.

Thanks!

On Wed 23-11-16 07:34:10, Michal Hocko wrote:
[...]
> commit b2ccdcb731b666aa28f86483656c39c5e53828c7
> Author: Michal Hocko <mhocko@suse.com>
> Date:   Wed Nov 23 07:26:30 2016 +0100
> 
>     mm, oom: stop pre-mature high-order OOM killer invocations
>     
>     31e49bfda184 ("mm, oom: protect !costly allocations some more for
>     !CONFIG_COMPACTION") was an attempt to reduce chances of pre-mature OOM
>     killer invocation for high order requests. It seemed to work for most
>     users just fine but it is far from bullet proof and obviously not
>     sufficient for Marc who has reported pre-mature OOM killer invocations
>     with 4.8 based kernels. 4.9 will all the compaction improvements seems
>     to be behaving much better but that would be too intrusive to backport
>     to 4.8 stable kernels. Instead this patch simply never declares OOM for
>     !costly high order requests. We rely on order-0 requests to do that in
>     case we are really out of memory. Order-0 requests are much more common
>     and so a risk of a livelock without any way forward is highly unlikely.
>     
>     Reported-by: Marc MERLIN <marc@merlins.org>
>     Signed-off-by: Michal Hocko <mhocko@suse.com>
> 
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index a2214c64ed3c..7401e996009a 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -3161,6 +3161,16 @@ should_compact_retry(struct alloc_context *ac, unsigned int order, int alloc_fla
>  	if (!order || order > PAGE_ALLOC_COSTLY_ORDER)
>  		return false;
>  
> +#ifdef CONFIG_COMPACTION
> +	/*
> +	 * This is a gross workaround to compensate a lack of reliable compaction
> +	 * operation. We cannot simply go OOM with the current state of the compaction
> +	 * code because this can lead to pre mature OOM declaration.
> +	 */
> +	if (order <= PAGE_ALLOC_COSTLY_ORDER)
> +		return true;
> +#endif
> +
>  	/*
>  	 * There are setups with compaction disabled which would prefer to loop
>  	 * inside the allocator rather than hit the oom killer prematurely.
> -- 
> Michal Hocko
> SUSE Labs

-- 
Michal Hocko
SUSE Labs

[toc] | [next] | [standalone]

#1531656 — Re: 4.8.8 kernel trigger OOM killer repeatedly when I have lots of RAM that should be free

[Multipart message — attachments visible in raw view] — view raw

On Mon, Nov 28, 2016 at 08:23:15AM +0100, Michal Hocko wrote:
> Marc, could you try this patch please? I think it should be pretty clear
> it should help you but running it through your use case would be more
> than welcome before I ask Greg to take this to the 4.8 stable tree.
 
This will take a little while, the whole copy took 5 days to finish and I'm a
bit hesitant about blowing it away and starting over :)
Let me see if I can come up with maybe another disk array for another test.

For now, as a reminder, I'm running that attached patch, and it works fine
I'll report back as soon as I can.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

[toc] | [prev] | [next] | [standalone]

#1532443

On Mon, Nov 28, 2016 at 08:23:15AM +0100, Michal Hocko wrote:
> Marc, could you try this patch please? I think it should be pretty clear
> it should help you but running it through your use case would be more
> than welcome before I ask Greg to take this to the 4.8 stable tree.

I ran it overnight and copied 1.4TB with it before it failed because
there wasn't enough disk space on the other side, so I think it fixes
the problem too.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

[toc] | [prev] | [next] | [standalone]

#1532454

On Tue 29-11-16 07:55:37, Marc MERLIN wrote:
> On Mon, Nov 28, 2016 at 08:23:15AM +0100, Michal Hocko wrote:
> > Marc, could you try this patch please? I think it should be pretty clear
> > it should help you but running it through your use case would be more
> > than welcome before I ask Greg to take this to the 4.8 stable tree.
> 
> I ran it overnight and copied 1.4TB with it before it failed because
> there wasn't enough disk space on the other side, so I think it fixes
> the problem too.

Can I add your Tested-by?

-- 
Michal Hocko
SUSE Labs

[toc] | [prev] | [next] | [standalone]

#1532486

On Tue, Nov 29, 2016 at 05:07:51PM +0100, Michal Hocko wrote:
> On Tue 29-11-16 07:55:37, Marc MERLIN wrote:
> > On Mon, Nov 28, 2016 at 08:23:15AM +0100, Michal Hocko wrote:
> > > Marc, could you try this patch please? I think it should be pretty clear
> > > it should help you but running it through your use case would be more
> > > than welcome before I ask Greg to take this to the 4.8 stable tree.
> > 
> > I ran it overnight and copied 1.4TB with it before it failed because
> > there wasn't enough disk space on the other side, so I think it fixes
> > the problem too.
> 
> Can I add your Tested-by?

Done.

Now, probably unrelated, but hard to be sure, doing those big copies
causes massive hangs on my system. I hit a few of the 120s hangs,
but more generally lots of things hang, including shells, my DNS server,
monitoring reading from USB and timing out, and so forth.
Examples below. 
I have a hard time telling what is the fault, but is there a chance it
might be memory allocation pressure?
I already have a preempt kernel, so I can't make it more preempt than
that.
Now, to be fair, this is not a new problem, it's just varying degrees of
bad and usually only happens when I do a lot of I/O with btrfs.
That said, btrfs may very well just be suffering from memory allocation
issues and hanging as a result, with everything else on my system also
hanging for similar reasons until the memory pressure goes away with the
copy or scrub are finished.

What do you think?

[28034.954435] INFO: task btrfs:5618 blocked for more than 120 seconds.
[28034.975471]       Tainted: G     U          4.8.10-amd64-preempt-sysrq-20161121vb3tj1 #12
[28035.000964] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[28035.025429] btrfs           D ffff91154d33fc70     0  5618   5372 0x00000080
[28035.047717]  ffff91154d33fc70 0000000000200246 ffff911842f880c0 ffff9115a4cf01c0
[28035.071020]  ffff91154d33fc58 ffff91154d340000 ffff91165493bca0 ffff9115623773f0
[28035.094252]  0000000000001000 0000000000000001 ffff91154d33fc88 ffffffffb86cf1a6
[28035.117538] Call Trace:
[28035.125791]  [<ffffffffb86cf1a6>] schedule+0x8b/0xa3
[28035.141550]  [<ffffffffb82bd18e>] btrfs_start_ordered_extent+0xce/0x122
[28035.162457]  [<ffffffffb809af6c>] ? wake_up_atomic_t+0x2c/0x2c
[28035.180891]  [<ffffffffb82bd434>] btrfs_wait_ordered_range+0xa9/0x10d
[28035.201723]  [<ffffffffb82aec04>] btrfs_truncate+0x40/0x24b
[28035.219269]  [<ffffffffb82af437>] btrfs_setattr+0x1da/0x2d7
[28035.237032]  [<ffffffffb81c7507>] notify_change+0x252/0x39c
[28035.254566]  [<ffffffffb81ad35b>] do_truncate+0x81/0xb4
[28035.271057]  [<ffffffffb81ad467>] vfs_truncate+0xd9/0xf9
[28035.287782]  [<ffffffffb81ad4ea>] do_sys_truncate+0x63/0xa7

I get other hangs like:

[10338.968912] perf: interrupt took too long (3927 > 3917), lowering kernel.perf_event_max_sample_rate to 50750

[12971.047705] ftdi_sio ttyUSB15: usb_serial_generic_read_bulk_callback - urb stopped: -32

[17761.122238] usb 4-1.4: USB disconnect, device number 39
[17761.141063] usb 4-1.4: usbfs: USBDEVFS_CONTROL failed cmd hub-ctrl rqt 160 rq 6 len 1024 ret -108
[17761.263252] usb 4-1: reset SuperSpeed USB device number 2 using xhci_hcd
[17761.938575] usb 4-1.4: new SuperSpeed USB device number 40 using xhci_hcd

[24130.574425] hpet1: lost 2306 rtc interrupts
[24156.034950] hpet1: lost 1628 rtc interrupts
[24173.314738] hpet1: lost 1104 rtc interrupts
[24180.129950] hpet1: lost 436 rtc interrupts
[24257.557955] hpet1: lost 4954 rtc interrupts
[24267.522656] hpet1: lost 637 rtc interrupts

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

[toc] | [prev] | [next] | [standalone]

#1532536

On Tue, Nov 29, 2016 at 8:34 AM, Marc MERLIN <marc@merlins.org> wrote:
> Now, to be fair, this is not a new problem, it's just varying degrees of
> bad and usually only happens when I do a lot of I/O with btrfs.

One situation where I've seen something like this happen is

 (a) lots and lots of dirty data queued up
 (b) horribly slow storage
 (c) filesystem that ends up serializing on writeback under certain
circumstances

The usual case for (b) in the modern world is big SSD's that have bad
worst-case behavior (ie they may do gbps speeds when doing well, and
then they come to a screeching halt when their buffers fill up and
they have to do rewrites, and their gbps throughput drops to mbps or
lower).

Generally you only find that kind of really nasty SSD in the USB stick
world these days.

The usual case for (c) is "fsync" or similar - often on a totally
unrelated file - which then ends up waiting for everything else to
flush too. Looks like btrfs_start_ordered_extent() does something kind
of like that, where it waits for data to be flushed.

The usual *fix* for this is to just not get into situation (a).

Sadly, our defaults for "how much dirty data do we allow" are somewhat
buggered. The global defaults are in "percent of memory", and are
generally _much_ too high for big-memory machines:

    [torvalds@i7 linux]$ cat /proc/sys/vm/dirty_ratio
    20
    [torvalds@i7 linux]$ cat /proc/sys/vm/dirty_background_ratio
    10

says that it only starts really throttling writes when you hit 20% of
all memory used. You don't say how much memory you have in that
machine, but if it's the same one you talked about earlier, it was
24GB. So you can have 4GB of dirty data waiting to be flushed out.

And we *try* to do this per-device backing-dev congestion thing to
make things work better, but it generally seems to not work very well.
Possibly because of inconsistent write speeds (ie _sometimes_ the SSD
does really well, and we want to open up, and then it shuts down).

One thing you can try is to just make the global limits much lower. As in

   echo 2 > /proc/sys/vm/dirty_ratio
   echo 1 > /proc/sys/vm/dirty_background_ratio

(if you want to go lower than 1%, you'll have to use the
"dirty_*ratio_bytes" byte limits instead of percentage limits).

Obviously you'll need to be root for this, and equally obviously it's
really a failure of the kernel. I'd *love* to get something like this
right automatically, but sadly it depends so much on memory size,
load, disk subsystem, etc etc that I despair at it.

On x86-32 we "fixed" this long ago by just saying "high memory is not
dirtyable", so you were always limited to a maximum of 10/20% of 1GB,
rather than the full memory range. It worked better, but it's a sad
kind of fix.

(See commit dc6e29da9162: "Fix balance_dirty_page() calculations with
CONFIG_HIGHMEM")

             Linus

[toc] | [prev] | [next] | [standalone]

#1532612

Thanks for the reply and suggestions.

On Tue, Nov 29, 2016 at 09:07:03AM -0800, Linus Torvalds wrote:
> On Tue, Nov 29, 2016 at 8:34 AM, Marc MERLIN <marc@merlins.org> wrote:
> > Now, to be fair, this is not a new problem, it's just varying degrees of
> > bad and usually only happens when I do a lot of I/O with btrfs.
> 
> One situation where I've seen something like this happen is
> 
>  (a) lots and lots of dirty data queued up
>  (b) horribly slow storage

In my case, it is a 5x 4TB HDD with 
software raid 5 < bcache < dmcrypt < btrfs
bcache is currently half disabled (as in I removed the actual cache) or
too many bcache requests pile up, and the kernel dies when too many
workqueues have piled up.
I'm just kind of worried that since I'm going through 4 subsystems
before my data can hit disk, that's a lot of memory allocations and
places where data can accumulate and cause bottlenecks if the next
subsystem isn't as fast.

But this shouldn't be "horribly slow", should it? (it does copy a few
terabytes per day, not fast, but not horrible, about 30MB/s or so)

> Sadly, our defaults for "how much dirty data do we allow" are somewhat
> buggered. The global defaults are in "percent of memory", and are
> generally _much_ too high for big-memory machines:
> 
>     [torvalds@i7 linux]$ cat /proc/sys/vm/dirty_ratio
>     20
>     [torvalds@i7 linux]$ cat /proc/sys/vm/dirty_background_ratio
>     10

I can confirm I have the same.

> says that it only starts really throttling writes when you hit 20% of
> all memory used. You don't say how much memory you have in that
> machine, but if it's the same one you talked about earlier, it was
> 24GB. So you can have 4GB of dirty data waiting to be flushed out.

Correct, 24GB and 4GB.

> And we *try* to do this per-device backing-dev congestion thing to
> make things work better, but it generally seems to not work very well.
> Possibly because of inconsistent write speeds (ie _sometimes_ the SSD
> does really well, and we want to open up, and then it shuts down).
> 
> One thing you can try is to just make the global limits much lower. As in
> 
>    echo 2 > /proc/sys/vm/dirty_ratio
>    echo 1 > /proc/sys/vm/dirty_background_ratio

I will give that a shot, thank you.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

[toc] | [prev] | [next] | [standalone]

#1532625

On Tue, Nov 29, 2016 at 9:40 AM, Marc MERLIN <marc@merlins.org> wrote:
>
> In my case, it is a 5x 4TB HDD with
> software raid 5 < bcache < dmcrypt < btrfs

It doesn't sound like the nasty situations I have seen (particularly
with large USB flash storage - often high momentary speed for
benchmarks, but slows down to a crawl after you've written a bit to
it, and doesn't have the smart garbage collection that modern "real"
SSDs have).

But while it doesn't sound like that nasty case, RAID5 will certainly
not help your write speed, and with spinning rust that potentially up
to 4GB (in fact, almost 5GB) of dirty pending data is going to take a
long time to write out if it's not all nice and contiguous (which it
won't be).

And btrfs might be weak on that case - I remember complaining about
fsync stuttering all IO a few years ago, exactly because it would
force-flush everything else too (ie you were doing non-synchronous
writes in one session, and then the browser did a "fsync" on the small
writes it did to the mysql database, and suddenly the browser paused
for ten seconds or more, because the fsync wasn't just waiting for the
small database update, but for _everythinig_ to be written back).

Your backtrace isn't for fsync, but it looks superficially similar:
"wait for write data to flush".

             Linus

[toc] | [prev] | [next] | [standalone]

#1533472

On Tue, Nov 29, 2016 at 10:01:10AM -0800, Linus Torvalds wrote:
> On Tue, Nov 29, 2016 at 9:40 AM, Marc MERLIN <marc@merlins.org> wrote:
> >
> > In my case, it is a 5x 4TB HDD with
> > software raid 5 < bcache < dmcrypt < btrfs
> 
> It doesn't sound like the nasty situations I have seen (particularly
> with large USB flash storage - often high momentary speed for
> benchmarks, but slows down to a crawl after you've written a bit to
> it, and doesn't have the smart garbage collection that modern "real"
> SSDs have).

I gave it a thought again, I think it is exactly the nasty situation you
described.
bcache takes I/O quickly while sending to SSD cache. SSD fills up, now
bcache can't handle IO as quickly and has to hang until the SSD has been
flushed to spinning rust drives.
This actually is exactly the same as filling up the cache on a USB key
and now you're waiting for slow writes to flash, is it not?

With your dirty ratio workaround, I was able to re-enable bcache and
have it not fall over, but only barely. I recorded over a hundred
workqueues in flight during the copy at some point (just not enough
to actually kill the kernel this time).

I've started a bcache followp on this here:
http://marc.info/?l=linux-bcache&m=148052441423532&w=2
http://marc.info/?l=linux-bcache&m=148052620524162&w=2

This message shows the huge pileup of workqueeues in bcache
just before the kernel dies with
Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3904 04/27/2013
task: ffff9ee0c2fa4180 task.stack: ffff9ee0c2fa8000
RIP: 0010:[<ffffffffbb57a128>]  [<ffffffffbb57a128>] cpuidle_enter_state+0x119/0x171
RSP: 0000:ffff9ee0c2fabea0  EFLAGS: 00000246
RAX: ffff9ee0de3d90c0 RBX: 0000000000000004 RCX: 000000000000001f
RDX: 0000000000000000 RSI: 0000000000000007 RDI: 0000000000000000
RBP: ffff9ee0c2fabed0 R08: 0000000000000f92 R09: 0000000000000f42
R10: ffff9ee0c2fabe50 R11: 071c71c71c71c71c R12: ffffe047bfdcb200
R13: 00000af626899577 R14: 0000000000000004 R15: 00000af6264cc557
FS:  0000000000000000(0000) GS:ffff9ee0de3c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000898b000 CR3: 000000045cc06000 CR4: 00000000001406e0
Stack:
 0000000000000f40 ffffe047bfdcb200 ffffffffbbccc060 ffff9ee0c2fac000
 ffff9ee0c2fa8000 ffff9ee0c2fac000 ffff9ee0c2fabee0 ffffffffbb57a1ac
 ffff9ee0c2fabf30 ffffffffbb09238d ffff9ee0c2fa8000 0000000700000004
Call Trace:
 [<ffffffffbb57a1ac>] cpuidle_enter+0x17/0x19
 [<ffffffffbb09238d>] cpu_startup_entry+0x210/0x28b
 [<ffffffffbb03de22>] start_secondary+0x13e/0x140
Code: 00 00 00 48 c7 c7 cd ae b2 bb c6 05 4b 8e 7a 00 01 e8 17 6c ae ff fa 66 0f 1f 44 00 00 31 ff e8 75 60 b4
44 00 00 <4c> 89 e8 b9 e8 03 00 00 4c 29 f8 48 99 48 f7 f9 ba ff ff ff 7f
Kernel panic - not syncing: Hard LOCKUP

A full traceback showing the pilup of requests is here:
http://marc.info/?l=linux-bcache&m=147949497808483&w=2

and there:
http://pastebin.com/rJ5RKUVm
(2 different ones but mostly the same result)

We can probably follow up on the bcache thread I Cc'ed you on since I'm
not sure if the fault here lies with bcache or the VM subsystem anymore.

Thanks.
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

[toc] | [prev] | [next] | [standalone]

#1533489

[Multipart message — attachments visible in raw view] — view raw

On Wed, Nov 30, 2016 at 9:47 AM, Marc MERLIN <marc@merlins.org> wrote:
>
> I gave it a thought again, I think it is exactly the nasty situation you
> described.
> bcache takes I/O quickly while sending to SSD cache. SSD fills up, now
> bcache can't handle IO as quickly and has to hang until the SSD has been
> flushed to spinning rust drives.
> This actually is exactly the same as filling up the cache on a USB key
> and now you're waiting for slow writes to flash, is it not?

It does sound like you might hit exactly the same kind of situation, yes.

And the fact that you have dmcrypt running too just makes things pile
up more. All those IO's end up slowed down by the scheduling too.

Anyway, none of this seems new per se. I'm adding Kent and Jens to the
cc (Tejun already was), in the hope that maybe they have some idea how
to control the nasty worst-case behavior wrt workqueue lockup (it's
not really a "lockup", it looks like it's just hundreds of workqueues
all waiting for IO to complete and much too deep IO queues).

I think it's the traditional "throughput is much easier to measure and
improve" situation, where making queues big help some throughput
situation, but ends up causing chaos when things go south.

And I think your NMI watchdog then turns the "system is no longer
responsive" into an actual kernel panic.

> With your dirty ratio workaround, I was able to re-enable bcache and
> have it not fall over, but only barely. I recorded over a hundred
> workqueues in flight during the copy at some point (just not enough
> to actually kill the kernel this time).
>
> I've started a bcache followp on this here:
> http://marc.info/?l=linux-bcache&m=148052441423532&w=2
> http://marc.info/?l=linux-bcache&m=148052620524162&w=2
>
> A full traceback showing the pilup of requests is here:
> http://marc.info/?l=linux-bcache&m=147949497808483&w=2
>
> and there:
> http://pastebin.com/rJ5RKUVm
> (2 different ones but mostly the same result)

Tejun/Kent - any way to just limit the workqueue depth for bcache?
Because that really isn't helping, and things *will* time out and
cause those problems when you have hundreds of IO's queued on a disk
that likely as a write iops around ~100..

And I really wonder if we should do the "big hammer" approach to the
dirty limits on non-HIGHMEM machines too (approximate the
"vm_highmem_is_dirtyable" by just limiting global_dirtyable_memory()
to 1 GB).

That would make the default dirty limits be 100/200MB (for soft/hard
throttling), which really is much more reasonable than gigabytes and
gigabytes of dirty data.

Of course, no way do we do that during rc7..

                    Linus

[toc] | [prev] | [next] | [standalone]

#1533494

On Wed, Nov 30, 2016 at 10:14:50AM -0800, Linus Torvalds wrote:
> Anyway, none of this seems new per se. I'm adding Kent and Jens to the
> cc (Tejun already was), in the hope that maybe they have some idea how
> to control the nasty worst-case behavior wrt workqueue lockup (it's
> not really a "lockup", it looks like it's just hundreds of workqueues
> all waiting for IO to complete and much too deep IO queues).
 
I'll take your word for it, all I got in the end was
Kernel panic - not syncing: Hard LOCKUP
and the system stone dead when I woke up hours later.

> And I think your NMI watchdog then turns the "system is no longer
> responsive" into an actual kernel panic.

Ah, I see.

Thanks for the reply, and sorry for bringing in that separate thread
from the btrfs mailing list, which effectively was a suggestion similar
to what you're saying here too.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

[toc] | [prev] | [next] | [standalone]

#1533553

Hello,

On Wed, Nov 30, 2016 at 10:14:50AM -0800, Linus Torvalds wrote:
> Tejun/Kent - any way to just limit the workqueue depth for bcache?
> Because that really isn't helping, and things *will* time out and
> cause those problems when you have hundreds of IO's queued on a disk
> that likely as a write iops around ~100..

Yeah, easily.  I'm assuming it's gonna be the bcache_wq allocated in
from bcache_init().  It's currently using 0 as @max_active and it can
set to be any arbitrary number.  It'd be a very crude way to control
what looks like a buffer bloat with IOs tho.  We can make it a bit
more granular by splitting workqueues per bcache instance / purpose
but for the long term the right solution seems to be hooking into
writeback throttling mechanism that block layer just grew recently.

Thanks.

-- 
tejun

[toc] | [prev] | [next] | [standalone]

#1534107

On Wed, Nov 30, 2016 at 03:30:11PM -0500, Tejun Heo wrote:
> Hello,
> 
> On Wed, Nov 30, 2016 at 10:14:50AM -0800, Linus Torvalds wrote:
> > Tejun/Kent - any way to just limit the workqueue depth for bcache?
> > Because that really isn't helping, and things *will* time out and
> > cause those problems when you have hundreds of IO's queued on a disk
> > that likely as a write iops around ~100..
> 
> Yeah, easily.  I'm assuming it's gonna be the bcache_wq allocated in
> from bcache_init().  It's currently using 0 as @max_active and it can
> set to be any arbitrary number.  It'd be a very crude way to control
> what looks like a buffer bloat with IOs tho.  We can make it a bit
> more granular by splitting workqueues per bcache instance / purpose
> but for the long term the right solution seems to be hooking into
> writeback throttling mechanism that block layer just grew recently.

Agreed that the writeback code is the right place to do it. Within bcache we
can't really do anything smarter than just throw a hard limit on the number of
outstanding IOs and enforce it by blocking in generic_make_request(), and the
bcache code is the wrong place to do that - we don't know what the limit should
be there, and all the IOs look the same at that point so you'd probably still
end up with writeback starving everything else.

I could futz with the workqueue stuff, but that'd likely as not break some other
workload - I've spent enough time as it is fighting with workqueue concurrency
stuff in the past. My preference would be to just try and get Jens's stuff in.

That said, I'm not sure how I feel about Jens's exact approach... it seems to me
that this can really just live within the writeback code, I don't know why it
should involve the block layer at all. plus, if I understand correctly his code
has the effect of blocking in generic_make_request() to throttle, which means
due to the way the writeback code is structured we'll be blocking with page
locks held. I did my own thing in bcachefs, same idea but throttling in
writepages... it's dumb and simple but it's worked exceedingly well, as far as
actual usability and responsiveness:

https://evilpiepirate.org/git/linux-bcache.git/tree/drivers/md/bcache/fs-io.c?h=bcache-dev&id=acf766b2dd33b076fdce66c86363a3e26a9b70cf#n1002

that said - any kind of throttling for writeback will be a million times better
than the current situation...

[toc] | [prev] | [next] | [standalone]

#1534367

On Thu, Dec 1, 2016 at 5:50 AM, Kent Overstreet
<kent.overstreet@gmail.com> wrote:
>
> That said, I'm not sure how I feel about Jens's exact approach... it seems to me
> that this can really just live within the writeback code, I don't know why it
> should involve the block layer at all. plus, if I understand correctly his code
> has the effect of blocking in generic_make_request() to throttle, which means
> due to the way the writeback code is structured we'll be blocking with page
> locks held.

Yeah, I do *not* believe that throttling at the block layer is at all
the right thing to do.

I do think that the block layer needs to throttle, but it needs to be
seen as a "last resort" kind of thing, where the block layer just
needs to limit how much it will have oending. But it should be seen as
a failure mode, not as a write balancing issue.

Because the real throttling absolutely needs to happen when things are
marked dirty, because no block layer throttling will ever fix the
situation where you just have too much memory dirtied that you cannot
free because it will take a minute to write out.

So throttling at a VM level is sane. Throttling at a block layer level is not.

            Linus

[toc] | [prev] | [next] | [standalone]

#1534379

On Thu, Dec 1, 2016 at 10:30 AM, Jens Axboe <axboe@fb.com> wrote:
>
> It's two different kinds of throttling. The vm absolutely should
> throttle at dirty time, to avoid having insane amounts of memory dirty.
> On the block layer side, throttling is about avoid the device queues
> being too long. It's very similar to the buffer bloating on the
> networking side. The block layer throttling is not a fix for the vm
> allowing too much memory to be dirty and causing issues, it's about
> keeping the device response latencies in check.

Sure. But if we really do just end up blocking in the block layer (in
situations where we didn't used to), that may be a bad thing. It might
be better to feed that information back to the VM instead,
particularly for writes, where the VM layer already tries to ratelimit
the writes.

And frankly, it's almost purely writes that matter. There  just aren't
a lot of ways to get that many parallel reads in real life.

I haven't looked at your patches, so maybe you already do this.

             Linus

[toc] | [prev] | [next] | [standalone]

#1532838 — Re: 4.8.8 kernel trigger OOM killer repeatedly when I have lots of RAM that should be free

On Tue, Nov 29, 2016 at 09:40:19AM -0800, Marc MERLIN wrote:
> Thanks for the reply and suggestions.
> 
> On Tue, Nov 29, 2016 at 09:07:03AM -0800, Linus Torvalds wrote:
> > On Tue, Nov 29, 2016 at 8:34 AM, Marc MERLIN <marc@merlins.org> wrote:
> > > Now, to be fair, this is not a new problem, it's just varying degrees of
> > > bad and usually only happens when I do a lot of I/O with btrfs.
> > 
> > One situation where I've seen something like this happen is
> > 
> >  (a) lots and lots of dirty data queued up
> >  (b) horribly slow storage
> 
> In my case, it is a 5x 4TB HDD with 
> software raid 5 < bcache < dmcrypt < btrfs
> bcache is currently half disabled (as in I removed the actual cache) or
> too many bcache requests pile up, and the kernel dies when too many
> workqueues have piled up.
> I'm just kind of worried that since I'm going through 4 subsystems
> before my data can hit disk, that's a lot of memory allocations and
> places where data can accumulate and cause bottlenecks if the next
> subsystem isn't as fast.
> 
> But this shouldn't be "horribly slow", should it? (it does copy a few
> terabytes per day, not fast, but not horrible, about 30MB/s or so)
> 
> > Sadly, our defaults for "how much dirty data do we allow" are somewhat
> > buggered. The global defaults are in "percent of memory", and are
> > generally _much_ too high for big-memory machines:
> > 
> >     [torvalds@i7 linux]$ cat /proc/sys/vm/dirty_ratio
> >     20
> >     [torvalds@i7 linux]$ cat /proc/sys/vm/dirty_background_ratio
> >     10
> 
> I can confirm I have the same.
> 
> > says that it only starts really throttling writes when you hit 20% of
> > all memory used. You don't say how much memory you have in that
> > machine, but if it's the same one you talked about earlier, it was
> > 24GB. So you can have 4GB of dirty data waiting to be flushed out.
> 
> Correct, 24GB and 4GB.
> 
> > And we *try* to do this per-device backing-dev congestion thing to
> > make things work better, but it generally seems to not work very well.
> > Possibly because of inconsistent write speeds (ie _sometimes_ the SSD
> > does really well, and we want to open up, and then it shuts down).
> > 
> > One thing you can try is to just make the global limits much lower. As in
> > 
> >    echo 2 > /proc/sys/vm/dirty_ratio
> >    echo 1 > /proc/sys/vm/dirty_background_ratio
> 
> I will give that a shot, thank you.

And, after 5H of copying, not a single hang, or USB disconnect, or anything.
Obviously this seems to point to other problems in the code, and I have no
idea which layer is a culprit here, but reducing the buffers absolutely
helped a lot.

Thanks much,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

[toc] | [prev] | [next] | [standalone]

#1533313

On 2016/11/30 8:01, Marc MERLIN wrote:
> And, after 5H of copying, not a single hang, or USB disconnect, or anything.
> Obviously this seems to point to other problems in the code, and I have no
> idea which layer is a culprit here, but reducing the buffers absolutely
> helped a lot.

Maybe you can try commit 63f53dea0c9866e9 ("mm: warn about allocations which stall for too long")
or http://lkml.kernel.org/r/1478416501-10104-1-git-send-email-penguin-kernel@I-love.SAKURA.ne.jp
for finding the culprit.

[toc] | [prev] | [next] | [standalone]

#1532466

On Mon, Nov 28, 2016 at 08:23:15AM +0100, Michal Hocko wrote:
> Marc, could you try this patch please? I think it should be pretty clear
> it should help you but running it through your use case would be more
> than welcome before I ask Greg to take this to the 4.8 stable tree.
> 
> Thanks!
> 
> On Wed 23-11-16 07:34:10, Michal Hocko wrote:
> [...]
> > commit b2ccdcb731b666aa28f86483656c39c5e53828c7
> > Author: Michal Hocko <mhocko@suse.com>
> > Date:   Wed Nov 23 07:26:30 2016 +0100
> > 
> >     mm, oom: stop pre-mature high-order OOM killer invocations
> >     
> >     31e49bfda184 ("mm, oom: protect !costly allocations some more for
> >     !CONFIG_COMPACTION") was an attempt to reduce chances of pre-mature OOM
> >     killer invocation for high order requests. It seemed to work for most
> >     users just fine but it is far from bullet proof and obviously not
> >     sufficient for Marc who has reported pre-mature OOM killer invocations
> >     with 4.8 based kernels. 4.9 will all the compaction improvements seems
> >     to be behaving much better but that would be too intrusive to backport
> >     to 4.8 stable kernels. Instead this patch simply never declares OOM for
> >     !costly high order requests. We rely on order-0 requests to do that in
> >     case we are really out of memory. Order-0 requests are much more common
> >     and so a risk of a livelock without any way forward is highly unlikely.
> >     
> >     Reported-by: Marc MERLIN <marc@merlins.org>
> >     Signed-off-by: Michal Hocko <mhocko@suse.com>

Tested-by: Marc MERLIN <marc@merlins.org>

Marc

> > diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> > index a2214c64ed3c..7401e996009a 100644
> > --- a/mm/page_alloc.c
> > +++ b/mm/page_alloc.c
> > @@ -3161,6 +3161,16 @@ should_compact_retry(struct alloc_context *ac, unsigned int order, int alloc_fla
> >  	if (!order || order > PAGE_ALLOC_COSTLY_ORDER)
> >  		return false;
> >  
> > +#ifdef CONFIG_COMPACTION
> > +	/*
> > +	 * This is a gross workaround to compensate a lack of reliable compaction
> > +	 * operation. We cannot simply go OOM with the current state of the compaction
> > +	 * code because this can lead to pre mature OOM declaration.
> > +	 */
> > +	if (order <= PAGE_ALLOC_COSTLY_ORDER)
> > +		return true;
> > +#endif
> > +
> >  	/*
> >  	 * There are setups with compaction disabled which would prefer to loop
> >  	 * inside the allocator rather than hit the oom killer prematurely.
> > -- 
> > Michal Hocko
> > SUSE Labs
> 
> -- 
> Michal Hocko
> SUSE Labs
> 

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web