Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1310017 > unrolled thread

Re: [RFC 1/3] oom, sysrq: Skip over oom victims and killed tasks

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [RFC 1/3] oom, sysrq: Skip over oom victims and killed tasks Michal Hocko <mhocko@kernel.org> - 2016-01-15 11:20 +0100
    Re: [RFC 1/3] oom, sysrq: Skip over oom victims and killed tasks One Thousand Gnomes <gnomes@lxorguk.ukuu.org.uk> - 2016-01-15 16:40 +0100
      Re: [RFC 1/3] oom, sysrq: Skip over oom victims and killed tasks David Rientjes <rientjes@google.com> - 2016-01-20 00:10 +0100
    Re: [RFC 1/3] oom, sysrq: Skip over oom victims and killed tasks David Rientjes <rientjes@google.com> - 2016-01-20 00:00 +0100
      Re: [RFC 1/3] oom, sysrq: Skip over oom victims and killed tasks Michal Hocko <mhocko@kernel.org> - 2016-01-20 11:00 +0100
        Re: [RFC 1/3] oom, sysrq: Skip over oom victims and killed tasks David Rientjes <rientjes@google.com> - 2016-01-21 01:10 +0100
          Re: [RFC 1/3] oom, sysrq: Skip over oom victims and killed tasks Michal Hocko <mhocko@kernel.org> - 2016-01-21 10:20 +0100

#1310017 — Re: [RFC 1/3] oom, sysrq: Skip over oom victims and killed tasks

On Thu 14-01-16 13:51:16, David Rientjes wrote:
> I think it's time to kill sysrq+F and I'll send those two patches
> unless there is a usecase I'm not aware of.

I have described one in the part you haven't quoted here. Let me repeat:
: Your system might be trashing to the point you are not able to log in
: and resolve the situation in a reasonable time yet you are still not
: OOM. sysrq+f is your only choice then.

Could you clarify why it is better to ditch a potentially usefull
emergency tool rather than to make it work reliably and predictably?
-- 
Michal Hocko
SUSE Labs

[toc] | [next] | [standalone]

#1310224

On Fri, 15 Jan 2016 11:12:18 +0100
Michal Hocko <mhocko@kernel.org> wrote:

> On Thu 14-01-16 13:51:16, David Rientjes wrote:
> > I think it's time to kill sysrq+F and I'll send those two patches
> > unless there is a usecase I'm not aware of.
> 
> I have described one in the part you haven't quoted here. Let me repeat:
> : Your system might be trashing to the point you are not able to log in
> : and resolve the situation in a reasonable time yet you are still not
> : OOM. sysrq+f is your only choice then.
> 
> Could you clarify why it is better to ditch a potentially usefull
> emergency tool rather than to make it work reliably and predictably?

Even if it doesn't work reliably and predictably it is *still* better
than removing it as it works currently. Today we have "might save you a
reboot", the removal turns it into "you'll have to reboot". That's a
regression.

Alan

[toc] | [prev] | [next] | [standalone]

#1312576

On Fri, 15 Jan 2016, One Thousand Gnomes wrote:

> > > I think it's time to kill sysrq+F and I'll send those two patches
> > > unless there is a usecase I'm not aware of.
> > 
> > I have described one in the part you haven't quoted here. Let me repeat:
> > : Your system might be trashing to the point you are not able to log in
> > : and resolve the situation in a reasonable time yet you are still not
> > : OOM. sysrq+f is your only choice then.
> > 
> > Could you clarify why it is better to ditch a potentially usefull
> > emergency tool rather than to make it work reliably and predictably?
> 
> Even if it doesn't work reliably and predictably it is *still* better
> than removing it as it works currently. Today we have "might save you a
> reboot", the removal turns it into "you'll have to reboot". That's a
> regression.
> 

Under what circumstance are you supposing to use sysrq+f in your 
hypothetical?  If you have access to the shell, then you can kill any 
process at random (and you may even be able to make better realtime 
decisions than the oom killer) and it will gain access to memory reserves 
immediately under my proposal when it tries to allocate memory.  The net 
result is that calling the oom killer is no better than you issuing the 
SIGKILL yourself.

This doesn't work if your are supposing to use sysrq+f without the ability 
to get access to the shell.  That's the point, I believe, that Michal has 
raised in this thread.  I'd like to address that issue directly rather 
than requiring human intervention to fix.  If you have deployed a very 
large number of machines to your datacenters, you don't possibly have the 
resources to do this.

[toc] | [prev] | [next] | [standalone]

#1312572

On Fri, 15 Jan 2016, Michal Hocko wrote:

> > I think it's time to kill sysrq+F and I'll send those two patches
> > unless there is a usecase I'm not aware of.
> 
> I have described one in the part you haven't quoted here. Let me repeat:
> : Your system might be trashing to the point you are not able to log in
> : and resolve the situation in a reasonable time yet you are still not
> : OOM. sysrq+f is your only choice then.
> 
> Could you clarify why it is better to ditch a potentially usefull
> emergency tool rather than to make it work reliably and predictably?

I'm concerned about your usecase where the kernel requires admin 
intervention to resolve such an issue and there is nothing in the VM we 
can do to fix it.

If you have a specific test that demonstrates when your usecase is needed, 
please provide it so we can address the issue that it triggers.  I'd 
prefer to fix the issue in the VM rather than require human intervention, 
especially when we try to keep a very large number of machines running in 
our datacenters.

[toc] | [prev] | [next] | [standalone]

#1313047

On Tue 19-01-16 14:57:33, David Rientjes wrote:
> On Fri, 15 Jan 2016, Michal Hocko wrote:
> 
> > > I think it's time to kill sysrq+F and I'll send those two patches
> > > unless there is a usecase I'm not aware of.
> > 
> > I have described one in the part you haven't quoted here. Let me repeat:
> > : Your system might be trashing to the point you are not able to log in
> > : and resolve the situation in a reasonable time yet you are still not
> > : OOM. sysrq+f is your only choice then.
> > 
> > Could you clarify why it is better to ditch a potentially usefull
> > emergency tool rather than to make it work reliably and predictably?
> 
> I'm concerned about your usecase where the kernel requires admin 
> intervention to resolve such an issue and there is nothing in the VM we 
> can do to fix it.
> 
> If you have a specific test that demonstrates when your usecase is needed, 
> please provide it so we can address the issue that it triggers.

No, I do not have a specific load in mind. But let's be realistic. There
will _always_ be corner cases where the VM cannot react properly or in a
timely fashion.

> I'd prefer to fix the issue in the VM rather than require human
> intervention, especially when we try to keep a very large number of
> machines running in our datacenters.

It is always preferable to resolve the mm related issue automagically,
of course. We should strive for robustness as much as possible but that
doesn't mean we should get the only emergency tool out of administrator
hands.

To be honest I really fail to understand your line of argumentation
here. Just that you think that sysrq+f might be not helpful in large
datacenters which you seem to care about, doesn't mean that it is not
helpful in other setups.

Removing the functionality is out of question IMHO so can we please
start discussing how to make it more predictable please?
-- 
Michal Hocko
SUSE Labs

[toc] | [prev] | [next] | [standalone]

#1313715

On Wed, 20 Jan 2016, Michal Hocko wrote:

> No, I do not have a specific load in mind. But let's be realistic. There
> will _always_ be corner cases where the VM cannot react properly or in a
> timely fashion.
> 

Then let's identify it and fix it, like we do with any other bug?  I'm 99% 
certain you are not advocating that human intervention is the ideal 
solution to prevent lengthy stalls or livelocks.

I can't speak for all possible configurations and workloads; the only 
thing we use sysrq+f for is automated testing of the oom killer itself.  
It would help to know of any situations when people actually need to use 
this to solve issues and then fix those issues rather than insisting that 
this is the ideal solution.

> To be honest I really fail to understand your line of argumentation
> here. Just that you think that sysrq+f might be not helpful in large
> datacenters which you seem to care about, doesn't mean that it is not
> helpful in other setups.
> 

This type of message isn't really contributing anything.  You don't have a 
specific load in mind, you can't identify a pending bug that people have 
complained about, you presumably can't show a testcase that demonstrates 
how it's required, yet you're arguing that we should keep a debugging tool 
around because you think somebody somewhere sometime might use it.

 [ I would imagine that users would be unhappy they have to kill processes 
   already, and would have reported how ridiculous it is that they had to
   use sysrq+f, but I haven't seen those bug reports. ]

I want the VM to be responsive, I don't want it to thrash forever, and I 
want it to not require root to trigger a sysrq to have the kernel kill a 
process for the VM to work properly.  We either need to fix the issue that 
causes the unresponsiveness or oom kill processes earlier.  This is very 
simple.

[toc] | [prev] | [next] | [standalone]

#1314019

On Wed 20-01-16 16:01:54, David Rientjes wrote:
> On Wed, 20 Jan 2016, Michal Hocko wrote:
> 
> > No, I do not have a specific load in mind. But let's be realistic. There
> > will _always_ be corner cases where the VM cannot react properly or in a
> > timely fashion.
> > 
> 
> Then let's identify it and fix it, like we do with any other bug?  I'm 99% 
> certain you are not advocating that human intervention is the ideal 
> solution to prevent lengthy stalls or livelocks.

I didn't claim that! Please read what I have written. I consider sysrq+f
as a _last resort_ emergency tool when the system doesn't behave in the
expected way.

> I can't speak for all possible configurations and workloads; the only 
> thing we use sysrq+f for is automated testing of the oom killer itself.  

That is your use case and it is not the one why the this functionality
has been introduced. This is _not a debuggin_ tool. Back in 2005 it has
been added precisely to allow for an immediate intervention while the
system was trashing heavily.

> It would help to know of any situations when people actually need to use 
> this to solve issues and then fix those issues rather than insisting that 
> this is the ideal solution.

I fully agree that such an issues should be investigated and fixed. That
is nothing against having the emergency tool and allow the admin to
intervene right away when it happens.

> > To be honest I really fail to understand your line of argumentation
> > here. Just that you think that sysrq+f might be not helpful in large
> > datacenters which you seem to care about, doesn't mean that it is not
> > helpful in other setups.
> > 
> 
> This type of message isn't really contributing anything.  You don't have a 
> specific load in mind, you can't identify a pending bug that people have 
> complained about, you presumably can't show a testcase that demonstrates 
> how it's required, yet you're arguing that we should keep a debugging tool 
> around because you think somebody somewhere sometime might use it.

Look, I am getting tired of this discussion. You seem to completely
ignore the emergency aspect of sysrq+f just because it doesn't seem to
fit in _your_ particular usecase. I have seen admins using sysrq+f when
a large application got crazy and started trashing to the point when
even ssh to the machine took ages and sysrq+f over serial console was
the only deterministic way to make the system usable. Such things are
still real. Just look at linux-mm ML (just off hand
http://lkml.kernel.org/r/20151221123557.GE3060%40orkisz). You can argue
we should fix them, and I agree but swap/page cache trashing are real
for ages and those are hard problems and very likely to be with us for
some more. Until our MM subsystem and all others that might interfere
are perfect we need a sledge hammer. And if we have a hammer then we
should really make sure it hits something when used rather than hitting
the thin air.

The patch proposed here doesn't make the code more complicated or harder
to maintain. It even doesn't have any side effects outside of sysrq+f
triggered OOM. Your only argument so far was:
"
: It certainly would get TIF_MEMDIE set if it needs to allocate memory
: itself and it calls the oom killer.  That doesn't mean that we should
: kill a different process, though, when the killed process should exit
: and free its memory.  So NACK to the fatal_signal_pending() check here.
"

And that argument is fundamentally broken because killed process is not
guaranteed to exit and free its memory. Moreover sysrq+f is by
definition an async action which might race by passing killed task and
that should deactivate it. The race is quite unlikely but emergency
tools should be as robust/reliable as possible. You also have ignored
my question about what kind of regression would such a change cause.
-- 
Michal Hocko
SUSE Labs

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web