Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1645644 > unrolled thread

[PATCH] mm/oom_kill: count global and memory cgroup oom kills

Back to article view | Back to linux.kernel

Contents

  [PATCH] mm/oom_kill: count global and memory cgroup oom kills Konstantin Khlebnikov <khlebnikov@yandex-team.ru> - 2017-05-19 16:30 +0200
    Re: [PATCH] mm/oom_kill: count global and memory cgroup oom kills Michal Hocko <mhocko@kernel.org> - 2017-05-19 16:40 +0200
    Re: [PATCH] mm/oom_kill: count global and memory cgroup oom kills Roman Guschin <guroan@gmail.com> - 2017-05-19 18:40 +0200
      Re: [PATCH] mm/oom_kill: count global and memory cgroup oom kills Konstantin Khlebnikov <khlebnikov@yandex-team.ru> - 2017-05-22 11:20 +0200
        Re: [PATCH] mm/oom_kill: count global and memory cgroup oom kills Roman Guschin <guroan@gmail.com> - 2017-05-22 20:10 +0200
        Re: [PATCH] mm/oom_kill: count global and memory cgroup oom kills David Rientjes <rientjes@google.com> - 2017-05-23 10:00 +0200
          Re: [PATCH] mm/oom_kill: count global and memory cgroup oom kills Konstantin Khlebnikov <khlebnikov@yandex-team.ru> - 2017-05-23 12:40 +0200
            Re: [PATCH] mm/oom_kill: count global and memory cgroup oom kills David Rientjes <rientjes@google.com> - 2017-05-24 22:50 +0200
    Re: [PATCH] mm/oom_kill: count global and memory cgroup oom kills Michal Hocko <mhocko@kernel.org> - 2017-05-23 09:30 +0200
      Re: [PATCH] mm/oom_kill: count global and memory cgroup oom kills Konstantin Khlebnikov <khlebnikov@yandex-team.ru> - 2017-05-23 13:10 +0200
        Re: [PATCH] mm/oom_kill: count global and memory cgroup oom kills Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp> - 2017-05-23 13:30 +0200

#1645644 — [PATCH] mm/oom_kill: count global and memory cgroup oom kills

Show count of global oom killer invocations in /proc/vmstat and
count of oom kills inside memory cgroup in knob "memory.events"
(in memory.oom_control for v1 cgroup).

Also describe difference between "oom" and "oom_kill" in memory
cgroup documentation. Currently oom in memory cgroup kills tasks
iff shortage has happened inside page fault.

These counters helps in monitoring oom kills - for now
the only way is grepping for magic words in kernel log.

Signed-off-by: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
---
 Documentation/cgroup-v2.txt   |   12 +++++++++++-
 include/linux/memcontrol.h    |    1 +
 include/linux/vm_event_item.h |    1 +
 mm/memcontrol.c               |    2 ++
 mm/oom_kill.c                 |    6 ++++++
 mm/vmstat.c                   |    1 +
 6 files changed, 22 insertions(+), 1 deletion(-)

diff --git a/Documentation/cgroup-v2.txt b/Documentation/cgroup-v2.txt
index dc5e2dcdbef4..a742008d76aa 100644
--- a/Documentation/cgroup-v2.txt
+++ b/Documentation/cgroup-v2.txt
@@ -830,9 +830,19 @@ PAGE_SIZE multiple when read back.
 
 	  oom
 
+		The number of time the cgroup's memory usage was
+		reached the limit and allocation was about to fail.
+		Result could be oom kill, -ENOMEM from any syscall or
+		completely ignored in cases like disk readahead.
+		For now oom in memory cgroup kills tasks iff shortage
+		has happened inside page fault.
+
+	  oom_kill
+
 		The number of times the OOM killer has been invoked in
 		the cgroup.  This may not exactly match the number of
-		processes killed but should generally be close.
+		processes killed but should generally be close:	each
+		invocation could kill several processes at once.
 
   memory.stat
 
diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 899949bbb2f9..2cdcebb78b58 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -55,6 +55,7 @@ enum memcg_event_item {
 	MEMCG_HIGH,
 	MEMCG_MAX,
 	MEMCG_OOM,
+	MEMCG_OOM_KILL,
 	MEMCG_NR_EVENTS,
 };
 
diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h
index d84ae90ccd5c..1707e0a7d943 100644
--- a/include/linux/vm_event_item.h
+++ b/include/linux/vm_event_item.h
@@ -41,6 +41,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 		KSWAPD_LOW_WMARK_HIT_QUICKLY, KSWAPD_HIGH_WMARK_HIT_QUICKLY,
 		PAGEOUTRUN, PGROTATED,
 		DROP_PAGECACHE, DROP_SLAB,
+		OOM_KILL,
 #ifdef CONFIG_NUMA_BALANCING
 		NUMA_PTE_UPDATES,
 		NUMA_HUGE_PTE_UPDATES,
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 94172089f52f..416024837b81 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -3574,6 +3574,7 @@ static int mem_cgroup_oom_control_read(struct seq_file *sf, void *v)
 
 	seq_printf(sf, "oom_kill_disable %d\n", memcg->oom_kill_disable);
 	seq_printf(sf, "under_oom %d\n", (bool)memcg->under_oom);
+	seq_printf(sf, "oom_kill %lu\n", memcg_sum_events(memcg, MEMCG_OOM_KILL));
 	return 0;
 }
 
@@ -5165,6 +5166,7 @@ static int memory_events_show(struct seq_file *m, void *v)
 	seq_printf(m, "high %lu\n", memcg_sum_events(memcg, MEMCG_HIGH));
 	seq_printf(m, "max %lu\n", memcg_sum_events(memcg, MEMCG_MAX));
 	seq_printf(m, "oom %lu\n", memcg_sum_events(memcg, MEMCG_OOM));
+	seq_printf(m, "oom_kill %lu\n", memcg_sum_events(memcg, MEMCG_OOM_KILL));
 
 	return 0;
 }
diff --git a/mm/oom_kill.c b/mm/oom_kill.c
index 04c9143a8625..c50bff3c3409 100644
--- a/mm/oom_kill.c
+++ b/mm/oom_kill.c
@@ -873,6 +873,12 @@ static void oom_kill_process(struct oom_control *oc, const char *message)
 		victim = p;
 	}
 
+	/* Raise event before sending signal: reaper must see this */
+	if (!is_memcg_oom(oc))
+		count_vm_event(OOM_KILL);
+	else
+		mem_cgroup_event(oc->memcg, MEMCG_OOM_KILL);
+
 	/* Get a reference to safely compare mm after task_unlock(victim) */
 	mm = victim->mm;
 	mmgrab(mm);
diff --git a/mm/vmstat.c b/mm/vmstat.c
index 76f73670200a..fe80b81a86e0 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -1018,6 +1018,7 @@ const char * const vmstat_text[] = {
 
 	"drop_pagecache",
 	"drop_slab",
+	"oom_kill",
 
 #ifdef CONFIG_NUMA_BALANCING
 	"numa_pte_updates",

[toc] | [next] | [standalone]

#1645655

[CC Vladimir and Johannes as this involves memcg]

i will get to this early next week but from a quick look it seems
acceptable.

On Fri 19-05-17 17:22:30, Konstantin Khlebnikov wrote:
> Show count of global oom killer invocations in /proc/vmstat and
> count of oom kills inside memory cgroup in knob "memory.events"
> (in memory.oom_control for v1 cgroup).
> 
> Also describe difference between "oom" and "oom_kill" in memory
> cgroup documentation. Currently oom in memory cgroup kills tasks
> iff shortage has happened inside page fault.
> 
> These counters helps in monitoring oom kills - for now
> the only way is grepping for magic words in kernel log.
> 
> Signed-off-by: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
> ---
>  Documentation/cgroup-v2.txt   |   12 +++++++++++-
>  include/linux/memcontrol.h    |    1 +
>  include/linux/vm_event_item.h |    1 +
>  mm/memcontrol.c               |    2 ++
>  mm/oom_kill.c                 |    6 ++++++
>  mm/vmstat.c                   |    1 +
>  6 files changed, 22 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/cgroup-v2.txt b/Documentation/cgroup-v2.txt
> index dc5e2dcdbef4..a742008d76aa 100644
> --- a/Documentation/cgroup-v2.txt
> +++ b/Documentation/cgroup-v2.txt
> @@ -830,9 +830,19 @@ PAGE_SIZE multiple when read back.
>  
>  	  oom
>  
> +		The number of time the cgroup's memory usage was
> +		reached the limit and allocation was about to fail.
> +		Result could be oom kill, -ENOMEM from any syscall or
> +		completely ignored in cases like disk readahead.
> +		For now oom in memory cgroup kills tasks iff shortage
> +		has happened inside page fault.
> +
> +	  oom_kill
> +
>  		The number of times the OOM killer has been invoked in
>  		the cgroup.  This may not exactly match the number of
> -		processes killed but should generally be close.
> +		processes killed but should generally be close:	each
> +		invocation could kill several processes at once.
>  
>    memory.stat
>  
> diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
> index 899949bbb2f9..2cdcebb78b58 100644
> --- a/include/linux/memcontrol.h
> +++ b/include/linux/memcontrol.h
> @@ -55,6 +55,7 @@ enum memcg_event_item {
>  	MEMCG_HIGH,
>  	MEMCG_MAX,
>  	MEMCG_OOM,
> +	MEMCG_OOM_KILL,
>  	MEMCG_NR_EVENTS,
>  };
>  
> diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h
> index d84ae90ccd5c..1707e0a7d943 100644
> --- a/include/linux/vm_event_item.h
> +++ b/include/linux/vm_event_item.h
> @@ -41,6 +41,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
>  		KSWAPD_LOW_WMARK_HIT_QUICKLY, KSWAPD_HIGH_WMARK_HIT_QUICKLY,
>  		PAGEOUTRUN, PGROTATED,
>  		DROP_PAGECACHE, DROP_SLAB,
> +		OOM_KILL,
>  #ifdef CONFIG_NUMA_BALANCING
>  		NUMA_PTE_UPDATES,
>  		NUMA_HUGE_PTE_UPDATES,
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index 94172089f52f..416024837b81 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -3574,6 +3574,7 @@ static int mem_cgroup_oom_control_read(struct seq_file *sf, void *v)
>  
>  	seq_printf(sf, "oom_kill_disable %d\n", memcg->oom_kill_disable);
>  	seq_printf(sf, "under_oom %d\n", (bool)memcg->under_oom);
> +	seq_printf(sf, "oom_kill %lu\n", memcg_sum_events(memcg, MEMCG_OOM_KILL));
>  	return 0;
>  }
>  
> @@ -5165,6 +5166,7 @@ static int memory_events_show(struct seq_file *m, void *v)
>  	seq_printf(m, "high %lu\n", memcg_sum_events(memcg, MEMCG_HIGH));
>  	seq_printf(m, "max %lu\n", memcg_sum_events(memcg, MEMCG_MAX));
>  	seq_printf(m, "oom %lu\n", memcg_sum_events(memcg, MEMCG_OOM));
> +	seq_printf(m, "oom_kill %lu\n", memcg_sum_events(memcg, MEMCG_OOM_KILL));
>  
>  	return 0;
>  }
> diff --git a/mm/oom_kill.c b/mm/oom_kill.c
> index 04c9143a8625..c50bff3c3409 100644
> --- a/mm/oom_kill.c
> +++ b/mm/oom_kill.c
> @@ -873,6 +873,12 @@ static void oom_kill_process(struct oom_control *oc, const char *message)
>  		victim = p;
>  	}
>  
> +	/* Raise event before sending signal: reaper must see this */
> +	if (!is_memcg_oom(oc))
> +		count_vm_event(OOM_KILL);
> +	else
> +		mem_cgroup_event(oc->memcg, MEMCG_OOM_KILL);
> +
>  	/* Get a reference to safely compare mm after task_unlock(victim) */
>  	mm = victim->mm;
>  	mmgrab(mm);
> diff --git a/mm/vmstat.c b/mm/vmstat.c
> index 76f73670200a..fe80b81a86e0 100644
> --- a/mm/vmstat.c
> +++ b/mm/vmstat.c
> @@ -1018,6 +1018,7 @@ const char * const vmstat_text[] = {
>  
>  	"drop_pagecache",
>  	"drop_slab",
> +	"oom_kill",
>  
>  #ifdef CONFIG_NUMA_BALANCING
>  	"numa_pte_updates",
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

-- 
Michal Hocko
SUSE Labs

[toc] | [prev] | [next] | [standalone]

#1645753

2017-05-19 15:22 GMT+01:00 Konstantin Khlebnikov <khlebnikov@yandex-team.ru>:
> Show count of global oom killer invocations in /proc/vmstat and
> count of oom kills inside memory cgroup in knob "memory.events"
> (in memory.oom_control for v1 cgroup).
>
> Also describe difference between "oom" and "oom_kill" in memory
> cgroup documentation. Currently oom in memory cgroup kills tasks
> iff shortage has happened inside page fault.
>
> These counters helps in monitoring oom kills - for now
> the only way is grepping for magic words in kernel log.
>
> Signed-off-by: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
> ---
>  Documentation/cgroup-v2.txt   |   12 +++++++++++-
>  include/linux/memcontrol.h    |    1 +
>  include/linux/vm_event_item.h |    1 +
>  mm/memcontrol.c               |    2 ++
>  mm/oom_kill.c                 |    6 ++++++
>  mm/vmstat.c                   |    1 +
>  6 files changed, 22 insertions(+), 1 deletion(-)
>
> diff --git a/Documentation/cgroup-v2.txt b/Documentation/cgroup-v2.txt
> index dc5e2dcdbef4..a742008d76aa 100644
> --- a/Documentation/cgroup-v2.txt
> +++ b/Documentation/cgroup-v2.txt
> @@ -830,9 +830,19 @@ PAGE_SIZE multiple when read back.
>
>           oom
>
> +               The number of time the cgroup's memory usage was
> +               reached the limit and allocation was about to fail.
> +               Result could be oom kill, -ENOMEM from any syscall or
> +               completely ignored in cases like disk readahead.
> +               For now oom in memory cgroup kills tasks iff shortage
> +               has happened inside page fault.

From a user's point of view the difference between "oom" and "max"
becomes really vague here,
assuming that "max" is described almost in the same words:

"The number of times the cgroup's memory usage was
about to go over the max boundary.  If direct reclaim
fails to bring it down, the OOM killer is invoked."

I wonder, if it's better to fix the existing "oom" value  to show what
it has to show, according to docs,
rather than to introduce a new one?

> +
> +         oom_kill
> +
>                 The number of times the OOM killer has been invoked in
>                 the cgroup.  This may not exactly match the number of
> -               processes killed but should generally be close.
> +               processes killed but should generally be close: each
> +               invocation could kill several processes at once.
>
>    memory.stat
>

[toc] | [prev] | [next] | [standalone]

#1646636

On 19.05.2017 19:34, Roman Guschin wrote:
> 2017-05-19 15:22 GMT+01:00 Konstantin Khlebnikov <khlebnikov@yandex-team.ru>:
>> Show count of global oom killer invocations in /proc/vmstat and
>> count of oom kills inside memory cgroup in knob "memory.events"
>> (in memory.oom_control for v1 cgroup).
>>
>> Also describe difference between "oom" and "oom_kill" in memory
>> cgroup documentation. Currently oom in memory cgroup kills tasks
>> iff shortage has happened inside page fault.
>>
>> These counters helps in monitoring oom kills - for now
>> the only way is grepping for magic words in kernel log.
>>
>> Signed-off-by: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
>> ---
>>   Documentation/cgroup-v2.txt   |   12 +++++++++++-
>>   include/linux/memcontrol.h    |    1 +
>>   include/linux/vm_event_item.h |    1 +
>>   mm/memcontrol.c               |    2 ++
>>   mm/oom_kill.c                 |    6 ++++++
>>   mm/vmstat.c                   |    1 +
>>   6 files changed, 22 insertions(+), 1 deletion(-)
>>
>> diff --git a/Documentation/cgroup-v2.txt b/Documentation/cgroup-v2.txt
>> index dc5e2dcdbef4..a742008d76aa 100644
>> --- a/Documentation/cgroup-v2.txt
>> +++ b/Documentation/cgroup-v2.txt
>> @@ -830,9 +830,19 @@ PAGE_SIZE multiple when read back.
>>
>>            oom
>>
>> +               The number of time the cgroup's memory usage was
>> +               reached the limit and allocation was about to fail.
>> +               Result could be oom kill, -ENOMEM from any syscall or
>> +               completely ignored in cases like disk readahead.
>> +               For now oom in memory cgroup kills tasks iff shortage
>> +               has happened inside page fault.
> 
>  From a user's point of view the difference between "oom" and "max"
> becomes really vague here,
> assuming that "max" is described almost in the same words:
> 
> "The number of times the cgroup's memory usage was
> about to go over the max boundary.  If direct reclaim
> fails to bring it down, the OOM killer is invoked."
> 
> I wonder, if it's better to fix the existing "oom" value  to show what
> it has to show, according to docs,
> rather than to introduce a new one?
> 

Nope, they are different. I think we should rephase documentation somehow

low - count of reclaims below low level
high - count of post-allocation reclaims above high level
max - count of direct reclaims
oom - count of failed direct reclaims
oom_kill - count of oom killer invocations and killed processes

>> +
>> +         oom_kill
>> +
>>                  The number of times the OOM killer has been invoked in
>>                  the cgroup.  This may not exactly match the number of
>> -               processes killed but should generally be close.
>> +               processes killed but should generally be close: each
>> +               invocation could kill several processes at once.
>>
>>     memory.stat
>>

[toc] | [prev] | [next] | [standalone]

#1647230

2017-05-22 10:11 GMT+01:00 Konstantin Khlebnikov <khlebnikov@yandex-team.ru>:
>
>
> On 19.05.2017 19:34, Roman Guschin wrote:
>>
>> 2017-05-19 15:22 GMT+01:00 Konstantin Khlebnikov
>> <khlebnikov@yandex-team.ru>:
>>  From a user's point of view the difference between "oom" and "max"
>> becomes really vague here,
>> assuming that "max" is described almost in the same words:
>>
>> "The number of times the cgroup's memory usage was
>> about to go over the max boundary.  If direct reclaim
>> fails to bring it down, the OOM killer is invoked."
>>
>> I wonder, if it's better to fix the existing "oom" value  to show what
>> it has to show, according to docs,
>> rather than to introduce a new one?
>>
>
> Nope, they are different. I think we should rephase documentation somehow
>
> low - count of reclaims below low level
> high - count of post-allocation reclaims above high level
> max - count of direct reclaims
> oom - count of failed direct reclaims
> oom_kill - count of oom killer invocations and killed processes

Definitely worth it.

Also, I would prefer to reserve "oom" for number of oom victims,
and introduce something like "reclaim_failed".
It will be consistent with existing vmstat.

Thanks!

[toc] | [prev] | [next] | [standalone]

#1647752

On Mon, 22 May 2017, Konstantin Khlebnikov wrote:

> Nope, they are different. I think we should rephase documentation somehow
> 
> low - count of reclaims below low level
> high - count of post-allocation reclaims above high level
> max - count of direct reclaims
> oom - count of failed direct reclaims
> oom_kill - count of oom killer invocations and killed processes
> 

In our kernel, we've maintained counts of oom kills per memcg for years as 
part of memory.oom_control for memcg v1, but we've also found it helpful 
to complement that with another count that specifies the number of 
processes oom killed that were attached to that exact memcg.

In your patch, oom_kill in memory.oom_control specifies that number of oom 
events that resulted in an oom kill of a process from that hierarchy, but 
not the number of processes killed from a specific memcg (the difference 
between oc->memcg and mem_cgroup_from_task(victim)).  Not sure if you 
would also find it helpful.

[toc] | [prev] | [next] | [standalone]

#1647925

On 23.05.2017 10:49, David Rientjes wrote:
> On Mon, 22 May 2017, Konstantin Khlebnikov wrote:
> 
>> Nope, they are different. I think we should rephase documentation somehow
>>
>> low - count of reclaims below low level
>> high - count of post-allocation reclaims above high level
>> max - count of direct reclaims
>> oom - count of failed direct reclaims
>> oom_kill - count of oom killer invocations and killed processes
>>
> 
> In our kernel, we've maintained counts of oom kills per memcg for years as
> part of memory.oom_control for memcg v1, but we've also found it helpful
> to complement that with another count that specifies the number of
> processes oom killed that were attached to that exact memcg.
> 
> In your patch, oom_kill in memory.oom_control specifies that number of oom
> events that resulted in an oom kill of a process from that hierarchy, but
> not the number of processes killed from a specific memcg (the difference
> between oc->memcg and mem_cgroup_from_task(victim)).  Not sure if you
> would also find it helpful.
> 

This is worth addition. Let's call it "oom_victim" for short.

It allows to locate leaky part if they are spread over sub-containers within common limit.
But doesn't tell which limit caused this kill. For hierarchical limits this might be not so easy.

I think oom_kill better suits for automatic actions - restart affected hierarchy, increase limits, e.t.c.
But oom_victim allows to determine container affected by global oom killer.

So, probably it's worth to merge them together and increment oom_kill by global killer for victim memcg:

	if (!is_memcg_oom(oc)) {
		count_vm_event(OOM_KILL);
		mem_cgroup_count_vm_event(mm, OOM_KILL);
	} else
		mem_cgroup_event(oc->memcg, OOM_KILL);

[toc] | [prev] | [next] | [standalone]

#1649940

On Tue, 23 May 2017, Konstantin Khlebnikov wrote:

> This is worth addition. Let's call it "oom_victim" for short.
> 
> It allows to locate leaky part if they are spread over sub-containers within
> common limit.
> But doesn't tell which limit caused this kill. For hierarchical limits this
> might be not so easy.
> 
> I think oom_kill better suits for automatic actions - restart affected
> hierarchy, increase limits, e.t.c.
> But oom_victim allows to determine container affected by global oom killer.
> 
> So, probably it's worth to merge them together and increment oom_kill by
> global killer for victim memcg:
> 
> 	if (!is_memcg_oom(oc)) {
> 		count_vm_event(OOM_KILL);
> 		mem_cgroup_count_vm_event(mm, OOM_KILL);
> 	} else
> 		mem_cgroup_event(oc->memcg, OOM_KILL);
> 

Our complete solution is that we have a complementary 
memory.oom_kill_control that allows users to register for eventfd(2) 
notification when the kernel oom killer kills a victim, but this is 
because we have had complete support for userspace oom handling for years.  
When read, it exports three classes of information:

 - the "total" (hierarchical) and "local" (memcg specific) number of oom
   kills for system oom conditions (overcommit),

 - the "total" and "local" number of oom kills for memcg oom conditions, 
   and
 
 - the total number of processes in the hierarchy where an oom victim was
   reaped successfully and unsuccessfully.

One benefit of this is that it prevents us from having to scrape the 
kernel log for oom events which has been troublesome in the past, but 
userspace can easily do so when the eventfd triggers for the kill 
notification.

[toc] | [prev] | [next] | [standalone]

#1647721

On Fri 19-05-17 17:22:30, Konstantin Khlebnikov wrote:
> Show count of global oom killer invocations in /proc/vmstat and
> count of oom kills inside memory cgroup in knob "memory.events"
> (in memory.oom_control for v1 cgroup).
> 
> Also describe difference between "oom" and "oom_kill" in memory
> cgroup documentation. Currently oom in memory cgroup kills tasks
> iff shortage has happened inside page fault.
> 
> These counters helps in monitoring oom kills - for now
> the only way is grepping for magic words in kernel log.

Have you considered adding memcg's oom alternative for the global case
as well. It would be useful to see how many times we hit the OOM
condition without killing anything. That could help debugging issues
when the OOM killer cannot be invoked (e.g. GFP_NO{FS,IO} contextx)
and the system cannot get out of the oom situation.
 
> Signed-off-by: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>

Acked-by: Michal Hocko <mhocko@suse.com>

> ---
>  Documentation/cgroup-v2.txt   |   12 +++++++++++-
>  include/linux/memcontrol.h    |    1 +
>  include/linux/vm_event_item.h |    1 +
>  mm/memcontrol.c               |    2 ++
>  mm/oom_kill.c                 |    6 ++++++
>  mm/vmstat.c                   |    1 +
>  6 files changed, 22 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/cgroup-v2.txt b/Documentation/cgroup-v2.txt
> index dc5e2dcdbef4..a742008d76aa 100644
> --- a/Documentation/cgroup-v2.txt
> +++ b/Documentation/cgroup-v2.txt
> @@ -830,9 +830,19 @@ PAGE_SIZE multiple when read back.
>  
>  	  oom
>  
> +		The number of time the cgroup's memory usage was
> +		reached the limit and allocation was about to fail.
> +		Result could be oom kill, -ENOMEM from any syscall or
> +		completely ignored in cases like disk readahead.
> +		For now oom in memory cgroup kills tasks iff shortage
> +		has happened inside page fault.
> +
> +	  oom_kill
> +
>  		The number of times the OOM killer has been invoked in
>  		the cgroup.  This may not exactly match the number of
> -		processes killed but should generally be close.
> +		processes killed but should generally be close:	each
> +		invocation could kill several processes at once.
>  
>    memory.stat
>  
> diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
> index 899949bbb2f9..2cdcebb78b58 100644
> --- a/include/linux/memcontrol.h
> +++ b/include/linux/memcontrol.h
> @@ -55,6 +55,7 @@ enum memcg_event_item {
>  	MEMCG_HIGH,
>  	MEMCG_MAX,
>  	MEMCG_OOM,
> +	MEMCG_OOM_KILL,
>  	MEMCG_NR_EVENTS,
>  };
>  
> diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h
> index d84ae90ccd5c..1707e0a7d943 100644
> --- a/include/linux/vm_event_item.h
> +++ b/include/linux/vm_event_item.h
> @@ -41,6 +41,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
>  		KSWAPD_LOW_WMARK_HIT_QUICKLY, KSWAPD_HIGH_WMARK_HIT_QUICKLY,
>  		PAGEOUTRUN, PGROTATED,
>  		DROP_PAGECACHE, DROP_SLAB,
> +		OOM_KILL,
>  #ifdef CONFIG_NUMA_BALANCING
>  		NUMA_PTE_UPDATES,
>  		NUMA_HUGE_PTE_UPDATES,
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index 94172089f52f..416024837b81 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -3574,6 +3574,7 @@ static int mem_cgroup_oom_control_read(struct seq_file *sf, void *v)
>  
>  	seq_printf(sf, "oom_kill_disable %d\n", memcg->oom_kill_disable);
>  	seq_printf(sf, "under_oom %d\n", (bool)memcg->under_oom);
> +	seq_printf(sf, "oom_kill %lu\n", memcg_sum_events(memcg, MEMCG_OOM_KILL));
>  	return 0;
>  }
>  
> @@ -5165,6 +5166,7 @@ static int memory_events_show(struct seq_file *m, void *v)
>  	seq_printf(m, "high %lu\n", memcg_sum_events(memcg, MEMCG_HIGH));
>  	seq_printf(m, "max %lu\n", memcg_sum_events(memcg, MEMCG_MAX));
>  	seq_printf(m, "oom %lu\n", memcg_sum_events(memcg, MEMCG_OOM));
> +	seq_printf(m, "oom_kill %lu\n", memcg_sum_events(memcg, MEMCG_OOM_KILL));
>  
>  	return 0;
>  }
> diff --git a/mm/oom_kill.c b/mm/oom_kill.c
> index 04c9143a8625..c50bff3c3409 100644
> --- a/mm/oom_kill.c
> +++ b/mm/oom_kill.c
> @@ -873,6 +873,12 @@ static void oom_kill_process(struct oom_control *oc, const char *message)
>  		victim = p;
>  	}
>  
> +	/* Raise event before sending signal: reaper must see this */
> +	if (!is_memcg_oom(oc))
> +		count_vm_event(OOM_KILL);
> +	else
> +		mem_cgroup_event(oc->memcg, MEMCG_OOM_KILL);
> +
>  	/* Get a reference to safely compare mm after task_unlock(victim) */
>  	mm = victim->mm;
>  	mmgrab(mm);
> diff --git a/mm/vmstat.c b/mm/vmstat.c
> index 76f73670200a..fe80b81a86e0 100644
> --- a/mm/vmstat.c
> +++ b/mm/vmstat.c
> @@ -1018,6 +1018,7 @@ const char * const vmstat_text[] = {
>  
>  	"drop_pagecache",
>  	"drop_slab",
> +	"oom_kill",
>  
>  #ifdef CONFIG_NUMA_BALANCING
>  	"numa_pte_updates",
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

-- 
Michal Hocko
SUSE Labs

[toc] | [prev] | [next] | [standalone]

#1647942

On 23.05.2017 10:27, Michal Hocko wrote:
> On Fri 19-05-17 17:22:30, Konstantin Khlebnikov wrote:
>> Show count of global oom killer invocations in /proc/vmstat and
>> count of oom kills inside memory cgroup in knob "memory.events"
>> (in memory.oom_control for v1 cgroup).
>>
>> Also describe difference between "oom" and "oom_kill" in memory
>> cgroup documentation. Currently oom in memory cgroup kills tasks
>> iff shortage has happened inside page fault.
>>
>> These counters helps in monitoring oom kills - for now
>> the only way is grepping for magic words in kernel log.
> 
> Have you considered adding memcg's oom alternative for the global case
> as well. It would be useful to see how many times we hit the OOM
> condition without killing anything. That could help debugging issues
> when the OOM killer cannot be invoked (e.g. GFP_NO{FS,IO} contextx)
> and the system cannot get out of the oom situation.

I think present warn_alloc() should be enough for debugging,
maybe it should taint kernel in some cases to give a hint for future warnings/bugs.

>   
>> Signed-off-by: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
> 
> Acked-by: Michal Hocko <mhocko@suse.com>
> 
>> ---
>>   Documentation/cgroup-v2.txt   |   12 +++++++++++-
>>   include/linux/memcontrol.h    |    1 +
>>   include/linux/vm_event_item.h |    1 +
>>   mm/memcontrol.c               |    2 ++
>>   mm/oom_kill.c                 |    6 ++++++
>>   mm/vmstat.c                   |    1 +
>>   6 files changed, 22 insertions(+), 1 deletion(-)
>>
>> diff --git a/Documentation/cgroup-v2.txt b/Documentation/cgroup-v2.txt
>> index dc5e2dcdbef4..a742008d76aa 100644
>> --- a/Documentation/cgroup-v2.txt
>> +++ b/Documentation/cgroup-v2.txt
>> @@ -830,9 +830,19 @@ PAGE_SIZE multiple when read back.
>>   
>>   	  oom
>>   
>> +		The number of time the cgroup's memory usage was
>> +		reached the limit and allocation was about to fail.
>> +		Result could be oom kill, -ENOMEM from any syscall or
>> +		completely ignored in cases like disk readahead.
>> +		For now oom in memory cgroup kills tasks iff shortage
>> +		has happened inside page fault.
>> +
>> +	  oom_kill
>> +
>>   		The number of times the OOM killer has been invoked in
>>   		the cgroup.  This may not exactly match the number of
>> -		processes killed but should generally be close.
>> +		processes killed but should generally be close:	each
>> +		invocation could kill several processes at once.
>>   
>>     memory.stat
>>   
>> diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
>> index 899949bbb2f9..2cdcebb78b58 100644
>> --- a/include/linux/memcontrol.h
>> +++ b/include/linux/memcontrol.h
>> @@ -55,6 +55,7 @@ enum memcg_event_item {
>>   	MEMCG_HIGH,
>>   	MEMCG_MAX,
>>   	MEMCG_OOM,
>> +	MEMCG_OOM_KILL,
>>   	MEMCG_NR_EVENTS,
>>   };
>>   
>> diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h
>> index d84ae90ccd5c..1707e0a7d943 100644
>> --- a/include/linux/vm_event_item.h
>> +++ b/include/linux/vm_event_item.h
>> @@ -41,6 +41,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
>>   		KSWAPD_LOW_WMARK_HIT_QUICKLY, KSWAPD_HIGH_WMARK_HIT_QUICKLY,
>>   		PAGEOUTRUN, PGROTATED,
>>   		DROP_PAGECACHE, DROP_SLAB,
>> +		OOM_KILL,
>>   #ifdef CONFIG_NUMA_BALANCING
>>   		NUMA_PTE_UPDATES,
>>   		NUMA_HUGE_PTE_UPDATES,
>> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
>> index 94172089f52f..416024837b81 100644
>> --- a/mm/memcontrol.c
>> +++ b/mm/memcontrol.c
>> @@ -3574,6 +3574,7 @@ static int mem_cgroup_oom_control_read(struct seq_file *sf, void *v)
>>   
>>   	seq_printf(sf, "oom_kill_disable %d\n", memcg->oom_kill_disable);
>>   	seq_printf(sf, "under_oom %d\n", (bool)memcg->under_oom);
>> +	seq_printf(sf, "oom_kill %lu\n", memcg_sum_events(memcg, MEMCG_OOM_KILL));
>>   	return 0;
>>   }
>>   
>> @@ -5165,6 +5166,7 @@ static int memory_events_show(struct seq_file *m, void *v)
>>   	seq_printf(m, "high %lu\n", memcg_sum_events(memcg, MEMCG_HIGH));
>>   	seq_printf(m, "max %lu\n", memcg_sum_events(memcg, MEMCG_MAX));
>>   	seq_printf(m, "oom %lu\n", memcg_sum_events(memcg, MEMCG_OOM));
>> +	seq_printf(m, "oom_kill %lu\n", memcg_sum_events(memcg, MEMCG_OOM_KILL));
>>   
>>   	return 0;
>>   }
>> diff --git a/mm/oom_kill.c b/mm/oom_kill.c
>> index 04c9143a8625..c50bff3c3409 100644
>> --- a/mm/oom_kill.c
>> +++ b/mm/oom_kill.c
>> @@ -873,6 +873,12 @@ static void oom_kill_process(struct oom_control *oc, const char *message)
>>   		victim = p;
>>   	}
>>   
>> +	/* Raise event before sending signal: reaper must see this */
>> +	if (!is_memcg_oom(oc))
>> +		count_vm_event(OOM_KILL);
>> +	else
>> +		mem_cgroup_event(oc->memcg, MEMCG_OOM_KILL);
>> +
>>   	/* Get a reference to safely compare mm after task_unlock(victim) */
>>   	mm = victim->mm;
>>   	mmgrab(mm);
>> diff --git a/mm/vmstat.c b/mm/vmstat.c
>> index 76f73670200a..fe80b81a86e0 100644
>> --- a/mm/vmstat.c
>> +++ b/mm/vmstat.c
>> @@ -1018,6 +1018,7 @@ const char * const vmstat_text[] = {
>>   
>>   	"drop_pagecache",
>>   	"drop_slab",
>> +	"oom_kill",
>>   
>>   #ifdef CONFIG_NUMA_BALANCING
>>   	"numa_pte_updates",
>>
>> --
>> To unsubscribe, send a message with 'unsubscribe linux-mm' in
>> the body to majordomo@kvack.org.  For more info on Linux MM,
>> see: http://www.linux-mm.org/ .
>> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
> 

[toc] | [prev] | [next] | [standalone]

#1647955

On 2017/05/23 20:05, Konstantin Khlebnikov wrote:
> On 23.05.2017 10:27, Michal Hocko wrote:
>> On Fri 19-05-17 17:22:30, Konstantin Khlebnikov wrote:
>>> Show count of global oom killer invocations in /proc/vmstat and
>>> count of oom kills inside memory cgroup in knob "memory.events"
>>> (in memory.oom_control for v1 cgroup).
>>>
>>> Also describe difference between "oom" and "oom_kill" in memory
>>> cgroup documentation. Currently oom in memory cgroup kills tasks
>>> iff shortage has happened inside page fault.
>>>
>>> These counters helps in monitoring oom kills - for now
>>> the only way is grepping for magic words in kernel log.
>>
>> Have you considered adding memcg's oom alternative for the global case
>> as well. It would be useful to see how many times we hit the OOM
>> condition without killing anything. That could help debugging issues
>> when the OOM killer cannot be invoked (e.g. GFP_NO{FS,IO} contextx)
>> and the system cannot get out of the oom situation.
> 
> I think present warn_alloc() should be enough for debugging,
> maybe it should taint kernel in some cases to give a hint for future warnings/bugs.
> 

I don't think warn_alloc() is enough. We can fail to warn using warn_alloc(), see
http://lkml.kernel.org/r/1495331504-12480-1-git-send-email-penguin-kernel@I-love.SAKURA.ne.jp .

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web