Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1244702 > unrolled thread

[PATCH 1/1] mm: vmstat: Add OOM victims count in vmstat counter

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  [PATCH 1/1] mm: vmstat: Add OOM victims count in vmstat counter Pintu Kumar <pintu.k@samsung.com> - 2015-10-12 15:50 +0200
    RE: [PATCH 1/1] mm: vmstat: Add OOM victims count in vmstat counter PINTU KUMAR <pintu.k@samsung.com> - 2015-10-12 16:50 +0200
    [RESEND PATCH 1/1] mm: vmstat: Add OOM victims count in vmstat counter Pintu Kumar <pintu.k@samsung.com> - 2015-10-12 16:50 +0200
      Re: [RESEND PATCH 1/1] mm: vmstat: Add OOM victims count in vmstat  counter David Rientjes <rientjes@google.com> - 2015-10-14 05:10 +0200
        RE: [RESEND PATCH 1/1] mm: vmstat: Add OOM victims count in vmstat  counter PINTU KUMAR <pintu.k@samsung.com> - 2015-10-14 15:50 +0200
          RE: [RESEND PATCH 1/1] mm: vmstat: Add OOM victims count in vmstat  counter David Rientjes <rientjes@google.com> - 2015-10-15 00:10 +0200
            RE: [RESEND PATCH 1/1] mm: vmstat: Add OOM victims count in vmstat  counter PINTU KUMAR <pintu.k@samsung.com> - 2015-10-15 16:40 +0200

#1244702 — [PATCH 1/1] mm: vmstat: Add OOM victims count in vmstat counter

This patch maintains the number of oom victims kill count in
/proc/vmstat.
Currently, we are dependent upon kernel logs when the kernel OOM occurs.
But kernel OOM can went passed unnoticed by the developer as it can
silently kill some background applications/services.
In some small embedded system, it might be possible that OOM is captured
in the logs but it was over-written due to ring-buffer.
Thus this interface can quickly help the user in analyzing, whether there
were any OOM kill happened in the past, or whether the system have ever
entered the oom kill stage till date.

Thus, it can be beneficial under following cases:
1. User can monitor kernel oom kill scenario without looking into the
   kernel logs.
2. It can help in tuning the watermark level in the system.
3. It can help in tuning the low memory killer behavior in user space.
4. It can be helpful on a logless system or if klogd logging
   (/var/log/messages) are disabled.

A snapshot of the result of 3 days of over night test is shown below:
System: ARM Cortex A7, 1GB RAM, 8GB EMMC
Linux: 3.10.xx
Category: reference smart phone device
Loglevel: 7
Conditions: Fully loaded, BT/WiFi/GPS ON
Tests: auto launching of ~30+ apps using test scripts, in a loop for
3 days.
At the end of tests, check:
$ cat /proc/vmstat
nr_oom_victims 6

As we noticed, there were around 6 oom kill victims.

The OOM is bad for any system. So, this counter can help in quickly
tuning the OOM behavior of the system, without depending on the logs.

Signed-off-by: Pintu Kumar <pintu.k@samsung.com>
---
 include/linux/vm_event_item.h |    1 +
 mm/oom_kill.c                 |    2 ++
 mm/page_alloc.c               |    1 -
 mm/vmstat.c                   |    1 +
 4 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h
index 2b1cef8..dd2600d 100644
--- a/include/linux/vm_event_item.h
+++ b/include/linux/vm_event_item.h
@@ -57,6 +57,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 #ifdef CONFIG_HUGETLB_PAGE
 		HTLB_BUDDY_PGALLOC, HTLB_BUDDY_PGALLOC_FAIL,
 #endif
+		NR_OOM_VICTIMS,
 		UNEVICTABLE_PGCULLED,	/* culled to noreclaim list */
 		UNEVICTABLE_PGSCANNED,	/* scanned for reclaimability */
 		UNEVICTABLE_PGRESCUED,	/* rescued from noreclaim list */
diff --git a/mm/oom_kill.c b/mm/oom_kill.c
index 03b612b..802b8a1 100644
--- a/mm/oom_kill.c
+++ b/mm/oom_kill.c
@@ -570,6 +570,7 @@ void oom_kill_process(struct oom_control *oc, struct task_struct *p,
 	 * space under its control.
 	 */
 	do_send_sig_info(SIGKILL, SEND_SIG_FORCED, victim, true);
+	count_vm_event(NR_OOM_VICTIMS);
 	mark_oom_victim(victim);
 	pr_err("Killed process %d (%s) total-vm:%lukB, anon-rss:%lukB, file-rss:%lukB\n",
 		task_pid_nr(victim), victim->comm, K(victim->mm->total_vm),
@@ -600,6 +601,7 @@ void oom_kill_process(struct oom_control *oc, struct task_struct *p,
 				task_pid_nr(p), p->comm);
 			task_unlock(p);
 			do_send_sig_info(SIGKILL, SEND_SIG_FORCED, p, true);
+			count_vm_event(NR_OOM_VICTIMS);
 		}
 	rcu_read_unlock();
 
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 9bcfd70..fafb09d 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2761,7 +2761,6 @@ __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order,
 		schedule_timeout_uninterruptible(1);
 		return NULL;
 	}
-
 	/*
 	 * Go through the zonelist yet one more time, keep very high watermark
 	 * here, this is only to catch a parallel oom killing, we must fail if
diff --git a/mm/vmstat.c b/mm/vmstat.c
index 1fd0886..8503a2e 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -808,6 +808,7 @@ const char * const vmstat_text[] = {
 	"htlb_buddy_alloc_success",
 	"htlb_buddy_alloc_fail",
 #endif
+	"nr_oom_victims",
 	"unevictable_pgs_culled",
 	"unevictable_pgs_scanned",
 	"unevictable_pgs_rescued",
-- 
1.7.9.5

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1244784

Hi,

Sorry, I forgot to mention the V2 update.
I will highlight the V2 changes and RESEND.

> -----Original Message-----
> From: Pintu Kumar [mailto:pintu.k@samsung.com]
> Sent: Monday, October 12, 2015 7:03 PM
> To: akpm@linux-foundation.org; minchan@kernel.org; dave@stgolabs.net;
> pintu.k@samsung.com; mhocko@suse.cz; koct9i@gmail.com;
> rientjes@google.com; hannes@cmpxchg.org; penguin-kernel@i-
> love.sakura.ne.jp; bywxiaobai@163.com; mgorman@suse.de; vbabka@suse.cz;
> js1304@gmail.com; kirill.shutemov@linux.intel.com;
> alexander.h.duyck@redhat.com; sasha.levin@oracle.com; cl@linux.com;
> fengguang.wu@intel.com; linux-kernel@vger.kernel.org; linux-mm@kvack.org
> Cc: cpgs@samsung.com; pintu_agarwal@yahoo.com; pintu.ping@gmail.com;
> vishnu.ps@samsung.com; rohit.kr@samsung.com; c.rajkumar@samsung.com;
> sreenathd@samsung.com
> Subject: [PATCH 1/1] mm: vmstat: Add OOM victims count in vmstat counter
> 
> This patch maintains the number of oom victims kill count in /proc/vmstat.
> Currently, we are dependent upon kernel logs when the kernel OOM occurs.
> But kernel OOM can went passed unnoticed by the developer as it can silently
> kill some background applications/services.
> In some small embedded system, it might be possible that OOM is captured in
> the logs but it was over-written due to ring-buffer.
> Thus this interface can quickly help the user in analyzing, whether there were
> any OOM kill happened in the past, or whether the system have ever entered
> the oom kill stage till date.
> 
> Thus, it can be beneficial under following cases:
> 1. User can monitor kernel oom kill scenario without looking into the
>    kernel logs.
> 2. It can help in tuning the watermark level in the system.
> 3. It can help in tuning the low memory killer behavior in user space.
> 4. It can be helpful on a logless system or if klogd logging
>    (/var/log/messages) are disabled.
> 
> A snapshot of the result of 3 days of over night test is shown below:
> System: ARM Cortex A7, 1GB RAM, 8GB EMMC
> Linux: 3.10.xx
> Category: reference smart phone device
> Loglevel: 7
> Conditions: Fully loaded, BT/WiFi/GPS ON
> Tests: auto launching of ~30+ apps using test scripts, in a loop for
> 3 days.
> At the end of tests, check:
> $ cat /proc/vmstat
> nr_oom_victims 6
> 
> As we noticed, there were around 6 oom kill victims.
> 
> The OOM is bad for any system. So, this counter can help in quickly tuning the
> OOM behavior of the system, without depending on the logs.
> 
> Signed-off-by: Pintu Kumar <pintu.k@samsung.com>
> ---
>  include/linux/vm_event_item.h |    1 +
>  mm/oom_kill.c                 |    2 ++
>  mm/page_alloc.c               |    1 -
>  mm/vmstat.c                   |    1 +
>  4 files changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h
> index 2b1cef8..dd2600d 100644
> --- a/include/linux/vm_event_item.h
> +++ b/include/linux/vm_event_item.h
> @@ -57,6 +57,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN,
> PSWPOUT,  #ifdef CONFIG_HUGETLB_PAGE
>  		HTLB_BUDDY_PGALLOC, HTLB_BUDDY_PGALLOC_FAIL,  #endif
> +		NR_OOM_VICTIMS,
>  		UNEVICTABLE_PGCULLED,	/* culled to noreclaim list */
>  		UNEVICTABLE_PGSCANNED,	/* scanned for reclaimability */
>  		UNEVICTABLE_PGRESCUED,	/* rescued from noreclaim list */
> diff --git a/mm/oom_kill.c b/mm/oom_kill.c index 03b612b..802b8a1 100644
> --- a/mm/oom_kill.c
> +++ b/mm/oom_kill.c
> @@ -570,6 +570,7 @@ void oom_kill_process(struct oom_control *oc, struct
> task_struct *p,
>  	 * space under its control.
>  	 */
>  	do_send_sig_info(SIGKILL, SEND_SIG_FORCED, victim, true);
> +	count_vm_event(NR_OOM_VICTIMS);
>  	mark_oom_victim(victim);
>  	pr_err("Killed process %d (%s) total-vm:%lukB, anon-rss:%lukB, file-
> rss:%lukB\n",
>  		task_pid_nr(victim), victim->comm, K(victim->mm->total_vm),
> @@ -600,6 +601,7 @@ void oom_kill_process(struct oom_control *oc, struct
> task_struct *p,
>  				task_pid_nr(p), p->comm);
>  			task_unlock(p);
>  			do_send_sig_info(SIGKILL, SEND_SIG_FORCED, p, true);
> +			count_vm_event(NR_OOM_VICTIMS);
>  		}
>  	rcu_read_unlock();
> 
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 9bcfd70..fafb09d 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -2761,7 +2761,6 @@ __alloc_pages_may_oom(gfp_t gfp_mask, unsigned
> int order,
>  		schedule_timeout_uninterruptible(1);
>  		return NULL;
>  	}
> -
>  	/*
>  	 * Go through the zonelist yet one more time, keep very high watermark
>  	 * here, this is only to catch a parallel oom killing, we must fail if
diff --git
> a/mm/vmstat.c b/mm/vmstat.c index 1fd0886..8503a2e 100644
> --- a/mm/vmstat.c
> +++ b/mm/vmstat.c
> @@ -808,6 +808,7 @@ const char * const vmstat_text[] = {
>  	"htlb_buddy_alloc_success",
>  	"htlb_buddy_alloc_fail",
>  #endif
> +	"nr_oom_victims",
>  	"unevictable_pgs_culled",
>  	"unevictable_pgs_scanned",
>  	"unevictable_pgs_rescued",
> --
> 1.7.9.5

Regards,
Pintu

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1244785 — [RESEND PATCH 1/1] mm: vmstat: Add OOM victims count in vmstat counter

This patch maintains the number of oom victims kill count in
/proc/vmstat.
Currently, we are dependent upon kernel logs when the kernel OOM occurs.
But kernel OOM can went passed unnoticed by the developer as it can
silently kill some background applications/services.
In some small embedded system, it might be possible that OOM is captured
in the logs but it was over-written due to ring-buffer.
Thus this interface can quickly help the user in analyzing, whether there
were any OOM kill happened in the past, or whether the system have ever
entered the oom kill stage till date.

Thus, it can be beneficial under following cases:
1. User can monitor kernel oom kill scenario without looking into the
   kernel logs.
2. It can help in tuning the watermark level in the system.
3. It can help in tuning the low memory killer behavior in user space.
4. It can be helpful on a logless system or if klogd logging
   (/var/log/messages) are disabled.

A snapshot of the result of 3 days of over night test is shown below:
System: ARM Cortex A7, 1GB RAM, 8GB EMMC
Linux: 3.10.xx
Category: reference smart phone device
Loglevel: 7
Conditions: Fully loaded, BT/WiFi/GPS ON
Tests: auto launching of ~30+ apps using test scripts, in a loop for
3 days.
At the end of tests, check:
$ cat /proc/vmstat
nr_oom_victims 6

As we noticed, there were around 6 oom kill victims.

The OOM is bad for any system. So, this counter can help in quickly
tuning the OOM behavior of the system, without depending on the logs.

Signed-off-by: Pintu Kumar <pintu.k@samsung.com>
---
V2: Removed oom_stall, Suggested By: Michal Hocko <mhocko@kernel.org>
    Renamed oom_kill_count to nr_oom_victims,
    Suggested By: Michal Hocko <mhocko@kernel.org>
    Suggested By: Anshuman Khandual <khandual@linux.vnet.ibm.com>

 include/linux/vm_event_item.h |    1 +
 mm/oom_kill.c                 |    2 ++
 mm/page_alloc.c               |    1 -
 mm/vmstat.c                   |    1 +
 4 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h
index 2b1cef8..dd2600d 100644
--- a/include/linux/vm_event_item.h
+++ b/include/linux/vm_event_item.h
@@ -57,6 +57,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 #ifdef CONFIG_HUGETLB_PAGE
 		HTLB_BUDDY_PGALLOC, HTLB_BUDDY_PGALLOC_FAIL,
 #endif
+		NR_OOM_VICTIMS,
 		UNEVICTABLE_PGCULLED,	/* culled to noreclaim list */
 		UNEVICTABLE_PGSCANNED,	/* scanned for reclaimability */
 		UNEVICTABLE_PGRESCUED,	/* rescued from noreclaim list */
diff --git a/mm/oom_kill.c b/mm/oom_kill.c
index 03b612b..802b8a1 100644
--- a/mm/oom_kill.c
+++ b/mm/oom_kill.c
@@ -570,6 +570,7 @@ void oom_kill_process(struct oom_control *oc, struct task_struct *p,
 	 * space under its control.
 	 */
 	do_send_sig_info(SIGKILL, SEND_SIG_FORCED, victim, true);
+	count_vm_event(NR_OOM_VICTIMS);
 	mark_oom_victim(victim);
 	pr_err("Killed process %d (%s) total-vm:%lukB, anon-rss:%lukB, file-rss:%lukB\n",
 		task_pid_nr(victim), victim->comm, K(victim->mm->total_vm),
@@ -600,6 +601,7 @@ void oom_kill_process(struct oom_control *oc, struct task_struct *p,
 				task_pid_nr(p), p->comm);
 			task_unlock(p);
 			do_send_sig_info(SIGKILL, SEND_SIG_FORCED, p, true);
+			count_vm_event(NR_OOM_VICTIMS);
 		}
 	rcu_read_unlock();
 
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 9bcfd70..fafb09d 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2761,7 +2761,6 @@ __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order,
 		schedule_timeout_uninterruptible(1);
 		return NULL;
 	}
-
 	/*
 	 * Go through the zonelist yet one more time, keep very high watermark
 	 * here, this is only to catch a parallel oom killing, we must fail if
diff --git a/mm/vmstat.c b/mm/vmstat.c
index 1fd0886..8503a2e 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -808,6 +808,7 @@ const char * const vmstat_text[] = {
 	"htlb_buddy_alloc_success",
 	"htlb_buddy_alloc_fail",
 #endif
+	"nr_oom_victims",
 	"unevictable_pgs_culled",
 	"unevictable_pgs_scanned",
 	"unevictable_pgs_rescued",
-- 
1.7.9.5

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1246272 — Re: [RESEND PATCH 1/1] mm: vmstat: Add OOM victims count in vmstat counter

On Mon, 12 Oct 2015, Pintu Kumar wrote:

> This patch maintains the number of oom victims kill count in
> /proc/vmstat.
> Currently, we are dependent upon kernel logs when the kernel OOM occurs.
> But kernel OOM can went passed unnoticed by the developer as it can
> silently kill some background applications/services.
> In some small embedded system, it might be possible that OOM is captured
> in the logs but it was over-written due to ring-buffer.
> Thus this interface can quickly help the user in analyzing, whether there
> were any OOM kill happened in the past, or whether the system have ever
> entered the oom kill stage till date.
> 
> Thus, it can be beneficial under following cases:
> 1. User can monitor kernel oom kill scenario without looking into the
>    kernel logs.

I'm not sure how helpful that would be since we don't know anything about 
the oom kill itself, only that at some point during the uptime there were 
oom kills.

> 2. It can help in tuning the watermark level in the system.

I disagree with this one, because we can encounter oom kills due to 
fragmentation rather than low memory conditions for high-order 
allocations.  The amount of free memory may be substantially higher than 
all zone watermarks.

> 3. It can help in tuning the low memory killer behavior in user space.

Same reason as above.

> 4. It can be helpful on a logless system or if klogd logging
>    (/var/log/messages) are disabled.
> 

This would be similar to point (1) above, and I question how helpful it 
would be.  I notice that all oom kills (system, cpuset, mempolicy, and 
memcg) are treated equally in this case and there's no way to 
differentiate them.  That would lead me to believe that you are targeting 
this change for systems that don't use mempolicies or cgroups.  That's 
fine, but I doubt it will be helpful for anybody else.

> A snapshot of the result of 3 days of over night test is shown below:
> System: ARM Cortex A7, 1GB RAM, 8GB EMMC
> Linux: 3.10.xx
> Category: reference smart phone device
> Loglevel: 7
> Conditions: Fully loaded, BT/WiFi/GPS ON
> Tests: auto launching of ~30+ apps using test scripts, in a loop for
> 3 days.
> At the end of tests, check:
> $ cat /proc/vmstat
> nr_oom_victims 6
> 
> As we noticed, there were around 6 oom kill victims.
> 
> The OOM is bad for any system. So, this counter can help in quickly
> tuning the OOM behavior of the system, without depending on the logs.
> 

NACK to the patch since it isn't justified.

We've long had a desire to have a better oom reporting mechanism rather 
than just the kernel log.  It seems like you're feeling the same pain.  I 
think it would be better to have an eventfd notifier for system oom 
conditions so we can track kernel oom kills (and conditions) in 
userspace.  I have a patch for that, and it works quite well when 
userspace is mlocked with a buffer in memory.

If you are only interested in a strict count of system oom kills, this 
could then easily be implemented without adding vmstat counters.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1246760 — RE: [RESEND PATCH 1/1] mm: vmstat: Add OOM victims count in vmstat counter

Hi,

Thank you very much for your review and comments.

> -----Original Message-----
> From: David Rientjes [mailto:rientjes@google.com]
> Sent: Wednesday, October 14, 2015 8:36 AM
> To: Pintu Kumar
> Cc: akpm@linux-foundation.org; minchan@kernel.org; dave@stgolabs.net;
> mhocko@suse.cz; koct9i@gmail.com; hannes@cmpxchg.org; penguin-kernel@i-
> love.sakura.ne.jp; bywxiaobai@163.com; mgorman@suse.de; vbabka@suse.cz;
> js1304@gmail.com; kirill.shutemov@linux.intel.com;
> alexander.h.duyck@redhat.com; sasha.levin@oracle.com; cl@linux.com;
> fengguang.wu@intel.com; linux-kernel@vger.kernel.org; linux-mm@kvack.org;
> cpgs@samsung.com; pintu_agarwal@yahoo.com; pintu.ping@gmail.com;
> vishnu.ps@samsung.com; rohit.kr@samsung.com; c.rajkumar@samsung.com;
> sreenathd@samsung.com
> Subject: Re: [RESEND PATCH 1/1] mm: vmstat: Add OOM victims count in vmstat
> counter
> 
> On Mon, 12 Oct 2015, Pintu Kumar wrote:
> 
> > This patch maintains the number of oom victims kill count in
> > /proc/vmstat.
> > Currently, we are dependent upon kernel logs when the kernel OOM occurs.
> > But kernel OOM can went passed unnoticed by the developer as it can
> > silently kill some background applications/services.
> > In some small embedded system, it might be possible that OOM is
> > captured in the logs but it was over-written due to ring-buffer.
> > Thus this interface can quickly help the user in analyzing, whether
> > there were any OOM kill happened in the past, or whether the system
> > have ever entered the oom kill stage till date.
> >
> > Thus, it can be beneficial under following cases:
> > 1. User can monitor kernel oom kill scenario without looking into the
> >    kernel logs.
> 
> I'm not sure how helpful that would be since we don't know anything about the
> oom kill itself, only that at some point during the uptime there were oom
kills.
> 
Not sure about others.
For me it was very helpful during sluggish and long duration ageing tests.
With this, I don't have to look into the logs manually.
I just monitor this count in a script. 
The moment I get nr_oom_victims > 1, I know that kernel OOM would have happened
and I need to take the log dump.
So, then I do: dmesg >> oom_logs.txt
Or, even stop the tests for further tuning.

> > 2. It can help in tuning the watermark level in the system.
> 
> I disagree with this one, because we can encounter oom kills due to
> fragmentation rather than low memory conditions for high-order allocations.
> The amount of free memory may be substantially higher than all zone
> watermarks.
> 
AFAIK, kernel oom happens only for lower-order (PAGE_ALLOC_COSTLY_ORDER).
For higher-order we get page allocation failure.

> > 3. It can help in tuning the low memory killer behavior in user space.
> 
> Same reason as above.
> 
> > 4. It can be helpful on a logless system or if klogd logging
> >    (/var/log/messages) are disabled.
> >
> 
> This would be similar to point (1) above, and I question how helpful it would
be.
> I notice that all oom kills (system, cpuset, mempolicy, and
> memcg) are treated equally in this case and there's no way to differentiate
them.
> That would lead me to believe that you are targeting this change for systems
> that don't use mempolicies or cgroups.  That's fine, but I doubt it will be
helpful
> for anybody else.
> 
No, we are not targeting any specific category.
Our goal is simple, track and report kernel oom kill as soon as it occurs.

> > A snapshot of the result of 3 days of over night test is shown below:
> > System: ARM Cortex A7, 1GB RAM, 8GB EMMC
> > Linux: 3.10.xx
> > Category: reference smart phone device
> > Loglevel: 7
> > Conditions: Fully loaded, BT/WiFi/GPS ON
> > Tests: auto launching of ~30+ apps using test scripts, in a loop for
> > 3 days.
> > At the end of tests, check:
> > $ cat /proc/vmstat
> > nr_oom_victims 6
> >
> > As we noticed, there were around 6 oom kill victims.
> >
> > The OOM is bad for any system. So, this counter can help in quickly
> > tuning the OOM behavior of the system, without depending on the logs.
> >
> 
> NACK to the patch since it isn't justified.
> 
> We've long had a desire to have a better oom reporting mechanism rather than
> just the kernel log.  It seems like you're feeling the same pain.  I think it
would be
> better to have an eventfd notifier for system oom conditions so we can track
> kernel oom kills (and conditions) in userspace.  I have a patch for that, and
it
> works quite well when userspace is mlocked with a buffer in memory.
> 
Ok, this would be interesting.
Can you point me to the patches?
I will quickly check if it is useful for us.

> If you are only interested in a strict count of system oom kills, this could
then
> easily be implemented without adding vmstat counters.
>
We are interested only to know when kernel OOM occurs and not even the oom
victim count. So that we can tune something is user space to avoid or delay it
as far as possible.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1247218 — RE: [RESEND PATCH 1/1] mm: vmstat: Add OOM victims count in vmstat counter

On Wed, 14 Oct 2015, PINTU KUMAR wrote:

> For me it was very helpful during sluggish and long duration ageing tests.
> With this, I don't have to look into the logs manually.
> I just monitor this count in a script. 
> The moment I get nr_oom_victims > 1, I know that kernel OOM would have happened
> and I need to take the log dump.
> So, then I do: dmesg >> oom_logs.txt
> Or, even stop the tests for further tuning.
> 

I think eventfd(2) was created for that purpose, to avoid the constant 
polling that you would have to do to check nr_oom_victims and then take a 
snapshot.

> > I disagree with this one, because we can encounter oom kills due to
> > fragmentation rather than low memory conditions for high-order allocations.
> > The amount of free memory may be substantially higher than all zone
> > watermarks.
> > 
> AFAIK, kernel oom happens only for lower-order (PAGE_ALLOC_COSTLY_ORDER).
> For higher-order we get page allocation failure.
> 

Order-3 is included.  I've seen machines with _gigabytes_ of free memory 
in ZONE_NORMAL on a node and have an order-3 page allocation failure that 
called the oom killer.

> > We've long had a desire to have a better oom reporting mechanism rather than
> > just the kernel log.  It seems like you're feeling the same pain.  I think it
> would be
> > better to have an eventfd notifier for system oom conditions so we can track
> > kernel oom kills (and conditions) in userspace.  I have a patch for that, and
> it
> > works quite well when userspace is mlocked with a buffer in memory.
> > 
> Ok, this would be interesting.
> Can you point me to the patches?
> I will quickly check if it is useful for us.
> 

https://lwn.net/Articles/589404.  It's invasive and isn't upstream.  I 
would like to restructure that patchset to avoid the memcg trickery and 
allow for a root-only eventfd(2) notification through procfs on system 
oom.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1247852 — RE: [RESEND PATCH 1/1] mm: vmstat: Add OOM victims count in vmstat counter

Hi,

> -----Original Message-----
> From: David Rientjes [mailto:rientjes@google.com]
> Sent: Thursday, October 15, 2015 3:35 AM
> To: PINTU KUMAR
> Cc: akpm@linux-foundation.org; minchan@kernel.org; dave@stgolabs.net;
> mhocko@suse.cz; koct9i@gmail.com; hannes@cmpxchg.org; penguin-kernel@i-
> love.sakura.ne.jp; bywxiaobai@163.com; mgorman@suse.de; vbabka@suse.cz;
> js1304@gmail.com; kirill.shutemov@linux.intel.com;
> alexander.h.duyck@redhat.com; sasha.levin@oracle.com; cl@linux.com;
> fengguang.wu@intel.com; linux-kernel@vger.kernel.org; linux-mm@kvack.org;
> cpgs@samsung.com; pintu_agarwal@yahoo.com; pintu.ping@gmail.com;
> vishnu.ps@samsung.com; rohit.kr@samsung.com; c.rajkumar@samsung.com
> Subject: RE: [RESEND PATCH 1/1] mm: vmstat: Add OOM victims count in vmstat
> counter
> 
> On Wed, 14 Oct 2015, PINTU KUMAR wrote:
> 
> > For me it was very helpful during sluggish and long duration ageing tests.
> > With this, I don't have to look into the logs manually.
> > I just monitor this count in a script.
> > The moment I get nr_oom_victims > 1, I know that kernel OOM would have
> > happened and I need to take the log dump.
> > So, then I do: dmesg >> oom_logs.txt
> > Or, even stop the tests for further tuning.
> >
> 
> I think eventfd(2) was created for that purpose, to avoid the constant polling
> that you would have to do to check nr_oom_victims and then take a snapshot.
> 
> > > I disagree with this one, because we can encounter oom kills due to
> > > fragmentation rather than low memory conditions for high-order
allocations.
> > > The amount of free memory may be substantially higher than all zone
> > > watermarks.
> > >
> > AFAIK, kernel oom happens only for lower-order
> (PAGE_ALLOC_COSTLY_ORDER).
> > For higher-order we get page allocation failure.
> >
> 
> Order-3 is included.  I've seen machines with _gigabytes_ of free memory in
> ZONE_NORMAL on a node and have an order-3 page allocation failure that
> called the oom killer.
> 
Yes, if PAGE_ALLOC_COSTLY_ORDER is defined as 3, then order-3 will be included
for OOM. But that's fine. We are just interested to know if system entered oom
state.
That's the reason, earlier I added even _oom_stall_ to know if system ever
entered oom but resulted into page allocation failure instead of oom killing.

> > > We've long had a desire to have a better oom reporting mechanism
> > > rather than just the kernel log.  It seems like you're feeling the
> > > same pain.  I think it
> > would be
> > > better to have an eventfd notifier for system oom conditions so we
> > > can track kernel oom kills (and conditions) in userspace.  I have a
> > > patch for that, and
> > it
> > > works quite well when userspace is mlocked with a buffer in memory.
> > >
> > Ok, this would be interesting.
> > Can you point me to the patches?
> > I will quickly check if it is useful for us.
> >
> 
> https://lwn.net/Articles/589404.  It's invasive and isn't upstream.  I would
like to
> restructure that patchset to avoid the memcg trickery and allow for a
root-only
> eventfd(2) notification through procfs on system oom.

I am interested only in global oom case and not memcg. We have memcg enabled but
I think even memcg_oom will finally invoke _oom_kill_process_.
So, I am interested in a patchset that can trigger notifications from
oom_kill_process, as soon as any victim is killed.
Sorry, from your patchset, I could not actually local the system_oom
notification patch.
If you have similar patchset please point me to it.
It will be really helpful.
Thank you!

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web