RE: [PATCH 1/2] sched/wait: Break up long wake list walk

From	"Liang, Kan" <kan.liang@intel.com>
Newsgroups	linux.kernel
Subject	RE: [PATCH 1/2] sched/wait: Break up long wake list walk
Date	2017-08-18 15:10 +0200
Message-ID	<ufPce-48Q-23@gated-at.bofh.it> (permalink)
References	(1 earlier) <uez9v-3gj-7@gated-at.bofh.it> <ufvGy-7iQ-9@gated-at.bofh.it> <ufvQd-7mK-7@gated-at.bofh.it> <ufzqO-1nj-23@gated-at.bofh.it> <ufzTP-1Ak-1@gated-at.bofh.it>
Organization	linux.* mail to news gateway

Show all headers | View raw


> On Thu, Aug 17, 2017 at 1:18 PM, Liang, Kan <kan.liang@intel.com> wrote:
> >
> > Here is the call stack of wait_on_page_bit_common when the queue is
> > long (entries >1000).
> >
> > # Overhead  Trace output
> > # ........  ..................
> > #
> >    100.00%  (ffffffff931aefca)
> >             |
> >             ---wait_on_page_bit
> >                __migration_entry_wait
> >                migration_entry_wait
> >                do_swap_page
> >                __handle_mm_fault
> >                handle_mm_fault
> >                __do_page_fault
> >                do_page_fault
> >                page_fault
> 
> Hmm. Ok, so it does seem to very much be related to migration. Your
> wake_up_page_bit() profile made me suspect that, but this one seems to
> pretty much confirm it.
> 
> So it looks like that wait_on_page_locked() thing in __migration_entry_wait(),
> and what probably happens is that your load ends up triggering a lot of
> migration (or just migration of a very hot page), and then *every* thread
> ends up waiting for whatever page that ended up getting migrated.
> 
> And so the wait queue for that page grows hugely long.
> 
> Looking at the other profile, the thing that is locking the page (that everybody
> then ends up waiting on) would seem to be
> migrate_misplaced_transhuge_page(), so this is _presumably_ due to NUMA
> balancing.
> 
> Does the problem go away if you disable the NUMA balancing code?
> 

Yes, the problem goes away when NUMA balancing is disabled.


Thanks,
Kan

Back to linux.kernel | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

[PATCH 1/2] sched/wait: Break up long wake list walk Tim Chen <tim.c.chen@linux.intel.com> - 2017-08-15 03:20 +0200
  Re: [PATCH 1/2] sched/wait: Break up long wake list walk Linus Torvalds <torvalds@linux-foundation.org> - 2017-08-15 03:50 +0200
    Re: [PATCH 1/2] sched/wait: Break up long wake list walk Andi Kleen <ak@linux.intel.com> - 2017-08-15 04:30 +0200
      Re: [PATCH 1/2] sched/wait: Break up long wake list walk Linus Torvalds <torvalds@linux-foundation.org> - 2017-08-15 05:00 +0200
        Re: [PATCH 1/2] sched/wait: Break up long wake list walk Andi Kleen <ak@linux.intel.com> - 2017-08-15 05:20 +0200
          Re: [PATCH 1/2] sched/wait: Break up long wake list walk Linus Torvalds <torvalds@linux-foundation.org> - 2017-08-15 05:30 +0200
            Re: [PATCH 1/2] sched/wait: Break up long wake list walk Tim Chen <tim.c.chen@linux.intel.com> - 2017-08-15 21:10 +0200
              Re: [PATCH 1/2] sched/wait: Break up long wake list walk Linus Torvalds <torvalds@linux-foundation.org> - 2017-08-15 21:50 +0200
              Re: [PATCH 1/2] sched/wait: Break up long wake list walk Linus Torvalds <torvalds@linux-foundation.org> - 2017-08-15 21:50 +0200
            Re: [PATCH 1/2] sched/wait: Break up long wake list walk Davidlohr Bueso <dave@stgolabs.net> - 2017-08-16 00:50 +0200
              Re: [PATCH 1/2] sched/wait: Break up long wake list walk Linus Torvalds <torvalds@linux-foundation.org> - 2017-08-16 01:00 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Linus Torvalds <torvalds@linux-foundation.org> - 2017-08-16 02:00 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk ebiederm@xmission.com (Eric W. Biederman) - 2017-08-17 01:30 +0200
              Re: [PATCH 1/2] sched/wait: Break up long wake list walk Linus Torvalds <torvalds@linux-foundation.org> - 2017-08-16 01:00 +0200
    RE: [PATCH 1/2] sched/wait: Break up long wake list walk "Liang, Kan" <kan.liang@intel.com> - 2017-08-17 18:20 +0200
      Re: [PATCH 1/2] sched/wait: Break up long wake list walk Linus Torvalds <torvalds@linux-foundation.org> - 2017-08-17 18:30 +0200
        RE: [PATCH 1/2] sched/wait: Break up long wake list walk "Liang, Kan" <kan.liang@intel.com> - 2017-08-17 22:20 +0200
          Re: [PATCH 1/2] sched/wait: Break up long wake list walk Linus Torvalds <torvalds@linux-foundation.org> - 2017-08-17 22:50 +0200
            Re: [PATCH 1/2] sched/wait: Break up long wake list walk Mel Gorman <mgorman@techsingularity.net> - 2017-08-18 14:30 +0200
              RE: [PATCH 1/2] sched/wait: Break up long wake list walk "Liang, Kan" <kan.liang@intel.com> - 2017-08-18 16:30 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Mel Gorman <mgorman@techsingularity.net> - 2017-08-18 16:50 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Tim Chen <tim.c.chen@linux.intel.com> - 2017-08-18 18:40 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Andi Kleen <ak@linux.intel.com> - 2017-08-18 18:50 +0200
                RE: [PATCH 1/2] sched/wait: Break up long wake list walk "Liang, Kan" <kan.liang@intel.com> - 2017-08-18 19:00 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Linus Torvalds <torvalds@linux-foundation.org> - 2017-08-18 19:50 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Mel Gorman <mgorman@techsingularity.net> - 2017-08-18 21:00 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Linus Torvalds <torvalds@linux-foundation.org> - 2017-08-18 21:20 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Andi Kleen <ak@linux.intel.com> - 2017-08-18 22:00 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Linus Torvalds <torvalds@linux-foundation.org> - 2017-08-18 22:20 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Mel Gorman <mgorman@techsingularity.net> - 2017-08-21 20:40 +0200
                RE: [PATCH 1/2] sched/wait: Break up long wake list walk "Liang, Kan" <kan.liang@intel.com> - 2017-08-21 21:00 +0200
                RE: [PATCH 1/2] sched/wait: Break up long wake list walk "Liang, Kan" <kan.liang@intel.com> - 2017-08-22 19:30 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Linus Torvalds <torvalds@linux-foundation.org> - 2017-08-22 20:20 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Linus Torvalds <torvalds@linux-foundation.org> - 2017-08-22 20:30 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Peter Zijlstra <peterz@infradead.org> - 2017-08-22 21:00 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Linus Torvalds <torvalds@linux-foundation.org> - 2017-08-22 21:20 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Peter Zijlstra <peterz@infradead.org> - 2017-08-22 21:10 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Andi Kleen <ak@linux.intel.com> - 2017-08-22 21:40 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Christopher Lameter <cl@linux.com> - 2017-08-22 23:10 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Andi Kleen <ak@linux.intel.com> - 2017-08-22 23:30 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Linus Torvalds <torvalds@linux-foundation.org> - 2017-08-23 01:00 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Linus Torvalds <torvalds@linux-foundation.org> - 2017-08-23 01:20 +0200
                RE: [PATCH 1/2] sched/wait: Break up long wake list walk "Liang, Kan" <kan.liang@intel.com> - 2017-08-23 17:00 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Linus Torvalds <torvalds@linux-foundation.org> - 2017-08-22 21:40 +0200
                RE: [PATCH 1/2] sched/wait: Break up long wake list walk "Liang, Kan" <kan.liang@intel.com> - 2017-08-22 22:00 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Linus Torvalds <torvalds@linux-foundation.org> - 2017-08-22 22:50 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Linus Torvalds <torvalds@linux-foundation.org> - 2017-08-22 23:00 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Peter Zijlstra <peterz@infradead.org> - 2017-08-22 23:00 +0200
                RE: [PATCH 1/2] sched/wait: Break up long wake list walk "Liang, Kan" <kan.liang@intel.com> - 2017-08-23 16:50 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Tim Chen <tim.c.chen@linux.intel.com> - 2017-08-23 18:00 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Linus Torvalds <torvalds@linux-foundation.org> - 2017-08-23 20:20 +0200
                RE: [PATCH 1/2] sched/wait: Break up long wake list walk "Liang, Kan" <kan.liang@intel.com> - 2017-08-23 23:00 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Linus Torvalds <torvalds@linux-foundation.org> - 2017-08-24 01:40 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Tim Chen <tim.c.chen@linux.intel.com> - 2017-08-24 19:50 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Linus Torvalds <torvalds@linux-foundation.org> - 2017-08-24 20:20 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Mel Gorman <mgorman@techsingularity.net> - 2017-08-24 22:50 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Mel Gorman <mgorman@techsingularity.net> - 2017-08-23 18:10 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Andi Kleen <ak@linux.intel.com> - 2017-08-18 22:10 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Linus Torvalds <torvalds@linux-foundation.org> - 2017-08-18 22:40 +0200
                RE: [PATCH 1/2] sched/wait: Break up long wake list walk "Liang, Kan" <kan.liang@intel.com> - 2017-08-18 22:30 +0200
                Re: [PATCH 1/2] sched/wait: Break up long wake list walk Linus Torvalds <torvalds@linux-foundation.org> - 2017-08-18 22:40 +0200
              Re: [PATCH 1/2] sched/wait: Break up long wake list walk Linus Torvalds <torvalds@linux-foundation.org> - 2017-08-18 19:00 +0200
            RE: [PATCH 1/2] sched/wait: Break up long wake list walk "Liang, Kan" <kan.liang@intel.com> - 2017-08-18 15:10 +0200

csiph-web