Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1218704

Re: [4.2, Regression] Queued spinlocks cause major XFS performance regression

Show all headers | View raw

On 09/04/2015 09:11 AM, Dave Chinner wrote:
> On Thu, Sep 03, 2015 at 11:39:21PM -0700, Linus Torvalds wrote:
>> On Thu, Sep 3, 2015 at 10:48 PM, Dave Chinner <david@fromorbit.com> wrote:
>>>
>>> When I turned spinlock debugging off on 4.2 to get some perf numbers
>>> a request from Linus, I got this:
>>
>> [ ugly numbers deleted ]
>>
>>> And then a quick call graph sample to find the lock:
>>>
>>>     37.19%    37.19%  [kernel]         [k] queued_spin_lock_slowpath
>>>     - queued_spin_lock_slowpath
>>>        - 99.98% _raw_spin_lock
>>>           - 89.16% xfs_log_commit_cil
>> [ snip ]
>>>
>>> This shows that we have catastrophic spinlock contention in the
>>> transaction commit path. The cil->xc_cil_lock spin lock as it's the
>>> only spinlock in that path. And while it's the hot lock in the
>>> commit path, turning spinlock debugging back on (and no other
>>> changes) shows that it shouldn't be contended:
>>>
>>>     8.92%  [kernel]  [k] _xfs_buf_find
>> [ snip ]
>>
>> So you basically have almost no spinlock overhead at all even when
>> debugging is on.
>
> *nod*
>
>> That's unusual, as usually the debug code makes the contention much much worse.
>
> Right. The debug behaviour is completely unchanged, that's why I
> didn't notice this earlier. And it's not until I scale this workload
> to >32p that is tend to see and significant level of contention on
> the cil->xc_cil_lock when the basic spin lock debugging is enabled.
>
>>> To confirm that this is indeed caused by the queued spinlocks, I
>>> removed the the spinlock debugging and did this to arch/x86/Kconfig:
>>>
>>> -       select ARCH_USE_QUEUED_SPINLOCK
>>>
>>> And the results are:
>>
>> Ok, that's pretty conclusive. It doesn't seem to make much _sense_,
>> but numbers talk, BS walks.
>>
>> If I read things right, the actual spinlock is the "cil->xc_cil_lock"
>> that is taken in xlog_cil_insert_items(), and it justr shows up in
>> xfs_log_commit_cil() in the call graph due to inlining. Correct?
>
> Yup, that's how I read it, too.
>
>> There doesn't seem to be anything even remotely strange going on in that area.
>>
>> Is this a PARAVIRT configuration? There were issues with PV
>> interaction at some point. If it is PV, and you don't actually use PV,
>> can you test with PV support disabled?
>
> $ grep PARAVIRT .config
> CONFIG_PARAVIRT=y
> # CONFIG_PARAVIRT_DEBUG is not set
> # CONFIG_PARAVIRT_SPINLOCKS is not set
> CONFIG_PARAVIRT_TIME_ACCOUNTING=y
> CONFIG_PARAVIRT_CLOCK=y
> $
>
> I'll retest with CONFIG_PARAVIRT=n....

Shouldn't matter at all. CONFIG_PARAVIRT_SPINLOCKS isn't set, so the
locks aren't para-virtualized.


Juergen
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

[4.2, Regression] Queued spinlocks cause major XFS performance  regression Dave Chinner <david@fromorbit.com> - 2015-09-04 07:50 +0200
  Re: [4.2, Regression] Queued spinlocks cause major XFS performance regression Linus Torvalds <torvalds@linux-foundation.org> - 2015-09-04 08:40 +0200
    Re: [4.2, Regression] Queued spinlocks cause major XFS performance  regression Dave Chinner <david@fromorbit.com> - 2015-09-04 09:20 +0200
      Re: [4.2, Regression] Queued spinlocks cause major XFS performance  regression Juergen Gross <jgross@suse.com> - 2015-09-04 09:40 +0200
      Re: [4.2, Regression] Queued spinlocks cause major XFS performance  regression Peter Zijlstra <peterz@infradead.org> - 2015-09-04 10:00 +0200
      Re: [4.2, Regression] Queued spinlocks cause major XFS performance  regression Dave Chinner <david@fromorbit.com> - 2015-09-04 10:40 +0200
        Re: [4.2, Regression] Queued spinlocks cause major XFS performance regression Linus Torvalds <torvalds@linux-foundation.org> - 2015-09-04 17:10 +0200
          Re: [4.2, Regression] Queued spinlocks cause major XFS performance  regression Peter Zijlstra <peterz@infradead.org> - 2015-09-04 17:20 +0200
            Re: [4.2, Regression] Queued spinlocks cause major XFS performance  regression Peter Zijlstra <peterz@infradead.org> - 2015-09-04 17:30 +0200
              Re: [4.2, Regression] Queued spinlocks cause major XFS performance  regression Dave Chinner <david@fromorbit.com> - 2015-09-07 01:40 +0200
              Re: [4.2, Regression] Queued spinlocks cause major XFS performance  regression Davidlohr Bueso <dave@stgolabs.net> - 2015-09-07 02:10 +0200
                Re: [4.2, Regression] Queued spinlocks cause major XFS performance  regression Peter Zijlstra <peterz@infradead.org> - 2015-09-07 09:00 +0200
            Re: [4.2, Regression] Queued spinlocks cause major XFS performance regression Linus Torvalds <torvalds@linux-foundation.org> - 2015-09-04 17:30 +0200
              Re: [4.2, Regression] Queued spinlocks cause major XFS performance  regression Peter Zijlstra <peterz@infradead.org> - 2015-09-04 17:40 +0200
                Re: [4.2, Regression] Queued spinlocks cause major XFS performance  regression Peter Zijlstra <peterz@infradead.org> - 2015-09-04 18:00 +0200
                Re: [4.2, Regression] Queued spinlocks cause major XFS performance regression Linus Torvalds <torvalds@linux-foundation.org> - 2015-09-04 18:00 +0200
                Re: [4.2, Regression] Queued spinlocks cause major XFS performance  regression Peter Zijlstra <peterz@infradead.org> - 2015-09-05 19:50 +0200
    Re: [4.2, Regression] Queued spinlocks cause major XFS performance  regression Peter Zijlstra <peterz@infradead.org> - 2015-09-04 09:40 +0200
      Re: [4.2, Regression] Queued spinlocks cause major XFS performance  regression Peter Zijlstra <peterz@infradead.org> - 2015-09-04 13:40 +0200
        Re: [4.2, Regression] Queued spinlocks cause major XFS performance  regression Dave Chinner <david@fromorbit.com> - 2015-09-05 00:10 +0200
        Re: [4.2, Regression] Queued spinlocks cause major XFS performance  regression Dave Chinner <david@fromorbit.com> - 2015-09-07 01:50 +0200

csiph-web