Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1721857 > unrolled thread

Re: [PATCH v3 0/5] fs/dcache: Limit # of negative dentries

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH v3 0/5] fs/dcache: Limit # of negative dentries Waiman Long <longman@redhat.com> - 2017-08-28 20:00 +0200
    Re: [PATCH v3 0/5] fs/dcache: Limit # of negative dentries Waiman Long <longman@redhat.com> - 2017-08-28 21:00 +0200

#1721857 — Re: [PATCH v3 0/5] fs/dcache: Limit # of negative dentries

On 07/28/2017 02:34 PM, Waiman Long wrote:
>  v2->v3:
>   - Add a faster pruning rate when the free pool is closed to depletion.
>   - As suggested by James Bottomley, add an artificial delay waiting
>     loop before killing a negative dentry and properly clear the
>     DCACHE_KILL_NEGATIVE flag if killing doesn't happen.
>   - Add a new patch to track number of negative dentries that are
>     forcifully killed.
>
>  v1->v2:
>   - Move the new nr_negative field to the end of dentry_stat_t structure
>     as suggested by Matthew Wilcox.
>   - With the help of Miklos Szeredi, fix incorrect locking order in
>     dentry_kill() by using lock_parent() instead of locking the parent's
>     d_lock directly.
>   - Correctly account for positive to negative dentry transitions.
>   - Automatic pruning of negative dentries will now ignore the reference
>     bit in negative dentries but not the regular shrinking.
>
> A rogue application can potentially create a large number of negative
> dentries in the system consuming most of the memory available. This
> can impact performance of other applications running on the system.
>
> This patchset introduces changes to the dcache subsystem to limit
> the number of negative dentries allowed to be created thus limiting
> the amount of memory that can be consumed by negative dentries.
>
> Patch 1 tracks the number of negative dentries used and disallow
> the creation of more when the limit is reached.
>
> Patch 2 enables /proc/sys/fs/dentry-state to report the number of
> negative dentries in the system.
>
> Patch 3 enables automatic pruning of negative dentries when it is
> close to the limit so that we won't end up killing recently used
> negative dentries.
>
> Patch 4 prevents racing between negative dentry pruning and umount
> operation.
>
> Patch 5 shows the number of forced negative dentry killings in
> /proc/sys/fs/dentry-state. End users can then tune the neg_dentry_pc=
> kernel boot parameter if they want to reduce forced negative dentry
> killings.
>
> Waiman Long (5):
>   fs/dcache: Limit numbers of negative dentries
>   fs/dcache: Report negative dentry number in dentry-state
>   fs/dcache: Enable automatic pruning of negative dentries
>   fs/dcache: Protect negative dentry pruning from racing with umount
>   fs/dcache: Track count of negative dentries forcibly killed
>
>  Documentation/admin-guide/kernel-parameters.txt |   7 +
>  fs/dcache.c                                     | 451 ++++++++++++++++++++++--
>  include/linux/dcache.h                          |   8 +-
>  include/linux/list_lru.h                        |   1 +
>  mm/list_lru.c                                   |   4 +-
>  5 files changed, 435 insertions(+), 36 deletions(-)
>
With a 4.13 based kernel, the positive & negative dentries lookup rates
(lookups per second) after initial boot on a 32GB memory VM with and
without the patch were as follows:

  Metric                    w/o patch        with patch
  ------                    ---------        ----------
  Positive dentry lookup      844881           842618
  Negative dentry lookup     1865158          1901875
  Negative dentry creation    609887           617215

The last row refers to the creation rate of 10 millions negative
dentries with the negative dentry limit set to 50% (> 80M dentries).
Ignoring some inherent noise in the test results, there wasn't any
noticeable difference in term of lookup and negative dentry creation
performance with or without this patch.

If the limit was set to 5% (the default), the 10M negative dentry
creation rate dropped to 199565 and the dentry-state was:

2344754 2326486 45      0       2316533 7494261

This was expected as negative dentry creation throttling with forced
dentry deletion happened in this case.

IOW, this patch does not cause any regression in term of lookup and
negative dentry creation performance as long as the limit hasn't been
reached.

Cheers,
Longman

[toc] | [next] | [standalone]

#1721907

On 08/28/2017 01:58 PM, Waiman Long wrote:
> On 07/28/2017 02:34 PM, Waiman Long wrote:
>>  v2->v3:
>>   - Add a faster pruning rate when the free pool is closed to depletion.
>>   - As suggested by James Bottomley, add an artificial delay waiting
>>     loop before killing a negative dentry and properly clear the
>>     DCACHE_KILL_NEGATIVE flag if killing doesn't happen.
>>   - Add a new patch to track number of negative dentries that are
>>     forcifully killed.
>>
>>  v1->v2:
>>   - Move the new nr_negative field to the end of dentry_stat_t structure
>>     as suggested by Matthew Wilcox.
>>   - With the help of Miklos Szeredi, fix incorrect locking order in
>>     dentry_kill() by using lock_parent() instead of locking the parent's
>>     d_lock directly.
>>   - Correctly account for positive to negative dentry transitions.
>>   - Automatic pruning of negative dentries will now ignore the reference
>>     bit in negative dentries but not the regular shrinking.
>>
>> A rogue application can potentially create a large number of negative
>> dentries in the system consuming most of the memory available. This
>> can impact performance of other applications running on the system.
>>
>> This patchset introduces changes to the dcache subsystem to limit
>> the number of negative dentries allowed to be created thus limiting
>> the amount of memory that can be consumed by negative dentries.
>>
>> Patch 1 tracks the number of negative dentries used and disallow
>> the creation of more when the limit is reached.
>>
>> Patch 2 enables /proc/sys/fs/dentry-state to report the number of
>> negative dentries in the system.
>>
>> Patch 3 enables automatic pruning of negative dentries when it is
>> close to the limit so that we won't end up killing recently used
>> negative dentries.
>>
>> Patch 4 prevents racing between negative dentry pruning and umount
>> operation.
>>
>> Patch 5 shows the number of forced negative dentry killings in
>> /proc/sys/fs/dentry-state. End users can then tune the neg_dentry_pc=
>> kernel boot parameter if they want to reduce forced negative dentry
>> killings.
>>
>> Waiman Long (5):
>>   fs/dcache: Limit numbers of negative dentries
>>   fs/dcache: Report negative dentry number in dentry-state
>>   fs/dcache: Enable automatic pruning of negative dentries
>>   fs/dcache: Protect negative dentry pruning from racing with umount
>>   fs/dcache: Track count of negative dentries forcibly killed
>>
>>  Documentation/admin-guide/kernel-parameters.txt |   7 +
>>  fs/dcache.c                                     | 451 ++++++++++++++++++++++--
>>  include/linux/dcache.h                          |   8 +-
>>  include/linux/list_lru.h                        |   1 +
>>  mm/list_lru.c                                   |   4 +-
>>  5 files changed, 435 insertions(+), 36 deletions(-)
>>
> With a 4.13 based kernel, the positive & negative dentries lookup rates
> (lookups per second) after initial boot on a 32GB memory VM with and
> without the patch were as follows:
>
>   Metric                    w/o patch        with patch
>   ------                    ---------        ----------
>   Positive dentry lookup      844881           842618
>   Negative dentry lookup     1865158          1901875
>   Negative dentry creation    609887           617215
>
> The last row refers to the creation rate of 10 millions negative
> dentries with the negative dentry limit set to 50% (> 80M dentries).
> Ignoring some inherent noise in the test results, there wasn't any
> noticeable difference in term of lookup and negative dentry creation
> performance with or without this patch.
>
> If the limit was set to 5% (the default), the 10M negative dentry
> creation rate dropped to 199565 and the dentry-state was:
>
> 2344754 2326486 45      0       2316533 7494261
>
> This was expected as negative dentry creation throttling with forced
> dentry deletion happened in this case.
>
> IOW, this patch does not cause any regression in term of lookup and
> negative dentry creation performance as long as the limit hasn't been
> reached.

Another performance data point about running the AIM7 highsystime
workload on a 36-core 32G VM is as follows:

Running the AIM7 high-systime workload on the VM, the baseline
performance was 186770 jobs/min. By running a single-thread rogue
negative dentry creation program in the background until the patched
kernel with 5% limit started throttling, the performance was 183746
jobs/min. On an unpatched kernel with memory almost exhausted and
memory shrinker was kicked in, the performance was 148997 jobs/min.

So the patch does protect the system from suffering significant
performance degradation in case a negative dentry creation rogue
program is runninig in the background.

Cheers,
Longman

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web