Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1200344 > unrolled thread

Re: [PATCH 0/3] Make workingset detection logic memcg aware

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH 0/3] Make workingset detection logic memcg aware Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> - 2015-08-05 03:40 +0200
    Re: [PATCH 0/3] Make workingset detection logic memcg aware Vladimir Davydov <vdavydov@parallels.com> - 2015-08-06 11:00 +0200
      Re: [PATCH 0/3] Make workingset detection logic memcg aware Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> - 2015-08-07 03:40 +0200
        Re: [PATCH 0/3] Make workingset detection logic memcg aware Vladimir Davydov <vdavydov@parallels.com> - 2015-08-08 15:10 +0200
          Re: [PATCH 0/3] Make workingset detection logic memcg aware Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> - 2015-08-09 16:20 +0200
            Re: [PATCH 0/3] Make workingset detection logic memcg aware Vladimir Davydov <vdavydov@parallels.com> - 2015-08-10 10:20 +0200
              Re: [PATCH 0/3] Make workingset detection logic memcg aware Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> - 2015-08-11 18:10 +0200

#1200344 — Re: [PATCH 0/3] Make workingset detection logic memcg aware

On 2015/08/03 21:04, Vladimir Davydov wrote:
> Hi,
> 
> Currently, workingset detection logic is not memcg aware - inactive_age
> is maintained per zone. As a result, if memory cgroups are used,
> refaulted file pages are activated randomly. This patch set makes
> inactive_age per lruvec so that workingset detection will work correctly
> for memory cgroup reclaim.
> 
> Thanks,
> 

Reading discussion, I feel storing more data is difficult, too.

I wonder, rather than collecting more data, rough calculation can help the situation.
for example,

   (refault_disatance calculated in zone) * memcg_reclaim_ratio < memcg's active list

If one of per-zone calc or per-memcg calc returns true, refault should be true.

memcg_reclaim_ratio is the percentage of scan in a memcg against in a zone.


Thanks,
-Kame



--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1201590

On Wed, Aug 05, 2015 at 10:34:58AM +0900, Kamezawa Hiroyuki wrote:

> Reading discussion, I feel storing more data is difficult, too.

Yep, even with the current 16-bit memcg id. Things would get even worse
if we wanted to extend it one day (will we?)

> 
> I wonder, rather than collecting more data, rough calculation can help the situation.
> for example,
> 
>    (refault_disatance calculated in zone) * memcg_reclaim_ratio < memcg's active list
> 
> If one of per-zone calc or per-memcg calc returns true, refault should be true.
> 
> memcg_reclaim_ratio is the percentage of scan in a memcg against in a zone.

This particular formula wouldn't work I'm afraid. If there are two
isolated cgroups issuing local reclaim on the same zone, the refault
distance needed for activation would be reduced by half for no apparent
reason.

The thing is that there is no need in inventing anything if refaults
from different cgroups are infrequent - it is enough to store only
zone/node ids in shadow entries then, as this patch set does. The
question remains, can we disregard them? Sometimes we need to sacrifice
accuracy for the sake of performance and/or code simplicity. E.g.
inter-cgroup concurrent file writes are not supported in the
implementation of the blkio writeback accounting AFAIK. May be, we could
neglect inter-cgroup refaults too? My point is that even if two cgroups
are actively sharing the same file, its pages will end up in the cgroup
which experiences less memory pressure (most likely, the one with the
greater limit), so inter-cgroup refaults should be rare. Am I wrong?

Anyway, workingset detection is broken for local reclaim (activations
are random) and needs to be fixed. What is worse, shadow entries are
accounted per memcg, but reclaimed only on global memory pressure, so
that they can eat all RAM available to a container w/o giving it a
chance to reclaim it. That said, even this patch set is a huge step
forward, because it makes activations much more deterministic and fixes
per memcg shadow nodes reclaim.

Thanks,
Vladimir
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1202190

On 2015/08/06 17:59, Vladimir Davydov wrote:
> On Wed, Aug 05, 2015 at 10:34:58AM +0900, Kamezawa Hiroyuki wrote:
>
>> Reading discussion, I feel storing more data is difficult, too.
>
> Yep, even with the current 16-bit memcg id. Things would get even worse
> if we wanted to extend it one day (will we?)
>
>>
>> I wonder, rather than collecting more data, rough calculation can help the situation.
>> for example,
>>
>>     (refault_disatance calculated in zone) * memcg_reclaim_ratio < memcg's active list
>>
>> If one of per-zone calc or per-memcg calc returns true, refault should be true.
>>
>> memcg_reclaim_ratio is the percentage of scan in a memcg against in a zone.
>
> This particular formula wouldn't work I'm afraid. If there are two
> isolated cgroups issuing local reclaim on the same zone, the refault
> distance needed for activation would be reduced by half for no apparent
> reason.

Hmm, you mean activation in memcg means activation in global LRU, and it's not a
valid reason. Current implementation does have the same issue, right ?

i.e. when a container has been hitting its limit for a while, and then, a file cache is
pushed out but came back soon, it can be easily activated.

I'd like to confirm what you want to do.

  1) avoid activating a file cache when it was kicked out because of memcg's local limit.
  2) maintain acitve/inactive ratio in memcg properly as global LRU does.
  3) reclaim shadow entry at proper timing.

All ? hmm. It seems that mixture of record of global memory pressure and of local memory
pressure is just wrong.

Now, the record is
　　　　
　　　　eviction | node | zone | 2bit.

How about changing this as

         0 |eviction | node | zone | 2bit
         1 |eviction |  memcgid    | 2bit

Assume each memcg has an eviction counter, which ignoring node/zone.
i.e. memcg local reclaim happens against memcg not against zone.

At page-in,
         if (the 1st bit is 0)
                 compare eviction counter with zone's counter and activate the page if needed.
         else if (the 1st bit is 1)
                 compare eviction counter with the memcg (if exists)
                 if (current memcg == recorded memcg && eviction distance is okay)
                      activate page.
                 else
                      inactivate
       
At page-out
         if (global memory pressure)
                 record eviction id with using zone's counter.
         else if (memcg local memory pressure)
                 record eviction id with memcg's counter.

By this,
    1) locally reclaimed pages cannot be activated unless it's refaulted in the same memcg.
       In this case, activating in the memcg has some meaning.

    2) At global memory pressure, distance is properly calculated based on global system status.
       global memory pressure can ignore memcg's behavior.

about shadow entries, kmemcg should take care of it....


Thanks,
-Kame




--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1203267

On Fri, Aug 07, 2015 at 10:38:16AM +0900, Kamezawa Hiroyuki wrote:
> On 2015/08/06 17:59, Vladimir Davydov wrote:
> >On Wed, Aug 05, 2015 at 10:34:58AM +0900, Kamezawa Hiroyuki wrote:
> >>I wonder, rather than collecting more data, rough calculation can help the situation.
> >>for example,
> >>
> >>    (refault_disatance calculated in zone) * memcg_reclaim_ratio < memcg's active list
> >>
> >>If one of per-zone calc or per-memcg calc returns true, refault should be true.
> >>
> >>memcg_reclaim_ratio is the percentage of scan in a memcg against in a zone.
> >
> >This particular formula wouldn't work I'm afraid. If there are two
> >isolated cgroups issuing local reclaim on the same zone, the refault
> >distance needed for activation would be reduced by half for no apparent
> >reason.
> 
> Hmm, you mean activation in memcg means activation in global LRU, and it's not a
> valid reason. Current implementation does have the same issue, right ?
> 
> i.e. when a container has been hitting its limit for a while, and then, a file cache is
> pushed out but came back soon, it can be easily activated.
> 
> I'd like to confirm what you want to do.
> 
>  1) avoid activating a file cache when it was kicked out because of memcg's local limit.

No, that's not what I want. I want pages of the workingset to get
activated on refault no matter if they were evicted on global memory
pressure or due to hitting a memory cgroup limit.

>  2) maintain acitve/inactive ratio in memcg properly as global LRU does.
>  3) reclaim shadow entry at proper timing.
> 
> All ? hmm. It seems that mixture of record of global memory pressure and of local memory
> pressure is just wrong.

What makes you think so? An example of misbehavior caused by this would
be nice to have.

> 
> Now, the record is
> 　　　　
> 　　　　eviction | node | zone | 2bit.
> 
> How about changing this as
> 
>         0 |eviction | node | zone | 2bit
>         1 |eviction |  memcgid    | 2bit
> 
> Assume each memcg has an eviction counter, which ignoring node/zone.
> i.e. memcg local reclaim happens against memcg not against zone.
> 
> At page-in,
>         if (the 1st bit is 0)
>                 compare eviction counter with zone's counter and activate the page if needed.
>         else if (the 1st bit is 1)
>                 compare eviction counter with the memcg (if exists)

Having a single counter per memcg won't scale with the number of NUMA
nodes.

>                 if (current memcg == recorded memcg && eviction distance is okay)
>                      activate page.
>                 else
>                      inactivate
> At page-out
>         if (global memory pressure)
>                 record eviction id with using zone's counter.
>         else if (memcg local memory pressure)
>                 record eviction id with memcg's counter.
> 

I don't understand how this is supposed to work when a memory cgroup
experiences both local and global pressure simultaneously.

Also, what if a memory cgroup is protected by memory.low? Such a cgroup
may have all its pages in the active list, because it is never scanned.
This will affect the refault distance of other cgroups, making
activations unpredictable.

Thanks,
Vladimir
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1203645

On 2015/08/08 22:05, Vladimir Davydov wrote:
> On Fri, Aug 07, 2015 at 10:38:16AM +0900, Kamezawa Hiroyuki wrote:
>> On 2015/08/06 17:59, Vladimir Davydov wrote:
>>> On Wed, Aug 05, 2015 at 10:34:58AM +0900, Kamezawa Hiroyuki wrote:
>>>> I wonder, rather than collecting more data, rough calculation can help the situation.
>>>> for example,
>>>>
>>>>     (refault_disatance calculated in zone) * memcg_reclaim_ratio < memcg's active list
>>>>
>>>> If one of per-zone calc or per-memcg calc returns true, refault should be true.
>>>>
>>>> memcg_reclaim_ratio is the percentage of scan in a memcg against in a zone.
>>>
>>> This particular formula wouldn't work I'm afraid. If there are two
>>> isolated cgroups issuing local reclaim on the same zone, the refault
>>> distance needed for activation would be reduced by half for no apparent
>>> reason.
>>
>> Hmm, you mean activation in memcg means activation in global LRU, and it's not a
>> valid reason. Current implementation does have the same issue, right ?
>>
>> i.e. when a container has been hitting its limit for a while, and then, a file cache is
>> pushed out but came back soon, it can be easily activated.
>>
>> I'd like to confirm what you want to do.
>>
>>   1) avoid activating a file cache when it was kicked out because of memcg's local limit.
>
> No, that's not what I want. I want pages of the workingset to get
> activated on refault no matter if they were evicted on global memory
> pressure or due to hitting a memory cgroup limit.
>

Sure.

>>   2) maintain acitve/inactive ratio in memcg properly as global LRU does.
>>   3) reclaim shadow entry at proper timing.
>>
>> All ? hmm. It seems that mixture of record of global memory pressure and of local memory
>> pressure is just wrong.
>
> What makes you think so? An example of misbehavior caused by this would
> be nice to have.
>

By design, memcg's LRU aging logic is independent from global memory allocation/pressure.


Assume there are 4 containers(using much page-cache) with 1GB limit on 4GB server,
   # contaienr A  workingset=600M   limit=1G (sleepy)
   # contaienr B  workingset=300M   limit=1G (work often)
   # container C  workingset=500M   limit=1G (work slowly)
   # container D  workingset=1.2G   limit=1G (work hard)
  
container D can drive the zone's distance counter because of local memory reclaim.
If active/inactive = 1:1, container D page can be activated.
At kswapd(global reclaim) runs, all container's LRU will rotate.

Possibility of refault in A, B, C is reduced by conainer D's counter updates.

But yes, some _real_  test are required.

>>
>> Now, the record is
>> 　　　　
>> 　　　　eviction | node | zone | 2bit.
>>
>> How about changing this as
>>
>>          0 |eviction | node | zone | 2bit
>>          1 |eviction |  memcgid    | 2bit
>>
>> Assume each memcg has an eviction counter, which ignoring node/zone.
>> i.e. memcg local reclaim happens against memcg not against zone.
>>
>> At page-in,
>>          if (the 1st bit is 0)
>>                  compare eviction counter with zone's counter and activate the page if needed.
>>          else if (the 1st bit is 1)
>>                  compare eviction counter with the memcg (if exists)
>
> Having a single counter per memcg won't scale with the number of NUMA
> nodes.
>
It doesn't matter, we can use lazy counter like pcpu counter because it's not needed to be very accurate.


>>                  if (current memcg == recorded memcg && eviction distance is okay)
>>                       activate page.
>>                  else
>>                       inactivate
>> At page-out
>>          if (global memory pressure)
>>                  record eviction id with using zone's counter.
>>          else if (memcg local memory pressure)
>>                  record eviction id with memcg's counter.
>>
>
> I don't understand how this is supposed to work when a memory cgroup
> experiences both local and global pressure simultaneously.
>

I think updating global distance counter by local reclaim may update counter too much.
Above is to avoid updating zone's counter and keep memcg's LRU active/inactive balanced.

> Also, what if a memory cgroup is protected by memory.low? Such a cgroup
> may have all its pages in the active list, because it is never scanned.

If LRU never scanned, all file caches tend to be in INACTIVE...it never refaults.

> This will affect the refault distance of other cgroups, making
> activations unpredictable.
>



Thanks,
-Kame







--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1203882

On Sun, Aug 09, 2015 at 11:12:25PM +0900, Kamezawa Hiroyuki wrote:
> On 2015/08/08 22:05, Vladimir Davydov wrote:
> >On Fri, Aug 07, 2015 at 10:38:16AM +0900, Kamezawa Hiroyuki wrote:
...
> >>All ? hmm. It seems that mixture of record of global memory pressure and of local memory
> >>pressure is just wrong.
> >
> >What makes you think so? An example of misbehavior caused by this would
> >be nice to have.
> >
> 
> By design, memcg's LRU aging logic is independent from global memory allocation/pressure.
> 
> 
> Assume there are 4 containers(using much page-cache) with 1GB limit on 4GB server,
>   # contaienr A  workingset=600M   limit=1G (sleepy)
>   # contaienr B  workingset=300M   limit=1G (work often)
>   # container C  workingset=500M   limit=1G (work slowly)
>   # container D  workingset=1.2G   limit=1G (work hard)
> container D can drive the zone's distance counter because of local memory reclaim.
> If active/inactive = 1:1, container D page can be activated.
> At kswapd(global reclaim) runs, all container's LRU will rotate.
> 
> Possibility of refault in A, B, C is reduced by conainer D's counter updates.

This does not necessarily mean we have to use different inactive_age
counter for global and local memory pressure. In your example, having
inactive_age per lruvec and using it for evictions on both global and
local memory pressure would work just fine.

> 
> But yes, some _real_  test are required.
> 
> >>
> >>Now, the record is
> >>　　　　
> >>　　　　eviction | node | zone | 2bit.
> >>
> >>How about changing this as
> >>
> >>         0 |eviction | node | zone | 2bit
> >>         1 |eviction |  memcgid    | 2bit
> >>
> >>Assume each memcg has an eviction counter, which ignoring node/zone.
> >>i.e. memcg local reclaim happens against memcg not against zone.
> >>
> >>At page-in,
> >>         if (the 1st bit is 0)
> >>                 compare eviction counter with zone's counter and activate the page if needed.
> >>         else if (the 1st bit is 1)
> >>                 compare eviction counter with the memcg (if exists)
> >
> >Having a single counter per memcg won't scale with the number of NUMA
> >nodes.
> >
> It doesn't matter, we can use lazy counter like pcpu counter because it's not needed to be very accurate.

Fair enough.

> 
> 
> >>                 if (current memcg == recorded memcg && eviction distance is okay)
> >>                      activate page.
> >>                 else
> >>                      inactivate
> >>At page-out
> >>         if (global memory pressure)
> >>                 record eviction id with using zone's counter.
> >>         else if (memcg local memory pressure)
> >>                 record eviction id with memcg's counter.
> >>
> >
> >I don't understand how this is supposed to work when a memory cgroup
> >experiences both local and global pressure simultaneously.
> >
> 
> I think updating global distance counter by local reclaim may update counter too much.

But if the inactive_age counter was per lruvec, then we wouldn't need to
bother about it.

> Above is to avoid updating zone's counter and keep memcg's LRU active/inactive balanced.
> 
> >Also, what if a memory cgroup is protected by memory.low? Such a cgroup
> >may have all its pages in the active list, because it is never scanned.
> 
> If LRU never scanned, all file caches tend to be in INACTIVE...it never refaults.

This is not true - there still may be activations from
mark_page_accessed.

Thanks,
Vladimir
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1205245

On 2015/08/10 17:14, Vladimir Davydov wrote:
> On Sun, Aug 09, 2015 at 11:12:25PM +0900, Kamezawa Hiroyuki wrote:
>> On 2015/08/08 22:05, Vladimir Davydov wrote:
>>> On Fri, Aug 07, 2015 at 10:38:16AM +0900, Kamezawa Hiroyuki wrote:
> ...
>>>> All ? hmm. It seems that mixture of record of global memory pressure and of local memory
>>>> pressure is just wrong.
>>>
>>> What makes you think so? An example of misbehavior caused by this would
>>> be nice to have.
>>>
>>
>> By design, memcg's LRU aging logic is independent from global memory allocation/pressure.
>>
>>
>> Assume there are 4 containers(using much page-cache) with 1GB limit on 4GB server,
>>    # contaienr A  workingset=600M   limit=1G (sleepy)
>>    # contaienr B  workingset=300M   limit=1G (work often)
>>    # container C  workingset=500M   limit=1G (work slowly)
>>    # container D  workingset=1.2G   limit=1G (work hard)
>> container D can drive the zone's distance counter because of local memory reclaim.
>> If active/inactive = 1:1, container D page can be activated.
>> At kswapd(global reclaim) runs, all container's LRU will rotate.
>>
>> Possibility of refault in A, B, C is reduced by conainer D's counter updates.
>
> This does not necessarily mean we have to use different inactive_age
> counter for global and local memory pressure. In your example, having
> inactive_age per lruvec and using it for evictions on both global and
> local memory pressure would work just fine.
>

you're right.

  
>>
>>
>>>>                  if (current memcg == recorded memcg && eviction distance is okay)
>>>>                       activate page.
>>>>                  else
>>>>                       inactivate
>>>> At page-out
>>>>          if (global memory pressure)
>>>>                  record eviction id with using zone's counter.
>>>>          else if (memcg local memory pressure)
>>>>                  record eviction id with memcg's counter.
>>>>
>>>
>>> I don't understand how this is supposed to work when a memory cgroup
>>> experiences both local and global pressure simultaneously.
>>>
>>
>> I think updating global distance counter by local reclaim may update counter too much.
>
> But if the inactive_age counter was per lruvec, then we wouldn't need to
> bother about it.
>
yes.

Anyway, what I understand now is that we need to reduce influence from a memcg's behavior
against other memcgs. Your way is dividing counter completely, my idea was implementing
different counter. Doing it by calculation will be good because we can't have enough record
space.


Thanks,
-Kame

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web