Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1378168

Re: [BUG] cgroup/workques/fork: deadlock when moving cgroups

Show all headers | View raw

Hello, Petr.

(cc'ing Johannes)

On Wed, Apr 13, 2016 at 11:42:16AM +0200, Petr Mladek wrote:
...
> By other words, "memcg_move_char/2860" flushes a work. But it cannot
> get flushed because one worker is blocked and another one could not
> get created. All these operations are blocked by the very same
> "memcg_move_char/2860".
> 
> Note that also "systemd/1" is waiting for "cgroup_mutex" in
> proc_cgroup_show(). But it seems that it is not in the main
> cycle causing the deadlock.
> 
> I am able to reproduce this problem quite easily (within few minutes).
> There are often even more tasks waiting for the cgroups-related locks
> but they are not causing the deadlock.
> 
> 
> The question is how to solve this problem. I see several possibilities:
> 
>   + avoid using workqueues in lru_add_drain_all()
> 
>   + make lru_add_drain_all() killable and restartable
> 
>   + do not block fork() when lru_add_drain_all() is running,
>     e.g. using some lazy techniques like RCU, workqueues
> 
>   + at least do not block fork of workers; AFAIK, they have a limited
>      cgroups usage anyway because they are marked with PF_NO_SETAFFINITY
> 
> 
> I am willing to test any potential fix or even work on the fix.
> But I do not have that big insight into the problem, so I would
> need some pointers.

An easy solution would be to make lru_add_drain_all() use a
WQ_MEM_RECLAIM workqueue.  A better way would be making charge moving
asynchronous similar to cpuset node migration but I don't know whether
that's realistic.  Will prep a patch to add a rescuer to
lru_add_drain_all().

Thanks.

-- 
tejun

Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

[BUG] cgroup/workques/fork: deadlock when moving cgroups Petr Mladek <pmladek@suse.com> - 2016-04-13 11:50 +0200
  Re: [BUG] cgroup/workques/fork: deadlock when moving cgroups Tejun Heo <tj@kernel.org> - 2016-04-13 20:40 +0200
    Re: [BUG] cgroup/workques/fork: deadlock when moving cgroups Tejun Heo <tj@kernel.org> - 2016-04-13 21:00 +0200
    Re: [BUG] cgroup/workques/fork: deadlock when moving cgroups Michal Hocko <mhocko@kernel.org> - 2016-04-13 21:30 +0200
    Re: [BUG] cgroup/workques/fork: deadlock when moving cgroups Michal Hocko <mhocko@kernel.org> - 2016-04-13 21:30 +0200
      Re: [BUG] cgroup/workques/fork: deadlock when moving cgroups Michal Hocko <mhocko@kernel.org> - 2016-04-13 21:50 +0200
        Re: [BUG] cgroup/workques/fork: deadlock when moving cgroups Michal Hocko <mhocko@kernel.org> - 2016-04-14 09:10 +0200
          Re: [BUG] cgroup/workques/fork: deadlock when moving cgroups Tejun Heo <tj@kernel.org> - 2016-04-14 17:40 +0200
      Re: [BUG] cgroup/workques/fork: deadlock when moving cgroups Tejun Heo <tj@kernel.org> - 2016-04-13 21:50 +0200
      Re: [BUG] cgroup/workques/fork: deadlock when moving cgroups Johannes Weiner <hannes@cmpxchg.org> - 2016-04-14 20:00 +0200
        Re: [BUG] cgroup/workques/fork: deadlock when moving cgroups Michal Hocko <mhocko@kernel.org> - 2016-04-15 09:10 +0200
          Re: [BUG] cgroup/workques/fork: deadlock when moving cgroups Tejun Heo <tj@kernel.org> - 2016-04-15 16:40 +0200
            Re: [BUG] cgroup/workques/fork: deadlock when moving cgroups Michal Hocko <mhocko@kernel.org> - 2016-04-15 17:10 +0200
              Re: [BUG] cgroup/workques/fork: deadlock when moving cgroups Tejun Heo <tj@kernel.org> - 2016-04-15 17:30 +0200

csiph-web