Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #211703 > unrolled thread

Slab Unreclaimable is continually growing

Started byMatthias Böttcher <matthias.boettcher@gmail.com>
First post2019-07-24 19:00 +0200
Last post2019-09-19 09:40 +0200
Articles 8 — 3 participants

Back to article view | Back to linux.debian.user

Contents

  Slab Unreclaimable is continually growing Matthias Böttcher <matthias.boettcher@gmail.com> - 2019-07-24 19:00 +0200
    Re: Slab Unreclaimable is continually growing Shahryar Afifi <shahryarafifi@yahoo.com> - 2019-07-24 21:10 +0200
      Re: Slab Unreclaimable is continually growing Matthias Böttcher <matthias.boettcher@gmail.com> - 2019-07-26 20:10 +0200
    Re: Slab Unreclaimable is continually growing Reco <recoverym4n@enotuniq.net> - 2019-07-24 21:50 +0200
      Re: Slab Unreclaimable is continually growing Matthias Böttcher <matthias.boettcher@gmail.com> - 2019-07-26 21:00 +0200
        Re: Slab Unreclaimable is continually growing Reco <recoverym4n@enotuniq.net> - 2019-07-26 21:20 +0200
          Re: Slab Unreclaimable is continually growing Matthias Böttcher <matthias.boettcher@gmail.com> - 2019-07-27 12:00 +0200
            Re: Slab Unreclaimable is continually growing Matthias Böttcher <matthias.boettcher@gmail.com> - 2019-09-19 09:40 +0200

#211703 — Slab Unreclaimable is continually growing

FromMatthias Böttcher <matthias.boettcher@gmail.com>
Date2019-07-24 19:00 +0200
SubjectSlab Unreclaimable is continually growing
Message-ID<ynsMp-4t4-1@gated-at.bofh.it>
Hi list,

my buster is using more and more memory until the oom-killer is
invoked. This occurs after two days of uptime, but no process is using
that memory. Slab seems suspicious to me.

Kernel: 4.19.37-5+deb10u1 (2019-07-19) x86_64

from /proc/meminfo after 8 hours uptime:

MemTotal:        4041116 kB
MemFree:         3165992 kB
MemAvailable:    3246560 kB
Buffers:           30144 kB
Cached:           240356 kB
SwapCached:            0 kB
Active:           162056 kB
Inactive:         144268 kB
SwapTotal:       4194300 kB
SwapFree:        4194300 kB

Slab:             521364 kB
SReclaimable:      66776 kB
SUnreclaim:       454588 kB

My Top 5 in slabtop:

$ sudo slabtop --sort c --once | head -n12

 Active / Total Objects (% used)    : 2234301 / 2270100 (98,4%)
 Active / Total Slabs (% used)      : 70832 / 70832 (100,0%)
 Active / Total Caches (% used)     : 98 / 124 (79,0%)
 Active / Total Size (% used)       : 501034,52K / 509459,17K (98,3%)
 Minimum / Average / Maximum Object : 0,01K / 0,22K / 8,00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
307534 304741  99%    0,20K  16186       19     64744K vm_area_struct
 14280  14274  99%    3,69K   1785        8     57120K task_struct
178048 152224  85%    0,25K  11128       16     44512K filp
  8536   8536 100%    4,00K   1067        8     34144K kmalloc-4096
 14640  14640 100%    2,06K    976       15     31232K sighand_cache

Active / Total Size is growing continually about 60 MB per hour.

How can I detect what is eating up my memory in SUnreclaim (slab unreclaimable)?

Thanks in advance
Matthias Böttcher

[toc] | [next] | [standalone]

#211709

FromShahryar Afifi <shahryarafifi@yahoo.com>
Date2019-07-24 21:10 +0200
Message-ID<ynuOe-5ZS-5@gated-at.bofh.it>
In reply to#211703
What happens  when you run sync in terminal? 



On Wed, 2019-07-24 at 18:54 +0200, Matthias B??ttcher wrote:
> Hi list,
> 
> my buster is using more and more memory until the oom-killer is
> invoked. This occurs after two days of uptime, but no process is
> using
> that memory. Slab seems suspicious to me.
> 
> Kernel: 4.19.37-5+deb10u1 (2019-07-19) x86_64
> 
> from /proc/meminfo after 8 hours uptime:
> 
> MemTotal:        4041116 kB
> MemFree:         3165992 kB
> MemAvailable:    3246560 kB
> Buffers:           30144 kB
> Cached:           240356 kB
> SwapCached:            0 kB
> Active:           162056 kB
> Inactive:         144268 kB
> SwapTotal:       4194300 kB
> SwapFree:        4194300 kB
> 
> Slab:             521364 kB
> SReclaimable:      66776 kB
> SUnreclaim:       454588 kB
> 
> My Top 5 in slabtop:
> 
> $ sudo slabtop --sort c --once | head -n12
> 
>  Active / Total Objects (% used)    : 2234301 / 2270100 (98,4%)
>  Active / Total Slabs (% used)      : 70832 / 70832 (100,0%)
>  Active / Total Caches (% used)     : 98 / 124 (79,0%)
>  Active / Total Size (% used)       : 501034,52K / 509459,17K (98,3%)
>  Minimum / Average / Maximum Object : 0,01K / 0,22K / 8,00K
> 
>   OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
> 307534 304741  99%    0,20K  16186       19     64744K vm_area_struct
>  14280  14274  99%    3,69K   1785        8     57120K task_struct
> 178048 152224  85%    0,25K  11128       16     44512K filp
>   8536   8536 100%    4,00K   1067        8     34144K kmalloc-4096
>  14640  14640 100%    2,06K    976       15     31232K sighand_cache
> 
> Active / Total Size is growing continually about 60 MB per hour.
> 
> How can I detect what is eating up my memory in SUnreclaim (slab
> unreclaimable)?
> 
> Thanks in advance
> Matthias B??ttcher
>

[toc] | [prev] | [next] | [standalone]

#211779

FromMatthias Böttcher <matthias.boettcher@gmail.com>
Date2019-07-26 20:10 +0200
Message-ID<yocPg-DL-13@gated-at.bofh.it>
In reply to#211709
Shahryar Afifi <shahryarafifi@yahoo.com>:
>
> What happens  when you run sync in terminal?

sync exits immediately and slabtop gives the same output like before sync.

[toc] | [prev] | [next] | [standalone]

#211710

FromReco <recoverym4n@enotuniq.net>
Date2019-07-24 21:50 +0200
Message-ID<ynvqV-6dM-1@gated-at.bofh.it>
In reply to#211703
	Hi.

On Wed, Jul 24, 2019 at 06:54:42PM +0200, Matthias Böttcher wrote:
>   OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
> 307534 304741  99%    0,20K  16186       19     64744K vm_area_struct
>  14280  14274  99%    3,69K   1785        8     57120K task_struct
> 178048 152224  85%    0,25K  11128       16     44512K filp
>   8536   8536 100%    4,00K   1067        8     34144K kmalloc-4096
>  14640  14640 100%    2,06K    976       15     31232K sighand_cache
> 
> How can I detect what is eating up my memory in SUnreclaim (slab unreclaimable)?

You did it already.
"vm_area_struct" is a kernel structure for anonymous memory allocations.
"task_struct" is a kernel structure for maintaining process execution.
"filp" is a kernel structure for virtual memory.

My guess is - a small number of processes that constantly allocate
memory in small numbers by executing brk(2) or its modern equivalents.

Or a relatively large number of short-lived processes.


I'd start with "pidstat -rl 1 10".

Reco

[toc] | [prev] | [next] | [standalone]

#211782

FromMatthias Böttcher <matthias.boettcher@gmail.com>
Date2019-07-26 21:00 +0200
Message-ID<yodBD-WI-9@gated-at.bofh.it>
In reply to#211710
Reco <recoverym4n@enotuniq.net>:
>
>         Hi.
>
> On Wed, Jul 24, 2019 at 06:54:42PM +0200, Matthias Böttcher wrote:
> >   OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
> > 307534 304741  99%    0,20K  16186       19     64744K vm_area_struct
> >  14280  14274  99%    3,69K   1785        8     57120K task_struct
> > 178048 152224  85%    0,25K  11128       16     44512K filp
> >   8536   8536 100%    4,00K   1067        8     34144K kmalloc-4096
> >  14640  14640 100%    2,06K    976       15     31232K sighand_cache
> >
> > How can I detect what is eating up my memory in SUnreclaim (slab unreclaimable)?
>
> You did it already.
> "vm_area_struct" is a kernel structure for anonymous memory allocations.
> "task_struct" is a kernel structure for maintaining process execution.
> "filp" is a kernel structure for virtual memory.
>
> My guess is - a small number of processes that constantly allocate
> memory in small numbers by executing brk(2) or its modern equivalents.
>
> Or a relatively large number of short-lived processes.
>
>
> I'd start with "pidstat -rl 1 10".

Now with an uptime of 2 days, 9 hours all counters of slabtop are
growing no more.

$ slabtop --sort c --once | head -n12
 Active / Total Objects (% used)    : 10336938 / 10484768 (98,6%)
 Active / Total Slabs (% used)      : 328327 / 328327 (100,0%)
 Active / Total Caches (% used)     : 98 / 124 (79,0%)
 Active / Total Size (% used)       : 2443615,58K / 2479644,41K (98,5%)
 Minimum / Average / Maximum Object : 0,01K / 0,24K / 8,00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
308520 307456  99%    1,05K  20568       15    329088K ext4_inode_cache
1285388 1269730  98%    0,20K  67652       19    270608K vm_area_struct
 61712  61692  99%    3,69K   7714        8    246848K task_struct
765472 649628  84%    0,25K  47842       16    191368K filp
 36088  36083  99%    4,00K   4511        8    144352K kmalloc-4096

What I saw with "pidstat -rl 1 10" was systemd-journald and nmbd, so I did:

    sudo apt purge samba     # Samba was not needed

    sudo systemctl stop systemd-journald-dev-log.socket \
         systemd-journald-audit.socket systemd-journald.socket

and additionally I stopped the socket for the Check_MK agent:

    sudo systemctl stop check_mk.socket

OK, these are 3 changes at once, so I have to figure out which change does it.

Matthias

[toc] | [prev] | [next] | [standalone]

#211785

FromReco <recoverym4n@enotuniq.net>
Date2019-07-26 21:20 +0200
Message-ID<yodV0-1jC-7@gated-at.bofh.it>
In reply to#211782
On Fri, Jul 26, 2019 at 08:55:19PM +0200, Matthias Böttcher wrote:
> Reco <recoverym4n@enotuniq.net>:
> >
> >         Hi.
> >
> > On Wed, Jul 24, 2019 at 06:54:42PM +0200, Matthias Böttcher wrote:
> > >   OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
> > > 307534 304741  99%    0,20K  16186       19     64744K vm_area_struct
> > >  14280  14274  99%    3,69K   1785        8     57120K task_struct
> > > 178048 152224  85%    0,25K  11128       16     44512K filp
> > >   8536   8536 100%    4,00K   1067        8     34144K kmalloc-4096
> > >  14640  14640 100%    2,06K    976       15     31232K sighand_cache
> > >
> > > How can I detect what is eating up my memory in SUnreclaim (slab unreclaimable)?
> >
> > You did it already.
> > "vm_area_struct" is a kernel structure for anonymous memory allocations.
> > "task_struct" is a kernel structure for maintaining process execution.
> > "filp" is a kernel structure for virtual memory.
> >
> > My guess is - a small number of processes that constantly allocate
> > memory in small numbers by executing brk(2) or its modern equivalents.
> >
> > Or a relatively large number of short-lived processes.
> >
> >
> > I'd start with "pidstat -rl 1 10".
> 
> Now with an uptime of 2 days, 9 hours all counters of slabtop are
> growing no more.
> 
> $ slabtop --sort c --once | head -n12
>  Active / Total Objects (% used)    : 10336938 / 10484768 (98,6%)
>  Active / Total Slabs (% used)      : 328327 / 328327 (100,0%)
>  Active / Total Caches (% used)     : 98 / 124 (79,0%)
>  Active / Total Size (% used)       : 2443615,58K / 2479644,41K (98,5%)
>  Minimum / Average / Maximum Object : 0,01K / 0,24K / 8,00K
> 
>   OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
> 308520 307456  99%    1,05K  20568       15    329088K ext4_inode_cache
> 1285388 1269730  98%    0,20K  67652       19    270608K vm_area_struct
>  61712  61692  99%    3,69K   7714        8    246848K task_struct
> 765472 649628  84%    0,25K  47842       16    191368K filp
>  36088  36083  99%    4,00K   4511        8    144352K kmalloc-4096

"ext4_inode_cache" is the usual top consumer of SLAB. I kind of
surprised that "dentry" did not make it to the top, but that can be
attributed to the filesystem usage on that server.


> What I saw with "pidstat -rl 1 10" was systemd-journald and nmbd, so I did:
> 
>     sudo apt purge samba     # Samba was not needed

Possible, but unlikely. Samba is popular, such abnormal memory
allocations would be widely known.


>     sudo systemctl stop systemd-journald-dev-log.socket \
>          systemd-journald-audit.socket systemd-journald.socket

It's redundant if you have rsyslog anyway, but again, it's a popular
software, such things would be noticed. Unless of course, you have
something that's spamming audit records - some Apparmor profile in
complain state.



> and additionally I stopped the socket for the Check_MK agent:
> 
>     sudo systemctl stop check_mk.socket

I do not know this one. What's its purpose? Monitoring, backup,
something else? Is there a source available.


On a side note, you seem to have enough used memory to consider turning
on transparent hugepages. It should help with the size of
"vm_area_struct" and "filp".

Reco

[toc] | [prev] | [next] | [standalone]

#211789

FromMatthias Böttcher <matthias.boettcher@gmail.com>
Date2019-07-27 12:00 +0200
Message-ID<yorEB-1DK-1@gated-at.bofh.it>
In reply to#211785
On Fri, Jul 26, 2019 at 10:12:20PM +0300, Reco wrote:
> On Fri, Jul 26, 2019 at 08:55:19PM +0200, Matthias Böttcher wrote:

[...]

> > and additionally I stopped the socket for the Check_MK agent:
> > 
> >     sudo systemctl stop check_mk.socket
> 
> I do not know this one. What's its purpose? Monitoring, backup,
> something else? Is there a source available.

Checkmk, formerly Check_MK, is a monitoring software[1].
I am using the Checkmk Raw Edition (CRE) open-source.
Per default it gets data from an agent on a host with connect to 6556/tcp
every 60 seconds. All counters are fetched in one connect,
nothing is sent to the host. The agent for Linux is one big bash script[2].

The deb with the agent brings systemd integration:

--------------------------------------------
$ cat "/etc/systemd/system/check_mk@.service"

# systemd service definition file
[Unit]
Description=Check_MK

[Service]
ExecStart=/usr/bin/check_mk_agent
KillMode=process

User=root
Group=root

StandardInput=socket

--------------------------------------------
$ cat /etc/systemd/system/check_mk.socket

# systemd socket definition file
[Unit]
Description=Check_MK Agent Socket

[Socket]
ListenStream=6556
Accept=true

[Install]
WantedBy=sockets.target
--------------------------------------------

With check_mk.socket started, SUnreclaim and vm_area_struct are growing
endlessly. So I will investigate the checkmk agent.

> Reco

Reco, Thank you for your explanations and bumping me in the right direction.

Matthias

[1] https://checkmk.com/
[2] https://github.com/tribe29/checkmk/blob/master/agents/check_mk_agent.linux

[toc] | [prev] | [next] | [standalone]

#214165

FromMatthias Böttcher <matthias.boettcher@gmail.com>
Date2019-09-19 09:40 +0200
Message-ID<yHZcJ-Kh-1@gated-at.bofh.it>
In reply to#211789
Just for reference:

check_mk_agent caused the endlessly growing of SUnreclaim in
/proc/meminfo and similary vm_area_struct in slabtop if the typical
conditions are met in a Debian Buster with check-mk-agent:

- Debian 10 Buster
    systemd        241-7~deb10u1 amd64
    Kernel 4.19.67-2, but with newer kernels too.

- check-mk-agent
    my version is 1.5.0p18, but with newer versions too

- check_mk.socket is active in systemd and used via 6556/tcp

- one or more checks are cached
    for example ntp in check_mk_agent or the simple
    echo "0 myservice - OK: This is my custom output"

tribe29 as creators of checkmk is aware of this, has observed it only
in Debian Buster and recommends in Buster xinetd for the listening
socket for its agent.

https://lists.mathias-kettner.de/pipermail/checkmk-en/2019-September/028675.html
https://lists.mathias-kettner.de/pipermail/checkmk-en/2019-September/028676.html

Matthias

Am Sa., 27. Juli 2019 um 11:59 Uhr schrieb Matthias Böttcher
<matthias.boettcher@gmail.com>:
>
> On Fri, Jul 26, 2019 at 10:12:20PM +0300, Reco wrote:
> > On Fri, Jul 26, 2019 at 08:55:19PM +0200, Matthias Böttcher wrote:
>
> [...]
>
> > > and additionally I stopped the socket for the Check_MK agent:
> > >
> > >     sudo systemctl stop check_mk.socket
> >
> > I do not know this one. What's its purpose? Monitoring, backup,
> > something else? Is there a source available.
>
> Checkmk, formerly Check_MK, is a monitoring software[1].
> I am using the Checkmk Raw Edition (CRE) open-source.
> Per default it gets data from an agent on a host with connect to 6556/tcp
> every 60 seconds. All counters are fetched in one connect,
> nothing is sent to the host. The agent for Linux is one big bash script[2].
>
> The deb with the agent brings systemd integration:
>
> --------------------------------------------
> $ cat "/etc/systemd/system/check_mk@.service"
>
> # systemd service definition file
> [Unit]
> Description=Check_MK
>
> [Service]
> ExecStart=/usr/bin/check_mk_agent
> KillMode=process
>
> User=root
> Group=root
>
> StandardInput=socket
>
> --------------------------------------------
> $ cat /etc/systemd/system/check_mk.socket
>
> # systemd socket definition file
> [Unit]
> Description=Check_MK Agent Socket
>
> [Socket]
> ListenStream=6556
> Accept=true
>
> [Install]
> WantedBy=sockets.target
> --------------------------------------------
>
> With check_mk.socket started, SUnreclaim and vm_area_struct are growing
> endlessly. So I will investigate the checkmk agent.
>
> > Reco
>
> Reco, Thank you for your explanations and bumping me in the right direction.
>
> Matthias
>
> [1] https://checkmk.com/
> [2] https://github.com/tribe29/checkmk/blob/master/agents/check_mk_agent.linux
>

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web