Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #274119 > unrolled thread

Small server hanging after some days - top, load avg and Mem buff/cache question

Back to article view | Back to linux.debian.user

Contents

  Small server hanging after some days - top,  load avg and Mem buff/cache question "B.M." <b-misc@gmx.ch> - 2024-10-28 11:50 +0100
    Re: Small server hanging after some days - top, load avg and Mem  buff/cache question "Alexander V. Makartsev" <avbetev@gmail.com> - 2024-10-28 12:40 +0100
    Re: Small server hanging after some days - top, load avg and Mem  buff/cache question Dan Ritter <dsr@randomstring.org> - 2024-10-28 15:40 +0100
      Re: Small server hanging after some days - top,  load avg and Mem buff/cache question "B.M." <b-misc@gmx.ch> - 2024-10-28 15:40 +0100
        Re: Small server hanging after some days - top, load avg and Mem  buff/cache question Dan Ritter <dsr@randomstring.org> - 2024-10-28 16:10 +0100

#274119 — Small server hanging after some days - top, load avg and Mem buff/cache question

[Multipart message — attachments visible in raw view] — view raw

Hi,

I have a small server (Raspberry Pi 4 in fact) and since a couple of weeks it
repeatedly hangs after some days until I reboot it (after months of uptime
without any problem - but I changed a few things in the meantime, so maybe
load is now higher than before).

At least after installing watchdog it reboots now automatically.

Here some top output just before reboot:

load avg 20 18 12 (so: much higher than normal, e.g. 1.1 1.2 1.5 or so)
MiB Mem: 3835 total, 618 free, 1194 used, 2264 buff/cache
MiB Swap: 1024 total, 600 free, 430 used, 2640 avail

Processes with highest CPU usage:
kswapd0 with 80%
java (openhab) 41%
kworker/08:10+events_unbound 21%
pg_dump 18%

Processes with highest Mem usage:
java (i.e. openhab) with 626678 virt, 22%
postgres 338824 virt
postgre 338800 virt

From my understanding, there is enough memory available, even swap usage
wouldn't be necessary, since buff/cache is about 50% of physical memory. Is
this correct? But than: why is it going to hang afterwards, or why this
extreme load?

The only swap device available is zram, no swap partition, no swap file. The
system runs on btrfs.

Any ideas?

Thank you,
Bernd

[toc] | [next] | [standalone]

#274121 — Re: Small server hanging after some days - top, load avg and Mem buff/cache question

[Multipart message — attachments visible in raw view] — view raw

On 28.10.2024 15:41, B.M. wrote:
> Hi,
>
> I have a small server (Raspberry Pi 4 in fact) and since a couple of weeks it
> repeatedly hangs after some days until I reboot it (after months of uptime
> without any problem - but I changed a few things in the meantime, so maybe
> load is now higher than before).
>
> At least after installing watchdog it reboots now automatically.
>
> Here some top output just before reboot:
>
> load avg 20 18 12 (so: much higher than normal, e.g. 1.1 1.2 1.5 or so)
> MiB Mem: 3835 total, 618 free, 1194 used, 2264 buff/cache
> MiB Swap: 1024 total, 600 free, 430 used, 2640 avail
>
> Processes with highest CPU usage:
> kswapd0 with 80%
> java (openhab) 41%
> kworker/08:10+events_unbound 21%
> pg_dump 18%
>
> Processes with highest Mem usage:
> java (i.e. openhab) with 626678 virt, 22%
> postgres 338824 virt
> postgre 338800 virt
>
>  From my understanding, there is enough memory available, even swap usage
> wouldn't be necessary, since buff/cache is about 50% of physical memory. Is
> this correct? But than: why is it going to hang afterwards, or why this
> extreme load?
>
> The only swap device available is zram, no swap partition, no swap file. The
> system runs on btrfs.
>
> Any ideas?
Does it hangs completely, i.e. no ssh, no video, no ping?
It is hard to suggest anything concrete without logs. You can list logs 
from previous boot with journalctl:
# journalctl -b -1
# journalctl -b -2
# journalctl -b -3
etc.

zram probably is a bad idea, in your case, because your limited hardware 
wastes CPU cycles on compression and
active processes will eventually run out of free memory resulting in 
heavy swapping and storage IO.
And in your case there is no swap partition "safety net", which is, as 
they say, better to have it and don't need it, than need it and don't 
have it.
btrfs in my experience is also too resource hungry and requires manually 
disabling CoW on write intensive files for better storage IO performance.
Monitoring temperatures is a good way to rule out another possible culprit.

-- 

  With kindest regards, Alexander.

  Debian - The universal operating system
  https://www.debian.org

[toc] | [prev] | [next] | [standalone]

#274124 — Re: Small server hanging after some days - top, load avg and Mem buff/cache question

B.M. wrote: 
> Hi,
> 
> I have a small server (Raspberry Pi 4 in fact) and since a couple of weeks it
> repeatedly hangs after some days until I reboot it (after months of uptime
> without any problem - but I changed a few things in the meantime, so maybe
> load is now higher than before).
> 
> At least after installing watchdog it reboots now automatically.
> 
> Here some top output just before reboot:
> 
> load avg 20 18 12 (so: much higher than normal, e.g. 1.1 1.2 1.5 or so)
> MiB Mem: 3835 total, 618 free, 1194 used, 2264 buff/cache
> MiB Swap: 1024 total, 600 free, 430 used, 2640 avail
> 
> Processes with highest CPU usage:
> kswapd0 with 80%

Swapping once is fine. Swapping continuously is bad. 

> java (openhab) 41%
> pg_dump 18%

pg_dump should not be running continuously; if it is running too
long, you need a better way of backing up PG. Replication to
another server is usually very efficient.

> Processes with highest Mem usage:
> java (i.e. openhab) with 626678 virt, 22%
> postgres 338824 virt
> postgre 338800 virt
> 
> From my understanding, there is enough memory available, even swap usage
> wouldn't be necessary, since buff/cache is about 50% of physical memory. Is
> this correct? But than: why is it going to hang afterwards, or why this
> extreme load?
> 
> The only swap device available is zram, no swap partition, no swap file. The
> system runs on btrfs

Ooch. Your system is likely thrashing between using RAM for the
applications (java, PG, pg-dump) and using RAM to swap out from
RAM to compressed RAM (this does not buy you as much as you
think).

The problem, then, is that you don't have enough RAM and you don't
have enough I/O speed to solve the RAM issue temporarily, so it
becomes permanent.

Try disabling swap entirely. 

If that doesn't work, you need a machine with more RAM, or you
need to be using less RAM.

-dsr- 

[toc] | [prev] | [next] | [standalone]

#274125

[Multipart message — attachments visible in raw view] — view raw

On Montag, 28. Oktober 2024 15:21:02 CET Dan Ritter wrote:
> B.M. wrote:
> > Hi,
> >
> > I have a small server (Raspberry Pi 4 in fact) and since a couple of weeks
> > it repeatedly hangs after some days until I reboot it (after months of
> > uptime without any problem - but I changed a few things in the meantime,
> > so maybe load is now higher than before).
> >
> > At least after installing watchdog it reboots now automatically.
> >
> > Here some top output just before reboot:
> >
> > load avg 20 18 12 (so: much higher than normal, e.g. 1.1 1.2 1.5 or so)
> > MiB Mem: 3835 total, 618 free, 1194 used, 2264 buff/cache
> > MiB Swap: 1024 total, 600 free, 430 used, 2640 avail
> >
> > Processes with highest CPU usage:
> > kswapd0 with 80%
>
> Swapping once is fine. Swapping continuously is bad.
>
> > java (openhab) 41%
> > pg_dump 18%
>
> pg_dump should not be running continuously; if it is running too
> long, you need a better way of backing up PG. Replication to
> another server is usually very efficient.
>
> > Processes with highest Mem usage:
> > java (i.e. openhab) with 626678 virt, 22%
> > postgres 338824 virt
> > postgre 338800 virt
> >
> > From my understanding, there is enough memory available, even swap usage
> > wouldn't be necessary, since buff/cache is about 50% of physical memory.
> > Is
> > this correct? But than: why is it going to hang afterwards, or why this
> > extreme load?
> >
> > The only swap device available is zram, no swap partition, no swap file.
> > The system runs on btrfs
>
> Ooch. Your system is likely thrashing between using RAM for the
> applications (java, PG, pg-dump) and using RAM to swap out from
> RAM to compressed RAM (this does not buy you as much as you
> think).
>
> The problem, then, is that you don't have enough RAM and you don't
> have enough I/O speed to solve the RAM issue temporarily, so it
> becomes permanent.
>
> Try disabling swap entirely.
>
> If that doesn't work, you need a machine with more RAM, or you
> need to be using less RAM.
>
> -dsr-

Thanks a lot for your inputs. In the meantime I disabled zram and added a 2
GiB swapfile.

What I don't understand: how can the system have not enough memory problems,
while it's showing ~ 2 GiB cached/buffer? I'd expect it to free some of that
and be fine again... 2 GiB is about 50% of the physical RAM of 4 GiB?

[toc] | [prev] | [next] | [standalone]

#274127 — Re: Small server hanging after some days - top, load avg and Mem buff/cache question

B.M. wrote: 
> Thanks a lot for your inputs. In the meantime I disabled zram and added a 2
> GiB swapfile.
> 
> What I don't understand: how can the system have not enough memory problems,
> while it's showing ~ 2 GiB cached/buffer? I'd expect it to free some of that
> and be fine again... 2 GiB is about 50% of the physical RAM of 4 GiB?

That 2GB is being used by zram as swap and cannot be freed.

You probably managed to increase your usable memory by 10-20%
with zram -- at the cost of constantly swapping.

(That's my hypothesis, anyway.)

Let us know what happens in a day or two.

-dsr-

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web