Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #224429 > unrolled thread

Swappiness in Buster

Back to article view | Back to linux.debian.user

Contents

  Swappiness in Buster Martin Reissner <mreissner@wavecon.de> - 2020-07-06 09:20 +0200
    Re: Swappiness in Buster songbird <songbird@anthive.com> - 2020-07-06 19:10 +0200
      Re: Swappiness in Buster Martin Reissner <mreissner@wavecon.de> - 2020-07-06 23:00 +0200
        Re: Swappiness in Buster deloptes <deloptes@gmail.com> - 2020-07-06 23:30 +0200
          Re: Swappiness in Buster Martin Reissner <mreissner@wavecon.de> - 2020-07-07 09:10 +0200
            Re: Swappiness in Buster Martin Reissner <mreissner@wavecon.de> - 2020-07-08 08:30 +0200
              minimizing buffer size (i.e. page cache) for bulk copy/rsync -- Re:  Swappiness in Buster Zenaan Harkness <zenaan@freedbms.net> - 2020-07-08 10:00 +0200
                Re: minimizing buffer size (i.e. page cache) for bulk copy/rsync --  Re: Swappiness in Buster Reco <recoverym4n@enotuniq.net> - 2020-07-08 11:50 +0200
                Re: minimizing buffer size (i.e. page cache) for bulk copy/rsync --  Re: Swappiness in Buster Dan Ritter <dsr@randomstring.org> - 2020-07-08 14:10 +0200
                  Re: minimizing buffer size (i.e. page cache) for bulk copy/rsync --  Re: Swappiness in Buster Greg Wooledge <wooledg@eeg.ccf.org> - 2020-07-08 14:10 +0200
                    Re: minimizing buffer size (i.e. page cache) for bulk copy/rsync --  Re: Swappiness in Buster Zenaan Harkness <zenaan@freedbms.net> - 2020-07-08 16:10 +0200
                      Re: minimizing buffer size (i.e. page cache) for bulk copy/rsync --  Re: Swappiness in Buster Greg Wooledge <wooledg@eeg.ccf.org> - 2020-07-08 16:20 +0200

#224429 — Swappiness in Buster

Hello,

ever since upgrading machines to Buster the vm.swappiness sysctl
parameter doesn't seem to do anything anymore and regardless on how I
set it via sysctl or directly in /proc the system behaves as it would
have a pretty high swappiness and thus is swapping out quite a bit under
load, using the memory mostly for buff/cache.

To make this clear, I'm seeing no performance degradation and there is
not much swapping in done, I'm merely wondering why this behaviour has
changed and if there's a way to make it work as before again. Also I'd
be happy to know if anybody else is experiencing this as Buster has been
out for a while now and I found some information on this ie. other users
reporting the same issue, but not really a lot.

Now if you're asking why I would configure swap if I don't want it to be
used, the answer is monitoring and failsafe. I usually set the
swappiness to 1 so to only swap if the box runs out of memory and then
trigger an alarm to let us know something is up. This is much more
convenient than the oomkiller striking down a mysqld process just
because it was allowed to use a bit too much memory.

Cheers,

Martin

[toc] | [next] | [standalone]

#224448

Martin Reissner wrote:

> Hello,
>
> ever since upgrading machines to Buster the vm.swappiness sysctl
> parameter doesn't seem to do anything anymore and regardless on how I
> set it via sysctl or directly in /proc the system behaves as it would
> have a pretty high swappiness and thus is swapping out quite a bit under
> load, using the memory mostly for buff/cache.
>
> To make this clear, I'm seeing no performance degradation and there is
> not much swapping in done, I'm merely wondering why this behaviour has
> changed and if there's a way to make it work as before again. Also I'd
> be happy to know if anybody else is experiencing this as Buster has been
> out for a while now and I found some information on this ie. other users
> reporting the same issue, but not really a lot.
>
> Now if you're asking why I would configure swap if I don't want it to be
> used, the answer is monitoring and failsafe. I usually set the
> swappiness to 1 so to only swap if the box runs out of memory and then
> trigger an alarm to let us know something is up. This is much more
> convenient than the oomkiller striking down a mysqld process just
> because it was allowed to use a bit too much memory.

  i haven't noticed any issues with my desktop system set
to 5, but i rarely if ever get much swapping going on 
anyways.  i'm not using buster but using testing.

  if you are running a database oriented server i think
the recommendation would be to set it to 0 or 1.


  songbird

[toc] | [prev] | [next] | [standalone]

#224458

On 06/07/2020 18:11, songbird wrote:
> Martin Reissner wrote:
> 
>> Hello,
>>
>> ever since upgrading machines to Buster the vm.swappiness sysctl
>> parameter doesn't seem to do anything anymore and regardless on how I
>> set it via sysctl or directly in /proc the system behaves as it would
>> have a pretty high swappiness and thus is swapping out quite a bit under
>> load, using the memory mostly for buff/cache.
>>
>> To make this clear, I'm seeing no performance degradation and there is
>> not much swapping in done, I'm merely wondering why this behaviour has
>> changed and if there's a way to make it work as before again. Also I'd
>> be happy to know if anybody else is experiencing this as Buster has been
>> out for a while now and I found some information on this ie. other users
>> reporting the same issue, but not really a lot.
>>
>> Now if you're asking why I would configure swap if I don't want it to be
>> used, the answer is monitoring and failsafe. I usually set the
>> swappiness to 1 so to only swap if the box runs out of memory and then
>> trigger an alarm to let us know something is up. This is much more
>> convenient than the oomkiller striking down a mysqld process just
>> because it was allowed to use a bit too much memory.
> 
>   i haven't noticed any issues with my desktop system set
> to 5, but i rarely if ever get much swapping going on 
> anyways.  i'm not using buster but using testing.
> 
>   if you are running a database oriented server i think
> the recommendation would be to set it to 0 or 1.
> 
> 
>   songbird
> 


Yeah, only talking about server and mostly database applications. I
usually set it to 1, but even tried 0 which disabled swap completely on
Stretch but on Buster it didn't make a difference at all, the setting
seems to be ignored while using default swappiness (60?) since Buster.

Following output is from a mariadb server that didn't swap at all
running Stretch.

root@server:~# cat /proc/sys/vm/swappiness
1
root@server:~# free -tm
              total     used     free   shared  buff/cache   available
Mem:          48284     9763      707       59       37814       38147
Swap:         11443     5020     6423
Total:        59728    14783     7131

[toc] | [prev] | [next] | [standalone]

#224462

Martin Reissner wrote:

> Yeah, only talking about server and mostly database applications. I
> usually set it to 1, but even tried 0 which disabled swap completely on
> Stretch but on Buster it didn't make a difference at all, the setting
> seems to be ignored while using default swappiness (60?) since Buster.
> 
> Following output is from a mariadb server that didn't swap at all
> running Stretch.
> 
> root@server:~# cat /proc/sys/vm/swappiness
> 1
> root@server:~# free -tm
> total     used     free   shared  buff/cache   available
> Mem:          48284     9763      707       59       37814       38147
> Swap:         11443     5020     6423
> Total:        59728    14783     7131

May be look deeper in documentation - I recall asking few years ago and was
answered that now it would cache whatever it can and will free on demand.
swap is done only if memory is really insufficient.

I don't recall when or where I asked read this

[toc] | [prev] | [next] | [standalone]

#224479

On 06/07/2020 23:27, deloptes wrote:

> May be look deeper in documentation - I recall asking few years ago and was
> answered that now it would cache whatever it can and will free on demand.
> swap is done only if memory is really insufficient.
> 
> I don't recall when or where I asked read this
> 

I probably wouldn't understand much delving deeper into the
documentation or even code^^
Using up all the available RAM for disk cache and freeing cache when
memory is needed is what Linux did for as long as I can remember and
still does.
Swapping being only done when there is no memory available is exactly
what the "swappiness" parameter did when set to "1" in Stretch, but this
doesn't seem to work in Buster anymore as it swaps out the diskcache
even when there is memory available.

[toc] | [prev] | [next] | [standalone]

#224577

For the sake of completeness, I seem to have solved it after some more
research and it turned out to be systemd, as following bug that was
reported for Centos7 seems to be applying to Debian Buster as well:

https://github.com/systemd/systemd/issues/9276

Luckily the workaround mentioned in:

https://github.com/systemd/systemd/issues/9276#issuecomment-442514543

applies here as well and by setting this to the desired swappiness value
and rebooting the system so far it seems to work as before and swapping
out is only done if it can't be avoided.

Martin

[toc] | [prev] | [next] | [standalone]

#224578 — minimizing buffer size (i.e. page cache) for bulk copy/rsync -- Re: Swappiness in Buster

On Wed, Jul 08, 2020 at 08:26:18AM +0200, Martin Reissner wrote:
> For the sake of completeness, I seem to have solved it after some more
> research and it turned out to be systemd, as following bug that was
> reported for Centos7 seems to be applying to Debian Buster as well:
> 
> https://github.com/systemd/systemd/issues/9276
> 
> Luckily the workaround mentioned in:
> 
> https://github.com/systemd/systemd/issues/9276#issuecomment-442514543
> 
> applies here as well and by setting this to the desired swappiness value
> and rebooting the system so far it seems to work as before and swapping
> out is only done if it can't be avoided.


Your thread brings to mind an "$AGE_OF_LINUX_KERNEL + 1 year" years old
Linux kernel bona fydee, bulk copy bug bear:

Doing a bulk copy e.g. using cp or rsync, to copy GiBs far greater than
available RAM from one disk to another, results in ...

  <sinister musak please>
    Total Annihilation of Desktop Latency
   <sinister laugh>

This happens to this day even on the latest Linux kernels on Sid such as
5.7.0-1-amd64 to pick an entirely random example... and has been
literally bugging me since the beginnings of my Debian journey ~20 years
ago.

Note that in this extremely humble user's experience, even

    ionice --class Idle ...

is not even able to quell the Memory Beast's all consuming hunger games
for all possible RAM and then some, in it's insatiable appetite for all
the page cache in...

  <more sinister musak please>
     The  Entire  Universe
   <sinister laugh>


What needs to happen ™©®

What is needed is an option somewhere, strictly abided by, where "For
the following command" only say 1MiB of buffer and Page Cache, in TOTAL,
is able to be used by that command ever, until its completion.

In this way my 16GiBs of RAM will __finally__ be left to its rightful
owners Firefox, mutt, and my XOrg's cursor, since to me as an extremely
humble desktop user, these are the only things in the world that I ever
care about, and I esPECIALLY want to NOT care that a 1,700 GiB movie
disk backup takes an extra 5 or 10 % longer than the NINE hours it
already takes across its USB 2 connection!

Seriously, is there a way to stop bulk copies from eternally flushing my
$Desktop's cached pages down the drain minute after blessful minute,
hour after gratitude filled hour?

The heavens will truly be pleased ..

[toc] | [prev] | [next] | [standalone]

#224584 — Re: minimizing buffer size (i.e. page cache) for bulk copy/rsync -- Re: Swappiness in Buster

	Hi.

On Wed, Jul 08, 2020 at 05:51:54PM +1000, Zenaan Harkness wrote:
> What needs to happen ™©®
> 
> What is needed is an option somewhere, strictly abided by, where "For
> the following command" only say 1MiB of buffer and Page Cache, in TOTAL,
> is able to be used by that command ever, until its completion.

This works for me in stable, it's systemwide though.

sysctl -w vm.dirty_bytes=16777216
sysctl -w vm.dirty_background_bytes=16777216

Reco

[toc] | [prev] | [next] | [standalone]

#224587 — Re: minimizing buffer size (i.e. page cache) for bulk copy/rsync -- Re: Swappiness in Buster

Zenaan Harkness wrote: 
> 
> Seriously, is there a way to stop bulk copies from eternally flushing my
> $Desktop's cached pages down the drain minute after blessful minute,
> hour after gratitude filled hour?

softlimit is packaged in daemontools.

NAME
       softlimit - runs another program with new resource
limits.

SYNOPSIS
       softlimit [ opts ] child

DESCRIPTION
       opts is a series of getopt-style options.  child consists
of one or more arguments.

       softlimit sets soft resource limits as specified by opts.
It then runs child.

OPTIONS
       In  each  of  the  following  opts, n may be =,
indicating that the soft limit should be set equal to the hard
       limit.

       opts controlling memory use:

       -m n   Same as -d n -s n -l n -a n.

       -d n   Limit the data segment per process to n bytes.

       -s n   Limit the stack segment per process to n bytes.

       -l n   Limit the locked physical pages per process to n
bytes. This option has no  effect  on  some  operating
              systems.

       -a n   Limit  the  total  of  all segments per process to
n bytes. This option has no effect on some operating
              systems.

.... and so on for other resources to limit.

-dsr-

[toc] | [prev] | [next] | [standalone]

#224588 — Re: minimizing buffer size (i.e. page cache) for bulk copy/rsync -- Re: Swappiness in Buster

On Wed, Jul 08, 2020 at 08:00:38AM -0400, Dan Ritter wrote:
> Zenaan Harkness wrote: 
> > 
> > Seriously, is there a way to stop bulk copies from eternally flushing my
> > $Desktop's cached pages down the drain minute after blessful minute,
> > hour after gratitude filled hour?
> 
> softlimit is packaged in daemontools.
> 
> NAME
>        softlimit - runs another program with new resource
> limits.

I haven't been following this thread because I don't know anything
about the kernel's "swappiness" settings.

But I'm a little bit confused about the intent here -- setting a
resource limit on memory is just going to make the program crash
with an "out of memory" error when it tries to exceed the limit.
It's not going to discipline a memory hog program into using
different algorithms.

[toc] | [prev] | [next] | [standalone]

#224595 — Re: minimizing buffer size (i.e. page cache) for bulk copy/rsync -- Re: Swappiness in Buster

On Wed, Jul 08, 2020 at 08:05:23AM -0400, Greg Wooledge wrote:
> On Wed, Jul 08, 2020 at 08:00:38AM -0400, Dan Ritter wrote:
> > Zenaan Harkness wrote: 
> > > 
> > > Seriously, is there a way to stop bulk copies from eternally flushing my
> > > $Desktop's cached pages down the drain minute after blessful minute,
> > > hour after gratitude filled hour?
> > 
> > softlimit is packaged in daemontools.
> > 
> > NAME
> >        softlimit - runs another program with new resource
> > limits.
> 
> I haven't been following this thread because I don't know anything
> about the kernel's "swappiness" settings.
> 
> But I'm a little bit confused about the intent here -- setting a
> resource limit on memory is just going to make the program crash
> with an "out of memory" error when it tries to exceed the limit.
> It's not going to discipline a memory hog program into using
> different algorithms.

I also know nothing about swappiness.

But this soft limit tool sounds exceedingly powerful (<busily rubs
hands together in anticipation of heavy use of limits to gleefully bring
misbehaving programs to screeching core dumping halts without remorse
MWUAHAHAHAHAHHAHAHHHHAHAAAAAAAAAAAAAA!!!!!>)

Hmm. Excuse me ... where was I?

Oh yes, powERRR!

What I imagine, and surely hope is that this softlimit program works as
advertised.  You see, the copy program (normally aliased as `cp`) when
copying say 900 Gigabytes, does not, or certainly should not, consume
more than say 1 Megabyte of memory as a read buffer.  Of course _some_
buffer is needed, since otherwise each read call reads only one byte,
which would result in a ridiculous number of kernel calls and the
performance that doing so entails.  But half a MiB being written to the
destination, whilst another half MiB is being read from the source
"should" be more than enough, since, roughly speaking, those buffers can
simply swap over when they're both ready for another round.

Since if it's not written in a totally insane way, cp (and to a lesser
extent rsync) should work like this already, what we really need to
limit is that pig with the strange "Linux" name (I think it's called
fake lipschtick on an even-toed ungulate family Suidae, I mean kernel).

You see Linux in its eternal and fruitless desire to make user's happy,
gladly tells cp's read side that reading is finished, and also tells its
write side that writing is finished, and so cp races as fast as it
possibly can even if, and especially if, one of the drives is a
different speed than the other (almost always the case) - and the bigger
the difference, the quicker RAM is filled up, and Firefox's tabs evicted
from memory.

What has been happening since the dawn of time is that this kernel
caches _every_ disk page read by a program such as cp (I know, I know,
this is previously unheard of and amazing information I'm leaking) into
something Linux calls the 'page cache' (totally bizzare thing to call an
in memory cache of disk pages, but hey, what do I know...).

And Linux, ever generous with other peoples' resources, hands out your
RAM, ballooning the page cache as though the world will end if it does
not do this to the greatest extent possible "because you might refer to
one of those pages in the near future" and of course you do, when you
write it to the destination disk, at which point, like a pregnant Sow on
meth, Linux almost faints in excitement as its prediction that you would
use that page again, comes true.

And so now Linux is ecstaticly hyping the utility of all these "read
once, write once" pages in the vain hope that evicting all your firefox
Tab's cached memory pages is The Right Thing To Do ™©®.  (And Firefox,
unlike cp and rsync, is helpful enough to tag most of its RAM pages with
"Totally ephemeral dude, feel free to dump to disk or even completely
destroy at any time").

Firefox is super helpful like that :(

And, cp and rsync are evidently nowhere as helpful as Firefox is to
Linux's desire to please the user, and so your desktop experience goes
straight down the drain.

This has been happening since I discovered Linux - 20 years ago, the
symptom was the entire desktop stuttering, the X cursor jumping with a
latency of 10 seconds, if you were lucky.

Alas, every one of the 4000+ kernel developers works for Google or
Amazon ECS, and throughput is the only thing the datacenter needs - that
and the ability to compile Linux kernels.

[toc] | [prev] | [next] | [standalone]

#224596 — Re: minimizing buffer size (i.e. page cache) for bulk copy/rsync -- Re: Swappiness in Buster

On Thu, Jul 09, 2020 at 12:02:39AM +1000, Zenaan Harkness wrote:
> What I imagine, and surely hope is that this softlimit program works as
> advertised.

It's hardly unique.  You could do the same thing with a shell script
wrapper that calls ulimit and then exec's the target program.

> [stuff about buffers]

> Since if it's not written in a totally insane way, cp (and to a lesser
> extent rsync) should work like this already, what we really need to
> limit is that pig with the strange "Linux" name

rsync uses a shitload of memory not because it's copying a file with
an enormous buffer size, but because it's tracking *all* of the file
metadata in the entire hierarchy that you feed to it.

> What has been happening since the dawn of time is that this kernel
> caches _every_ disk page read by a program such as cp [...]

Well, prematurely killing rsync is not going to affect the kernel's
retention or discarding of specific disk cache pages that are more
precious to you.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web