Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #236884 > unrolled thread

Limit memory consumption of an ad hoc process

Back to article view | Back to linux.debian.user

Contents

  Limit memory consumption of an ad hoc process Victor Sudakov <vas@sibptus.ru> - 2021-07-05 05:40 +0200
    Re: Limit memory consumption of an ad hoc process Greg Wooledge <greg@wooledge.org> - 2021-07-05 14:00 +0200
      Re: Limit memory consumption of an ad hoc process Victor Sudakov <vas@sibptus.ru> - 2021-07-05 18:00 +0200
        Re: Limit memory consumption of an ad hoc process Victor Sudakov <vas@sibptus.ru> - 2021-07-05 18:10 +0200
          Re: Limit memory consumption of an ad hoc process Greg Wooledge <greg@wooledge.org> - 2021-07-05 19:00 +0200
            Re: Limit memory consumption of an ad hoc process Victor Sudakov <vas@sibptus.ru> - 2021-07-06 04:50 +0200
              Re: Limit memory consumption of an ad hoc process The Wanderer <wanderer@fastmail.fm> - 2021-07-06 05:10 +0200
                Re: Limit memory consumption of an ad hoc process Greg Wooledge <greg@wooledge.org> - 2021-07-06 05:20 +0200
                Re: Limit memory consumption of an ad hoc process Greg Wooledge <greg@wooledge.org> - 2021-07-06 05:20 +0200
                  Re: Limit memory consumption of an ad hoc process Victor Sudakov <vas@sibptus.ru> - 2021-07-06 06:30 +0200
                Re: Limit memory consumption of an ad hoc process Victor Sudakov <vas@sibptus.ru> - 2021-07-06 06:40 +0200
    Re: Limit memory consumption of an ad hoc process Victor Sudakov <vas@sibptus.ru> - 2021-07-06 06:30 +0200

#236884 — Limit memory consumption of an ad hoc process

[Multipart message — attachments visible in raw view] — view raw

Dear Colleagues,

FreeBSD has a simple way to run some ad-hoc program with memory limits:

$ limits -m 2G ./mytest
  memoryuse             2097152 kB
  vmemoryuse           infinity kB
$ limits -m 1G ./mytest
  memoryuse             1048576 kB
  vmemoryuse           infinity kB

How do I do the same in Linux (without root permissions)?

-- 
Victor Sudakov VAS4-RIPE
http://vas.tomsk.ru/
2:5005/49@fidonet

[toc] | [next] | [standalone]

#236899

On Mon, Jul 05, 2021 at 10:15:19AM +0700, Victor Sudakov wrote:
> FreeBSD has a simple way to run some ad-hoc program with memory limits:
> 
> $ limits -m 2G ./mytest
>   memoryuse             2097152 kB
>   vmemoryuse           infinity kB
> $ limits -m 1G ./mytest
>   memoryuse             1048576 kB
>   vmemoryuse           infinity kB
> 
> How do I do the same in Linux (without root permissions)?

One way would be:

(ulimit -m 2097152; ./mytest)

This assumes you're using bash as your shell.

You might want to review the resource limits and how they work --
see setrlimit(2) for the kernel's documentation.  See "help ulimit"
for bash's brief-form documentation.  The longer documentation is in
the bash(1) man page.

You never need root privileges to *reduce* one of your resource limits
to a lower value.  Nor do you need root privs to raise your soft limit
up to the hard limit.  However, you do need root privs to raise your
hard limit.

[toc] | [prev] | [next] | [standalone]

#236922

[Multipart message — attachments visible in raw view] — view raw

Greg Wooledge wrote:
> On Mon, Jul 05, 2021 at 10:15:19AM +0700, Victor Sudakov wrote:
> > FreeBSD has a simple way to run some ad-hoc program with memory limits:
> > 
> > $ limits -m 2G ./mytest
> >   memoryuse             2097152 kB
> >   vmemoryuse           infinity kB
> > $ limits -m 1G ./mytest
> >   memoryuse             1048576 kB
> >   vmemoryuse           infinity kB
> > 
> > How do I do the same in Linux (without root permissions)?
> 
> One way would be:
> 
> (ulimit -m 2097152; ./mytest)

Is the value in KB or MB? Seems KB. Anyway, the suggested method does not seem to work:

$ ulimit -m 640 ; stress-ng --vm 1 --vm-bytes 2G
-bash: ulimit: max memory size: cannot modify limit: Operation not permitted
stress-ng: info:  [10272] defaulting to a 86400 second (1 day, 0.00 secs) run per stressor
stress-ng: info:  [10272] dispatching hogs: 1 vm
^C

> 
> This assumes you're using bash as your shell.

I am, as you see above. I expect stress-ng to be killed when it tries to
allocate 2G of memory but it's not happening. Can you reproduce your
advice?

> 
> You never need root privileges to *reduce* one of your resource limits
> to a lower value.  Nor do you need root privs to raise your soft limit
> up to the hard limit.  However, you do need root privs to raise your
> hard limit.
> 

Then, why does it say "cannot modify limit: Operation not permitted" ?

-- 
Victor Sudakov VAS4-RIPE
http://vas.tomsk.ru/
2:5005/49@fidonet

[toc] | [prev] | [next] | [standalone]

#236923

[Multipart message — attachments visible in raw view] — view raw

Victor Sudakov wrote:
> > This assumes you're using bash as your shell.
> 
> I am, as you see above. I expect stress-ng to be killed when it tries to
> allocate 2G of memory but it's not happening. Can you reproduce your
> advice?

Even as root, there is no "Operation not permitted" message, but there
is no working limit either:


# ulimit -m 1048576 ; stress-ng --vm 1 --vm-bytes 8G
stress-ng: info:  [10961] defaulting to a 86400 second (1 day, 0.00 secs) run per stressor
stress-ng: info:  [10961] dispatching hogs: 1 vm

stress-ng is not killed. Swap is not enabled on this test host.

-- 
Victor Sudakov VAS4-RIPE
http://vas.tomsk.ru/
2:5005/49@fidonet

[toc] | [prev] | [next] | [standalone]

#236926

On Mon, Jul 05, 2021 at 10:57:03PM +0700, Victor Sudakov wrote:
> Greg Wooledge wrote:
> > One way would be:
> > 
> > (ulimit -m 2097152; ./mytest)
> 
> Is the value in KB or MB? Seems KB.

Please read the documentation.

unicorn:~$ ulimit -a | grep -- -m
max memory size             (kbytes, -m) unlimited

> Anyway, the suggested method does not seem to work:
> 
> $ ulimit -m 640 ; stress-ng --vm 1 --vm-bytes 2G
> -bash: ulimit: max memory size: cannot modify limit: Operation not permitted
> stress-ng: info:  [10272] defaulting to a 86400 second (1 day, 0.00 secs) run per stressor
> stress-ng: info:  [10272] dispatching hogs: 1 vm

Have you been omitting the parentheses in your previous testing?

unicorn:~$ (ulimit -m 640; /bin/echo hi)
hi

Setting the resource limit DOWNWARD should always be allowed.  This
particular one is set to infinity by default in Debian, so the only way
setting it to 640 kbytes would be disallowed is if you had previously
set it lower than that.  Which, judging by the lack of parentheses in
your sample call, is a thing you might have done.

You don't want to nuke resources limits in your interactive shell.  That's
what the explicit subshells are for.


On Mon, Jul 05, 2021 at 11:06:31PM +0700, Victor Sudakov wrote:
> Even as root, there is no "Operation not permitted" message, but there
> is no working limit either:
> 
> 
> # ulimit -m 1048576 ; stress-ng --vm 1 --vm-bytes 8G
> stress-ng: info:  [10961] defaulting to a 86400 second (1 day, 0.00 secs) run per stressor
> stress-ng: info:  [10961] dispatching hogs: 1 vm
> 
> stress-ng is not killed. Swap is not enabled on this test host.

Perhaps -m is not the correct resource limit for what you're trying to
achieve.  You've been assuming that it is, but I told you to read
setrlimit(3) to see what each of them is and what it actually controls.

>From ulimit's output:

max memory size             (kbytes, -m) unlimited

>From bash(1):

              -m     The maximum resident set size (many systems do not  honor
                     this limit)

>From setrlimit(3):

       RLIMIT_RSS
              This is a limit (in bytes) on the process's  resident  set  (the
              number of virtual pages resident in RAM).  This limit has effect
              only in Linux 2.4.x, x < 30, and there  affects  only  calls  to
              madvise(2) specifying MADV_WILLNEED.

Perhaps you want "ulimit -v" instead?

And in any case, when you test this stuff, use the subshells.  Or be
prepared to discard and re-spawn a whole lot of terminal windows.

[toc] | [prev] | [next] | [standalone]

#236944

[Multipart message — attachments visible in raw view] — view raw

Greg Wooledge wrote:

[dd]

> > 
> > 
> > # ulimit -m 1048576 ; stress-ng --vm 1 --vm-bytes 8G
> > stress-ng: info:  [10961] defaulting to a 86400 second (1 day, 0.00 secs) run per stressor
> > stress-ng: info:  [10961] dispatching hogs: 1 vm
> > 
> > stress-ng is not killed. Swap is not enabled on this test host.
> 
> Perhaps -m is not the correct resource limit for what you're trying to
> achieve.  You've been assuming that it is, but I told you to read
> setrlimit(3) to see what each of them is and what it actually controls.
> 
> >From ulimit's output:
> 
> max memory size             (kbytes, -m) unlimited
> 
> >From bash(1):
> 
>               -m     The maximum resident set size (many systems do not  honor
>                      this limit)
> 
> >From setrlimit(3):
> 
>        RLIMIT_RSS
>               This is a limit (in bytes) on the process's  resident  set  (the
>               number of virtual pages resident in RAM).  This limit has effect
>               only in Linux 2.4.x, x < 30, and there  affects  only  calls  to
>               madvise(2) specifying MADV_WILLNEED.
> 
> Perhaps you want "ulimit -v" instead?

You are correct, `ulimit -v` works:

$ ulimit -v 1048576 ; stress-ng --vm 1 --vm-bytes 8G
stress-ng: info:  [11081] defaulting to a 86400 second (1 day, 0.00 secs) run per stressor
stress-ng: info:  [11081] dispatching hogs: 1 vm
stress-ng: error: [11083] stress-ng-vm: gave up trying to mmap, no available memory
stress-ng: info:  [11081] successful run completed in 10.01s
$ 

> 
> And in any case, when you test this stuff, use the subshells.  

Well, my goal is to find a way to limit programs run from cron, PHP's
proc_open() etc., not from the interactive shell. What would you advise?

The equivalent of FreeBSD's `/usr/bin/limits` wrapper seems to be
`/usr/bin/prlimit` but I'm really at a loss what limit to specify for
testing (the equivalent of `ulimit -v`). Should be RLIMIT_AS but --as 
crashes prlimit:

$ prlimit --as=1048576 /bin/ls
/bin/ls: error while loading shared libraries: libselinux.so.1: failed to map segment from shared object
$
$ prlimit --as=1048576 stress-ng
Segmentation fault
$

-- 
Victor Sudakov VAS4-RIPE
http://vas.tomsk.ru/
2:5005/49@fidonet

[toc] | [prev] | [next] | [standalone]

#236945

[Multipart message — attachments visible in raw view] — view raw

On 2021-07-05 at 22:42, Victor Sudakov wrote:

> Greg Wooledge wrote:

>> And in any case, when you test this stuff, use the subshells.  
> 
> Well, my goal is to find a way to limit programs run from cron, PHP's
> proc_open() etc., not from the interactive shell. What would you advise?

Is there any reason to think that the ulimit approach wouldn't work in
that type of context, as well as when invoked from within a shell?

> The equivalent of FreeBSD's `/usr/bin/limits` wrapper seems to be
> `/usr/bin/prlimit` but I'm really at a loss what limit to specify for
> testing (the equivalent of `ulimit -v`). Should be RLIMIT_AS but --as 
> crashes prlimit:
> 
> $ prlimit --as=1048576 /bin/ls
> /bin/ls: error while loading shared libraries: libselinux.so.1: failed to map segment from shared object
> $
> $ prlimit --as=1048576 stress-ng
> Segmentation fault
> $

I'm not familiar with prlimit, but just looking at this output and at
the man page, what unit do you think the value of the --as argument is
in? If it's in bytes, then what you're doing here is specirying an
address-space size of only one megabyte, and I don't find it at all
surprising if that's too small for the combination of ls plus its libraries.

On my machine, ls is 144K, and its dependent libraries (as reported by
ldd, and exclusing linux-vdso.so.1 which doesn't seem to be an actual
file in the filesystem) range in size from 19K (libdl) to 1.8M (libc),
for a total of 2.4M. That's more than twice the hypothesized
one-megabyte threshold.

-- 
   The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man.         -- George Bernard Shaw

[toc] | [prev] | [next] | [standalone]

#236946

On Mon, Jul 05, 2021 at 11:11:06PM -0400, Greg Wooledge wrote:
> > > $ prlimit --as=1048576 /bin/ls
> > > /bin/ls: error while loading shared libraries: libselinux.so.1: failed to map segment from shared object
> 
> Sounds like you set it too low.  Remember, setrlimit(2) and prlimit(2)
> say that RLIMIT_DATA is specified in bytes.  This is not the same as
> bash's ulimit.

... and the same for RLIMIT_AS which is probably what you're using.

It would be incredibly useful if the various shell-interface tools
(ulimit, prlimit, and so on) would explicitly say which setrlimit(2)
resource they're actually using, instead of forcing us to guess.

[toc] | [prev] | [next] | [standalone]

#236947

On Mon, Jul 05, 2021 at 11:01:25PM -0400, The Wanderer wrote:
> On 2021-07-05 at 22:42, Victor Sudakov wrote:
> 
> > Greg Wooledge wrote:
> 
> >> And in any case, when you test this stuff, use the subshells.  
> > 
> > Well, my goal is to find a way to limit programs run from cron, PHP's
> > proc_open() etc., not from the interactive shell. What would you advise?
> 
> Is there any reason to think that the ulimit approach wouldn't work in
> that type of context, as well as when invoked from within a shell?

Depends on how the cron job is set up.  If the cron job never calls
bash (but rather, sh) then you can't use bash's ulimit.

Likewise, I would guess that PHP's proc_open() calls /bin/sh, not bash,
but you'd have to ask a (more experienced) PHP user.

I advised using subshells during testing because you don't want to
irrevocably cripple the shell you're using to launch the tests, as the
OP did earlier.  Once you've screwed up your interactive shell, you have
to throw it away and start over, which is a pain in the ass.

> > The equivalent of FreeBSD's `/usr/bin/limits` wrapper seems to be
> > `/usr/bin/prlimit` but I'm really at a loss what limit to specify for
> > testing (the equivalent of `ulimit -v`). Should be RLIMIT_AS but --as 
> > crashes prlimit:
> > 
> > $ prlimit --as=1048576 /bin/ls
> > /bin/ls: error while loading shared libraries: libselinux.so.1: failed to map segment from shared object

Sounds like you set it too low.  Remember, setrlimit(2) and prlimit(2)
say that RLIMIT_DATA is specified in bytes.  This is not the same as
bash's ulimit.

[toc] | [prev] | [next] | [standalone]

#236949

[Multipart message — attachments visible in raw view] — view raw

Greg Wooledge wrote:
> On Mon, Jul 05, 2021 at 11:01:25PM -0400, The Wanderer wrote:
> > On 2021-07-05 at 22:42, Victor Sudakov wrote:
> > 
> > > Greg Wooledge wrote:
> > 
> > >> And in any case, when you test this stuff, use the subshells.  
> > > 
> > > Well, my goal is to find a way to limit programs run from cron, PHP's
> > > proc_open() etc., not from the interactive shell. What would you advise?
> > 
> > Is there any reason to think that the ulimit approach wouldn't work in
> > that type of context, as well as when invoked from within a shell?
> 
> Depends on how the cron job is set up.  If the cron job never calls
> bash (but rather, sh) then you can't use bash's ulimit.
> 
> Likewise, I would guess that PHP's proc_open() calls /bin/sh, not bash,
> but you'd have to ask a (more experienced) PHP user.

I suppose PHP's proc_open() does not call any shell at all, it executes
the binary directly. To execute command in a shell, there is
shell_exec() but it does not give access to the executed program's
stdin/stdout.

I think I can call the prlimit wrapper from proc_open() and that should do.

> 
> > > The equivalent of FreeBSD's `/usr/bin/limits` wrapper seems to be
> > > `/usr/bin/prlimit` but I'm really at a loss what limit to specify for
> > > testing (the equivalent of `ulimit -v`). Should be RLIMIT_AS but --as 
> > > crashes prlimit:
> > > 
> > > $ prlimit --as=1048576 /bin/ls
> > > /bin/ls: error while loading shared libraries: libselinux.so.1: failed to map segment from shared object
> 
> Sounds like you set it too low.  Remember, setrlimit(2) and prlimit(2)
> say that RLIMIT_DATA is specified in bytes.  This is not the same as
> bash's ulimit.

I've never thought 1MB would be too low for /bin/ls, but it seems you are
right. Over 2MB required for a simple /bin/true!

$ prlimit --as=2000000 /bin/true
/bin/true: error while loading shared libraries: libc.so.6: failed to map segment from shared object
$ prlimit --as=2500000 /bin/true
$ 

But prlimit works mostly as expected:

$ prlimit --as=1073741824 stress-ng stress-ng --vm 1 --vm-bytes 2G
stress-ng: info:  [19042] defaulting to a 86400 second (1 day, 0.00 secs) run per stressor
stress-ng: info:  [19042] dispatching hogs: 1 vm
stress-ng: error: [19044] stress-ng-vm: gave up trying to mmap, no available memory
stress-ng: info:  [19042] successful run completed in 10.01s
$

-- 
Victor Sudakov VAS4-RIPE
http://vas.tomsk.ru/
2:5005/49@fidonet

[toc] | [prev] | [next] | [standalone]

#236950

[Multipart message — attachments visible in raw view] — view raw

The Wanderer wrote:
> On 2021-07-05 at 22:42, Victor Sudakov wrote:
> 
> > Greg Wooledge wrote:
> 
> >> And in any case, when you test this stuff, use the subshells.  
> > 
> > Well, my goal is to find a way to limit programs run from cron, PHP's
> > proc_open() etc., not from the interactive shell. What would you advise?
> 
> Is there any reason to think that the ulimit approach wouldn't work in
> that type of context, as well as when invoked from within a shell?

That depends how you are going to *use* the ulimit approach in that type
of context. For example, there is no way to specify ulimit in PHP's
proc_open(), maybe because setrlimit() is OS-specific.

There are other cases when binaries are executed directly, without a
shell, hence the need for a wrapper.

-- 
Victor Sudakov VAS4-RIPE
http://vas.tomsk.ru/
2:5005/49@fidonet

[toc] | [prev] | [next] | [standalone]

#236948

[Multipart message — attachments visible in raw view] — view raw

Victor Sudakov wrote:
> 
> FreeBSD has a simple way to run some ad-hoc program with memory limits:
> 
> $ limits -m 2G ./mytest
>   memoryuse             2097152 kB
>   vmemoryuse           infinity kB
> $ limits -m 1G ./mytest
>   memoryuse             1048576 kB
>   vmemoryuse           infinity kB
> 
> How do I do the same in Linux (without root permissions)?

The short answer is /usr/bin/prlimit, it's the direct equivalent of
FreeBSD's /usr/bin/limits wrapper.

Thanks to all who replied especially Greg Wooledge!


-- 
Victor Sudakov VAS4-RIPE
http://vas.tomsk.ru/
2:5005/49@fidonet

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web