Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #270042 > unrolled thread

Having ten thousands of mount bind causes various processes to go into loops

Back to article view | Back to linux.debian.user

Contents

  Having ten thousands of mount bind causes various processes to go  into loops Julien Petit <julien@nethik.fr> - 2024-06-12 12:20 +0200
    Re: Having ten thousands of mount bind causes various processes to go  into loops Max Nikulin <manikulin@gmail.com> - 2024-06-12 18:20 +0200
    Re: Having ten thousands of mount bind causes various processes to go  into loops Richard <rrosner5@gmail.com> - 2024-06-12 19:10 +0200
      Re: Having ten thousands of mount bind causes various processes to go  into loops Julien Petit <julien@nethik.fr> - 2024-06-14 12:00 +0200
        Re: Having ten thousands of mount bind causes various processes to go  into loops Richard <rrosner5@gmail.com> - 2024-06-14 13:00 +0200
          Re: Having ten thousands of mount bind causes various processes to go  into loops Julien Petit <julien@nethik.fr> - 2024-06-19 11:10 +0200

#270042 — Having ten thousands of mount bind causes various processes to go into loops

Dear,

Not sure i should report a bug so here is a report first. For more
than 10 years now, we've been using mount binds to create shares rw or
ro. It's been working perfectly under older Debian. A few months ago,
we migrated to Ubuntu Jammy and started having processes running 100%
non stop. While examining the processes in question, we could see the
same thing: it seemed to be reading all the mounts indefinitely.
It started with the phpsessionclean.service. We managed to fix it
editing /lib/systemd/system/phpsessionclean.service and disabling
sandboxing entries. But then it started to happen with other
processes.
Anything related to systemd seems affected in a way. For instance, we
cannot start haproxy if the mounts are mounted.
We tested with the last Debian and it is affected too.

We understand that 14 000 mounts is a lot. So maybe our usage will be
questioned. But this has been working for ages so why not now?

The problem can be very easily reproduced:

1. Launch the latest Debian stable
2. Execute the following script to create mounts:
#!/bin/bash
mkdir /home/test/directories
mkdir /home/test/mounts

for i in {1..14000}
do
   echo "Mounting dir $i"
   mkdir "/home/test/directories/dir_$i"
   mkdir "/home/test/mounts/dir_$i"
   mount --bind -o rw "/home/test/directories/dir_$i" "/home/test/mounts/dir_$i"
done

After that, the "top" command will show processes getting stuck using
100% of CPU never ending.

Has anyone a clue if this is fixable? Should i report a bug?
Thanks for your help.

[toc] | [next] | [standalone]

#270047

On 12/06/2024 17:02, Julien Petit wrote:
> for i in {1..14000}
> do
>     echo "Mounting dir $i"
>     mkdir "/home/test/directories/dir_$i"
>     mkdir "/home/test/mounts/dir_$i"
>     mount --bind -o rw "/home/test/directories/dir_$i" "/home/test/mounts/dir_$i"
> done
> 
> After that, the "top" command will show processes getting stuck using
> 100% of CPU never ending.

What processes are CPU hungry?

> Has anyone a clue if this is fixable? Should i report a bug?

Perhaps it is not a Debian-specific bug, just more active usage of 
sandboxing in systemd. If some applications have troubles parsing 
/proc/mounts then bugs should be filed against them.

However do you need shared subtrees? It may cause exponential growth of 
number of moutpoints, see
https://docs.kernel.org/filesystems/sharedsubtree.html
https://manpages.debian.org/bookworm/mount/mount.8.en.html#Shared_subtree_operations

[toc] | [prev] | [next] | [standalone]

#270049

[Multipart message — attachments visible in raw view] — view raw

Best question probably is: what exactly are you needing 14.000 mounts for?
Even snaps shouldn't be that ridiculous. So what's your use case? Maybe
there's a better solution to what you are doing. If it's just about having
a place that is rw only without execution permissions, just crate a
separate partition, mount it somewhere - e.g. /home/test/mounts and tell
mount/fstab to use the option noexec. No need for for your script. Or if
it's a more advanced file system like btrfs you may be able to simply
create a subvolume with the same capabilities, no need to tinker around
with partitions.

It's true this issue should be looked into, but it doesn't look urgent as
long as there are alternatives.

Richard

Am Mi., 12. Juni 2024 um 16:33 Uhr schrieb Julien Petit <julien@nethik.fr>:

> Dear,
>
> Not sure i should report a bug so here is a report first. For more
> than 10 years now, we've been using mount binds to create shares rw or
> ro. It's been working perfectly under older Debian. A few months ago,
> we migrated to Ubuntu Jammy and started having processes running 100%
> non stop. While examining the processes in question, we could see the
> same thing: it seemed to be reading all the mounts indefinitely.
> It started with the phpsessionclean.service. We managed to fix it
> editing /lib/systemd/system/phpsessionclean.service and disabling
> sandboxing entries. But then it started to happen with other
> processes.
> Anything related to systemd seems affected in a way. For instance, we
> cannot start haproxy if the mounts are mounted.
> We tested with the last Debian and it is affected too.
>
> We understand that 14 000 mounts is a lot. So maybe our usage will be
> questioned. But this has been working for ages so why not now?
>
> The problem can be very easily reproduced:
>
> 1. Launch the latest Debian stable
> 2. Execute the following script to create mounts:
> #!/bin/bash
> mkdir /home/test/directories
> mkdir /home/test/mounts
>
> for i in {1..14000}
> do
>    echo "Mounting dir $i"
>    mkdir "/home/test/directories/dir_$i"
>    mkdir "/home/test/mounts/dir_$i"
>    mount --bind -o rw "/home/test/directories/dir_$i"
> "/home/test/mounts/dir_$i"
> done
>
> After that, the "top" command will show processes getting stuck using
> 100% of CPU never ending.
>
> Has anyone a clue if this is fixable? Should i report a bug?
> Thanks for your help.
>
>

[toc] | [prev] | [next] | [standalone]

#270098

> Best question probably is: what exactly are you needing 14.000 mounts for? Even snaps shouldn't be that ridiculous. So what's your use case? Maybe there's a better solution to what you are doing. If it's just about having a place that is rw only without execution permissions, just crate a separate partition, mount it somewhere - e.g. /home/test/mounts and tell mount/fstab to use the option noexec. No need for for your script. Or if it's a more advanced file system like btrfs you may be able to simply create a subvolume with the same capabilities, no need to tinker around with partitions.

We use the mounts to share an initial folder with either rw or ro
wrights in a user directory. The user directory is then accessible
through a web interface, sftp, webdav and rsync. There is probably
better ways to do that now but that's a legacy app (2009) that we'd
rather leave alone :)

> It's true this issue should be looked into, but it doesn't look urgent as long as there are alternatives.

Yes, not urgent and very specific. I'm going to try to set the mounts
to private as Max suggested and see how it goes.
Thanks for your help.

[toc] | [prev] | [next] | [standalone]

#270099

[Multipart message — attachments visible in raw view] — view raw

On 14.06.24 11:38, Julien Petit wrote:
> We use the mounts to share an initial folder with either rw or ro
> wrights in a user directory. The user directory is then accessible
> through a web interface, sftp, webdav and rsync. There is probably
> better ways to do that now but that's a legacy app (2009) that we'd
> rather leave alone :)
If there's a better way should be judged on what exactly that app expects. For the web interface, maybe the http server - or whatever makes the web interface accessible to the users - can limit permissions. For the rest of the use cases it would be interesting which circumstances would need to be fulfilled for a user to be able to change permissions on a file they own. And if they could even change the permissions through sftp, webdav or rsync. Because if not, the simplest fix would be a cron job that peridoically sets the permissions on the directory, so you don't need a dedicated mount. But Maybe you want to create a separate topic where you describe exatcly what the basic requirements are and ask for suggestions what the best solution may be. Maybe something like AppArmor rules or other methods that aren't known by your typical user could be a better solution.
> Yes, not urgent and very specific. I'm going to try to set the mounts
> to private as Max suggested and see how it goes.
> Thanks for your help.

If you haven't already, remember to create a bug report and include as much detail and logs as you can gather, as people will need to be able to tell what the actual issue is. Maybe it's a limitation of the file system, of the hardware or something else.

Richard

[toc] | [prev] | [next] | [standalone]

#270251

> If there's a better way should be judged on what exactly that app expects. For the web interface, maybe the http server - or whatever makes the web interface accessible to the users - can limit permissions. For the rest of the use cases it would be interesting which circumstances would need to be fulfilled for a user to be able to change permissions on a file they own. And if they could even change the permissions through sftp, webdav or rsync. Because if not, the simplest fix would be a cron job that peridoically sets the permissions on the directory, so you don't need a dedicated mount. But Maybe you want to create a separate topic where you describe exatcly what the basic requirements are and ask for suggestions what the best solution may be. Maybe something like AppArmor rules or other methods that aren't known by your typical user could be a better solution.

Rights are not the challenge here. It's to be able to share a
directory across multiple users.
For instance you would have : /users/bob/accounting shared with Alice
and accessible in her home directory /users/alice/accounting

> If you haven't already, remember to create a bug report and include as much detail and logs as you can gather, as people will need to be able to tell what the actual issue is. Maybe it's a limitation of the file system, of the hardware or something else.

I haven't already. I want to test the using private mounts first.

Thanks again for your input.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web