Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1199320 > unrolled thread

Re: kdbus: to merge or not to merge?

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: kdbus: to merge or not to merge? Andy Lutomirski <luto@kernel.org> - 2015-08-04 01:10 +0200
    Re: kdbus: to merge or not to merge? David Herrmann <dh.herrmann@gmail.com> - 2015-08-04 11:00 +0200
      Re: kdbus: to merge or not to merge? Linus Torvalds <torvalds@linux-foundation.org> - 2015-08-04 15:50 +0200
        Re: kdbus: to merge or not to merge? David Herrmann <dh.herrmann@gmail.com> - 2015-08-04 16:20 +0200
          Re: kdbus: to merge or not to merge? Andy Lutomirski <luto@amacapital.net> - 2015-08-04 16:50 +0200
            Re: kdbus: to merge or not to merge? Andy Lutomirski <luto@amacapital.net> - 2015-08-05 02:20 +0200
              Re: kdbus: to merge or not to merge? Daniel Mack <daniel@zonque.org> - 2015-08-06 09:10 +0200
                Re: kdbus: to merge or not to merge? Andy Lutomirski <luto@amacapital.net> - 2015-08-06 17:30 +0200
                  Re: kdbus: to merge or not to merge? Daniel Mack <daniel@zonque.org> - 2015-08-06 19:30 +0200
            Re: kdbus: to merge or not to merge? David Herrmann <dh.herrmann@gmail.com> - 2015-08-05 09:20 +0200
              Re: kdbus: to merge or not to merge? Andy Lutomirski <luto@amacapital.net> - 2015-08-05 22:20 +0200
                Re: kdbus: to merge or not to merge? David Herrmann <dh.herrmann@gmail.com> - 2015-08-06 10:10 +0200
                  Re: kdbus: to merge or not to merge? Martin Steigerwald <martin@lichtvoll.de> - 2015-08-06 10:30 +0200
                  Re: kdbus: to merge or not to merge? Andy Lutomirski <luto@amacapital.net> - 2015-08-06 17:30 +0200
                    Re: kdbus: to merge or not to merge? Daniel Mack <daniel@zonque.org> - 2015-08-06 20:20 +0200
                      Re: kdbus: to merge or not to merge? Andy Lutomirski <luto@amacapital.net> - 2015-08-06 20:50 +0200
                        Re: kdbus: to merge or not to merge? Daniel Mack <daniel@zonque.org> - 2015-08-07 16:50 +0200
                          Re: kdbus: to merge or not to merge? Andy Lutomirski <luto@amacapital.net> - 2015-08-07 17:10 +0200
                          Re: kdbus: to merge or not to merge? Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2015-08-09 21:10 +0200
                            Re: kdbus: to merge or not to merge? Daniel Mack <daniel@zonque.org> - 2015-08-10 00:20 +0200
                              Re: kdbus: to merge or not to merge? Andy Lutomirski <luto@amacapital.net> - 2015-08-10 04:20 +0200
                              Re: kdbus: to merge or not to merge? Linus Torvalds <torvalds@linux-foundation.org> - 2015-08-10 19:10 +0200
                            Re: kdbus: to merge or not to merge? David Lang <david@lang.hm> - 2015-08-10 04:50 +0200
                        Re: kdbus: to merge or not to merge? cee1 <fykcee1@gmail.com> - 2015-08-07 17:40 +0200

Page 1 of 2  [1] 2  Next page →

#1199320 — Re: kdbus: to merge or not to merge?

On Mon, Jun 22, 2015 at 11:06 PM, Andy Lutomirski <luto@amacapital.net> wrote:
> 2. Kdbus introduces a novel buffering model.  Receivers allocate a big
> chunk of what's essentially tmpfs space.  Assuming that space is
> available (in a virtual memory sense), senders synchronously write to
> the receivers' tmpfs space.  Broadcast senders synchronously write to
> *all* receivers' tmpfs space.  I think that, regardless of
> implementation, this is problematic if the sender and the receiver are
> in different memcgs.  Suppose that the message is to be written to a
> page in the receivers' tmpfs space that is not currently resident.  If
> the write happens in the sender's memcg context, then a receiver can
> effectively allocate an unlimited number of pages in the sender's
> memcg, which will, in practice, be the init memcg if the sender is
> systemd.  This breaks the memcg model.  If, on the other hand, the
> sender writes to the receiver's tmpfs space in the receiver's memcg
> context, then the sender will block (or fail?  presumably
> unpredictable failures are a bad thing) if the receiver's memcg is at
> capacity.

I realize that everyone is sick of this thread.  Nonetheless, I should
emphasize that I'm actually serious about this issue.  I got Fedora
Rawhide working under kdbus (thanks, everyone!), and I ran this little
program:

#include <systemd/sd-bus.h>
#include <err.h>

int main(int argc, char *argv[])
{
    while (1) {
        sd_bus *bus;
        if (sd_bus_open_system(&bus) < 0) {
            /* warn("sd_bus_open_system"); */
            continue;
        }
        sd_bus_close(bus);
    }
}

under both userspace dbus and under kdbus.  Userspace dbus burns some
CPU -- no big deal.  I expected kdbus to fail to scale and burn a
disproportionate amount of CPU (because I don't see how it /can/
scale).  Instead it fell over completely.  I didn't bother debugging
it, but offhand I'd guess that the system OOMed and didn't come back.

On very brief inspection, Rawhide seems to have a lot of kdbus
connections with 16MiB of mapped tmpfs stuff each.  (53 of them
mapped, and I don't know how many exist with tmpfs backing but aren't
mapped.  Presumably the number only goes up as the degree of reliance
on the userspace proxy goes down.  As it stands, that's over 3GB of
uncommitted backing store that my test is likely to forcibly commit
very quickly.)

Frankly, I don't understand how it's possible to cleanly implement
kdbus' broadcast or lifetime semantics* in an environment with bounded
CPU or bounded memory.  (And unbounded memory just changes the
problem, since the message backlog can just get worse and worse.)

I work in an industry in which lots of parties broadcast lots of data
to lots of people.  If you try to drink from the firehose and you
can't swallow fast enough, either you need to throw something out (and
test your recovery code!) or you fail.  At least in finance, no one
pretends that a global order of events in different cities is
practical.

* Detecting when when your peer goes away is, of course, a widely
encountered and widely solved problem.  I don't know of any deployed
systems that solve it by broadcasting the lifetime of everything to
everyone and relying on those broadcasts going through, though.

--Andy
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1199561

Hi

On Tue, Aug 4, 2015 at 1:02 AM, Andy Lutomirski <luto@kernel.org> wrote:
>                                                I got Fedora
> Rawhide working under kdbus (thanks, everyone!), and I ran this little
> program:
>
> #include <systemd/sd-bus.h>
> #include <err.h>
>
> int main(int argc, char *argv[])
> {
>     while (1) {
>         sd_bus *bus;
>         if (sd_bus_open_system(&bus) < 0) {
>             /* warn("sd_bus_open_system"); */
>             continue;
>         }
>         sd_bus_close(bus);

You lack a call to sd_bus_unref() here. Without it, your loop contains:

        while (1)
                malloc(1024);

This simple malloc-loop already hogs your system. If I add the
required call to _unref(), your tool runs smoothly on my machine.

>     }
> }
>
> under both userspace dbus and under kdbus.  Userspace dbus burns some
> CPU -- no big deal.  I expected kdbus to fail to scale and burn a
> disproportionate amount of CPU (because I don't see how it /can/
> scale).  Instead it fell over completely.  I didn't bother debugging
> it, but offhand I'd guess that the system OOMed and didn't come back.

I cannot see the relation to kdbus.

> On very brief inspection, Rawhide seems to have a lot of kdbus
> connections with 16MiB of mapped tmpfs stuff each.  (53 of them
> mapped, and I don't know how many exist with tmpfs backing but aren't
> mapped.  Presumably the number only goes up as the degree of reliance
> on the userspace proxy goes down.

What does this have to do with the proxy? Why would resource
consumption go *up* as the proxy users decline? Please elaborate.

>                                         I don't know of any deployed
> systems that solve it by broadcasting the lifetime of everything to
> everyone and relying on those broadcasts going through, though.

Luckily, kdbus does not do this.

Thanks
David
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1199844

On Tue, Aug 4, 2015 at 1:58 AM, David Herrmann <dh.herrmann@gmail.com> wrote:
>
> You lack a call to sd_bus_unref() here.

I assume it was intentional. Why would Andy talk about "scaling" otherwise?

And the worry was why the kdbus version killed the machine, but the
userspace version did not. That's a rather big difference, and not a
good one.

Possibly the kdbus version ends up not just allocating user space
memory (which we should handle gracefully), but kernel allocations too
(which absolutely have to be explicitly resource-managed).

               Linus
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1199902

Hi

On Tue, Aug 4, 2015 at 3:46 PM, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
> On Tue, Aug 4, 2015 at 1:58 AM, David Herrmann <dh.herrmann@gmail.com> wrote:
>>
>> You lack a call to sd_bus_unref() here.
>
> I assume it was intentional. Why would Andy talk about "scaling" otherwise?
>
> And the worry was why the kdbus version killed the machine, but the
> userspace version did not. That's a rather big difference, and not a
> good one.

Neither test 'kills' the machine:

 * The userspace version will be killed by the OOM killer after about
20s running (depending how much memory you have).

 * The kernel version runs for 1024 iterations (maximum kdbus
connections per user) and then produces errors.

In fact, the kernel version is even more stable than the user-space
version, and bails out much earlier. Run it on a VT and everything
works just fine.

The only issue you get with kdbus is the compat-bus-daemon, which
assert()s as a side-effect of accept4() failing. In other words, the
compat bus-daemon gets ENFILE if you open that many connections, then
assert()s and thus kills all other proxy connections. This has the
side effect, that Xorg loses access to your graphics device and thus
your screen 'freezes'. Also networkmanager bails out and stops network
connections.

This is a bug in the proxy (which is already fixed).

Thanks
David
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1199954

On Tue, Aug 4, 2015 at 7:09 AM, David Herrmann <dh.herrmann@gmail.com> wrote:
> Hi
>
> On Tue, Aug 4, 2015 at 3:46 PM, Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
>> On Tue, Aug 4, 2015 at 1:58 AM, David Herrmann <dh.herrmann@gmail.com> wrote:
>>>
>>> You lack a call to sd_bus_unref() here.
>>
>> I assume it was intentional. Why would Andy talk about "scaling" otherwise?

It was actually an error.  I assumed that, since the user version
worked fine (at least for as long as I ran it) and the kernel version
didn't (killed X and left a blinking cursor, no visible log messages
even when run from a text console, and no obvious OOM recovery after a
long wait) that it was a kdbus issue or issue with other kdbus
clients.

I'll play with it more today.

>>
>> And the worry was why the kdbus version killed the machine, but the
>> userspace version did not. That's a rather big difference, and not a
>> good one.
>
> Neither test 'kills' the machine:
>
>  * The userspace version will be killed by the OOM killer after about
> 20s running (depending how much memory you have).

Not on my system.  Maybe too much memory?

>
>  * The kernel version runs for 1024 iterations (maximum kdbus
> connections per user) and then produces errors.
>
> In fact, the kernel version is even more stable than the user-space
> version, and bails out much earlier. Run it on a VT and everything
> works just fine.


On my system, everything died as described above.

>
> The only issue you get with kdbus is the compat-bus-daemon, which
> assert()s as a side-effect of accept4() failing. In other words, the
> compat bus-daemon gets ENFILE if you open that many connections, then
> assert()s and thus kills all other proxy connections. This has the
> side effect, that Xorg loses access to your graphics device and thus
> your screen 'freezes'. Also networkmanager bails out and stops network
> connections.

Ah, interesting.

>
> This is a bug in the proxy (which is already fixed).

Should I expect to see it in Rawhide soon?

Anyway, the broadcasts that I intended to exercise were
KDBUS_ITEM_ID_REMOVE.  Those appear to be broadcast to everyone,
irrespective of "policy", so long as the "match" thingy allows it.  As
far as I can tell, that's the default behavior (i.e. receivers accept
KDBUS_DST_ID_BROADCAST), but even if it's not default, we'll still
fail to scale as long as the number of receivers accepting
KDBUS_DST_ID_BROADCAST grows as systems become more kdbus-integrated.

The bloom filter thing won't help at all according to the docs: bloom
filters don't apply to kernel-generated notifications.

So yes, as far as I can tell, kdbus really does track object lifetime
by broadcasting every single destruction event to every single
receiver (subject to caveats above) and pokes the data into every
receiver's tmpfs space (via kdbus_bus_broadcast ->
kdbus_conn_entry_insert -> lots of other stuff -> vfs_iter_write).  At
that point, there's well over a gigabyte of tmpfs space that can be
scribbled on (and thus committed and thus needs to be read) by rogue
broadcasters even on Rawhide, and Rawhide seems to have barely started
converting all the kdbus clients from using the proxy to using kdbus
directly.

IIUC, once gdbus switches over to using kdbus directly, with current
buffer sizing, the average laptop will have more kdbus pool tmpfs
space mapped than total RAM.  I still don't see how this will work
well.

I guess my test didn't exercise what I meant it to.  I wrote it,
userspace survived (on my system) and kdbus didn't.  Apparently I blew
up the bus proxy, not the pool mechanism.  Next time I'll try to
better characterize exactly what it is I'm doing to my poor VM...

--Andy
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1200321

On Tue, Aug 4, 2015 at 7:47 AM, Andy Lutomirski <luto@amacapital.net> wrote:
> On Tue, Aug 4, 2015 at 7:09 AM, David Herrmann <dh.herrmann@gmail.com> wrote:
>> Hi
>>
>> On Tue, Aug 4, 2015 at 3:46 PM, Linus Torvalds
>> <torvalds@linux-foundation.org> wrote:
>>> On Tue, Aug 4, 2015 at 1:58 AM, David Herrmann <dh.herrmann@gmail.com> wrote:
>>>>
>>>> You lack a call to sd_bus_unref() here.
>>>
>>> I assume it was intentional. Why would Andy talk about "scaling" otherwise?
>
> It was actually an error.  I assumed that, since the user version
> worked fine (at least for as long as I ran it) and the kernel version
> didn't (killed X and left a blinking cursor, no visible log messages
> even when run from a text console, and no obvious OOM recovery after a
> long wait) that it was a kdbus issue or issue with other kdbus
> clients.
>
> I'll play with it more today.
>

I added the missing sd_bus_unref call.

With userspace dbus, my program takes 95% CPU and dbus-daemon takes
88% CPU or so.

With kdbus, I see abuse-bus (my test), systemd-journald,
systemd-bus-proxy, auditd, gnome-shell, mission-control, sedispatch,
firewalld, polkitd, NetworkManager, systemd, avahi-daemon, audisp,
abrt-dump-jour* (whatever it's called -- it truncated), upowerd, and
systemd-logind all taking tons of CPU.  I've listed them in decreasing
order of amount of CPU burned -- the top several are taking about as
much as is possible.  Load average is over 13.  That's if I run it
from a text console while I'm logged in to gnome in a different VT.

If I run the program from a graphical terminal, everything freezes so
hard that the cursor doesn't even make it to the next line when I hit
enter.

So I still claim that kdbus doesn't scale.  I'm not even just saying
that it doesn't scale to large systems -- somewhat to my surprise, it
doesn't even seem to scale well enough for a mostly empty Rawhide
workstation system running just a graphical terminal.  And I didn't
even try to find stress tests more interesting than connecting and
disconnecting in a loop.

FWIW, the old test (without the unref) appeared to be allocating 16M
of mapped kdbus pool every iteration, which seems unlikely to have
helped matters.

--Andy
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1201503

Hi Andy,

On 08/05/2015 02:18 AM, Andy Lutomirski wrote:
> I added the missing sd_bus_unref call.
> 
> With userspace dbus, my program takes 95% CPU and dbus-daemon takes
> 88% CPU or so.
> 
> With kdbus, I see abuse-bus (my test), systemd-journald,
> systemd-bus-proxy, auditd, gnome-shell, mission-control, sedispatch,
> firewalld, polkitd, NetworkManager, systemd, avahi-daemon, audisp,
> abrt-dump-jour* (whatever it's called -- it truncated), upowerd, and
> systemd-logind all taking tons of CPU.  I've listed them in decreasing
> order of amount of CPU burned -- the top several are taking about as
> much as is possible.  Load average is over 13.  That's if I run it
> from a text console while I'm logged in to gnome in a different VT.

That's right, I can reproduce this here. To explain what's going on, let
me provide some background.

Every time a client connects to kdbus, a new ID is assigned to the
connection, and other connections which have previously subscribed to
notifications of type KDBUS_ITEM_ID_ADD or _REMOVE get a notification
and are woken up so they can dispatch it. By default, no such matches
exists, applicaions have to explicitly opt-in if they are interested in
these events.

In DBus (both kdbus and DBus1), such matches are installed on the
NameOwnerChanged signal, and they can be either specific to a single ID,
or broad, which will make them match on any ID. There's actually no
reason for applications to install unspecific matches, but if they do,
they will of course get what they asked for, and are woken up on every
ID that is added to or removed from the bus. What you're seeing in your
system profile is that some applications misbehave and install
unspecific matches when they shouldn't. That's a userspace bug that
needs fixing. Two candidates were actually in the systemd code base
(logind and PID1), and both are now patched.

Note that these applications are actually affected on both DBus1 and
kdbus. The reason you didn't see them trip up in your test is that
sd_bus_open() behaves differently in the two worlds. In kdbus, it will
immediately call into the kernel and register a new connection, hence
triggering the behavior described above. On DBus1, however, the HELLO
message will not be transmitted to the daemon until the first message is
sent, so no ID is assigned, and no notifications are sent. When
augmenting the test program a little so it reads its own ID on the bus,
you'll see similar behavior on DBus1 as well, but the bottleneck in this
case is the daemon, which significantly mitigates the load caused by
other tasks.

So, to wrap it up: you've triggered an existing userspace bug. The
userspace components under our control have now been fixed, and we'll
talk to other people to make them aware of the issue too. However, these
issues are not directly related to kdbus, but rather show more impact as
a side-effect now.

You've raised a valid point here. Thanks a lot for providing this test,
much appreciated!


Daniel

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1201844

On Thu, Aug 6, 2015 at 12:06 AM, Daniel Mack <daniel@zonque.org> wrote:
> Hi Andy,
>
> On 08/05/2015 02:18 AM, Andy Lutomirski wrote:
>> I added the missing sd_bus_unref call.
>>
>> With userspace dbus, my program takes 95% CPU and dbus-daemon takes
>> 88% CPU or so.
>>
>> With kdbus, I see abuse-bus (my test), systemd-journald,
>> systemd-bus-proxy, auditd, gnome-shell, mission-control, sedispatch,
>> firewalld, polkitd, NetworkManager, systemd, avahi-daemon, audisp,
>> abrt-dump-jour* (whatever it's called -- it truncated), upowerd, and
>> systemd-logind all taking tons of CPU.  I've listed them in decreasing
>> order of amount of CPU burned -- the top several are taking about as
>> much as is possible.  Load average is over 13.  That's if I run it
>> from a text console while I'm logged in to gnome in a different VT.
>
> That's right, I can reproduce this here. To explain what's going on, let
> me provide some background.
>
> Every time a client connects to kdbus, a new ID is assigned to the
> connection, and other connections which have previously subscribed to
> notifications of type KDBUS_ITEM_ID_ADD or _REMOVE get a notification
> and are woken up so they can dispatch it. By default, no such matches
> exists, applicaions have to explicitly opt-in if they are interested in
> these events.
>
> In DBus (both kdbus and DBus1), such matches are installed on the
> NameOwnerChanged signal, and they can be either specific to a single ID,
> or broad, which will make them match on any ID. There's actually no
> reason for applications to install unspecific matches, but if they do,
> they will of course get what they asked for, and are woken up on every
> ID that is added to or removed from the bus. What you're seeing in your
> system profile is that some applications misbehave and install
> unspecific matches when they shouldn't. That's a userspace bug that
> needs fixing. Two candidates were actually in the systemd code base
> (logind and PID1), and both are now patched.

Can you point me at the patch?

It sounds like that will reduce the scalability issue with this
particular test from whatever userspace overhead exists * number of
clients to just the overhead of looping over all clients and their
matches in the kernel.

--Andy
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1201924

On 08/06/2015 05:27 PM, Andy Lutomirski wrote:
>> In DBus (both kdbus and DBus1), such matches are installed on the
>> > NameOwnerChanged signal, and they can be either specific to a single ID,
>> > or broad, which will make them match on any ID. There's actually no
>> > reason for applications to install unspecific matches, but if they do,
>> > they will of course get what they asked for, and are woken up on every
>> > ID that is added to or removed from the bus. What you're seeing in your
>> > system profile is that some applications misbehave and install
>> > unspecific matches when they shouldn't. That's a userspace bug that
>> > needs fixing. Two candidates were actually in the systemd code base
>> > (logind and PID1), and both are now patched.
>
> Can you point me at the patch?

  https://github.com/systemd/systemd/pull/876
  https://github.com/systemd/systemd/pull/887

firewalld and possibly some other applications in the Fedora default
install use python-slip, a convenience library that currently
unconditionally installs the broad matches. I filed a bug with patches here:

  https://fedorahosted.org/python-slip/ticket/2


And I filed more bugs for some GNOME components.


Thanks,
Daniel

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1200471

Hi

On Tue, Aug 4, 2015 at 4:47 PM, Andy Lutomirski <luto@amacapital.net> wrote:
> On Tue, Aug 4, 2015 at 7:09 AM, David Herrmann <dh.herrmann@gmail.com> wrote:
>> This is a bug in the proxy (which is already fixed).
>
> Should I expect to see it in Rawhide soon?

Use this workaround until it does:

    $ DBUS_SYSTEM_BUS_ADDRESS="kernel:path=/sys/fs/kdbus/0-system/bus"
./your-binary

> Anyway, the broadcasts that I intended to exercise were
> KDBUS_ITEM_ID_REMOVE.  Those appear to be broadcast to everyone,
> irrespective of "policy", so long as the "match" thingy allows it.

Matches are opt-in, not opt-out. Nobody will get this message unless
they opt in.

> The bloom filter thing won't help at all according to the docs: bloom
> filters don't apply to kernel-generated notifications.

Bloom filters apply to message payloads. Kernel notifications do not
carry a message payload. Message metadata can be filtered for
explicitly (without false-positives).

> So yes, as far as I can tell, kdbus really does track object lifetime
> by broadcasting every single destruction event to every single
> receiver (subject to caveats above) and pokes the data into every
> receiver's tmpfs space.

Broadcast reception is opt-in.

Thanks
David
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1201094

On Wed, Aug 5, 2015 at 12:10 AM, David Herrmann <dh.herrmann@gmail.com> wrote:
> Hi
>
> On Tue, Aug 4, 2015 at 4:47 PM, Andy Lutomirski <luto@amacapital.net> wrote:
>> On Tue, Aug 4, 2015 at 7:09 AM, David Herrmann <dh.herrmann@gmail.com> wrote:
>>> This is a bug in the proxy (which is already fixed).
>>
>> Should I expect to see it in Rawhide soon?
>
> Use this workaround until it does:
>
>     $ DBUS_SYSTEM_BUS_ADDRESS="kernel:path=/sys/fs/kdbus/0-system/bus"
> ./your-binary
>

Which binary is supposed to be run like that?

>> Anyway, the broadcasts that I intended to exercise were
>> KDBUS_ITEM_ID_REMOVE.  Those appear to be broadcast to everyone,
>> irrespective of "policy", so long as the "match" thingy allows it.
>
> Matches are opt-in, not opt-out. Nobody will get this message unless
> they opt in.
>

And what opts in?  Either something's broken, or there's a different
scalabilty problem, or a whole pile of kdbus-using programs in Fedora
Rawhide do, in fact, opt in.

My interest in instrumenting kdbus and systemd to figure out the exact
mechanism by which my tiny test case causes my system to freeze is
near zero.  I bet I'm actually right about the mechanism, but that's
sort of beside the point.  It freezes, so /something's/ wrong.  The
only real relevance of my suspicion about the failure mode is that I
think it's a design issue that isn't going to be easy to fix.

>
>> So yes, as far as I can tell, kdbus really does track object lifetime
>> by broadcasting every single destruction event to every single
>> receiver (subject to caveats above) and pokes the data into every
>> receiver's tmpfs space.
>
> Broadcast reception is opt-in.

I've pointed out several times that there a feature in kdbus that
doesn't work well and I get told that the problematic feature is
opt-in.  Given that all existing prototype userspace that I'm aware of
(systemd and its consumers) apparently opts in, I don't really care
that the feature is opt-in.

Also, given things like this:

commit d27c8057699d164648b7d8c1559fa6529998f89d
Author: David Herrmann <dh.herrmann@gmail.com>
Date:   Tue May 26 09:30:14 2015 +0200

    kdbus: forward ID notifications to everyone

it really does seem to me that the point of these ID notifications is
for everyone to get them.

Also, you haven't addressed the memory usage issues -- I don't see how
a full kdbus-using desktop system can be expected to fit into RAM on
anything short of the biggest and beefiest laptops.  I also don't see
how a kdbus-using xdg-app-happy kdbus-using system (with
correspondingly many pools) will fit into RAM on even the biggest
laptops.

--Andy
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1201536

Hi

On Wed, Aug 5, 2015 at 10:11 PM, Andy Lutomirski <luto@amacapital.net> wrote:
> On Wed, Aug 5, 2015 at 12:10 AM, David Herrmann <dh.herrmann@gmail.com> wrote:
>> Hi
>>
>> On Tue, Aug 4, 2015 at 4:47 PM, Andy Lutomirski <luto@amacapital.net> wrote:
>>> On Tue, Aug 4, 2015 at 7:09 AM, David Herrmann <dh.herrmann@gmail.com> wrote:
>>>> This is a bug in the proxy (which is already fixed).
>>>
>>> Should I expect to see it in Rawhide soon?
>>
>> Use this workaround until it does:
>>
>>     $ DBUS_SYSTEM_BUS_ADDRESS="kernel:path=/sys/fs/kdbus/0-system/bus"
>> ./your-binary
>>
>
> Which binary is supposed to be run like that?

Your test.

>>> Anyway, the broadcasts that I intended to exercise were
>>> KDBUS_ITEM_ID_REMOVE.  Those appear to be broadcast to everyone,
>>> irrespective of "policy", so long as the "match" thingy allows it.
>>
>> Matches are opt-in, not opt-out. Nobody will get this message unless
>> they opt in.
>>
>
> And what opts in?  Either something's broken, or there's a different
> scalabilty problem, or a whole pile of kdbus-using programs in Fedora
> Rawhide do, in fact, opt in.

See Daniel's explanation. If applications subscribe to all
notifications, they get what they asked for. I recommend filing bug
reports for the applications in question.

>          Given that all existing prototype userspace that I'm aware of
> (systemd and its consumers) apparently opts in, I don't really care
> that the feature is opt-in.

This is just plain wrong. Out of the dozens of dbus applications, you
found like 9 which are buggy? Two of them are already fixed, the
maintainers of the other ones notified.
I'd be interested where you got this notion that "all existing
prototype userspace [...] opts in".

> Also, given things like this:
>
> commit d27c8057699d164648b7d8c1559fa6529998f89d
> Author: David Herrmann <dh.herrmann@gmail.com>
> Date:   Tue May 26 09:30:14 2015 +0200
>
>     kdbus: forward ID notifications to everyone
>
> it really does seem to me that the point of these ID notifications is
> for everyone to get them.

It's not. This patch just opens the policy so everyone can see those
notifications. By default, it's not delivered to anyone.

> Also, you haven't addressed the memory usage issues --

..because it doesn't change anything. If your IPC is message based and
async, _someone_ needs to buffer. I don't see the difference between
buffering locally on !EPOLLOUT or buffering in a shmem pool. In both
cases, clients have control over the buffer size. If you disagree,
please _elaborate_.

Thanks
David
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1201554

Am Donnerstag, 6. August 2015, 10:04:57 schrieb David Herrmann:
> >          Given that all existing prototype userspace that I'm aware of
> >
> > (systemd and its consumers) apparently opts in, I don't really care
> > that the feature is opt-in.
> 
> This is just plain wrong. Out of the dozens of dbus applications, you
> found like 9 which are buggy? Two of them are already fixed, the
> maintainers of the other ones notified.
> I'd be interested where you got this notion that "all existing
> prototype userspace [...] opts in".

But these few can create the issues Andy described?

Sure, one can argue I can setup a stress or stress-ng command line invocation 
as root user that will basically grind a Linux system to a halt – and in a way 
I consider this to be a bug in the kernel as well, but one that exists since a 
long time. But a GUI application running as a user?

How about some robustness regarding what you see as bugs in userspace here?

I think "The bug is not mine" is exactly the same language we have seen here 
before. If the kernel relies on bug-free userspace applications in order to do 
its job properly I think it has robustness issues. One certainly wouldn´t want 
this with any mission critical realtime OS. I think it is the kernel that 
should be in control.

Thanks,
-- 
Martin
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1201846

On Aug 6, 2015 1:04 AM, "David Herrmann" <dh.herrmann@gmail.com> wrote:
> >          Given that all existing prototype userspace that I'm aware of
> > (systemd and its consumers) apparently opts in, I don't really care
> > that the feature is opt-in.
>
> This is just plain wrong. Out of the dozens of dbus applications, you
> found like 9 which are buggy? Two of them are already fixed, the
> maintainers of the other ones notified.
> I'd be interested where you got this notion that "all existing
> prototype userspace [...] opts in".
>

I would say instead that, out of one in-use kdbus library, I found one
that was buggy.  Maybe gdbus really does use kdbus already, but on
very brief inspection it looked like it didn't at least on my test VM.

>
> > Also, you haven't addressed the memory usage issues --
>
> ..because it doesn't change anything. If your IPC is message based and
> async, _someone_ needs to buffer. I don't see the difference between
> buffering locally on !EPOLLOUT or buffering in a shmem pool. In both
> cases, clients have control over the buffer size. If you disagree,
> please _elaborate_.

If the client buffers on !EPOLLOUT and has a monster buffer, then
that's the client's problem.

If every single program has a monster buffer, then it's everyone's
problem, and the size of the problem gets multiplied by the number of
programs.

Also, sensible clients that produce bulk data will throttle on
!EPOLLOUT rather than blindly buffering, but that's not an option when
the huge buffer is on the receiver's end.  Read up on "bufferbloat".

--Andy
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1201960

On 08/06/2015 05:21 PM, Andy Lutomirski wrote:
> Maybe gdbus really does use kdbus already, but on
> very brief inspection it looked like it didn't at least on my test VM.

No, it's not in any released version yet. The patches for that are being
worked on though and look promising.

> If the client buffers on !EPOLLOUT and has a monster buffer, then
> that's the client's problem.
> 
> If every single program has a monster buffer, then it's everyone's
> problem, and the size of the problem gets multiplied by the number of
> programs.

The size of the memory pool of a bus client is chosen by the client
itself individually during the HELLO call. It's pretty much the same as
if the client allocated the buffer itself, except that the kernel does
it on their behalf.

Also note that kdbus features a peer-to-peer based quota accounting
logic, so a single bus connection can not DOS another one by filling its
buffer.


Thanks,
Daniel
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1201976

On Thu, Aug 6, 2015 at 11:14 AM, Daniel Mack <daniel@zonque.org> wrote:
> On 08/06/2015 05:21 PM, Andy Lutomirski wrote:
>> Maybe gdbus really does use kdbus already, but on
>> very brief inspection it looked like it didn't at least on my test VM.
>
> No, it's not in any released version yet. The patches for that are being
> worked on though and look promising.
>
>> If the client buffers on !EPOLLOUT and has a monster buffer, then
>> that's the client's problem.
>>
>> If every single program has a monster buffer, then it's everyone's
>> problem, and the size of the problem gets multiplied by the number of
>> programs.
>
> The size of the memory pool of a bus client is chosen by the client
> itself individually during the HELLO call. It's pretty much the same as
> if the client allocated the buffer itself, except that the kernel does
> it on their behalf.
>
> Also note that kdbus features a peer-to-peer based quota accounting
> logic, so a single bus connection can not DOS another one by filling its
> buffer.

I haven't looked at the quota code at all.

Nonetheless, it looks like the slice logic (aside: it looks *way* more
complicated than necessary -- what's wrong with circular buffers)
will, under most (but not all!) workloads, concentrate access to a
smallish fraction of the pool.  This is IMO bad, since it means that
most of the time most of the pool will remain uncommitted.  If, at
some point, something causes the access pattern to change and hit all
the pages (even just once), suddenly all of the pools get committed,
and your memory usage blows up.

Again, please stop blaming the clients.  In practice, kdbus is a
system involving the kernel, systemd, sd-bus, and other stuff, mostly
written by the same people.  If kdbus gets merged and it survives but
half the clients blow up and peoples' systems fall over, that's not
okay.

--Andy
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1202767

On 08/06/2015 08:43 PM, Andy Lutomirski wrote:
> Nonetheless, it looks like the slice logic (aside: it looks *way* more
> complicated than necessary -- what's wrong with circular buffers)
> will, under most (but not all!) workloads, concentrate access to a
> smallish fraction of the pool.  This is IMO bad, since it means that
> most of the time most of the pool will remain uncommitted.  If, at
> some point, something causes the access pattern to change and hit all
> the pages (even just once), suddenly all of the pools get committed,
> and your memory usage blows up.

That's a general problem with memory overcommitment, and not specific to
kdbus. IOW: You'd have the same problem with a similar logic implemented
in userspace, right?


Daniel

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1202804

On Fri, Aug 7, 2015 at 7:40 AM, Daniel Mack <daniel@zonque.org> wrote:
> On 08/06/2015 08:43 PM, Andy Lutomirski wrote:
>> Nonetheless, it looks like the slice logic (aside: it looks *way* more
>> complicated than necessary -- what's wrong with circular buffers)
>> will, under most (but not all!) workloads, concentrate access to a
>> smallish fraction of the pool.  This is IMO bad, since it means that
>> most of the time most of the pool will remain uncommitted.  If, at
>> some point, something causes the access pattern to change and hit all
>> the pages (even just once), suddenly all of the pools get committed,
>> and your memory usage blows up.
>
> That's a general problem with memory overcommitment, and not specific to
> kdbus. IOW: You'd have the same problem with a similar logic implemented
> in userspace, right?
>

Sure, except that, if it's in userspace and it starts causing
problems, then userspace can fix it without running into kernel ABI
stability issues.

--Andy
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1203688

On Fri, Aug 07, 2015 at 06:26:31PM +0300, Linus Torvalds wrote:
> User space memory allocation is not AT ALL the same thing as kdbus.
> Kernel allocations are very very different from user allocations. We
> have reasonable, fairly tested, and generic models for handling user
> space memory allocation issues - limiting, debugging, failing, and
> handling catastrophes (ie oom). And no, even that doesn't always work
> perfectly, but at least there is a *lot* of support for it, and this
> is not some special case.

The memory in this case is a shmem file that is created by the kernel,
but on behalf of the bus client task, which will eventually own it. As
discussed with the mm developers, the same logic for accounting, OOM
handling, etc. applies to the kdbus shmem buffers, as they are written
to from the context of another task.  If this is mistaken, then yes, you
are right, and the code will have to be changed.

> This discussion has been full of kdbus people ignoring Andy saying "it
> worked with the user space version, it killed the machine with kdbus".
> And now people trying to claim the issues are the same. HELL NO.

Andy found some great bugs with regards to flooding the bus with
requests, which has not been ignored at all.  The same issue is present
in dbus today, but the kdbus code runs faster and more messages were
being sent than the current userspace dbus daemon, so the machine
becomes unresponsive easier.

The issue is with userspace clients opting in to receive all
NameOwnerChanged messages on the bus, which is not a good idea as they
constantly get woken up and process them, which is why the CPU was
pegged.  This issue should now be fixed in Rawhide for some of the
packages we found that were doing this. Maintainers of other packages
have been informed.  End result, no one has ever really tested sending
"bad" messages to the current system as all existing dbus users try to
be "good actors", thanks to Andy's testing, these apps should all now
become much more robust.

In chatting with Daniel on IRC, he is writing up a summary of how the
kdbus memory pools work in more detail, and he said he would sent that
out in a day or so, so that everyone can review.

thanks,

greg k-h

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1203704

On 08/09/2015 09:00 PM, Greg Kroah-Hartman wrote:
> In chatting with Daniel on IRC, he is writing up a summary of how the
> kdbus memory pools work in more detail, and he said he would sent that
> out in a day or so, so that everyone can review.

Yes, let me quickly describe again how the kdbus pool logic works.

Every bus connection (peer) owns a buffer which is used in order to
receive payloads. Such payloads are either messages sent from other
connections, notifications or returned answer structures in return of
query commands (name lists, etc).

In order to avoid the kernel having to maintaining an internal buffer
the connections then read from with an extra command, we decided to let
the connections own their buffer directly, so they can mmap() the memory
into their task. Allocating a local buffer to collect asynchronous
messages is what they would need to do anyway, so we implemented a
short-cut that allows the kernel to directly access the memory and write
to it. The size of this buffer pool is configured by each connection
individually, during the HELLO call, so the kernel interface is as
flexible as any other memory allocation scheme the kernel provides and
is subject to the same limits.

Internally, the connection pool is simply a shmem backed file. From the
context of the HELLO ioctl, we are calling into shmem_file_setup(), so
the file is eventually owned by the task which created the bus task
connecting to the bus. One reason why we do the shmem file allocation in
the kernel and on behalf of a the userspace task is that we clear the
VM_MAYWRITE bit to prevent the task from writing to the pool through its
mapped buffer. We also do not set VM_NORESERVE, so the entire buffer is
pre-accounted for the task that created the connection.

The pool implementation uses an r/b tree to organize the buffer into
slices. Those slices can be kept by userspace as long as the parsing
implementation needs to have access to them. When finished, the slices
are freed. A simple ring buffer cannot cope with the gaps that emerge by
that.

When a connection buffer is written to, it is done from the context of
another task which calls into the kdbus code through one of the ioctls.
The memcg implementation should hence charge the task that acts as
writer, which is maybe not ideal but can be changed easily with some
addition to the internal APIs. We omitted it for the current version,
which is non-intrusive with regards to other kernel subsystems.

The kdbus implementation is actually comparable to two tasks X and Y
which both have their own buffer file open and mmap()ed, and they both
pass their FD to the other side. If X now writes to Y's file, and that
is causing a page fault, X is accounted for it, correct?

The kernel does *not* do any memory allocation to buffer payload, and
all other allocations (for instance, to keep around the internal state
of a connection, names etc) are subject to conservatively chosen
limitations. There is no unbounded memory allocation in kdbus that I am
aware of. If there was, it would clearly be a bug.

Addressing the point Andy made earlier: yes, due to memory
overcommitment, OOM situations may happen with certain patterns, but the
kernel should have the same measures to deal with them that it already
has with other types of shared userspace memory. Right?

Hope that all makes sense, we're open to discussions around the desired
accounting details. I've copied linux-mm to let more people have a look
into this again.


Thanks,
Daniel
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.kernel

csiph-web