Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1200321 > unrolled thread

Re: kdbus: to merge or not to merge?

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: kdbus: to merge or not to merge? Andy Lutomirski <luto@amacapital.net> - 2015-08-05 02:20 +0200
    Re: kdbus: to merge or not to merge? Daniel Mack <daniel@zonque.org> - 2015-08-06 09:10 +0200
      Re: kdbus: to merge or not to merge? Andy Lutomirski <luto@amacapital.net> - 2015-08-06 17:30 +0200
        Re: kdbus: to merge or not to merge? Daniel Mack <daniel@zonque.org> - 2015-08-06 19:30 +0200

#1200321 — Re: kdbus: to merge or not to merge?

On Tue, Aug 4, 2015 at 7:47 AM, Andy Lutomirski <luto@amacapital.net> wrote:
> On Tue, Aug 4, 2015 at 7:09 AM, David Herrmann <dh.herrmann@gmail.com> wrote:
>> Hi
>>
>> On Tue, Aug 4, 2015 at 3:46 PM, Linus Torvalds
>> <torvalds@linux-foundation.org> wrote:
>>> On Tue, Aug 4, 2015 at 1:58 AM, David Herrmann <dh.herrmann@gmail.com> wrote:
>>>>
>>>> You lack a call to sd_bus_unref() here.
>>>
>>> I assume it was intentional. Why would Andy talk about "scaling" otherwise?
>
> It was actually an error.  I assumed that, since the user version
> worked fine (at least for as long as I ran it) and the kernel version
> didn't (killed X and left a blinking cursor, no visible log messages
> even when run from a text console, and no obvious OOM recovery after a
> long wait) that it was a kdbus issue or issue with other kdbus
> clients.
>
> I'll play with it more today.
>

I added the missing sd_bus_unref call.

With userspace dbus, my program takes 95% CPU and dbus-daemon takes
88% CPU or so.

With kdbus, I see abuse-bus (my test), systemd-journald,
systemd-bus-proxy, auditd, gnome-shell, mission-control, sedispatch,
firewalld, polkitd, NetworkManager, systemd, avahi-daemon, audisp,
abrt-dump-jour* (whatever it's called -- it truncated), upowerd, and
systemd-logind all taking tons of CPU.  I've listed them in decreasing
order of amount of CPU burned -- the top several are taking about as
much as is possible.  Load average is over 13.  That's if I run it
from a text console while I'm logged in to gnome in a different VT.

If I run the program from a graphical terminal, everything freezes so
hard that the cursor doesn't even make it to the next line when I hit
enter.

So I still claim that kdbus doesn't scale.  I'm not even just saying
that it doesn't scale to large systems -- somewhat to my surprise, it
doesn't even seem to scale well enough for a mostly empty Rawhide
workstation system running just a graphical terminal.  And I didn't
even try to find stress tests more interesting than connecting and
disconnecting in a loop.

FWIW, the old test (without the unref) appeared to be allocating 16M
of mapped kdbus pool every iteration, which seems unlikely to have
helped matters.

--Andy
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1201503

Hi Andy,

On 08/05/2015 02:18 AM, Andy Lutomirski wrote:
> I added the missing sd_bus_unref call.
> 
> With userspace dbus, my program takes 95% CPU and dbus-daemon takes
> 88% CPU or so.
> 
> With kdbus, I see abuse-bus (my test), systemd-journald,
> systemd-bus-proxy, auditd, gnome-shell, mission-control, sedispatch,
> firewalld, polkitd, NetworkManager, systemd, avahi-daemon, audisp,
> abrt-dump-jour* (whatever it's called -- it truncated), upowerd, and
> systemd-logind all taking tons of CPU.  I've listed them in decreasing
> order of amount of CPU burned -- the top several are taking about as
> much as is possible.  Load average is over 13.  That's if I run it
> from a text console while I'm logged in to gnome in a different VT.

That's right, I can reproduce this here. To explain what's going on, let
me provide some background.

Every time a client connects to kdbus, a new ID is assigned to the
connection, and other connections which have previously subscribed to
notifications of type KDBUS_ITEM_ID_ADD or _REMOVE get a notification
and are woken up so they can dispatch it. By default, no such matches
exists, applicaions have to explicitly opt-in if they are interested in
these events.

In DBus (both kdbus and DBus1), such matches are installed on the
NameOwnerChanged signal, and they can be either specific to a single ID,
or broad, which will make them match on any ID. There's actually no
reason for applications to install unspecific matches, but if they do,
they will of course get what they asked for, and are woken up on every
ID that is added to or removed from the bus. What you're seeing in your
system profile is that some applications misbehave and install
unspecific matches when they shouldn't. That's a userspace bug that
needs fixing. Two candidates were actually in the systemd code base
(logind and PID1), and both are now patched.

Note that these applications are actually affected on both DBus1 and
kdbus. The reason you didn't see them trip up in your test is that
sd_bus_open() behaves differently in the two worlds. In kdbus, it will
immediately call into the kernel and register a new connection, hence
triggering the behavior described above. On DBus1, however, the HELLO
message will not be transmitted to the daemon until the first message is
sent, so no ID is assigned, and no notifications are sent. When
augmenting the test program a little so it reads its own ID on the bus,
you'll see similar behavior on DBus1 as well, but the bottleneck in this
case is the daemon, which significantly mitigates the load caused by
other tasks.

So, to wrap it up: you've triggered an existing userspace bug. The
userspace components under our control have now been fixed, and we'll
talk to other people to make them aware of the issue too. However, these
issues are not directly related to kdbus, but rather show more impact as
a side-effect now.

You've raised a valid point here. Thanks a lot for providing this test,
much appreciated!


Daniel

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1201844

On Thu, Aug 6, 2015 at 12:06 AM, Daniel Mack <daniel@zonque.org> wrote:
> Hi Andy,
>
> On 08/05/2015 02:18 AM, Andy Lutomirski wrote:
>> I added the missing sd_bus_unref call.
>>
>> With userspace dbus, my program takes 95% CPU and dbus-daemon takes
>> 88% CPU or so.
>>
>> With kdbus, I see abuse-bus (my test), systemd-journald,
>> systemd-bus-proxy, auditd, gnome-shell, mission-control, sedispatch,
>> firewalld, polkitd, NetworkManager, systemd, avahi-daemon, audisp,
>> abrt-dump-jour* (whatever it's called -- it truncated), upowerd, and
>> systemd-logind all taking tons of CPU.  I've listed them in decreasing
>> order of amount of CPU burned -- the top several are taking about as
>> much as is possible.  Load average is over 13.  That's if I run it
>> from a text console while I'm logged in to gnome in a different VT.
>
> That's right, I can reproduce this here. To explain what's going on, let
> me provide some background.
>
> Every time a client connects to kdbus, a new ID is assigned to the
> connection, and other connections which have previously subscribed to
> notifications of type KDBUS_ITEM_ID_ADD or _REMOVE get a notification
> and are woken up so they can dispatch it. By default, no such matches
> exists, applicaions have to explicitly opt-in if they are interested in
> these events.
>
> In DBus (both kdbus and DBus1), such matches are installed on the
> NameOwnerChanged signal, and they can be either specific to a single ID,
> or broad, which will make them match on any ID. There's actually no
> reason for applications to install unspecific matches, but if they do,
> they will of course get what they asked for, and are woken up on every
> ID that is added to or removed from the bus. What you're seeing in your
> system profile is that some applications misbehave and install
> unspecific matches when they shouldn't. That's a userspace bug that
> needs fixing. Two candidates were actually in the systemd code base
> (logind and PID1), and both are now patched.

Can you point me at the patch?

It sounds like that will reduce the scalability issue with this
particular test from whatever userspace overhead exists * number of
clients to just the overhead of looping over all clients and their
matches in the kernel.

--Andy
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1201924

On 08/06/2015 05:27 PM, Andy Lutomirski wrote:
>> In DBus (both kdbus and DBus1), such matches are installed on the
>> > NameOwnerChanged signal, and they can be either specific to a single ID,
>> > or broad, which will make them match on any ID. There's actually no
>> > reason for applications to install unspecific matches, but if they do,
>> > they will of course get what they asked for, and are woken up on every
>> > ID that is added to or removed from the bus. What you're seeing in your
>> > system profile is that some applications misbehave and install
>> > unspecific matches when they shouldn't. That's a userspace bug that
>> > needs fixing. Two candidates were actually in the systemd code base
>> > (logind and PID1), and both are now patched.
>
> Can you point me at the patch?

  https://github.com/systemd/systemd/pull/876
  https://github.com/systemd/systemd/pull/887

firewalld and possibly some other applications in the Fedora default
install use python-slip, a convenience library that currently
unconditionally installs the broad matches. I filed a bug with patches here:

  https://fedorahosted.org/python-slip/ticket/2


And I filed more bugs for some GNOME components.


Thanks,
Daniel

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web