Path: csiph.com!3.eu.feeder.erje.net!feeder.erje.net!news.roellig-ltd.de!open-news-network.org!weretis.net!feeder7.news.weretis.net!usenet.goja.nl.eu.org!aioe.org!bofh.it!news.nic.it!robomod
From: Reco <recoverym4n@enotuniq.net>
Newsgroups: linux.debian.user
Subject: Re: high cpu load systemd-journal, nobody logs
Date: Wed, 01 Jul 2020 11:10:05 +0200
Message-ID: <AnGUI-6eP-77@gated-at.bofh.it>
References: <AnG8i-5BK-5@gated-at.bofh.it>
Old-Return-Path: <recoverym4n@enotuniq.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.10.1 (2018-07-13)
Approved: robomod@news.nic.it
Lines: 54
Organization: linux.* mail to news gateway
Sender: robomod@news.nic.it
Xref: csiph.com linux.debian.user:224246

	Hi.

On Wed, Jul 01, 2020 at 09:48:00AM +0200, Nico Becker wrote:
> Can somebody give me some hints?

Sure. First, the poll(2) by itself is interesting, but useless if you
don't know what file descriptors 3 and 5 are referring to. So,

lsof -Pnp (pid from systemd-journal)

Or at least:

ls -al /proc/(pid from systemd-journal)/fd/3 /proc/(pid from systemd-journal)/fd/5


> i have no idea why the fork poll endless,

Second, the fork by itself, combined with accept(2) is the usual way to
process incoming connections (TCP connections in this case).
It's the amount of such connections that gives you the trouble.


> and produce high cpu load.

Third, and here's the most interesting part.
Your strace blurb shows multiple incoming connections from 192.168.2.247
(different source ports for each accept(2)). If the memory serves,
systemd-journal-remote should use HTTP for transferring the journal
records. While it's a pretty dumb design decision in general, it's
possible to use it to an advantage in a case like this, meaning:

tcpdump -pni any -c1024 -s0 -w /tmp/journal.pcap tcp port 19531

View the resulting file with Wireshark (or whatever equivalent), it
should contain the hints about the behaviour you're seeing.


The way I see it, systemd-journal-remote uses short-living HTTP to
"transfer" journal records to another system. "Transfer", because
contrary to the logic they've implemented "pull" mechanism for this, not
a "push" one.
While it gives you the ability to view journald records
(systemd-journald is a server, a browser is a client), it also mean that
your browser is forced to send numerous HTTP requests to the
systemd-journal-remote, effectively DOS'ing it.


> i cant upgrade/update the system to an newer debian.

Hardly relevant here, see above. Personally I'd say "screw it", and
would use conventional rsyslogd remote logging, but since it's a
embedded system - it may be impossible for you to follow this advice.

Reco