Re: high cpu load systemd-journal, nobody logs

Path	csiph.com!3.eu.feeder.erje.net!feeder.erje.net!news.roellig-ltd.de!open-news-network.org!weretis.net!feeder7.news.weretis.net!usenet.goja.nl.eu.org!aioe.org!bofh.it!news.nic.it!robomod
From	Reco <recoverym4n@enotuniq.net>
Newsgroups	linux.debian.user
Subject	Re: high cpu load systemd-journal, nobody logs
Date	Wed, 01 Jul 2020 11:10:05 +0200
Message-ID	<AnGUI-6eP-77@gated-at.bofh.it> (permalink)
References	<AnG8i-5BK-5@gated-at.bofh.it>
X-Original-To	debian-user@lists.debian.org
X-Mailbox-Line	From debian-user-request@lists.debian.org Wed Jul 1 09:09:09 2020
Old-Return-Path	<recoverym4n@enotuniq.net>
X-Amavis-Spam-Status	No, score=-7.1 tagged_above=-10000 required=5.3 tests=[BAYES_00=-2, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FOURLA=0.1, LDO_WHITELIST=-5] autolearn=ham autolearn_force=no
X-Policyd-Weight	using cached result; rate: -4.6
Dkim-Signature	v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=enotuniq.net; s=dkim; h=Message-ID:User-Agent:In-Reply-To:Content-Type: MIME-Version:References:Subject:To:From:Date:Sender:Cc: Content-Transfer-Encoding:Content-ID:Content-Description; bh=MiQMoGBH750MEP8d8Bim8Trzc0L8MULRKvI5LSxG2eU=; b=IC3wbhrgiATdXH4IJ5JpwOTovc hSRR9IyTQpomYvmMNxC8/XEP8FkW9gzg2PKkdPUFtbhHYr5z+047hwN21wYJxZ+WqFNwMOVY4bwnu x+N6vAknj/EvZaA8WJCvO603SrNafJncdLz7zi9TQU9cWFuA2ql5skJNp5RsEg7S3pUnX/homoPzN ZD0ZOZep1nhqFW+LqZFrrC6hsaAHedOg2xdy1mSBmgpdba7X4QxkiQu7GdtVy9yGsvxKZnrMeksrU orKTJXnIvsrNu4FAovy3Am8HJ/jPUZE190HlXTryrHwqf1fFfFdmLpYcJhbj3aNUUXzL849z9rMr2 SxOyxXKg==;
MIME-Version	1.0
Content-Type	text/plain; charset=us-ascii
Content-Disposition	inline
User-Agent	Mutt/1.10.1 (2018-07-13)
X-Mailing-List	<debian-user@lists.debian.org> archive/latest/764548
List-ID	<debian-user.lists.debian.org>
List-URL	<https://lists.debian.org/debian-user/>
List-Archive	https://lists.debian.org/msgid-search/E1jqYV5-0001QR-Mm@enotuniq.net
Approved	robomod@news.nic.it
Lines	54
Organization	linux.* mail to news gateway
Sender	robomod@news.nic.it
X-Original-Date	Wed, 1 Jul 2020 11:53:51 +0300
X-Original-Message-ID	<E1jqYV5-0001QR-Mm@enotuniq.net>
X-Original-References	<8e3b9c16b2a902ba02b16efb641115d5@gmx.de>
Xref	csiph.com linux.debian.user:224246

Show key headers only | View raw

	Hi.

On Wed, Jul 01, 2020 at 09:48:00AM +0200, Nico Becker wrote:
> Can somebody give me some hints?

Sure. First, the poll(2) by itself is interesting, but useless if you
don't know what file descriptors 3 and 5 are referring to. So,

lsof -Pnp (pid from systemd-journal)

Or at least:

ls -al /proc/(pid from systemd-journal)/fd/3 /proc/(pid from systemd-journal)/fd/5

> i have no idea why the fork poll endless,

Second, the fork by itself, combined with accept(2) is the usual way to
process incoming connections (TCP connections in this case).
It's the amount of such connections that gives you the trouble.

> and produce high cpu load.

Third, and here's the most interesting part.
Your strace blurb shows multiple incoming connections from 192.168.2.247
(different source ports for each accept(2)). If the memory serves,
systemd-journal-remote should use HTTP for transferring the journal
records. While it's a pretty dumb design decision in general, it's
possible to use it to an advantage in a case like this, meaning:

tcpdump -pni any -c1024 -s0 -w /tmp/journal.pcap tcp port 19531

View the resulting file with Wireshark (or whatever equivalent), it
should contain the hints about the behaviour you're seeing.

The way I see it, systemd-journal-remote uses short-living HTTP to
"transfer" journal records to another system. "Transfer", because
contrary to the logic they've implemented "pull" mechanism for this, not
a "push" one.
While it gives you the ability to view journald records
(systemd-journald is a server, a browser is a client), it also mean that
your browser is forced to send numerous HTTP requests to the
systemd-journal-remote, effectively DOS'ing it.

> i cant upgrade/update the system to an newer debian.

Hardly relevant here, see above. Personally I'd say "screw it", and
would use conventional rsyslogd remote logging, but since it's a
embedded system - it may be impossible for you to follow this advice.

Reco

Back to linux.debian.user | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

high cpu load systemd-journal, nobody logs Nico Becker <nico.becker@gmx.de> - 2020-07-01 10:20 +0200
  Re: high cpu load systemd-journal, nobody logs Reco <recoverym4n@enotuniq.net> - 2020-07-01 11:10 +0200

csiph-web