Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #267706 > unrolled thread

Selective rotation of journald logs

Back to article view | Back to linux.debian.user

Contents

  Selective rotation of journald logs Nicolas George <george@nsup.org> - 2024-02-23 11:20 +0100
    Re: Selective rotation of journald logs Mariusz Gronczewski <xani@devrandom.pl> - 2024-02-23 12:50 +0100
      Re: Selective rotation of journald logs Nicolas George <george@nsup.org> - 2024-02-23 13:10 +0100
        Re: Selective rotation of journald logs Mariusz Gronczewski <xani@devrandom.pl> - 2024-02-23 13:50 +0100
          Re: Selective rotation of journald logs Nicolas George <george@nsup.org> - 2024-02-23 13:50 +0100
            Re: Selective rotation of journald logs Greg Wooledge <greg@wooledge.org> - 2024-02-23 14:00 +0100
              Re: Selective rotation of journald logs Nicolas George <george@nsup.org> - 2024-02-23 14:10 +0100
                Re: Selective rotation of journald logs Greg Wooledge <greg@wooledge.org> - 2024-02-23 14:10 +0100
                  Re: Selective rotation of journald logs Nicolas George <george@nsup.org> - 2024-02-23 14:20 +0100
                    Re: Selective rotation of journald logs Mariusz Gronczewski <xani@devrandom.pl> - 2024-02-23 14:30 +0100
                Re: Selective rotation of journald logs Mariusz Gronczewski <xani@devrandom.pl> - 2024-02-23 14:20 +0100
                  Re: Selective rotation of journald logs Nicolas George <george@nsup.org> - 2024-02-23 14:30 +0100
                    Re: Selective rotation of journald logs Mariusz Gronczewski <xani@devrandom.pl> - 2024-02-23 14:40 +0100
            Re: Selective rotation of journald logs Mariusz Gronczewski <xani@devrandom.pl> - 2024-02-23 14:10 +0100
          Journald's qualities (was: Selective rotation of journald logs) Stefan Monnier <monnier@iro.umontreal.ca> - 2024-02-23 16:30 +0100
            Re: Journald's qualities (was: Selective rotation of journald logs) Mariusz Gronczewski <xani@devrandom.pl> - 2024-02-23 18:20 +0100
              Re: Journald's qualities Stefan Monnier <monnier@iro.umontreal.ca> - 2024-02-23 21:40 +0100
                Re: Journald's qualities The Wanderer <wanderer@fastmail.fm> - 2024-02-24 01:20 +0100
                  Re: Journald's qualities Jeffrey Walton <noloader@gmail.com> - 2024-02-24 05:10 +0100
                    Re: Journald's qualities Mariusz Gronczewski <xani@devrandom.pl> - 2024-02-24 11:00 +0100
                  Re: Journald's qualities Stefan Monnier <monnier@iro.umontreal.ca> - 2024-02-24 05:20 +0100
            Re: Journald's qualities (was: Selective rotation of journald logs) Dan Ritter <dsr@randomstring.org> - 2024-02-23 19:00 +0100
              Re: Journald's qualities Stefan Monnier <monnier@iro.umontreal.ca> - 2024-02-23 21:50 +0100
              Re: Journald's qualities (was: Selective rotation of journald logs) Nicholas Geovanis <nickgeovanis@gmail.com> - 2024-02-23 22:10 +0100
                Re: Journald's qualities (was: Selective rotation of journald logs) Mariusz Gronczewski <xani@devrandom.pl> - 2024-02-24 11:10 +0100
              Re: Journald's qualities (was: Selective rotation of journald logs) Mariusz Gronczewski <xani@devrandom.pl> - 2024-02-24 10:40 +0100
                Re: Journald's qualities Anssi Saari <anssi.saari@debian-user.mail.kapsi.fi> - 2024-02-26 12:20 +0100
                  Re: Journald's qualities Mariusz Gronczewski <xani@devrandom.pl> - 2024-02-26 12:40 +0100
                    Re: Journald's qualities Jeffrey Walton <noloader@gmail.com> - 2024-02-26 17:10 +0100
                    Re: Journald's qualities Anssi Saari <anssi.saari@debian-user.mail.kapsi.fi> - 2024-02-27 10:50 +0100
    Re: Selective rotation of journald logs Max Nikulin <manikulin@gmail.com> - 2024-02-23 16:40 +0100
      Re: Selective rotation of journald logs Max Nikulin <manikulin@gmail.com> - 2024-02-25 04:00 +0100
      Re: Selective rotation of journald logs Nicolas George <george@nsup.org> - 2024-02-26 15:50 +0100

Page 2 of 2 — ← Prev page 1 [2]

#267763 — Re: Journald's qualities

>>>> but what are the advantages of journald's representation compared
>>>> to a naive one?
>>> 
>>> in short: querability without text parsing. That's about it.
>> 
>> They have to parse the binary format, so that's not in and of itself
>> an upside compared to parsing CSV.
>> 
>> I've made my share of bad design decisions that don't pan out. But
>> there's always an upside to my decision (even when it turns out it 
>> speeds up only those cases which can never occur, because of some
>> other aspect of the system).
>> 
>> AFAICT the format is *not* just a plain sequence of log entries, so 
>> there's some additional structure which is intended to speed up some
>> operations.
>> 
>> IOW, even if contrived, there should be *some* use case where it
>> does better than CSV, no?
>
> I can think of two possibilities, just offhand, in no particular order:
>
> * No need to parse the timestamps, et cetera, and take the risk that
> someone put in one that's in a format you don't expect; the times are
> stored internally in a consistent guaranteed format, so you can just use
> internal reader functions (paired with, and updated alongside, the
> internal writer functions) and be done with it.

Can't think of any reason why the same wouldn't apply to CSV: if someone
messes up the timestamps by hand, they're on their own.

> * No need to worry about handling log entries that *contain* commas, or
> whatever other element was chosen as the separator.

That's just a very minor convenience issue and it does not require
a structure any more complex than a plain sequence of log entries.

Same for FSS, it doesn't seem to require the more complex structure used
by journald.  There must have been some other use-case they had in mind
where they thought they could avoid the linear-time scan or something in
a way that they expected would be algorithmically beneficial.
I just can't see what it is they had in mind.


        Stefan

[toc] | [prev] | [next] | [standalone]

#267741 — Re: Journald's qualities (was: Selective rotation of journald logs)

Stefan Monnier wrote: 
> Makes one wonder why they don't use naive append-only "plain text" logs
> (tho with appropriate delimiters (maybe some kind of CSV) to make
> searches more reliable than with old-style plain text logs)?
> 
> What are the advantages of journald's representation?
> I mean, to justify the slow search and large disk space usage, there is
> presumably some upside for some use cases.  I can see some weak argument
> against Sqlite based on the size of Sqlite, but what are the advantages
> of journald's representation compared to a naive one?


systemd's design philosophy, observed from the outside, goes
like this:

- assume that the machine is a laptop or mobile device that is
always changing: moves from network to network, plugs and
unplugs devices, goes to sleep and is woken up

- disk space is limited, but cpu time is free

- the network knows better than local config

- all services serve the local user's desktop, primarily

- the local user doesn't know anything but can call on a remote
  sysadmin

- systemd is the best at doing anything, so it should do
everything.

If all of these assumptions match up with your particular use
case, systemd is happy.


Long-term logs are for servers, so systemd doesn't want them.
systemd thinks logs are for finding out what just happened
recently. If you wanted long-term logs, obviously you would
configure a central repository on some other machine and ship
them across the network.


I have nothing but praise for the Debian maintainers of rsyslog,
who have arranged it so that installing rsyslog immediately does
appropriate things to pull data out of systemd.

-dsr-

[toc] | [prev] | [next] | [standalone]

#267755 — Re: Journald's qualities

> systemd's design philosophy, observed from the outside, goes
> like this:

Let's try and stick to the subject of the log representation in
`journald`, because we all know about the varied opinions about SystemD.

Being an "old-hand", I'm not in love of SystemD, but the thing does have
its upsides.  In the case of the format of `journald` logs, I don't
see them.


        Stefan

[toc] | [prev] | [next] | [standalone]

#267756 — Re: Journald's qualities (was: Selective rotation of journald logs)

[Multipart message — attachments visible in raw view] — view raw

On Fri, Feb 23, 2024, 2:57 PM Dan Ritter <dsr@randomstring.org> wrote:

> Stefan Monnier wrote:
> > Makes one wonder why they don't use naive append-only "plain text" logs
> > (tho with appropriate delimiters (maybe some kind of CSV) to make
> > searches more reliable than with old-style plain text logs)?
> >
> > What are the advantages of journald's representation?
> > I mean, to justify the slow search and large disk space usage, there is
> > presumably some upside for some use cases.  I can see some weak argument
> > against Sqlite based on the size of Sqlite, but what are the advantages
> > of journald's representation compared to a naive one?
>
>
> systemd's design philosophy, observed from the outside, goes
> like this:
>

....bunch trimmed.....

Exactly correct in my view. Systemd's use-case is the desktop, not the
server in the datacenter. They will be using log-aggregation software in
the datacenter anyway so no use for systemd logging. We don't install
desktop software on servers either, no X Windows, no gnome, etc. Network
connections are stable, no roaming :-)

Long-term logs are for servers, so systemd doesn't want them.
> systemd thinks logs are for finding out what just happened
> recently. If you wanted long-term logs, obviously you would
> configure a central repository on some other machine and ship
> them across the network.
>
>
> I have nothing but praise for the Debian maintainers of rsyslog,
> who have arranged it so that installing rsyslog immediately does
> appropriate things to pull data out of systemd.
>
> -dsr-
>
>

[toc] | [prev] | [next] | [standalone]

#267767 — Re: Journald's qualities (was: Selective rotation of journald logs)

Dnia 2024-02-23, o godz. 15:05:52
Nicholas Geovanis <nickgeovanis@gmail.com> napisał(a):

> On Fri, Feb 23, 2024, 2:57 PM Dan Ritter <dsr@randomstring.org> wrote:
> 
> > Stefan Monnier wrote:  
> > > Makes one wonder why they don't use naive append-only "plain
> > > text" logs (tho with appropriate delimiters (maybe some kind of
> > > CSV) to make searches more reliable than with old-style plain
> > > text logs)?
> > >
> > > What are the advantages of journald's representation?
> > > I mean, to justify the slow search and large disk space usage,
> > > there is presumably some upside for some use cases.  I can see
> > > some weak argument against Sqlite based on the size of Sqlite,
> > > but what are the advantages of journald's representation compared
> > > to a naive one?  
> >
> >
> > systemd's design philosophy, observed from the outside, goes
> > like this:
> >  
> 
> ....bunch trimmed.....
> 
> Exactly correct in my view. Systemd's use-case is the desktop, not the
> server in the datacenter. They will be using log-aggregation software
> in the datacenter anyway so no use for systemd logging. We don't
> install desktop software on servers either, no X Windows, no gnome,
> etc. Network connections are stable, no roaming :-)
> 
> Long-term logs are for servers, so systemd doesn't want them.

Right but it would be nice if it could at least forward them upstream
then! Your choices are

* use rsyslog, which does everything better than journald, including
  writing to many different databases directly, just to forward to
  remote host.
* setup super special listener for super special journald remote log
  sending method that nothing but journald supports

If it just supported standard, common protocols like RFC5424 flavour
(that adds structured data to entries), or hell, just "send a JSONed
log to that address", I could just have a bunch of my "small" devices
run journald-only and send it to central logging server. But no dice,
even my router needs to run extra daemon just to forward logs.

Actually, having pluggable "backends" would solve a lot of that, dumb
little IoT device could just have "a text file" or remote output
implemented as write method and "grep the text file" as query method.
Or DBI plugin that could write to SQLite and query it back. Or just
remote log plugin that returned "sorry, your logs are in another
castle" on query. But that's just building rsyslog but worse at that
point...

I wish I disliked C less than I miss those features...
-- 
Mariusz Gronczewski (XANi) <xani666@gmail.com>
GnuPG: 0xEA8ACE64
https://devrandom.eu

[toc] | [prev] | [next] | [standalone]

#267765 — Re: Journald's qualities (was: Selective rotation of journald logs)

Dnia 2024-02-23, o godz. 12:34:34
Dan Ritter <dsr@randomstring.org> napisał(a):

> Stefan Monnier wrote: 
> > Makes one wonder why they don't use naive append-only "plain text"
> > logs (tho with appropriate delimiters (maybe some kind of CSV) to
> > make searches more reliable than with old-style plain text logs)?
> > 
> > What are the advantages of journald's representation?
> > I mean, to justify the slow search and large disk space usage,
> > there is presumably some upside for some use cases.  I can see some
> > weak argument against Sqlite based on the size of Sqlite, but what
> > are the advantages of journald's representation compared to a naive
> > one?  
> 
> 
> systemd's design philosophy, observed from the outside, goes
> like this:
> 
> - assume that the machine is a laptop or mobile device that is
> always changing: moves from network to network, plugs and
> unplugs devices, goes to sleep and is woken up

I had thought similar thoughts many times when using it. But some of
the stuff that derived from that is pretty useful; for example we had a
service tht downloaded encryption keys from central server to unlock
the hard drive and sysV version was a bit of a mess (if service didn't
ran on boot there was no unlocked volume to mount), while in systemd
the dependencies allow it to automatically bring it up when mount is
requested, and the mount will also be automatically done if service
using it is started (if it has correct RequiresMountsFor entries). So
some actually useful stuff for more "constant config" machines had
flown from that

> - disk space is limited, but cpu time is free

In which way ? It doesn't care about wasting space, but it does assume
you have blazing fast storage or else journal related ops are sluggish
(and waste disk cache as it just reads hundreds of MBs to find last few
lines of logfile)

> - the network knows better than local config

Offtopic but since Debian switched to systemd for DNS management on
VPNs and suc I need to restart it sometimes multiple times to just get
"right" DNS servers, because there appears to be no notion of priority:

https://github.com/systemd/systemd/issues/27543

so now any time I connect to work (just openvpn tunnel, nothing fancy)
I need to spam

systemclt restart systemd-resolved ; sleep 1 ; cat /etc/resolv.conf

few times till the dice rolls the right order of DNS servers...

XANi



-- 
Mariusz Gronczewski (XANi) <xani666@gmail.com>
GnuPG: 0xEA8ACE64
https://devrandom.eu

[toc] | [prev] | [next] | [standalone]

#267838 — Re: Journald's qualities

Mariusz Gronczewski <xani@devrandom.pl> writes:

> Offtopic but since Debian switched to systemd for DNS management on
> VPNs and suc I need to restart it sometimes multiple times to just get
> "right" DNS servers, because there appears to be no notion of priority:
>
> https://github.com/systemd/systemd/issues/27543
>
> so now any time I connect to work (just openvpn tunnel, nothing fancy)
> I need to spam
>
> systemclt restart systemd-resolved ; sleep 1 ; cat /etc/resolv.conf
>
> few times till the dice rolls the right order of DNS servers...

Interesting. I leaped on systemd-networkd and -resolved when I read
years ago it added interface specific DNS support. So now my local DNS
(dnsmasq in the router) handles my home network and what goes out via
the VPN (i.e. tun0 or wg0 these days) uses the VPN's DNS. Or if the VPN
is off, the local DNS forwards queries to DHCP assigned DNS. I see no
issues although I don't have the kind of VPN where some external traffic
goes through it only but might work for that too. For me the default was
that systemd-resolved dutifully spammed all DNS queries to all DNS
servers through all interfaces.

This interface specific DNS was a little hard to setup as I
recall. Easier with WG than OpenVPN.

[toc] | [prev] | [next] | [standalone]

#267840 — Re: Journald's qualities

Dnia 2024-02-26, o godz. 13:10:43
Anssi Saari <anssi.saari@debian-user.mail.kapsi.fi> napisał(a):

> Mariusz Gronczewski <xani@devrandom.pl> writes:
> 
> > Offtopic but since Debian switched to systemd for DNS management on
> > VPNs and suc I need to restart it sometimes multiple times to just
> > get "right" DNS servers, because there appears to be no notion of
> > priority:
> >
> > https://github.com/systemd/systemd/issues/27543
> >
> > so now any time I connect to work (just openvpn tunnel, nothing
> > fancy) I need to spam
> >
> > systemclt restart systemd-resolved ; sleep 1 ; cat /etc/resolv.conf
> >
> > few times till the dice rolls the right order of DNS servers...  
> 
> Interesting. I leaped on systemd-networkd and -resolved when I read
> years ago it added interface specific DNS support. So now my local DNS
> (dnsmasq in the router) handles my home network and what goes out via
> the VPN (i.e. tun0 or wg0 these days) uses the VPN's DNS. 

... in what way? You need to resolve DNS first before you know which
interface the traffic is going out of.

> Or if the
> VPN is off, the local DNS forwards queries to DHCP assigned DNS. I
> see no issues although I don't have the kind of VPN where some
> external traffic goes through it only but might work for that too.
> For me the default was that systemd-resolved dutifully spammed all
> DNS queries to all DNS servers through all interfaces.
> 
> This interface specific DNS was a little hard to setup as I
> recall. Easier with WG than OpenVPN.
> 

Our case is basically that:

* some of the records exist only on VPN DNS server (private domains
  pointing to private IPs)
* some of the records exist on outside but the VPN DNS returns private
  range IP addresses for it (so-called split-horizon DNS). 

So the only right way is to ask the first server on the list. That
worked before systemd-resolved came as Debian scripts just put the
VPN's DNS servers in the front. Now it is throw of the dice any time
the daemon is restarted.

The proper way would be either to:

* ask in order, with components registering the DNS server specifying
  that priority so the daemon can result the sorted list
* have a way to do per-domain exception and do "if domain is
  *.internal.example.com, ask VPN server's DNS"

The second is possible in dnsmasq but not (AFAIK) in systemd. And
currently neither "make systemd a DNS resolver" nor "use
systemd-resolved provided DNS config" work reliably.

-- 
Mariusz Gronczewski (XANi) <xani666@gmail.com>
GnuPG: 0xEA8ACE64
https://devrandom.eu

[toc] | [prev] | [next] | [standalone]

#267849 — Re: Journald's qualities

On Mon, Feb 26, 2024 at 10:42 AM Mariusz Gronczewski <xani@devrandom.pl> wrote:
>
> Dnia 2024-02-26, o godz. 13:10:43
> Anssi Saari <anssi.saari@debian-user.mail.kapsi.fi> napisał(a):
>
> > Mariusz Gronczewski <xani@devrandom.pl> writes:
> >
> > > Offtopic but since Debian switched to systemd for DNS management on
> > > VPNs and suc I need to restart it sometimes multiple times to just
> > > get "right" DNS servers, because there appears to be no notion of
> > > priority:
> > >
> > > https://github.com/systemd/systemd/issues/27543
> > >
> > > so now any time I connect to work (just openvpn tunnel, nothing
> > > fancy) I need to spam
> > >
> > > systemclt restart systemd-resolved ; sleep 1 ; cat /etc/resolv.conf
> > >
> > > few times till the dice rolls the right order of DNS servers...
> >
> > Interesting. I leaped on systemd-networkd and -resolved when I read
> > years ago it added interface specific DNS support. So now my local DNS
> > (dnsmasq in the router) handles my home network and what goes out via
> > the VPN (i.e. tun0 or wg0 these days) uses the VPN's DNS.
>
> ... in what way? You need to resolve DNS first before you know which
> interface the traffic is going out of.

I _think_ that depends on the configuration. You can use local DNS for
name resolution, or remote (VPN) DNS for name resolution.

Sometimes both are used at the same time. I think that's called "split
DNS" or "split brain DNS."

> > Or if the
> > VPN is off, the local DNS forwards queries to DHCP assigned DNS. I
> > see no issues although I don't have the kind of VPN where some
> > external traffic goes through it only but might work for that too.
> > For me the default was that systemd-resolved dutifully spammed all
> > DNS queries to all DNS servers through all interfaces.
> >
> > This interface specific DNS was a little hard to setup as I
> > recall. Easier with WG than OpenVPN.
> >
>
> Our case is basically that:
>
> * some of the records exist only on VPN DNS server (private domains
>   pointing to private IPs)
> * some of the records exist on outside but the VPN DNS returns private
>   range IP addresses for it (so-called split-horizon DNS).
>
> So the only right way is to ask the first server on the list. That
> worked before systemd-resolved came as Debian scripts just put the
> VPN's DNS servers in the front. Now it is throw of the dice any time
> the daemon is restarted.
>
> The proper way would be either to:
>
> * ask in order, with components registering the DNS server specifying
>   that priority so the daemon can result the sorted list
> * have a way to do per-domain exception and do "if domain is
>   *.internal.example.com, ask VPN server's DNS"
>
> The second is possible in dnsmasq but not (AFAIK) in systemd. And
> currently neither "make systemd a DNS resolver" nor "use
> systemd-resolved provided DNS config" work reliably.

Jeff

[toc] | [prev] | [next] | [standalone]

#267883 — Re: Journald's qualities

Mariusz Gronczewski <xani@devrandom.pl> writes:

> ... in what way? You need to resolve DNS first before you know which
> interface the traffic is going out of.

You can specify a domain via resolvectl domain for an interface. In my
simple case I specify just ~. as the domain for my VPN and for my local
domain it's lan. So everything not going to .lan goes to VPN DNS.

[toc] | [prev] | [next] | [standalone]

#267736

On 23/02/2024 17:15, Nicolas George wrote:
> 
> How do I tell systemd's logging system to keep authentication logs for
> one year and mail logs for one month?

I am realizing that the following is not an answer to the asked 
question. The thread is no more than useless arguing anyway.

Some ideas that might be useful in close cases:
- a bunch of filtering options and --output=export as a part of log 
rotation to have selective copy of the journal in an alternative directory
- systemd-journald.service(8) § JOURNAL NAMESPACES for some services.


P.S.
When I read the question I decided that syslog for auth facility is a 
kind of solution.

[toc] | [prev] | [next] | [standalone]

#267809

On 24/02/2024 21:09, Byunghee HWANG wrote:
>> When I read the question I decided that syslog for auth facility is a
>> kind of solution.
> 
> Really i would like to learn more. Can you show a more specific example?
> Also i'm using Debian (sid).

Install rsyslog and logrotate and read their configuration files. 
/var/log/auth is configured by default. Journal contains more metadata 
than syslog text files however.

[toc] | [prev] | [next] | [standalone]

#267845

Max Nikulin (12024-02-23):
> I am realizing that the following is not an answer to the asked question.
> The thread is no more than useless arguing anyway.
> 
> Some ideas that might be useful in close cases:
> - a bunch of filtering options and --output=export as a part of log rotation
> to have selective copy of the journal in an alternative directory

Hi.

You know what? I think it is exactly the right answer.

I think the principle for this thing is: journald maintains the current
logs in a best-effort manner, but when it comes to archival, the needs
and manners are too varied, it does not try to provide a ready-to-use
solution, but journalctl provides all the tools necessary to implement
what we want.

If I am right, if it is indeed the way it is meant to be used, I just
wish somebody would have bothered writing it in so many words at the
start of a documentation.

Thanks.

-- 
  Nicolas George

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.debian.user

csiph-web