Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #267706 > unrolled thread

Selective rotation of journald logs

Back to article view | Back to linux.debian.user

Contents

  Selective rotation of journald logs Nicolas George <george@nsup.org> - 2024-02-23 11:20 +0100
    Re: Selective rotation of journald logs Mariusz Gronczewski <xani@devrandom.pl> - 2024-02-23 12:50 +0100
      Re: Selective rotation of journald logs Nicolas George <george@nsup.org> - 2024-02-23 13:10 +0100
        Re: Selective rotation of journald logs Mariusz Gronczewski <xani@devrandom.pl> - 2024-02-23 13:50 +0100
          Re: Selective rotation of journald logs Nicolas George <george@nsup.org> - 2024-02-23 13:50 +0100
            Re: Selective rotation of journald logs Greg Wooledge <greg@wooledge.org> - 2024-02-23 14:00 +0100
              Re: Selective rotation of journald logs Nicolas George <george@nsup.org> - 2024-02-23 14:10 +0100
                Re: Selective rotation of journald logs Greg Wooledge <greg@wooledge.org> - 2024-02-23 14:10 +0100
                  Re: Selective rotation of journald logs Nicolas George <george@nsup.org> - 2024-02-23 14:20 +0100
                    Re: Selective rotation of journald logs Mariusz Gronczewski <xani@devrandom.pl> - 2024-02-23 14:30 +0100
                Re: Selective rotation of journald logs Mariusz Gronczewski <xani@devrandom.pl> - 2024-02-23 14:20 +0100
                  Re: Selective rotation of journald logs Nicolas George <george@nsup.org> - 2024-02-23 14:30 +0100
                    Re: Selective rotation of journald logs Mariusz Gronczewski <xani@devrandom.pl> - 2024-02-23 14:40 +0100
            Re: Selective rotation of journald logs Mariusz Gronczewski <xani@devrandom.pl> - 2024-02-23 14:10 +0100
          Journald's qualities (was: Selective rotation of journald logs) Stefan Monnier <monnier@iro.umontreal.ca> - 2024-02-23 16:30 +0100
            Re: Journald's qualities (was: Selective rotation of journald logs) Mariusz Gronczewski <xani@devrandom.pl> - 2024-02-23 18:20 +0100
              Re: Journald's qualities Stefan Monnier <monnier@iro.umontreal.ca> - 2024-02-23 21:40 +0100
                Re: Journald's qualities The Wanderer <wanderer@fastmail.fm> - 2024-02-24 01:20 +0100
                  Re: Journald's qualities Jeffrey Walton <noloader@gmail.com> - 2024-02-24 05:10 +0100
                    Re: Journald's qualities Mariusz Gronczewski <xani@devrandom.pl> - 2024-02-24 11:00 +0100
                  Re: Journald's qualities Stefan Monnier <monnier@iro.umontreal.ca> - 2024-02-24 05:20 +0100
            Re: Journald's qualities (was: Selective rotation of journald logs) Dan Ritter <dsr@randomstring.org> - 2024-02-23 19:00 +0100
              Re: Journald's qualities Stefan Monnier <monnier@iro.umontreal.ca> - 2024-02-23 21:50 +0100
              Re: Journald's qualities (was: Selective rotation of journald logs) Nicholas Geovanis <nickgeovanis@gmail.com> - 2024-02-23 22:10 +0100
                Re: Journald's qualities (was: Selective rotation of journald logs) Mariusz Gronczewski <xani@devrandom.pl> - 2024-02-24 11:10 +0100
              Re: Journald's qualities (was: Selective rotation of journald logs) Mariusz Gronczewski <xani@devrandom.pl> - 2024-02-24 10:40 +0100
                Re: Journald's qualities Anssi Saari <anssi.saari@debian-user.mail.kapsi.fi> - 2024-02-26 12:20 +0100
                  Re: Journald's qualities Mariusz Gronczewski <xani@devrandom.pl> - 2024-02-26 12:40 +0100
                    Re: Journald's qualities Jeffrey Walton <noloader@gmail.com> - 2024-02-26 17:10 +0100
                    Re: Journald's qualities Anssi Saari <anssi.saari@debian-user.mail.kapsi.fi> - 2024-02-27 10:50 +0100
    Re: Selective rotation of journald logs Max Nikulin <manikulin@gmail.com> - 2024-02-23 16:40 +0100
      Re: Selective rotation of journald logs Max Nikulin <manikulin@gmail.com> - 2024-02-25 04:00 +0100
      Re: Selective rotation of journald logs Nicolas George <george@nsup.org> - 2024-02-26 15:50 +0100

Page 1 of 2  [1] 2  Next page →

#267706 — Selective rotation of journald logs

Hi.

It might be an obvious question, but I do not manage to find the obvious
answer:

How do I tell systemd's logging system to keep authentication logs for
one year and mail logs for one month?

Regards,

-- 
  Nicolas George

[toc] | [next] | [standalone]

#267713

Dnia 2024-02-23, o godz. 11:15:29
Nicolas George <george@nsup.org> napisał(a):

> Hi.
> 
> It might be an obvious question, but I do not manage to find the
> obvious answer:
> 
> How do I tell systemd's logging system to keep authentication logs for
> one year and mail logs for one month?
> 
> Regards,
> 

That is not a feature systemd's logging have. You'd have to make a
rsyslogd rule to put it in one directory, another one for the other use
then tweak logrotate rules to rotate and keep each of them for
different length.

Most packages already log to separate files and have separate logrotate
rules so often it is just changing a single line, for example auth.log
is rotated with rest of the default logs:

/var/log/syslog
/var/log/mail.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/cron.log
{
	rotate 4
	weekly
	missingok
	notifempty
	compress
	delaycompress
	sharedscripts
	postrotate
		/usr/lib/rsyslog/rsyslog-rotate
	endscript
}

so if you wanted to change only auth.log you'd copy the section, and remove it from current one, like this:


/var/log/auth.log
{
	rotate 53 # a year in weeks
	weekly
	missingok
	notifempty
	compress
	delaycompress
	sharedscripts
	postrotate
		/usr/lib/rsyslog/rsyslog-rotate
	endscript
}


-- 
Mariusz Gronczewski (XANi) <xani666@gmail.com>
GnuPG: 0xEA8ACE64
https://devrandom.eu

[toc] | [prev] | [next] | [standalone]

#267715

Mariusz Gronczewski (12024-02-23):
> That is not a feature systemd's logging have.

That is what it seems, but I would like second opinions.

>						You'd have to make a
> rsyslogd rule to put it in one directory

Thanks, but my question was about systemd's infrastructure. Answers
about the old systlog/logrotate infrastructure are a waste of time since
I already know how they work and they are amply documented elsewhere.

-- 
  Nicolas George

[toc] | [prev] | [next] | [standalone]

#267719

Dnia 2024-02-23, o godz. 13:02:00
Nicolas George <george@nsup.org> napisał(a):

> Mariusz Gronczewski (12024-02-23):
> > That is not a feature systemd's logging have.  
> 
> That is what it seems, but I would like second opinions.
> 
> >						You'd have to make a
> > rsyslogd rule to put it in one directory  
> 
> Thanks, but my question was about systemd's infrastructure. Answers
> about the old systlog/logrotate infrastructure are a waste of time
> since I already know how they work and they are amply documented
> elsewhere.
> 

So to say it short: It is horrid. There is no any sensible indexing or
split between services or anything. Single spamming service can easily
make it rotate out more important but much more rare log entries and
there is nothing you can do about it.

It is also woefully inefficient, with each "line" of log taking few
times more than it does in text or even if thrown into SQLite database.

For example:

[13:37:01]cthulhu:/var/log/journal☠ ls -s -h /tmp/log.sqlite
79M /tmp/log.sqlite
[13:37:12]cthulhu:/var/log/journal☠ sqlite /tmp/log.sqlite 'select count(*) from log'
386351
journalctl |wc -l
381770
[13:37:48]cthulhu:/var/log/journal☠ journalctl |dd of=/dev/zero bs=1M
0+15442 records in
0+15442 records out
63643115 bytes (64 MB, 61 MiB) copied, 5,47791 s, 11,6 MB/s
du -h /var/log/journal/
337M	/var/log/journal/44cf6f547971fc33309d1e9e000002e7
337M	/var/log/journal/

(I've raised a bug 8 years ago about it https://github.com/systemd/systemd/issues/2460 )

To summarize: the ~61MB of resulting text of logs takes around 337MB 
on disk when saved by journald, ~80MB in sqlite (~103MB when I added 
index for time and process). Use the old ways you know, it's a waste 
of time to try to wrangle journald to do what you want.


-- 
Mariusz Gronczewski (XANi) <xani666@gmail.com>
GnuPG: 0xEA8ACE64
https://devrandom.eu

[toc] | [prev] | [next] | [standalone]

#267720

Mariusz Gronczewski (12024-02-23):
> So to say it short: It is horrid.

Generic bashing of systemd in favor of a blind cult of the good old ways
are not what I am looking for either, and the unbalanced tone of your
reply makes it look like precisely that.

-- 
  Nicolas George

[toc] | [prev] | [next] | [standalone]

#267721

On Fri, Feb 23, 2024 at 01:48:35PM +0100, Nicolas George wrote:
> Mariusz Gronczewski (12024-02-23):
> > So to say it short: It is horrid.
> 
> Generic bashing of systemd in favor of a blind cult of the good old ways
> are not what I am looking for either, and the unbalanced tone of your
> reply makes it look like precisely that.

What was "blind" about his anaylsis?  It looked pretty well thought out
to me.  He showed actual examples of how space-inefficient it is, and
provided a theoretical example of how one misbehaved service could
flush out the important logs of well-behaved services.

If anything, the person who's blindly following a path is *you*.  You're
looking to do something that multiple people have said is not possible,
and when they offer you an alternative, your claws come out.

[toc] | [prev] | [next] | [standalone]

#267723

Greg Wooledge (12024-02-23):
> What was "blind" about his anaylsis?  It looked pretty well thought out
> to me.  He showed actual examples of how space-inefficient it is, and
> provided a theoretical example of how one misbehaved service could
> flush out the important logs of well-behaved services.

The selective blindness here is to look only at the bad things. The
drawbacks mentioned exist, but they are in part there for reasons, in
order to fix the issues of the previous solutions.

An analysis that mentions only the bad things and conclude to recommend
using the previous solutions without even discussing their own drawbacks
is not worth our time.

> If anything, the person who's blindly following a path is *you*.  You're
> looking to do something that multiple people have said is not possible,

Multiple?

> and when they offer you an alternative, your claws come out.

When somebody spends one line answering the question and then pages
“offering an alternative” by explaining things a baby sysadmin would
already know, I deduce they are not much above the level of baby
sysadmin themselves, and it cancels any trust I could have put in the
one-line answer.

Regards,

-- 
  Nicolas George

[toc] | [prev] | [next] | [standalone]

#267725

On Fri, Feb 23, 2024 at 02:03:50PM +0100, Nicolas George wrote:
> When somebody spends one line answering the question and then pages
> “offering an alternative” by explaining things a baby sysadmin would
> already know, I deduce they are not much above the level of baby
> sysadmin themselves, and it cancels any trust I could have put in the
> one-line answer.

Have you even *read* this mailing list?  Most of the people who ask
for help here lack experience that you might consider "baby sysadmin"
level, and would greatly appreciate the explanations.

[toc] | [prev] | [next] | [standalone]

#267726

Greg Wooledge (12024-02-23):
> Have you even *read* this mailing list?  Most of the people who ask
> for help here lack experience that you might consider "baby sysadmin"
> level, and would greatly appreciate the explanations.

It is usually quite easy to tell the difference by the phrasing and
accuracy of the question.

Regards,

-- 
  Nicolas George

[toc] | [prev] | [next] | [standalone]

#267729

Dnia 2024-02-23, o godz. 14:09:45
Nicolas George <george@nsup.org> napisał(a):

> Greg Wooledge (12024-02-23):
> > Have you even *read* this mailing list?  Most of the people who ask
> > for help here lack experience that you might consider "baby
> > sysadmin" level, and would greatly appreciate the explanations.  
> 
> It is usually quite easy to tell the difference by the phrasing and
> accuracy of the question.
> 

It does. It looked like person that can't comprehend the manual they
have been given nor use Google



-- 
Mariusz Gronczewski (XANi) <xani666@gmail.com>
GnuPG: 0xEA8ACE64
https://devrandom.eu

[toc] | [prev] | [next] | [standalone]

#267727

Dnia 2024-02-23, o godz. 14:03:50
Nicolas George <george@nsup.org> napisał(a):

> Greg Wooledge (12024-02-23):
> > What was "blind" about his anaylsis?  It looked pretty well thought
> > out to me.  He showed actual examples of how space-inefficient it
> > is, and provided a theoretical example of how one misbehaved
> > service could flush out the important logs of well-behaved
> > services.  
> 
> The selective blindness here is to look only at the bad things. The
> drawbacks mentioned exist, but they are in part there for reasons, in
> order to fix the issues of the previous solutions.

I wrote about the stuff you explicitly asked, saving the logs on disk,
because you apparently can't stand a conversation about alternatives.

First you bitch I wrote about "the old stuff you know" now you bitch
about not going thru entire manpage of journalctl features. Decide

> 
> An analysis that mentions only the bad things and conclude to
> recommend using the previous solutions without even discussing their
> own drawbacks is not worth our time.
> 

The selective blindness is your problem, not "us". You decided "old
stuff" didn't work for you and got into shouting match when I described
you in excessive detail why what you want to do is impossible under
systemd.

And you got angry that someone described you why you can't do that.

> > If anything, the person who's blindly following a path is *you*.
> > You're looking to do something that multiple people have said is
> > not possible,  
> 
> Multiple?
> 
> > and when they offer you an alternative, your claws come out.  
> 
> When somebody spends one line answering the question and then pages
> “offering an alternative” by explaining things a baby sysadmin would
> already know, I deduce they are not much above the level of baby
> sysadmin themselves, and it cancels any trust I could have put in the
> one-line answer.
> 

I assumed that you are a beginner because if you read a manpage of
journald/ctl you would know what you are asking is not possible. And
the fact you haven't mentioned any other "traditional" way of doing it
made it look like that too, a newbie linux user that didn't knew it
existed.

So I wrote it with detail needed for beginner to do the job.

I would say I am sorry for mistaking your "skill" level but you already
proven abilty to read the documentation with understand is above you,
so I am not.

Like, really what kind of person gets angry when they get too much
details in instruction?


-- 
Mariusz Gronczewski (XANi) <xani666@gmail.com>
GnuPG: 0xEA8ACE64
https://devrandom.eu

[toc] | [prev] | [next] | [standalone]

#267728

Mariusz Gronczewski (12024-02-23):
> Like, really what kind of person gets angry when they get too much
> details in instruction?

What kind of person writes pages of angry mail when the details are not
liked?

-- 
  Nicolas George

[toc] | [prev] | [next] | [standalone]

#267730

Dnia 2024-02-23, o godz. 14:23:07
Nicolas George <george@nsup.org> napisał(a):

> Mariusz Gronczewski (12024-02-23):
> > Like, really what kind of person gets angry when they get too much
> > details in instruction?  
> 
> What kind of person writes pages of angry mail when the details are
> not liked?
> 

That would be you, the thing like "conversation" and "having common
courtesy to answer questions instead of ignoring ones you don't like"
seems like foreign concept to you too. Now please just be quiet as this
is just spam at that point and the question has already been answered.

-- 
Mariusz Gronczewski (XANi) <xani666@gmail.com>
GnuPG: 0xEA8ACE64
https://devrandom.eu

[toc] | [prev] | [next] | [standalone]

#267722

Dnia 2024-02-23, o godz. 13:48:35
Nicolas George <george@nsup.org> napisał(a):

> Mariusz Gronczewski (12024-02-23):
> > So to say it short: It is horrid.  
> 
> Generic bashing of systemd in favor of a blind cult of the good old
> ways are not what I am looking for either, and the unbalanced tone of
> your reply makes it look like precisely that.
> 

What was "blind" about it? I had now years of experience on few
hundreds systems dealing with it from basically since it was included
in Debian.

I assume you're one of those ignorants that assume any critique of 
product means that I somehow "hate" it  and everything around it without 
reason and that is frankly disgusting.

Systemd saved us thousands of lines of code thanks to its many useful
features and cut down whole swathes of bugs commonly made in "simple"
SysV systems and I'd recommend it on any alternative in a flash.

But journalctl is just not one of them and the way it
stores the data looks like an amateur really wanted towanted to write a
file format/database/querying system and failed at all of that, instead
of slapping SQLite (or any other existing established embedded
database) on it and calling it a day. Hell "a bunch of texfiles
prefixed with program name would've been an improvement. That's even
before we got into whole log duration management that I already
mentioned.

I *want* it to be good, system like that that is also efficient and
easy to query would be a dream for embedded system but it. is. just.
not. that. At the moment.

-- 
Mariusz Gronczewski (XANi) <xani666@gmail.com>
GnuPG: 0xEA8ACE64
https://devrandom.eu

[toc] | [prev] | [next] | [standalone]

#267735 — Journald's qualities (was: Selective rotation of journald logs)

> [13:37:48]cthulhu:/var/log/journal☠ journalctl |dd of=/dev/zero bs=1M
> 0+15442 records in
> 0+15442 records out
> 63643115 bytes (64 MB, 61 MiB) copied, 5,47791 s, 11,6 MB/s
> du -h /var/log/journal/
> 337M	/var/log/journal/44cf6f547971fc33309d1e9e000002e7
> 337M	/var/log/journal/
>
> (I've raised a bug 8 years ago about it
>  https://github.com/systemd/systemd/issues/2460 )

Oh, that bug report is quite interesting, thanks.
Makes one wonder why they don't use naive append-only "plain text" logs
(tho with appropriate delimiters (maybe some kind of CSV) to make
searches more reliable than with old-style plain text logs)?

What are the advantages of journald's representation?
I mean, to justify the slow search and large disk space usage, there is
presumably some upside for some use cases.  I can see some weak argument
against Sqlite based on the size of Sqlite, but what are the advantages
of journald's representation compared to a naive one?


        Stefan


PS: FWIW, the only time I thought about the design of a log format,
I was thinking along the lines of reducing redundancy, i.e. keeping
a kind of "suffix tree" of past messages so as to replace the many
repeated chunks of text with references to past log entries.
Then I realized that I could get the same result much more simply by
gzipping the files, which would naturally be done as part of
log rotation 🙂

[toc] | [prev] | [next] | [standalone]

#267740 — Re: Journald's qualities (was: Selective rotation of journald logs)

Dnia 2024-02-23, o godz. 10:26:49
Stefan Monnier <monnier@iro.umontreal.ca> napisał(a):

> > [13:37:48]cthulhu:/var/log/journal☠ journalctl |dd of=/dev/zero
> > bs=1M 0+15442 records in
> > 0+15442 records out
> > 63643115 bytes (64 MB, 61 MiB) copied, 5,47791 s, 11,6 MB/s
> > du -h /var/log/journal/
> > 337M	/var/log/journal/44cf6f547971fc33309d1e9e000002e7
> > 337M	/var/log/journal/
> >
> > (I've raised a bug 8 years ago about it
> >  https://github.com/systemd/systemd/issues/2460 )  
> 
> Oh, that bug report is quite interesting, thanks.
> Makes one wonder why they don't use naive append-only "plain text"
> logs (tho with appropriate delimiters (maybe some kind of CSV) to make
> searches more reliable than with old-style plain text logs)?
> 
> What are the advantages of journald's representation?
> I mean, to justify the slow search and large disk space usage, there
> is presumably some upside for some use cases.  

The advantage of binary format is that you have separate fields for
program, level etc and it is easier to filter by exactly what you want. 
So "systemctl status servicename" just asks log subsystem for exactly
what it needs.

But journald logs aren't build for querying speed, there is for example
no index (in memory or on disk) that tells you where last log entries
from apps are so you get pathological cases like in ticket.

The problem is that you kinda do 2 types of querying on logs most often:

* get me all/most of what happened in last x lines/hours
* get me everything recent (or just everything period) that this
  particular service returned.

Which kinda exclude the "simple" tricks" like "just hash over service
name so at worst you have to search only 1/64th of files" as they would
make the "get me everything" case slower. But maybe not by all that
much.

The size issue... it's appendable binary format which means that any
metadata systemd adds can't be deduplicated by just having join on
table with service info, it needs to be repeated every single line and
I'm guessing that's where the bloat comes from.

I rambled about SQLite as it is kinda closest comparison for embedded
binary format that can be searched and I think that being able to run
SQL queries (sqlite now even have full text search) against your local
logs with no database daemons running would be killer feature on some of
the smaller devices.


> argument against Sqlite based on the size of Sqlite, 

I'd argue that's not really technically relevant as in most cases you'd
have other app in system using sqlite for *something* and will have
installed the library anyway. And space savings gets wasted just on how
verbose journald format is.

> but what are the
> advantages of journald's representation compared to a naive one?

in short: querability without text parsing. That's about it.

Which *is* a great feature, do not get me wrong, I use it all the time,
but the slowdown caused by its unfortunate implementation detail is
also showing up when I use it pretty often as the most common use case
is "something broke, log in on server that has been running on
automation for last 3 months and see what is happening"

It does have some features around FSS (signing logs) so for folks that
need it there isn't many alternatives as it would be hard to make
similar feature on database.

But on flipside inability to separate log into log rotation groups is
*nasty*. You basically can't have verbose service in the system as that
will rotate away any old logs of other services away. And looking for
those logs is also the worst case:

[17:57:56]cthulhu:~☠ echo 3 > /proc/sys/vm/drop_caches
[17:58:14]cthulhu:~☠ time systemctl status nginx |grep Notice
Notice: journal has been rotated since unit was started, output may be incomplete.

real	0m1,645s
user	0m0,000s
sys	0m0,094s
[17:58:19]cthulhu:~☠ echo 3 > /proc/sys/vm/drop_caches
[17:58:28]cthulhu:~☠ time sqlite /tmp/log_indexed.sqlite 'select * from log where source = "nginx";'  >/dev/null

real	0m0,374s
user	0m0,000s
sys	0m0,009s


I guess another one being appendable log is that in case of crash it's easier to recover than embedded database...

...or so would I say if I didn't saw journald throwing away binary logs with some errors after crash.


> 
> 
>         Stefan
> 
> 
> PS: FWIW, the only time I thought about the design of a log format,
> I was thinking along the lines of reducing redundancy, i.e. keeping
> a kind of "suffix tree" of past messages so as to replace the many
> repeated chunks of text with references to past log entries.
> Then I realized that I could get the same result much more simply by
> gzipping the files, which would naturally be done as part of
> log rotation 🙂
> 

One-service-per-file approach is honestly good enough for most stuff.
PITA to get chronological order thought, every approach really have
some drawbacks and benefits.

But honestly ? Send it to central logger and worry there if you have a
chance, especially if you do some parsing there can be some amazing
productivity gains to be had.

We took a bunch of work to have every firewall log parsed into right
columns but now answering questions like "at which firewall this
traffic is blocked?" is a breeze.

Of course, not really an option for smaller stuff

-- 
Mariusz Gronczewski (XANi) <xani666@gmail.com>
GnuPG: 0xEA8ACE64
https://devrandom.eu

[toc] | [prev] | [next] | [standalone]

#267753 — Re: Journald's qualities

>> Oh, that bug report is quite interesting, thanks.
>> Makes one wonder why they don't use naive append-only "plain text"
>> logs (tho with appropriate delimiters (maybe some kind of CSV) to make
>> searches more reliable than with old-style plain text logs)?
>> 
>> What are the advantages of journald's representation?
>> I mean, to justify the slow search and large disk space usage, there
>> is presumably some upside for some use cases.  
>
> The advantage of binary format is that you have separate fields for
> program, level etc and it is easier to filter by exactly what you want. 
> So "systemctl status servicename" just asks log subsystem for exactly
> what it needs.

That's why I said "with appropriate delimiters (maybe some kind of
CSV)".  Really plain text is what we had before, but it's not rocket
science to have a "plain text" format with a reliably delimited set
of fields, which is what I'd assume as the obvious naive choice.

>> argument against Sqlite based on the size of Sqlite, 
> I'd argue that's not really technically relevant

I did say "weak" 🙂

>> but what are the advantages of journald's representation compared to
>> a naive one?
> in short: querability without text parsing. That's about it.

They have to parse the binary format, so that's not in and of itself an
upside compared to parsing CSV.

I've made my share of bad design decisions that don't pan out.
But there's always an upside to my decision (even when it turns out it
speeds up only those cases which can never occur, because of some other
aspect of the system).

AFAICT the format is *not* just a plain sequence of log entries, so
there's some additional structure which is intended to speed up
some operations.

IOW, even if contrived, there should be *some* use case where it does
better than CSV, no?

> It does have some features around FSS (signing logs) so for folks that
> need it there isn't many alternatives as it would be hard to make
> similar feature on database.

That should be just as easy to add to a CSV-style representation, tho.

> But on flipside inability to separate log into log rotation groups is
> *nasty*. You basically can't have verbose service in the system as that
> will rotate away any old logs of other services away. And looking for
> those logs is also the worst case:

It shouldn't be difficult to add a filtering step to the log rotation
(just extract all the log entries, filter out the unwanted ones, and
create a new log file with the result).
Beside the obvious usual issues of atomicity, the only real problem
I can imagine there is FSS for which you'd need to seal separately the
groups that share differently filtering rules.

> I guess another one being appendable log is that in case of crash it's
> easier to recover than embedded database...

AFAICT, CSV shares this property just as well.

> One-service-per-file approach is honestly good enough for most stuff.
> PITA to get chronological order though, every approach really have
> some drawbacks and benefits.

If you use a reasonable format with precise enough timestamps, you
should be able to "weave" entries from separate files back into
a coherent single chronology (assuming all those entries got their
timestamps from the same journald in the first place).


        Stefan

[toc] | [prev] | [next] | [standalone]

#267759 — Re: Journald's qualities

[Multipart message — attachments visible in raw view] — view raw

On 2024-02-23 at 15:35, Stefan Monnier wrote:

>>> but what are the advantages of journald's representation compared
>>> to a naive one?
>> 
>> in short: querability without text parsing. That's about it.
> 
> They have to parse the binary format, so that's not in and of itself
> an upside compared to parsing CSV.
> 
> I've made my share of bad design decisions that don't pan out. But
> there's always an upside to my decision (even when it turns out it 
> speeds up only those cases which can never occur, because of some
> other aspect of the system).
> 
> AFAICT the format is *not* just a plain sequence of log entries, so 
> there's some additional structure which is intended to speed up some
> operations.
> 
> IOW, even if contrived, there should be *some* use case where it
> does better than CSV, no?

I can think of two possibilities, just offhand, in no particular order:

* No need to parse the timestamps, et cetera, and take the risk that
someone put in one that's in a format you don't expect; the times are
stored internally in a consistent guaranteed format, so you can just use
internal reader functions (paired with, and updated alongside, the
internal writer functions) and be done with it.

* No need to worry about handling log entries that *contain* commas, or
whatever other element was chosen as the separator.

-- 
   The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man.         -- George Bernard Shaw

[toc] | [prev] | [next] | [standalone]

#267762 — Re: Journald's qualities

On Fri, Feb 23, 2024 at 10:17 PM The Wanderer <wanderer@fastmail.fm> wrote:
>
> On 2024-02-23 at 15:35, Stefan Monnier wrote:
>
> >>> but what are the advantages of journald's representation compared
> >>> to a naive one?
> >>
> >> in short: querability without text parsing. That's about it.
> >
> > They have to parse the binary format, so that's not in and of itself
> > an upside compared to parsing CSV.
> >
> > I've made my share of bad design decisions that don't pan out. But
> > there's always an upside to my decision (even when it turns out it
> > speeds up only those cases which can never occur, because of some
> > other aspect of the system).
> >
> > AFAICT the format is *not* just a plain sequence of log entries, so
> > there's some additional structure which is intended to speed up some
> > operations.
> >
> > IOW, even if contrived, there should be *some* use case where it
> > does better than CSV, no?
>
> I can think of two possibilities, just offhand, in no particular order:
>
> * No need to parse the timestamps, et cetera, and take the risk that
> someone put in one that's in a format you don't expect; the times are
> stored internally in a consistent guaranteed format, so you can just use
> internal reader functions (paired with, and updated alongside, the
> internal writer functions) and be done with it.
>
> * No need to worry about handling log entries that *contain* commas, or
> whatever other element was chosen as the separator.

Systemd also provides tamper-resistant logs. The property is often
desirable in the enterprise. See Forward Secure Sealing,
<https://lwn.net/Articles/512895/>.

Jeff

[toc] | [prev] | [next] | [standalone]

#267766 — Re: Journald's qualities

Dnia 2024-02-23, o godz. 23:02:49
Jeffrey Walton <noloader@gmail.com> napisał(a):

> 
> Systemd also provides tamper-resistant logs. The property is often
> desirable in the enterprise. See Forward Secure Sealing,
> <https://lwn.net/Articles/512895/>.
> 
> Jeff
> 

I had mentioned that feature. I haven't it seen in a wild or as an
requirement, ever, and we work with few of the local banks deploying
apps on their infrastructure.

It's *basically* always "send the logs to audit server". rm -rf
/var/log does not care about tamper proofing either.

IMO it's a feature that should be a separate plugin that the people
that need it can just load and use. There is no reason to have it in
default logging format or carry the burden of code for it in core.

Now, tamper-proof *wire* format, that could be useful (if enough other
software supported it). Rsyslog have RELP
(https://en.wikipedia.org/wiki/Reliable_Event_Logging_Protocol) that we
use as it fix few of the issues with sending logs via TCP/TLS
(interrupted connection can lose up to buffer's worth of logs), having
on top of that information "hey, some of the blocks of that log were
lost before being sent" would be useful. For that all it would be
needed is to FSS send queue of logger (which wouldn't be queried so it
could be nice and compressed), not entire on-disk format. 

Then again journald can't even send(AFAIK) using normal syslog protocol
as author decided to XKCD#927 that too...

-- 
Mariusz Gronczewski (XANi) <xani666@gmail.com>
GnuPG: 0xEA8ACE64
https://devrandom.eu

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.debian.user

csiph-web