Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.bugs.dist > #1231341 > unrolled thread

Bug#1094965: wtmpdb: provides no way to prune rotated logs

Back to article view | Back to linux.debian.bugs.dist

Contents

  Bug#1094965: wtmpdb: provides no way to prune rotated logs Andrew Bower <andrew@bower.uk> - 2025-02-01 22:30 +0100
    Bug#1094965: wtmpdb: provides no way to prune rotated logs Andrew Bower <andrew@bower.uk> - 2025-02-03 23:40 +0100
      Bug#1094965: wtmpdb: provides no way to prune rotated logs Chris Hofstaedtler <zeha@debian.org> - 2025-02-04 00:00 +0100
        Bug#1094965: wtmpdb: provides no way to prune rotated logs Andrew Bower <andrew@bower.uk> - 2025-02-04 01:40 +0100
          Bug#1094965: wtmpdb: provides no way to prune rotated logs Chris Hofstaedtler <zeha@debian.org> - 2025-02-04 10:40 +0100
            Bug#1094965: wtmpdb: provides no way to prune rotated logs Andrew Bower <andrew@bower.uk> - 2025-02-04 11:00 +0100

#1231341 — Bug#1094965: wtmpdb: provides no way to prune rotated logs

Package: wtmpdb
Version: 0.13.0-5
Severity: normal

wtmpdb will rotate logs but nothing will prune them (like logrotate would for
other types of log - unfortunately it doesn't fit in well with the wtmpdb way).

Also logs shouldn't be in a state directory like /var/lib. We can forgive the
live database being there, perhaps.

One solution I have in mind is:

 - add 'wtmpdb rotate --dest DIR' option
 - change timer/cron files to dump rotated files in /var/log
 - after rotation, prune files keeping a minimum age and count
 - configuration is by editing the unit/cron files

I think we can set some long defaults - we just don't want unbounded
accumulation!

[toc] | [next] | [standalone]

#1231660

Control: tags -1 patch

On Sat, Feb 01, 2025 at 09:24:31PM +0000, Andrew Bower wrote:
> wtmpdb will rotate logs but nothing will prune them (like logrotate would for
> other types of log - unfortunately it doesn't fit in well with the wtmpdb way).

I have come up with a solution whereby a 'prune' systemd unit is chained
from the 'rotate' service. This calls into a script to prune logs beyond
a configured limit. One can disable the prun unit or edit it to adjust
the parameters.

Any feedback much appreciated!

https://salsa.debian.org/abower/wtmpdb/-/commit/512fa747f9943e1b8b4e7c169dabc1c67c004313
(branch 'log-pruning')

One nicety of the old wtmp logrotate config is that it wouldn't even
rotate until the log hit 1MB.

What do we think of that idea? It seems logical - no point in rotating
tiny files!

[toc] | [prev] | [next] | [standalone]

#1231662

* Andrew Bower <andrew@bower.uk> [250203 23:39]:
> Control: tags -1 patch
> 
> On Sat, Feb 01, 2025 at 09:24:31PM +0000, Andrew Bower wrote:
> > wtmpdb will rotate logs but nothing will prune them (like logrotate would for
> > other types of log - unfortunately it doesn't fit in well with the wtmpdb way).
> 
> I have come up with a solution whereby a 'prune' systemd unit is chained
> from the 'rotate' service. This calls into a script to prune logs beyond
> a configured limit. One can disable the prun unit or edit it to adjust
> the parameters.

> Any feedback much appreciated!
> 
> https://salsa.debian.org/abower/wtmpdb/-/commit/512fa747f9943e1b8b4e7c169dabc1c67c004313
> (branch 'log-pruning')

ISTM this is a lot of new, potentially fragile sh code, possibly
with undefined semantics. Indeed shellcheck has a lot of things to
say, including:

| In prune-logs line 9:
| set -o pipefail
|        ^------^
| SC3040 (warning): In POSIX sh, set option pipefail is undefined.

I'm not an expert in sh scripting, but I'd try to make it shellcheck
clean. Generally I think whatever Debian ends up shipping should
also go upstream.

> One nicety of the old wtmp logrotate config is that it wouldn't even
> rotate until the log hit 1MB.
> 
> What do we think of that idea? It seems logical - no point in rotating
> tiny files!

It's a trade off to what should be kept. If you have strict
requirements on how many days can be kept, then also small files
must be treated.

Maybe a find call with -mtime <something> would be enough.

Chris

[toc] | [prev] | [next] | [standalone]

#1231664

Thanks for the review, Chris!

On Mon, Feb 03, 2025 at 11:52:08PM +0100, Chris Hofstaedtler wrote:
> ISTM this is a lot of new, potentially fragile sh code, possibly
> with undefined semantics. Indeed shellcheck has a lot of things to
> say, including:
> 
> | In prune-logs line 9:
> | set -o pipefail
> |        ^------^
> | SC3040 (warning): In POSIX sh, set option pipefail is undefined.
> 
> I'm not an expert in sh scripting, but I'd try to make it shellcheck
> clean.

Thanks for the prompt. Partly I made the mistake of treating "dash has
it" as a proxy for "POSIX has it"... Now cleaned, trimmed and safer on
my 'log-pruning' branch.

> Generally I think whatever Debian ends up shipping should
> also go upstream.

Surely it depends - some thing are highly distro-specific. The amount of
detritus that accumulates in upstream contrib directories over the
years... scripts that new integrators would surely rather rewrite for
their own context.

(As for the tool itself, however, upstreaming is going well so far...!)

In this case I'd so much rather use logrotate (locking concerns
acknowledged) because it has all the semantics you want, configurable
in a standard way. But that doesn't fit with wtmpdb's rotation scheme
and I didn't imagine dumping that entirely would go down well.

> > One nicety of the old wtmp logrotate config is that it wouldn't even
> > rotate until the log hit 1MB.

I did add a size gate to the cron job. The unit file could be conditional
too?

> > What do we think of that idea? It seems logical - no point in rotating
> > tiny files!
> 
> It's a trade off to what should be kept. If you have strict
> requirements on how many days can be kept, then also small files
> must be treated.

Not sure what you're suggesting there, I mean that there's no point in
splitting the master log file into tiny fragments in the first place,
not that they shouldn't be kept - to the contrary! I don't think we have
to be too tight on the storage, just to be bounded, not be pathological
and not leave a dog's breakfast on the user's filesystem!

> Maybe a find call with -mtime <something> would be enough.

I see what you mean, I did think of that of course, but I wasn't
convinced that relying on file metadata wasn't itself fragile; I am
concerned that we need to fail safe by deleting less, hence looking at
age and count. (Size is an additional gate.) By the time you've glued
those things together you're going to be growing that wrapper too.

I also wondered about shipping with the pruner disabled - we only really
need a way to make sure it can't grow endlessly. With nothing it rather
looks like it's not been considered.

Again, thanks for taking the trouble to feed back. I'm keen for users to
have a good, least-surprises experience.

[toc] | [prev] | [next] | [standalone]

#1231712

On Tue, Feb 04, 2025 at 12:33:04AM +0000, Andrew Bower wrote:
> 
> Surely it depends - some thing are highly distro-specific. The amount of
> detritus that accumulates in upstream contrib directories over the
> years... scripts that new integrators would surely rather rewrite for
> their own context.

I just don't see why different distros need to come up with
different log rotating strategies. If one distro does the work, then
all can and should benefit. Also it will be less surprising for
upstream, when they introduce a rotation-with-purging concept.

> > > What do we think of that idea? It seems logical - no point in rotating
> > > tiny files!
> > 
> > It's a trade off to what should be kept. If you have strict
> > requirements on how many days can be kept, then also small files
> > must be treated.
> 
> Not sure what you're suggesting there, I mean that there's no point in
> splitting the master log file into tiny fragments in the first place,
> not that they shouldn't be kept - to the contrary! I don't think we have
> to be too tight on the storage, just to be bounded, not be pathological
> and not leave a dog's breakfast on the user's filesystem!

I'm saying that in some jurisdictions you cannot keep such logs
longer than X days. If you skip over small files, you will keep them
longer than X days.

[toc] | [prev] | [next] | [standalone]

#1231714

On Tue, Feb 04, 2025 at 10:32:55AM +0100, Chris Hofstaedtler wrote:
> On Tue, Feb 04, 2025 at 12:33:04AM +0000, Andrew Bower wrote:
> > 
> > Surely it depends - some thing are highly distro-specific. The amount of
> > detritus that accumulates in upstream contrib directories over the
> > years... scripts that new integrators would surely rather rewrite for
> > their own context.
> 
> I just don't see why different distros need to come up with
> different log rotating strategies. If one distro does the work, then
> all can and should benefit. Also it will be less surprising for
> upstream, when they introduce a rotation-with-purging concept.

Funnily enough that was my initial intention - add pruning to and/or
alongside the rotate operation upstream.

The reason I didn't was because I thought it was a bit late to
co-ordinate at this stage and because I thought that would be vulnerable
to the criticism of "why are you doing this task in C that is a natural
fit for a system script". Food for thought though - I'll look into it.

> > > > What do we think of that idea? It seems logical - no point in rotating
> > > > tiny files!
> > > 
> > > It's a trade off to what should be kept. If you have strict
> > > requirements on how many days can be kept, then also small files
> > > must be treated.
> > 
> > Not sure what you're suggesting there, I mean that there's no point in
> > splitting the master log file into tiny fragments in the first place,
> > not that they shouldn't be kept - to the contrary! I don't think we have
> > to be too tight on the storage, just to be bounded, not be pathological
> > and not leave a dog's breakfast on the user's filesystem!
> 
> I'm saying that in some jurisdictions you cannot keep such logs
> longer than X days. If you skip over small files, you will keep them
> longer than X days.

Ah yes, then that makes sense. Max age.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.bugs.dist

csiph-web