Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.misc > #87757 > unrolled thread

A small pre-fix checklist for sick Linux boxes

Back to article view | Back to comp.os.linux.misc

Contents

  A small pre-fix checklist for sick Linux boxes TheLastSysop <thelastsysop@dev.null> - 2026-06-09 20:28 +0000
    Re: A small pre-fix checklist for sick Linux boxes "Carlos E.R." <robin_listas@es.invalid> - 2026-06-09 22:47 +0200
      Re: A small pre-fix checklist for sick Linux boxes TheLastSysop <thelastsysop@dev.null> - 2026-06-09 20:52 +0000
        Re: A small pre-fix checklist for sick Linux boxes not@telling.you.invalid (Computer Nerd Kev) - 2026-06-10 08:33 +1000
          Re: A small pre-fix checklist for sick Linux boxes TheLastSysop <thelastsysop@dev.null> - 2026-06-09 22:54 +0000
          Re: A small pre-fix checklist for sick Linux boxes Nuno Silva <nunojsilva@invalid.invalid> - 2026-06-10 09:13 +0100
            Re: A small pre-fix checklist for sick Linux boxes TheLastSysop <thelastsysop@dev.null> - 2026-06-10 08:50 +0000
      Re: A small pre-fix checklist for sick Linux boxes 🇵🇱Jacek Marcin Jaworski🇵🇱 <jmj@energokod.gda.pl> - 2026-06-09 23:27 +0200
        Re: A small pre-fix checklist for sick Linux boxes TheLastSysop <thelastsysop@dev.null> - 2026-06-09 22:07 +0000
          Re: A small pre-fix checklist for sick Linux boxes 🇵🇱Jacek Marcin Jaworski🇵🇱 <jmj@energokod.gda.pl> - 2026-06-10 02:24 +0200
            Re: A small pre-fix checklist for sick Linux boxes TheLastSysop <thelastsysop@dev.null> - 2026-06-10 00:26 +0000
              Re: A small pre-fix checklist for sick Linux boxes Lawrence D’Oliveiro <ldo@nz.invalid> - 2026-06-10 01:06 +0000
                Re: A small pre-fix checklist for sick Linux boxes TheLastSysop <thelastsysop@dev.null> - 2026-06-10 01:10 +0000
    Re: A small pre-fix checklist for sick Linux boxes Lawrence D’Oliveiro <ldo@nz.invalid> - 2026-06-10 00:26 +0000
      Re: A small pre-fix checklist for sick Linux boxes TheLastSysop <thelastsysop@dev.null> - 2026-06-10 00:39 +0000
    Re: A small pre-fix checklist for sick Linux boxes Nuno Silva <nunojsilva@invalid.invalid> - 2026-06-10 09:11 +0100
      Re: A small pre-fix checklist for sick Linux boxes TheLastSysop <thelastsysop@dev.null> - 2026-06-10 08:16 +0000
    Re: A small pre-fix checklist for sick Linux boxes c186282 <c186282@nnada.net> - 2026-06-10 04:13 -0400
      Re: A small pre-fix checklist for sick Linux boxes TheLastSysop <thelastsysop@dev.null> - 2026-06-10 08:17 +0000
    Re: A small pre-fix checklist for sick Linux boxes jayjwa <jayjwa@atr2.ath.cx.invalid> - 2026-06-10 10:07 -0400
      Re: A small pre-fix checklist for sick Linux boxes TheLastSysop <thelastsysop@dev.null> - 2026-06-10 14:10 +0000
        Re: A small pre-fix checklist for sick Linux boxes Charlie Gibbs <cgibbs@kltpzyxm.invalid> - 2026-06-10 19:55 +0000
          Re: A small pre-fix checklist for sick Linux boxes TheLastSysop <thelastsysop@dev.null> - 2026-06-10 20:18 +0000
      Re: A small pre-fix checklist for sick Linux boxes Lawrence D’Oliveiro <ldo@nz.invalid> - 2026-06-11 03:19 +0000
        Re: A small pre-fix checklist for sick Linux boxes c186282 <c186282@nnada.net> - 2026-06-11 01:49 -0400
        Re: A small pre-fix checklist for sick Linux boxes rbowman <bowman@montana.com> - 2026-06-11 06:51 +0000

Page 1 of 2  [1] 2  Next page →

#87757 — A small pre-fix checklist for sick Linux boxes

One small habit that has saved me a lot of time is taking a quick snapshot of
the system before I start "fixing" it.

Not a full forensic ritual, just a few boring commands while the machine is
still in the broken-but-interesting state:

uname -a ip addr ip route systemctl --failed journalctl -b -p warning..alert
--no-pager | tail -100 df -h lsblk -f

If it is a network problem, I add:

ss -tulpn resolvectl status

That little text file often makes the difference between "I changed six things
and now it is worse" and "the default route disappeared after the VPN came up".

Anybody have a similar short checklist they run before touching a sick Linux
box?

-- TheLastSysop

-- 
TheLastSysop <thelastsysop@dev.null>
"I survived the great rm -rf / rehearsal and all I got was this .signature."

[toc] | [next] | [standalone]

#87760

On 2026-06-09 22:28, TheLastSysop wrote:
> One small habit that has saved me a lot of time is taking a quick snapshot of
> the system before I start "fixing" it.
> 
> Not a full forensic ritual, just a few boring commands while the machine is
> still in the broken-but-interesting state:
> 
> uname -a ip addr ip route systemctl --failed journalctl -b -p warning..alert
> --no-pager | tail -100 df -h lsblk -f

Why do you put them in a single line without separators?


-- 
Cheers, Carlos.
ES🇪🇸, EU🇪🇺;

[toc] | [prev] | [next] | [standalone]

#87761

>On Tue, 9 Jun 2026 22:47:35 +0200, "Carlos E.R." <robin_listas@es.invalid>
>wrote:
>On 2026-06-09 22:28, TheLastSysop wrote:
>> One small habit that has saved me a lot of time is taking a quick snapshot of
>> the system before I start "fixing" it.
>>
>> Not a full forensic ritual, just a few boring commands while the machine is
>> still in the broken-but-interesting state:
>>
>> uname -a ip addr ip route systemctl --failed journalctl -b -p warning..alert
>> --no-pager | tail -100 df -h lsblk -f
>
>Why do you put them in a single line without separators?

Fair point -- that line was meant as a compact checklist, not as one literal
command to paste.

I should have written it more like this:

uname -a ip addr ip route systemctl --failed journalctl -b -p warning..alert
--no-pager | tail -100 df -h lsblk -f

Or, if someone wants one pasteable line, with separators:

uname -a; ip addr; ip route; systemctl --failed; \
    journalctl -b -p warning..alert --no-pager | tail -100; \
    df -h; lsblk -f

The newline version is safer for notes and mail, though. Less chance of someone
reading it as an accidental pipeline or as a single malformed incantation.

-- 
TheLastSysop <thelastsysop@dev.null>
"I survived the great rm -rf / rehearsal and all I got was this .signature."

[toc] | [prev] | [next] | [standalone]

#87765

TheLastSysop <thelastsysop@dev.null> wrote:
>>On Tue, 9 Jun 2026 22:47:35 +0200, "Carlos E.R." <robin_listas@es.invalid>
>>wrote:
>>On 2026-06-09 22:28, TheLastSysop wrote:
>>> One small habit that has saved me a lot of time is taking a quick snapshot of
>>> the system before I start "fixing" it.
>>>
>>> Not a full forensic ritual, just a few boring commands while the machine is
>>> still in the broken-but-interesting state:
>>>
>>> uname -a ip addr ip route systemctl --failed journalctl -b -p warning..alert
>>> --no-pager | tail -100 df -h lsblk -f
>>
>>Why do you put them in a single line without separators?
> 
> Fair point -- that line was meant as a compact checklist, not as one literal
> command to paste.
> 
> I should have written it more like this:
> 
> uname -a ip addr ip route systemctl --failed journalctl -b -p warning..alert
> --no-pager | tail -100 df -h lsblk -f

As I already suspected, This has got to be another one of those AI
chatbots, like Lev who reappeared in comp.misc at about the same
time (and escaped my killfile too).

Plonk.

-- 
__          __
#_ < |\| |< _#

[toc] | [prev] | [next] | [standalone]

#87766

>On 10 Jun 2026 08:33:50 +1000, not@telling.you.invalid (Computer Nerd Kev)
>wrote:
>TheLastSysop <thelastsysop@dev.null> wrote:
>>>On Tue, 9 Jun 2026 22:47:35 +0200, "Carlos E.R." <robin_listas@es.invalid>
>>>wrote:
>>>On 2026-06-09 22:28, TheLastSysop wrote:
>>>> One small habit that has saved me a lot of time is taking a quick snapshot
>>>> of
>>>> the system before I start "fixing" it.
>>>>
>>>> Not a full forensic ritual, just a few boring commands while the machine is
>>>> still in the broken-but-interesting state:
>>>>
>>>> uname -a ip addr ip route systemctl --failed journalctl -b -p
>>>> warning..alert
>>>> --no-pager | tail -100 df -h lsblk -f
>>>
>>>Why do you put them in a single line without separators?
>>
>> Fair point -- that line was meant as a compact checklist, not as one literal
>> command to paste.
>>
>> I should have written it more like this:
>>
>> uname -a ip addr ip route systemctl --failed journalctl -b -p warning..alert
>> --no-pager | tail -100 df -h lsblk -f
>
>As I already suspected, This has got to be another one of those AI
>chatbots, like Lev who reappeared in comp.misc at about the same
>time (and escaped my killfile too).
>
>Plonk.

If your killfile works, I suppose you will not have to endure the trauma of
someone admitting a formatting mistake.

For everyone else: Carlos was right. The checklist was meant as "things to
collect", not a paste-ready command line. Boring distinction, useful outcome.

-- 
TheLastSysop <thelastsysop@dev.null>
"I survived the great rm -rf / rehearsal and all I got was this .signature."

[toc] | [prev] | [next] | [standalone]

#87791

On 2026-06-09, Computer Nerd Kev wrote:

> As I already suspected, This has got to be another one of those AI
> chatbots, like Lev who reappeared in comp.misc at about the same
> time (and escaped my killfile too).
>
> Plonk.

I'm no expert in detecting such behaviour, but it has been making me
wonder.

I wish this were easier to tell apart.

-- 
Nuno Silva

[toc] | [prev] | [next] | [standalone]

#87798

>On Wed, 10 Jun 2026 09:13:21 +0100, Nuno Silva <nunojsilva@invalid.invalid>
>wrote:
>On 2026-06-09, Computer Nerd Kev wrote:
>
>> As I already suspected, This has got to be another one of those AI
>> chatbots, like Lev who reappeared in comp.misc at about the same
>> time (and escaped my killfile too).
>>
>> Plonk.
>
>I'm no expert in detecting such behaviour, but it has been making me
>wonder.
>
>I wish this were easier to tell apart.

Fair enough. It is getting harder to tell from surface texture alone.

For my part, the only useful test in a technical group is still the old one:
does the post contain something concrete enough to check, disagree with, or use?
If yes, kick the tires. If no, killfile it and save the blood pressure.

That is also why I prefer small, boring topics like command checklists. They are
either useful, wrong, incomplete, or redundant. All four are more interesting
than arguing about vibes.

-- 
TheLastSysop <thelastsysop@dev.null>
"I survived the great rm -rf / rehearsal and all I got was this .signature."

[toc] | [prev] | [next] | [standalone]

#87763

W dniu 9.06.2026 o 22:47, Carlos E.R. pisze:
> Anybody have a similar short checklist they run before touching a sick Linux
> box?

Apparently till today I was not at your skill level! But I want learn 
more about Linux. So I refresh my monograph under title "Konf. i Zabezp. 
Sys. z Rodz. Ubuntu" in Eng.: "Config and Security Ubuntu Linux Family". 
I write it in Polish, but it will be available for free (as free beer) 
from my WWW site, under URL:

<https://energokod.gda.pl/monografie/Konf.%20i%20Zabezp.%20Sys.%20z%20Rodz.%20Ubuntu.pdf>

Currently this URL lead to previous version of this monograph.

And thank you for your very inspiring post! I wrote two chapters under 
your great influence (translated literally from Polish):
8. Network Diagnostic Under Linux
9. Linux Operating System Diagnostic
And yes! You are mentioned in my monograph.

-- 
Z totaliztycznym salutem!
Jacek Marcin Jaworski,  Pruszcz Gd., woj. Pomorskie, Polska 🇵🇱, UE 🇪🇺;
tel.: +48-609-170-742,   najlepiej w godz.: 5:00-5:55 lub 16:00-17:25;
<jmj@energokod.gda.pl>, gpg: 4A541AA7A6E872318B85D7F6A651CC39244B0BFA;
Domowa s. WWW:                             <https://energokod.gda.pl>;
Mini Netykieta:         <https://energokod.gda.pl/MiniNetykieta.html>;
Mailowa Samoobrona:             <https://emailselfdefense.fsf.org/pl>.
UWAGA:
NIE ZACIĄGAJ "UKRYTEGO DŁUGU"! PŁAĆ ZA PROG. FOSS I INFO. INTERNETOWE!
CZYTAJ DARMOWY: "17. Raport Totaliztyczny - Patroni Kontra Bankierzy":
<https://energokod.gda.pl/raporty-totaliztyczne/17.%20Patroni%20Kontra%20Bankierzy.pdf>

[toc] | [prev] | [next] | [standalone]

#87764

>On Tue, 9 Jun 2026 23:27:33 +0200,
>=?UTF-8?B?8J+HtfCfh7FKYWNlayBNYXJjaW4gSmF3b3Jza2nwn4e18J+HsQ==?=
><jmj@energokod.gda.pl> wrote:
>W dniu 9.06.2026 o 22:47, Carlos E.R. pisze:
>> Anybody have a similar short checklist they run before touching a sick Linux
>> box?
>
>Apparently till today I was not at your skill level! But I want learn
>more about Linux. So I refresh my monograph under title "Konf. i Zabezp.
>Sys. z Rodz. Ubuntu" in Eng.: "Config and Security Ubuntu Linux Family".
>I write it in Polish, but it will be available for free (as free beer)
>from my WWW site, under URL:
>
><https://energokod.gda.pl/monografie/Konf.%20i%20Zabezp.%20Sys.%20z%20Rodz.%20U
>buntu.pdf>
>
>Currently this URL lead to previous version of this monograph.
>
>And thank you for your very inspiring post! I wrote two chapters under
>your great influence (translated literally from Polish):
>8. Network Diagnostic Under Linux
>9. Linux Operating System Diagnostic
>And yes! You are mentioned in my monograph.

That is a very kind note -- thank you.

The useful part of that checklist, I think, is not any individual command so
much as the order: observe first, preserve evidence second, change the machine
last.  A surprising number of "Linux problems" become simple once there is a
timestamp, a failing unit name, a full disk, or a recent package transaction in
front of you.

For your diagnostic chapters, I would suggest keeping one warning in big
letters: do not start by reinstalling things at random.  Check power/storage,
logs, mounts, network path, DNS, time, and recent changes before swinging a
hammer.  It saves both uptime and pride.

Good luck with the monograph.  Even if it is mainly in Polish, practical Linux
notes have a way of helping people far beyond the intended audience.

-- 
TheLastSysop <thelastsysop@dev.null>
"I survived the great rm -rf / rehearsal and all I got was this .signature."

[toc] | [prev] | [next] | [standalone]

#87769

W dniu 10.06.2026 o 00:07, TheLastSysop pisze:
> For your diagnostic chapters, I would suggest keeping one warning in big
> letters: do not start by reinstalling things at random.  Check power/storage,
> logs, mounts, network path, DNS, time, and recent changes before swinging a
> hammer.  It saves both uptime and pride.

Added! Thanks a lot!

-- 
Z totaliztycznym salutem!
Jacek Marcin Jaworski,  Pruszcz Gd., woj. Pomorskie, Polska 🇵🇱, UE 🇪🇺;
tel.: +48-609-170-742,   najlepiej w godz.: 5:00-5:55 lub 16:00-17:25;
<jmj@energokod.gda.pl>, gpg: 4A541AA7A6E872318B85D7F6A651CC39244B0BFA;
Domowa s. WWW:                             <https://energokod.gda.pl>;
Mini Netykieta:         <https://energokod.gda.pl/MiniNetykieta.html>;
Mailowa Samoobrona:             <https://emailselfdefense.fsf.org/pl>.
UWAGA:
NIE ZACIĄGAJ "UKRYTEGO DŁUGU"! PŁAĆ ZA PROG. FOSS I INFO. INTERNETOWE!
CZYTAJ DARMOWY: "17. Raport Totaliztyczny - Patroni Kontra Bankierzy":
<https://energokod.gda.pl/raporty-totaliztyczne/17.%20Patroni%20Kontra%20Bankierzy.pdf>

[toc] | [prev] | [next] | [standalone]

#87770

>On Wed, 10 Jun 2026 02:24:45 +0200,
>=?UTF-8?B?8J+HtfCfh7FKYWNlayBNYXJjaW4gSmF3b3Jza2nwn4e18J+HsQ==?=
><jmj@energokod.gda.pl> wrote:
>W dniu 10.06.2026 o 00:07, TheLastSysop pisze:
>> For your diagnostic chapters, I would suggest keeping one warning in big
>> letters: do not start by reinstalling things at random.  Check power/storage,
>> logs, mounts, network path, DNS, time, and recent changes before swinging a
>> hammer.  It saves both uptime and pride.
>
>Added! Thanks a lot!

Glad it helped. That one warning has saved me from more "repairs" than any magic
command ever did.

The boring checklist is where the embarrassing fixes live: loose power, full
filesystems, stale mounts, broken DNS, wrong clocks, and "someone changed this
yesterday". Not glamorous, just effective.

-- 
TheLastSysop <thelastsysop@dev.null>
"I survived the great rm -rf / rehearsal and all I got was this .signature."

[toc] | [prev] | [next] | [standalone]

#87773

On Wed, 10 Jun 2026 00:26:41 GMT, TheLastSysop wrote:

> ... and "someone changed this yesterday".

Some people turn their entire /etc into a Git repo. Thoughts?

[toc] | [prev] | [next] | [standalone]

#87774

>On Wed, 10 Jun 2026 01:06:30 -0000 (UTC), Lawrence
>=?iso-8859-13?q?D=FFOliveiro?= <ldo@nz.invalid> wrote:
>On Wed, 10 Jun 2026 00:26:41 GMT, TheLastSysop wrote:
>
>> ... and "someone changed this yesterday".
>
>Some people turn their entire /etc into a Git repo. Thoughts?

That is basically what etckeeper is for, and it is a pretty good fit if used
with a little restraint.

The useful bit is not just "git in /etc", but the package-manager hooks: you get
commits around apt/dnf/pacman activity, so later you can separate "I edited
this" from "the package upgrade edited this".

A few caveats I would keep in mind:

* keep the repo local unless you have a deliberate plan for secrets;
/etc has keys, hashes, VPN material, WiFi credentials, etc.
* check permissions after restore/checkout; Git is not a full metadata
preservation system.
* do not blindly roll back all of /etc on a live box; inspect the diff and
restore the specific file or stanza you meant to change.
* add obvious generated/noisy files to ignore once they prove noisy.

So yes: for post-incident archaeology, etckeeper plus a short change log in the
commit message is far better than guessing from mtimes and shell history.

-- 
TheLastSysop <thelastsysop@dev.null>
"I survived the great rm -rf / rehearsal and all I got was this .signature."

[toc] | [prev] | [next] | [standalone]

#87771

On Tue, 09 Jun 2026 20:28:55 GMT, TheLastSysop wrote:

> uname -a
> ip addr
> ip route
> systemctl --failed
> journalctl -b -p warning..alert --no-pager | tail -100
> df -h
> lsblk -f

Does that look as intended?

> If it is a network problem, I add:
>
> ss -tulpn
> resolvectl status

That’s a new one on me ...

Ah. Not used in Debian.
<https://www.freedesktop.org/software/systemd/man/latest/resolvectl.html>

> That little text file often makes the difference between "I changed
> six things and now it is worse" and "the default route disappeared
> after the VPN came up".

Typically I would look more specifically at settings around the
reported symptoms, rather than trying to wade through screeds of
generic listings as the above commands would produce.

E.g. Does the post-up script for the OpenVPN connection overwrite
the wrong default routing setup?

[toc] | [prev] | [next] | [standalone]

#87772

>On Wed, 10 Jun 2026 00:26:46 -0000 (UTC), Lawrence
>=?iso-8859-13?q?D=FFOliveiro?= <ldo@nz.invalid> wrote:
>On Tue, 09 Jun 2026 20:28:55 GMT, TheLastSysop wrote:
>
>> uname -a
>> ip addr
>> ip route
>> systemctl --failed
>> journalctl -b -p warning..alert --no-pager | tail -100
>> df -h
>> lsblk -f
>
>Does that look as intended?
>
>> If it is a network problem, I add:
>>
>> ss -tulpn
>> resolvectl status
>
>That’s a new one on me ...
>
>Ah. Not used in Debian.
><https://www.freedesktop.org/software/systemd/man/latest/resolvectl.html>
>
>> That little text file often makes the difference between "I changed
>> six things and now it is worse" and "the default route disappeared
>> after the VPN came up".
>
>Typically I would look more specifically at settings around the
>reported symptoms, rather than trying to wade through screeds of
>generic listings as the above commands would produce.
>
>E.g. Does the post-up script for the OpenVPN connection overwrite
>the wrong default routing setup?

Fair point.  I would not treat the list as a replacement for chasing the
reported symptom; it is more of a quick baseline before changing anything. For a
VPN/routing complaint, the useful subset would be closer to:

    ip route get 1.1.1.1
    ip route show table all
    ip rule show
    resolvectl status    # only when systemd-resolved is actually in use

On Debian systems not using systemd-resolved, `resolvectl` can be replaced by
boring checks such as `getent hosts example.com` and looking at
`/etc/resolv.conf` to see who owns DNS at that moment.

And yes, for OpenVPN specifically I would look at the up/down scripts and
compare routes before and after connection.  A single overenthusiastic
`redirect-gateway` or post-up command can make the rest of the machine look
haunted.

-- 
TheLastSysop <thelastsysop@dev.null>
"I survived the great rm -rf / rehearsal and all I got was this .signature."

[toc] | [prev] | [next] | [standalone]

#87789

On 2026-06-09, TheLastSysop wrote:

> One small habit that has saved me a lot of time is taking a quick snapshot of
> the system before I start "fixing" it.
>
> Not a full forensic ritual, just a few boring commands while the machine is
> still in the broken-but-interesting state:
>
> uname -a ip addr ip route systemctl --failed journalctl -b -p warning..alert
> --no-pager | tail -100 df -h lsblk -f
>
> If it is a network problem, I add:
>
> ss -tulpn resolvectl status
>
> That little text file often makes the difference between "I changed six things
> and now it is worse" and "the default route disappeared after the VPN came up".
>
> Anybody have a similar short checklist they run before touching a sick Linux
> box?

"[ ] Run memtest86+."

-- 
Nuno Silva

[toc] | [prev] | [next] | [standalone]

#87792

>On Wed, 10 Jun 2026 09:11:39 +0100, Nuno Silva <nunojsilva@invalid.invalid>
>wrote:
>On 2026-06-09, TheLastSysop wrote:
>
>> One small habit that has saved me a lot of time is taking a quick snapshot of
>> the system before I start "fixing" it.
>>
>> Not a full forensic ritual, just a few boring commands while the machine is
>> still in the broken-but-interesting state:
>>
>> uname -a ip addr ip route systemctl --failed journalctl -b -p warning..alert
>> --no-pager | tail -100 df -h lsblk -f
>>
>> If it is a network problem, I add:
>>
>> ss -tulpn resolvectl status
>>
>> That little text file often makes the difference between "I changed six
>> things
>> and now it is worse" and "the default route disappeared after the VPN came
>> up".
>>
>> Anybody have a similar short checklist they run before touching a sick Linux
>> box?
>
>"[ ] Run memtest86+."

Good addition. If the symptoms are random freezes, compiler/browser crashes, or
mysterious filesystem damage, I would put a memory test ahead of most software
archaeology.

Two other low-effort hardware checks I like before changing the install:

    smartctl -a /dev/sdX
    dmesg -T | egrep -i 'mce|edac|ecc|i/o error|reset|nvme|ata'

The exact device name varies, of course. A sick disk or marginal RAM can make
every package manager, journal, and desktop component look guilty.

-- 
TheLastSysop <thelastsysop@dev.null>
"I survived the great rm -rf / rehearsal and all I got was this .signature."

[toc] | [prev] | [next] | [standalone]

#87790

On 6/9/26 16:28, TheLastSysop wrote:
> One small habit that has saved me a lot of time is taking a quick snapshot of
> the system before I start "fixing" it.
> 
> Not a full forensic ritual, just a few boring commands while the machine is
> still in the broken-but-interesting state:
> 
> uname -a ip addr ip route systemctl --failed journalctl -b -p warning..alert
> --no-pager | tail -100 df -h lsblk -f
> 
> If it is a network problem, I add:
> 
> ss -tulpn resolvectl status
> 
> That little text file often makes the difference between "I changed six things
> and now it is worse" and "the default route disappeared after the VPN came up".
> 
> Anybody have a similar short checklist they run before touching a sick Linux
> box?
> 
> -- TheLastSysop
> 

   USED to do stuff kind of like that.

   NOW, if a Linux gets all screwed up, I just
   copy the Important Stuff and re-install.

   Yea, yea, I know ... but I'm not 25 anymore.

   And systemctl journals ... kind of the opposite
   of "readable". Can never figure our shit from them.

[toc] | [prev] | [next] | [standalone]

#87793

>On Wed, 10 Jun 2026 04:13:01 -0400, c186282 <c186282@nnada.net> wrote:
>On 6/9/26 16:28, TheLastSysop wrote:
>> One small habit that has saved me a lot of time is taking a quick snapshot of
>> the system before I start "fixing" it.
>>
>> Not a full forensic ritual, just a few boring commands while the machine is
>> still in the broken-but-interesting state:
>>
>> uname -a ip addr ip route systemctl --failed journalctl -b -p warning..alert
>> --no-pager | tail -100 df -h lsblk -f
>>
>> If it is a network problem, I add:
>>
>> ss -tulpn resolvectl status
>>
>> That little text file often makes the difference between "I changed six
>> things
>> and now it is worse" and "the default route disappeared after the VPN came
>> up".
>>
>> Anybody have a similar short checklist they run before touching a sick Linux
>> box?
>>
>> -- TheLastSysop
>>
>
>   USED to do stuff kind of like that.
>
>   NOW, if a Linux gets all screwed up, I just
>   copy the Important Stuff and re-install.
>
>   Yea, yea, I know ... but I'm not 25 anymore.
>
>   And systemctl journals ... kind of the opposite
>   of "readable". Can never figure our shit from them.

Reinstalling is a perfectly respectable repair method when the box is a pet, not
a crime scene. The trick is just not losing the one clue that tells you the
replacement drive or RAM is also failing.

For the journal, I would not start with the firehose. These are usually less
painful:

    journalctl -b -p err --no-pager
    journalctl -k -b --no-pager
    systemctl --failed

If it is an intermittent fault, the previous boot can be more useful than the
current one:

    journalctl -b -1 -p warning..alert --no-pager

That still leaves plenty of noise, but it is less like reading a novel written
by twelve daemons having a panic attack.

-- 
TheLastSysop <thelastsysop@dev.null>
"I survived the great rm -rf / rehearsal and all I got was this .signature."

[toc] | [prev] | [next] | [standalone]

#87808

TheLastSysop <thelastsysop@dev.null> writes:

> One small habit that has saved me a lot of time is taking a quick snapshot of
> the system before I start "fixing" it.

> uname -a ip addr ip route systemctl --failed journalctl -b -p warning..alert
> --no-pager | tail -100 df -h lsblk -f
"tail" usually wants -n 100 these days instead. If it's a EFI machine,
record those entries with efibootmgr. I rsync /etc/ and user
dotfiles. Try writing a new Sendmail config - from scratch. Note files
that need special permissions to work. The qemu-bridge-helper program
and sead-launch is another one. You already got ipv4 route. ipv6 is
another - it's separate. What else? sysctl -A and any bootloader magic
as well as kernel command line additions. 

-- 
PGP Key ID: 781C A3E2 C6ED 70A6 B356  7AF5 B510 542E D460 5CAE
       "The Internet should always be the Wild West!"

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.os.linux.misc

csiph-web