Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.devel > #116165 > unrolled thread

Improvement of headless server upgrades

Back to article view | Back to linux.debian.devel

Contents

  Improvement of headless server upgrades "Helmut K. C. Tessarek" <tessarek@evermeet.cx> - 2025-03-05 03:40 +0100
    Re: Improvement of headless server upgrades Fabio Fantoni <fantonifabio@tiscali.it> - 2025-03-05 10:30 +0100
    Re: Improvement of headless server upgrades Marc Haber <mh+debian-devel@zugschlus.de> - 2025-03-05 16:10 +0100
      Re: Improvement of headless server upgrades Michael Banck <mbanck@debian.org> - 2025-03-05 16:20 +0100
        Re: Improvement of headless server upgrades Bjørn Mork <bjorn@mork.no> - 2025-03-05 17:50 +0100
          Re: Improvement of headless server upgrades Matthias Urlichs <matthias@urlichs.de> - 2025-03-05 19:30 +0100
            Re: Improvement of headless server upgrades Marc Haber <mh+debian-devel@zugschlus.de> - 2025-03-05 20:40 +0100
              Re: Improvement of headless server upgrades Soren Stoutner <soren@debian.org> - 2025-03-05 21:00 +0100
            Re: Improvement of headless server upgrades "Helmut K. C. Tessarek" <tessarek@evermeet.cx> - 2025-03-05 20:40 +0100
            Re: Improvement of headless server upgrades Bjørn Mork <bjorn@mork.no> - 2025-03-05 21:10 +0100
              Re: Improvement of headless server upgrades Marc Haber <mh+debian-devel@zugschlus.de> - 2025-03-05 21:40 +0100
                Re: Improvement of headless server upgrades Matthias Urlichs <matthias@urlichs.de> - 2025-03-05 22:30 +0100
                  Re: Improvement of headless server upgrades Marc Haber <mh+debian-devel@zugschlus.de> - 2025-03-06 09:00 +0100
              Re: Improvement of headless server upgrades "Helmut K. C. Tessarek" <tessarek@evermeet.cx> - 2025-03-05 21:40 +0100
        Re: Improvement of headless server upgrades "Helmut K. C. Tessarek" <tessarek@evermeet.cx> - 2025-03-05 20:40 +0100
          Re: Improvement of headless server upgrades Marc Haber <mh+debian-devel@zugschlus.de> - 2025-03-06 10:10 +0100
      Re: Improvement of headless server upgrades Marc Haber <mh+debian-devel@zugschlus.de> - 2025-03-06 15:50 +0100
      Re: Improvement of headless server upgrades Richard Lewis <richard.lewis.debian@googlemail.com> - 2025-03-07 01:50 +0100
    Re: Improvement of headless server upgrades Helmut Grohne <helmut@subdivi.de> - 2025-03-06 06:50 +0100
      Re: Improvement of headless server upgrades "Helmut K. C. Tessarek" <tessarek@evermeet.cx> - 2025-03-07 00:10 +0100
    Re: Improvement of headless server upgrades Holger Levsen <holger@layer-acht.org> - 2025-03-06 09:50 +0100
      Re: Improvement of headless server upgrades Fabio Fantoni <fantonifabio@tiscali.it> - 2025-03-06 11:30 +0100
      Re: Improvement of headless server upgrades Wookey <wookey@wookware.org> - 2025-03-06 15:20 +0100
        Re: Improvement of headless server upgrades Andrey Rakhmatullin <wrar@debian.org> - 2025-03-06 17:40 +0100
        Re: Improvement of headless server upgrades Bjørn Mork <bjorn@mork.no> - 2025-03-06 18:00 +0100
        Re: Improvement of headless server upgrades "Helmut K. C. Tessarek" <tessarek@evermeet.cx> - 2025-03-07 00:00 +0100
          Re: Improvement of headless server upgrades "Helmut K. C. Tessarek" <tessarek@evermeet.cx> - 2025-03-07 00:20 +0100

Page 2 of 2 — ← Prev page 1 [2]

#116210

[Multipart message — attachments visible in raw view] — view raw

On Tue, Mar 04, 2025 at 08:39:43PM -0500, Helmut K. C. Tessarek wrote:
> Both network "outages" could have been prevented by adding a note at the end
> of the dist-upgrade output.

they could also have been prevented by reading the release notes and following
their advice.

that would also have prevented this wonderful bikeshedding thread.

sorry to state the obvious.


-- 
cheers,
	Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

It blows my mind that the people who rage about the spike proteins don't rage
about governments encouraging the spread of self-replicating auto-mutating
aerosolised spike proteins. (@1goodtern)

[toc] | [prev] | [next] | [standalone]

#116222

[Multipart message — attachments visible in raw view] — view raw

Il 06/03/2025 09:43, Holger Levsen ha scritto:
> On Tue, Mar 04, 2025 at 08:39:43PM -0500, Helmut K. C. Tessarek wrote:
>> Both network "outages" could have been prevented by adding a note at the end
>> of the dist-upgrade output.
> they could also have been prevented by reading the release notes and following
> their advice.
>
> that would also have prevented this wonderful bikeshedding thread.
>
> sorry to state the obvious.
>
>
I think the things to do can be summed up as follows: read release note 
of new major version and NEWS entries

If you are upgrading important or critical systems try also:

- test before in a testing system, or vm. or even just start from a less 
critical one

- try to have additional access in case of unforeseen events (there are 
not only the network ones mentioned in this discussion that can prevent 
the boot but also other undocumented and/or specific ones to the system 
or to a part of it). I mean for example an access from the host in case 
of vm or an ipmi system in case of physical server.

- if you don't have much time to read the whole release note read at 
least the "Upgrade specific items" part, for example for buster there is 
"5.1.6. Migrating from legacy network interface names".

- if you don't have time to read NEWS every update do it at least for 
the first time so you can see all the common ones on the basic packages, 
for example however it can be useful to have a quick look at each update 
and read those of specific programs to be aware of important or 
essential changes and save time rather than having problems later, 
perhaps not noticing the problems immediately or not being able to find 
the cause immediately


Regarding possible improvements I think the only thing that can be added 
is a check to the upgrade operations and if it is detected that you are 
upgrading to packages of the next major version add a simple initial 
warning (but not a big "image" as suggested in topic start) in which it 
is recommended to read the release notes, especially the "Upgrade 
specific items" part, and the NEWS of the packages.

[toc] | [prev] | [next] | [standalone]

#116230

[Multipart message — attachments visible in raw view] — view raw

On 2025-03-06 08:43 +0000, Holger Levsen wrote:
> On Tue, Mar 04, 2025 at 08:39:43PM -0500, Helmut K. C. Tessarek wrote:
> > Both network "outages" could have been prevented by adding a note at the end
> > of the dist-upgrade output.
> 
> they could also have been prevented by reading the release notes and following
> their advice.

Would it? Do we know why these things happened? K.C. does not say what
he was upgrading from/to (so it wasn't a very useful report in that
regard), but there has certainly been a long-term expectation that
headless upgrades will work (and in my experience of 25 years now they
always do (well done everyone - I am regularly impressed at how this
usually doesn't break).

Which entry in which release notes will warn that this time
(presumably - was this an upgrade to unstable K.C. or to bookworm, or
something else?) the (pretty old now) eth0 -> 'annoying, unmemorable,
but ordered and unique', renaming will/might actually break your
config? Or that dhcpd will be replaced by network manager in such a
way that things break?  (Is the mechanism here that the server was
running dhcpd to dish out addresses so now in fact the server is
working but other machines are not getting addresses?

I do read the release notes before the first upgrade to a new release,
but I wouldn't be expecting either of the mentioned things to break so
I'm not (yet) convinced that 'RTFM' is a fair response here. I do
vaguely recall that one version of DHCP (isc-dhcp?) was being retired
(did that happen for bookworm?). But normally debian upgrades do not
replace your existing packages, precisely because they might be doing
something important.

I've just looked over the release notes for upgrading to bookworm and
whilst here is loads of good advice about checking for obsolete
packages, noting removals, making backups etc, it is largely generic
and relies on the user knowing what removed packages do. I didn't see
anything specific which warned about the 2 issues noted, and the guide
is pretty long these days, so some skimming the 3rd time you upgrade
is inevitable. So yeah, would RTFM really have avoided these problems?
Maybe, maybe not.

In general I would echo Helmut's response: it is better to work out
why this happenned and try to prevent it, than to add more notices, as
we do have some notice mechanisms already. But they are old, and
expectations change so it's not crazy to ask if they are still sufficient.

But equally, starting with 'it's your fault' seems unhelpful and
possibly unfair. I look forward to some answers in response to Helmut,
and we'll find out if there are real issues to address or not.

Wookey
-- 
Principal hats:  Debian, Wookware
http://wookware.org/

[toc] | [prev] | [next] | [standalone]

#116236

[Multipart message — attachments visible in raw view] — view raw

On Thu, Mar 06, 2025 at 02:13:49PM +0000, Wookey wrote:
>Which entry in which release notes will warn that this time
>(presumably - was this an upgrade to unstable K.C. or to bookworm, or
>something else?) the (pretty old now) eth0 -> 'annoying, unmemorable,
>but ordered and unique', renaming will/might actually break your
>config?

4.1.6 in https://www.debian.org/releases/buster/amd64/release-notes.en.txt
(I cannot link to a HTML version because it doesn't look like it exists
anymore, or I cannot Google it, which is the same thing in our case).

-- 
WBR, wRAR

[toc] | [prev] | [next] | [standalone]

#116237

Wookey <wookey@wookware.org> writes:

> Which entry in which release notes will warn that this time
> (presumably - was this an upgrade to unstable K.C. or to bookworm, or
> something else?) the (pretty old now) eth0 -> 'annoying, unmemorable,
> but ordered and unique', renaming will/might actually break your
> config?

There probably hasn't been a warning since buster, when the release
notes [1] warned:

4.1.6. Verify network interface name support

    Systems upgraded from older releases that still use network
    interfaces with names like eth0 or wlan0 are at risk of losing
    networking once they switch to buster; see Section 5.1.6,
    “Migrating from legacy network interface names” for migration
    instructions.


Maybe all release notes after this should have warned about using
systemd unpredictable interface names on headless systems? Particularily
on systems with a single network interface, where such breakage is
completely unnecessary and "eth0" is guaranteed to be stable from kernel
to kernel.

We should probably also advise users of headless systems with multiple
interfaces that they need to manually configure names for their
interfaces.  Personally I prefer more descriptive names like "uplink",
"backend" etc.  Avoids the systemd breakage and makes network config
easier to read/debug.


Bjørn

[1] - https://www.debian.org/releases/buster/amd64/release-notes.en.txt

[toc] | [prev] | [next] | [standalone]

#116246

[Multipart message — attachments visible in raw view] — view raw

On 2025-03-06 09:13, Wookey wrote:
> Would it? Do we know why these things happened? K.C. does not say what
> he was upgrading from/to (so it wasn't a very useful report in that
> regard), but there has certainly been a long-term expectation that
> headless upgrades will work (and in my experience of 25 years now they
> always do (well done everyone - I am regularly impressed at how this
> usually doesn't break).

It was a Debian Buster on armv7l that I had forgotten, because it was 
running perfectly until now, but apps complained that the OS is no 
longer supported. That upgrade changed the ifname.
(The other issue happened either from bullseye to bookworm or after.)

I have done upgrades from buster to bullseye before (on amd64) and I 
never lost network connectivity even though the network interface name 
changed.
Maybe because those machines already used a different network setup. I 
don't really know. (It's been a while.)
This was the reason I was so startled that I couldn't connect anymore.
Yes, I had done updates where the ifname changed, but sorry that I 
forgot that fact, since it was rather a long time ago (epecially since I 
still had network access in my previous upgradres where that happened).

But I can only agree and I am equally impressed as Wookey how well 
dist-upgrades work.

I maybe should have mentioned that I have done countless updates before 
and never ran into any issues. Or at least I knew beforehand that I had 
to make changes that something like that might not happen.

Please note that I was not trying to blame Debian or package managers or 
anyone really. This was clearly my fault. I started this thread to begin 
a discussion whether there is an option to prevent something like that. 
An option that doesn't require me to read through 3000 lines of text. If 
not, it's fine.
If yes, it might be awesome to do so. That's all.

Cheers,
   K. C.

-- 
regards Helmut K. C. Tessarek              KeyID 0x172380A011EF4944
Key fingerprint = 8A55 70C1 BD85 D34E ADBC 386C 1723 80A0 11EF 4944

/*
    Thou shalt not follow the NULL pointer for chaos and madness
    await thee at its end.
*/

[toc] | [prev] | [next] | [standalone]

#116248

[Multipart message — attachments visible in raw view] — view raw

On 2025-03-06 17:56, Helmut K. C. Tessarek wrote:
> It was a Debian Buster on armv7l that I had forgotten, because it
> was running perfectly until now, but apps complained that the OS is
> no longer supported.

To clarify my previous statement:
The upgrade to bullseye changed the ifname.

Buster used eth0.
After the rebooot, bulsseye used a different ifname.


-- 
regards Helmut K. C. Tessarek              KeyID 0x172380A011EF4944
Key fingerprint = 8A55 70C1 BD85 D34E ADBC 386C 1723 80A0 11EF 4944

/*
    Thou shalt not follow the NULL pointer for chaos and madness
    await thee at its end.
*/

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.debian.devel

csiph-web