Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #176444 > unrolled thread

Systemd: no error but "maintenance mode"

Back to article view | Back to linux.debian.user

Contents

  Systemd: no error but "maintenance mode" Steffen Dettmer <steffen.dettmer@gmail.com> - 2017-01-08 14:50 +0100
    Re: Systemd: no error but "maintenance mode" Joe <joe@jretrading.com> - 2017-01-08 20:00 +0100
    Re: Systemd: no error but "maintenance mode" Joe <joe@jretrading.com> - 2017-01-08 20:10 +0100
      Re: Systemd: no error but "maintenance mode" Steffen Dettmer <steffen.dettmer@gmail.com> - 2017-01-10 20:30 +0100
        Re: Systemd: no error but "maintenance mode" Michael Biebl <email@michaelbiebl.de> - 2017-01-10 21:30 +0100
        Re: Systemd: no error but "maintenance mode" Joe <joe@jretrading.com> - 2017-01-10 21:40 +0100
        Re: Systemd: no error but "maintenance mode" David Wright <deblis@lionunicorn.co.uk> - 2017-01-10 21:50 +0100
          Re: Systemd: no error but "maintenance mode" Joe <joe@jretrading.com> - 2017-01-10 22:50 +0100

#176444 — Systemd: no error but "maintenance mode"

Hi,

What happened before:
I had issue with a Debian server SATA bus [1]. I noticed because
apt-get upgrade hung, because initramfs updater calls "sync" which
hang because of [1]. All operations accessing a certain (backup) disk
blocked. Shutdown over network. It was reported server power LED still
up, so probably shutdown hang, too. Server was powered off and disk
pulled.

Then server did not come up. Now I went to its location with monitor
and keyboard. I saw a screen full of boot messages, each line prefixed
with green "[OK]". On bottom, I got asked for root password. I though
Debian does not use root passwords? I'm old fashioned so fortunately I
set one, so I could log in.

I noticed /var/log/syslog had not been updated recently. Obviously
logging is not running / working. All file systems are correctly
mounted read-write.

What happens? Why is however dropping me only a shell when everything
from booting worked fine?
What is this for a bad idea to spawn a LOCAL shell on a server? Or is
Debian nowadays desktop only? I know ubuntu-server, is there also some
debian-server?

Any hint appreciated.

Steffen



[1]
task kworker/u8:0:27432 blocked for more than 120 seconds
Workqueue: writeback bdi_writeback_workfn (flush-8:48)
[<ffffffff81510d09>] ? io_schedule+0x99/0x120
...

[toc] | [next] | [standalone]

#176462

On Sun, 8 Jan 2017 14:28:09 +0100
Steffen Dettmer <steffen.dettmer@gmail.com> wrote:

> Hi,
> 
> What happened before:
> I had issue with a Debian server SATA bus [1]. I noticed because
> apt-get upgrade hung, because initramfs updater calls "sync" which
> hang because of [1]. All operations accessing a certain (backup) disk
> blocked. Shutdown over network. It was reported server power LED still
> up, so probably shutdown hang, too. Server was powered off and disk
> pulled.
> 
> Then server did not come up. Now I went to its location with monitor
> and keyboard. I saw a screen full of boot messages, each line prefixed
> with green "[OK]". On bottom, I got asked for root password. I though
> Debian does not use root passwords? I'm old fashioned so fortunately I
> set one, so I could log in.
> 
> I noticed /var/log/syslog had not been updated recently. Obviously
> logging is not running / working. All file systems are correctly
> mounted read-write.
> 
> What happens? Why is however dropping me only a shell when everything
> from booting worked fine?
> What is this for a bad idea to spawn a LOCAL shell on a server? Or is
> Debian nowadays desktop only? I know ubuntu-server, is there also some
> debian-server?
> 
> Any hint appreciated.
> 
> Steffen
> 
> 
> 
> [1]
> task kworker/u8:0:27432 blocked for more than 120 seconds
> Workqueue: writeback bdi_writeback_workfn (flush-8:48)
> [<ffffffff81510d09>] ? io_schedule+0x99/0x120
> ...
> 

[toc] | [prev] | [next] | [standalone]

#176464

On Sun, 8 Jan 2017 14:28:09 +0100
Steffen Dettmer <steffen.dettmer@gmail.com> wrote:

> Hi,
> 
> What happened before:
> I had issue with a Debian server SATA bus [1]. I noticed because
> apt-get upgrade hung, because initramfs updater calls "sync" which
> hang because of [1]. All operations accessing a certain (backup) disk
> blocked. Shutdown over network. It was reported server power LED still
> up, so probably shutdown hang, too. Server was powered off and disk
> pulled.

The disc you removed: did it have an entry in /etc/fstab? Does this
server use systemd? If yes to both, comment the /etc/fstab entry or
give it an extra option 'noauto'. Systemd assumes that any local drive
listed in /etc/fstab without 'noauto' is essential for booting, and will
drop you to maintenance mode if the drive cannot be found.
> 
> Then server did not come up. Now I went to its location with monitor
> and keyboard. I saw a screen full of boot messages, each line prefixed
> with green "[OK]". On bottom, I got asked for root password. I though
> Debian does not use root passwords? I'm old fashioned so fortunately I
> set one, so I could log in.
> 

Yes, a plain Debian installation will ask you to set a root password.

> Or is Debian nowadays desktop only? I know ubuntu-server, is there
> also some debian-server?

No, there is no special server variant, Debian Stable would normally be
used, with whatever additional software the user wishes.

-- 
Joe

[toc] | [prev] | [next] | [standalone]

#176566

On Sun, Jan 8, 2017 at 7:45 PM, Joe <joe@jretrading.com> wrote:
>> What happened before:
>> I had issue with a Debian server SATA bus [1]. I noticed because
>> apt-get upgrade hung, because initramfs updater calls "sync" which
>> hang because of [1]. All operations accessing a certain (backup) disk
>> blocked. Shutdown over network. It was reported server power LED still
>> up, so probably shutdown hang, too. Server was powered off and disk
>> pulled.
>
> The disc you removed: did it have an entry in /etc/fstab? Does this
> server use systemd? If yes to both, comment the /etc/fstab entry or
> give it an extra option 'noauto'.

In short: yes, this was the problem. I didn't found this first because
there was no easily visible error message.

Thanks you for your help!

Do you - or anyone - know how to mount a disk at boot without
failing when it is not there? Like normal "classic" behavior?
Of course I can write a simple script if I somehow manage
systemd to execute it. In the old days there was some rc.local,
surely I find something like this. I just wonder what to put
into fstab that script detects as to be mounted without
systemd aborting when it is missing (without patching
systemd of course)

Longer story (aka "what actually happened"):

The disk I removed had no entry. But I put it into
external USB dock, replacing the disk that there was before
(not assuming an USB disk could be required at all).

The removed USB disk had an entry in fstab, but with "noauto".
Yeah, but unfortunately in wrong format
(I think it was "<dev> <mnt> noauto noauto 0 0",
but of course noauto is not a valid file type).

And yes, this was the problem. systemd seems to ignore the
check flag and always checks (if auto or unless noauto).
That's why "fsck -a" worked. But systemd does not run fsck -a
but (I guess) implements an own parser for fstab, then does not
even call fsck (which gives a clear error message when it is called
with an non-existing device), so there was no error message
from fsck either (actually fsck would even pass on the
example, which does not matter, since it was not even started).

Then I was in "maintenance mode" with 0 failed units, because
"fs mount unit" does not count in here. Also on Console all
lines were prefixed with "OK".

> Systemd assumes that any local drive
> listed in /etc/fstab without 'noauto' is essential for booting, and will
> drop you to maintenance mode if the drive cannot be found.

Yes, maintenance mode without network. The network card
was available, but not IP configured (an error message about missing
firmware was also present in journal and I assumed it would be the real
problem, but this message seems to be normal).

So my problem was that systemd makes a wrong assumption,
does not report the error well, does not start fsck which would report
and then go to maintenance mode without without remote access
possibility and finally does not show this error as failed unit or on
console. Only in the journal there was an error together with several
others. These others are "normal" for systemd.

This worked well before systemd when some init script just started
"fsck -a && mount -a" :-)

>> Then server did not come up. Now I went to its location with monitor
>> and keyboard. I saw a screen full of boot messages, each line prefixed
>> with green "[OK]". On bottom, I got asked for root password. I though
>> Debian does not use root passwords? I'm old fashioned so fortunately I
>> set one, so I could log in.
>
> Yes, a plain Debian installation will ask you to set a root password.

Ahh good, so this is safe.

> No, there is no special server variant, Debian Stable would normally be
> used, with whatever additional software the user wishes.

Alright, so seems I chose correctly.

Steffen

[toc] | [prev] | [next] | [standalone]

#176569

[Multipart message — attachments visible in raw view] — view raw

Am 10.01.2017 um 20:08 schrieb Steffen Dettmer:
> On Sun, Jan 8, 2017 at 7:45 PM, Joe <joe@jretrading.com> wrote:
>>> What happened before:
>>> I had issue with a Debian server SATA bus [1]. I noticed because
>>> apt-get upgrade hung, because initramfs updater calls "sync" which
>>> hang because of [1]. All operations accessing a certain (backup) disk
>>> blocked. Shutdown over network. It was reported server power LED still
>>> up, so probably shutdown hang, too. Server was powered off and disk
>>> pulled.
>>
>> The disc you removed: did it have an entry in /etc/fstab? Does this
>> server use systemd? If yes to both, comment the /etc/fstab entry or
>> give it an extra option 'noauto'.
> 
> In short: yes, this was the problem. I didn't found this first because
> there was no easily visible error message.
> 
> Thanks you for your help!
> 
> Do you - or anyone - know how to mount a disk at boot without
> failing when it is not there?

https://www.debian.org/releases/stable/amd64/release-notes/ch-information.en.html#systemd-auto-mounts-incompat


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[toc] | [prev] | [next] | [standalone]

#176570

On Tue, 10 Jan 2017 20:08:10 +0100
Steffen Dettmer <steffen.dettmer@gmail.com> wrote:

> On Sun, Jan 8, 2017 at 7:45 PM, Joe <joe@jretrading.com> wrote:
> >> What happened before:
> >> I had issue with a Debian server SATA bus [1]. I noticed because
> >> apt-get upgrade hung, because initramfs updater calls "sync" which
> >> hang because of [1]. All operations accessing a certain (backup)
> >> disk blocked. Shutdown over network. It was reported server power
> >> LED still up, so probably shutdown hang, too. Server was powered
> >> off and disk pulled.  
> >
> > The disc you removed: did it have an entry in /etc/fstab? Does this
> > server use systemd? If yes to both, comment the /etc/fstab entry or
> > give it an extra option 'noauto'.  
> 
> In short: yes, this was the problem. I didn't found this first because
> there was no easily visible error message.
> 
> Thanks you for your help!
> 
> Do you - or anyone - know how to mount a disk at boot without
> failing when it is not there? Like normal "classic" behavior?
> Of course I can write a simple script if I somehow manage
> systemd to execute it. In the old days there was some rc.local,
> surely I find something like this. I just wonder what to put
> into fstab that script detects as to be mounted without
> systemd aborting when it is missing (without patching
> systemd of course)
> 

I ran into this problem on a system 'upgraded' to systemd-as-init,
because I had a few USB sticks named in /etc/fstab in order that the
nearly-brain-dead usbmount would actually mount them with a consistent
name and place, and mount the partitions rather than the entire drive,
as sometimes happened.

Fortunately, systemd has a rather better automounter, and was able to
mount the stick by name, which usbmount did not. So I could simply
remove the fstab entries and everything worked better than before. I
don't leave anything plugged in when I boot this machine, as I do
sometimes boot from USB, and if I leave a USB storage device plugged
in, the BIOS is not clever enough to realise that there is no bootloader
on it... even a Kindle will kill the boot. But it is possible that if
your BIOS does not try to boot from a pre-plugged USB medium, that
systemd will automount it when it is found during boot. This should do
what you want, if it happens.

Having said that, I have a few network shares which were sometimes
troublesome during boot, and now have 'noauto' in their fstab options.
They *also* have 'x-systemd.automount' options, which mounts them on the
first access attempt to the mount point. I've never needed to try
whether this will work with removable drives, but it might, if systemd
will not automount them (without an fstab entry) during boot.

-- 
Joe

[toc] | [prev] | [next] | [standalone]

#176573

On Tue 10 Jan 2017 at 20:08:10 (+0100), Steffen Dettmer wrote:
> On Sun, Jan 8, 2017 at 7:45 PM, Joe <joe@jretrading.com> wrote:
> >> What happened before:
> >> I had issue with a Debian server SATA bus [1]. I noticed because
> >> apt-get upgrade hung, because initramfs updater calls "sync" which
> >> hang because of [1]. All operations accessing a certain (backup) disk
> >> blocked. Shutdown over network. It was reported server power LED still
> >> up, so probably shutdown hang, too. Server was powered off and disk
> >> pulled.
> >
> > The disc you removed: did it have an entry in /etc/fstab? Does this
> > server use systemd? If yes to both, comment the /etc/fstab entry or
> > give it an extra option 'noauto'.
> 
> In short: yes, this was the problem. I didn't found this first because
> there was no easily visible error message.
> 
> Thanks you for your help!
> 
> Do you - or anyone - know how to mount a disk at boot without
> failing when it is not there? Like normal "classic" behavior?
> Of course I can write a simple script if I somehow manage
> systemd to execute it. In the old days there was some rc.local,
> surely I find something like this. I just wonder what to put
> into fstab that script detects as to be mounted without
> systemd aborting when it is missing (without patching
> systemd of course)

I thought I had pointed it out to you already:

  Similarly, both   man fstab   and   man mount   could benefit from
  more gloss on (no)auto and nofail. Compare the terse "nofail"
  entry with the following one on "relatime" which discusses
  kernel versions.

  How essential it was to read §5.6.1 in jessie's release notes!

§5.6.1 says: 'If it fails to mount an "auto" mount (without the
"nofail" option), systemd will drop to an emergency shell rather than
continuing the boot.

'We recommend that all removable or "optional" mount points
(e.g. non-critical network drives) listed in /etc/fstab either have
the "noauto" or the "nofail" option.'

> The removed USB disk had an entry in fstab, [...]

> (an error message about missing
> firmware was also present in journal and I assumed it would be the real
> problem, but this message seems to be normal).

> So my problem was that systemd makes a wrong assumption,

> This worked well before systemd when some init script just started
> "fsck [...]

I can see that a post-mortem of what went wrong is useful, but I can't
understand why you say that systemd made the "wrong" assumption.
If you specify something wrongly, you can't really expect it to do
what you want it to do, or what sysv chanced to do, just to match your
own assumptions.

Similarly, missing firmware is only "normal" in that so long as you
don't depend on the functionality provided by that firmware, you'll
be all right.

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#176577

On Tue, 10 Jan 2017 14:29:17 -0600
David Wright <deblis@lionunicorn.co.uk> wrote:


> 
> I can see that a post-mortem of what went wrong is useful, but I can't
> understand why you say that systemd made the "wrong" assumption.
> If you specify something wrongly, you can't really expect it to do
> what you want it to do, or what sysv chanced to do, just to match your
> own assumptions.
> 

I think the point here is that systemd introduced new behaviour which
caused a previously-working system to fail to boot in a way which
could not have happened in all the years before systemd was introduced.

This is exactly the kind of thing which gives server admins nightmares:
we can never be sure that we've spotted every possible gotcha that may
be lying in wait for us.

-- 
Joe

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web