Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #201627 > unrolled thread

Heck, why did the server freeze?

Started byAlessandro Vesely <vesely@tana.it>
First post2018-10-25 20:00 +0200
Last post2018-10-26 20:20 +0200
Articles 7 — 4 participants

Back to article view | Back to linux.debian.user

Contents

  Heck, why did the server freeze? Alessandro Vesely <vesely@tana.it> - 2018-10-25 20:00 +0200
    Re: Heck, why did the server freeze? Brian <ad44@cityscape.co.uk> - 2018-10-25 20:40 +0200
      Re: Heck, why did the server freeze? Alessandro Vesely <vesely@tana.it> - 2018-10-26 11:30 +0200
        Re: Heck, why did the server freeze? Reco <recoverym4n@enotuniq.net> - 2018-10-26 11:30 +0200
          Re: Heck, why did the server freeze? Joe <joe@jretrading.com> - 2018-10-26 18:10 +0200
          Re: Heck, why did the server freeze? Alessandro Vesely <vesely@tana.it> - 2018-10-26 19:40 +0200
            Re: Heck, why did the server freeze? Reco <recoverym4n@enotuniq.net> - 2018-10-26 20:20 +0200

#201627 — Heck, why did the server freeze?

FromAlessandro Vesely <vesely@tana.it>
Date2018-10-25 20:00 +0200
SubjectHeck, why did the server freeze?
Message-ID<wMS5k-1PA-15@gated-at.bofh.it>
Hi all,
early this morning a network card burned out.  A few hours later, the server
was not responding on any network address, nor on the system console.  I had to
power it down.

Upon rebooting, network errors were detected an I arranged the server to work
with the available hardware.  The last line logged was an incoming email from a
spammer in Brazil.  It shouldn't have triggered any severe damage.  I found no
breakdown hint in the logs.

My theory is that the system didn't realize that the card was broken, didn't
turn the interface down, and kept storing outgoing stuff until it blew off.  Is
that reasonable or should I be more paranoid?

TIA for any answer

Best
Ale

[toc] | [next] | [standalone]

#201630

FromBrian <ad44@cityscape.co.uk>
Date2018-10-25 20:40 +0200
Message-ID<wMSI1-2hB-1@gated-at.bofh.it>
In reply to#201627
On Thu 25 Oct 2018 at 19:53:26 +0200, Alessandro Vesely wrote:

> Hi all,
> early this morning a network card burned out.  A few hours later, the server
> was not responding on any network address, nor on the system console.  I had to
> power it down.
> 
> Upon rebooting, network errors were detected an I arranged the server to work
> with the available hardware.  The last line logged was an incoming email from a
> spammer in Brazil.  It shouldn't have triggered any severe damage.  I found no
> breakdown hint in the logs.
> 
> My theory is that the system didn't realize that the card was broken, didn't
> turn the interface down, and kept storing outgoing stuff until it blew off.  Is
> that reasonable or should I be more paranoid?

You have given an exact diagnosis of your problem - the network
card failed. What's your problem? Replace it instead of agonising
and theorising.

-- 
Brian.

[toc] | [prev] | [next] | [standalone]

#201669

FromAlessandro Vesely <vesely@tana.it>
Date2018-10-26 11:30 +0200
Message-ID<wN6Bj-2fr-1@gated-at.bofh.it>
In reply to#201630
On Thu 25/Oct/2018 20:30:27 +0200 Brian wrote:
> On Thu 25 Oct 2018 at 19:53:26 +0200, Alessandro Vesely wrote:
> 
>> Hi all,
>> early this morning a network card burned out.  A few hours later, the server
>> was not responding on any network address, nor on the system console.  I had to
>> power it down.
>> 
>> Upon rebooting, network errors were detected an I arranged the server to work
>> with the available hardware.  The last line logged was an incoming email from a
>> spammer in Brazil.  It shouldn't have triggered any severe damage.  I found no
>> breakdown hint in the logs.
>> 
>> My theory is that the system didn't realize that the card was broken, didn't
>> turn the interface down, and kept storing outgoing stuff until it blew off.  Is
>> that reasonable or should I be more paranoid?
> 
> You have given an exact diagnosis of your problem - the network
> card failed. What's your problem? Replace it instead of agonising
> and theorising.

The problem is that the server froze.  I don't think that's what it is supposed
to do when a card fails.

During the reboot, I had a plethora of RTNETLINK answers: Network is down /
Network is unreachable / File exists.  It took me several RJ45 swaps to know
which network and which side of the link were the culprit.

Contrast that with log lines about anything else, from non-redundant power
supplies to failed GPG signatures.  In part, the missing precise diagnosis must
be a shortcoming on part of the card vendor.  However, how come the kernel
didn't realize that the link had to go down, log something, and just fail any
subsequent call on that interface, instead of freezing?  Or did it freeze for
an unrelated reason?

Best
Ale
-- 

P.S. New card expected on Monday.

[toc] | [prev] | [next] | [standalone]

#201670

FromReco <recoverym4n@enotuniq.net>
Date2018-10-26 11:30 +0200
Message-ID<wN6Bk-2fr-7@gated-at.bofh.it>
In reply to#201669
	Hi.

On Fri, Oct 26, 2018 at 11:23:39AM +0200, Alessandro Vesely wrote:
> On Thu 25/Oct/2018 20:30:27 +0200 Brian wrote:
> > On Thu 25 Oct 2018 at 19:53:26 +0200, Alessandro Vesely wrote:
> > 
> >> Hi all,
> >> early this morning a network card burned out.  A few hours later, the server
> >> was not responding on any network address, nor on the system console.  I had to
> >> power it down.
> >> 
> >> Upon rebooting, network errors were detected an I arranged the server to work
> >> with the available hardware.  The last line logged was an incoming email from a
> >> spammer in Brazil.  It shouldn't have triggered any severe damage.  I found no
> >> breakdown hint in the logs.
> >> 
> >> My theory is that the system didn't realize that the card was broken, didn't
> >> turn the interface down, and kept storing outgoing stuff until it blew off.  Is
> >> that reasonable or should I be more paranoid?
> > 
> > You have given an exact diagnosis of your problem - the network
> > card failed. What's your problem? Replace it instead of agonising
> > and theorising.
> 
> The problem is that the server froze.  I don't think that's what it is supposed
> to do when a card fails.

It's my impression too.

> Contrast that with log lines about anything else, from non-redundant power
> supplies to failed GPG signatures.  In part, the missing precise diagnosis must
> be a shortcoming on part of the card vendor.  However, how come the kernel
> didn't realize that the link had to go down, log something, and just fail any
> subsequent call on that interface, instead of freezing?  Or did it freeze for
> an unrelated reason?

I believe that it's impossible to answer this question. It's highly
likely that it was kernel panic. Whenever it was related to failed NIC,
or no - it's impossible to tell since there's no kernel backtrace.
I'd install, say, kdump-tools for the future incidents like this.

Reco

[toc] | [prev] | [next] | [standalone]

#201693

FromJoe <joe@jretrading.com>
Date2018-10-26 18:10 +0200
Message-ID<wNcQp-61w-1@gated-at.bofh.it>
In reply to#201670
On Fri, 26 Oct 2018 12:27:36 +0300
Reco <recoverym4n@enotuniq.net> wrote:

> 	Hi.
> 

> > 
> > The problem is that the server froze.  I don't think that's what it
> > is supposed to do when a card fails.  
> 
> It's my impression too.
> 

A bad peripheral can cause almost any problem. If the bit of the card
connecting with the outside world fails, it probably won't affect the
rest of the computer. If a bit involving e.g. the PCI bus fails, it may
well stop other stuff from working, maybe the CPU itself. To be fair,
it's usually the outside bit that dies, often because of the fact that
it's connected to other equipment.

I had a dead PC a week ago. No PSU fan, even. Reseating the PATA
cable leading to the optical drives fixed it... and I've seen, at least
twice, a bad optical drive preventing start-up.

-- 
Joe

[toc] | [prev] | [next] | [standalone]

#201697

FromAlessandro Vesely <vesely@tana.it>
Date2018-10-26 19:40 +0200
Message-ID<wNefw-6Js-3@gated-at.bofh.it>
In reply to#201670
On Fri 26/Oct/2018 11:27:36 +0200 Reco wrote:
> On Fri, Oct 26, 2018 at 11:23:39AM +0200, Alessandro Vesely wrote:
>> The problem is that the server froze.  I don't think that's what it is supposed
>> to do when a card fails.
> 
> It's my impression too.

In general, it is too difficult to know if a link is good, at least on the
local side.  I found nothing better than running pings by cron.


>> Contrast that with log lines about anything else, from non-redundant power
>> supplies to failed GPG signatures.  In part, the missing precise diagnosis must
>> be a shortcoming on part of the card vendor.  However, how come the kernel
>> didn't realize that the link had to go down, log something, and just fail any
>> subsequent call on that interface, instead of freezing?  Or did it freeze for
>> an unrelated reason?
> 
> I believe that it's impossible to answer this question. It's highly
> likely that it was kernel panic. Whenever it was related to failed NIC,
> or no - it's impossible to tell since there's no kernel backtrace.

Right.  I should have tried Ctrl-Alt-F1 or some of the SysRq hacks[*], but I
was too upset by services not responding...

[*] https://www.kernel.org/doc/html/latest/admin-guide/sysrq.html


> I'd install, say, kdump-tools for the future incidents like this.

Just installed, thank you!  (I'll reboot when the new card arrives).

Best
Ale
--

[toc] | [prev] | [next] | [standalone]

#201703

FromReco <recoverym4n@enotuniq.net>
Date2018-10-26 20:20 +0200
Message-ID<wNeSe-7bh-11@gated-at.bofh.it>
In reply to#201697
On Fri, Oct 26, 2018 at 07:38:56PM +0200, Alessandro Vesely wrote:
> On Fri 26/Oct/2018 11:27:36 +0200 Reco wrote:
> > On Fri, Oct 26, 2018 at 11:23:39AM +0200, Alessandro Vesely wrote:
> >> The problem is that the server froze.  I don't think that's what it is supposed
> >> to do when a card fails.
> > 
> > It's my impression too.
> 
> In general, it is too difficult to know if a link is good, at least on the
> local side.  I found nothing better than running pings by cron.

lldpd, there's no substitutes for that in a single L2 segment.
Taking WAN into account, pings seem the most reasonable way.

Reco

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web