Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.bugs.dist > #1013105 > unrolled thread

Bug#962454: Link failures after upgrade to +deb10u1

Started byAlberto Gonzalez Iniesta <agi@inittab.org>
First post2020-06-08 12:40 +0200
Last post2020-07-04 14:00 +0200
Articles 4 — 3 participants

Back to article view | Back to linux.debian.bugs.dist

Contents

  Bug#962454: Link failures after upgrade to +deb10u1 Alberto Gonzalez Iniesta <agi@inittab.org> - 2020-06-08 12:40 +0200
    Bug#962454: Link failures after upgrade to +deb10u1 Valentin Vidic <vvidic@debian.org> - 2020-06-11 11:10 +0200
      Bug#962454: Link failures after upgrade to +deb10u1 Alberto Gonzalez Iniesta <agi@inittab.org> - 2020-06-11 14:30 +0200
    Bug#962454: Link failures after upgrade to +deb10u1 wferi@niif.hu - 2020-07-04 14:00 +0200

#1013105 — Bug#962454: Link failures after upgrade to +deb10u1

FromAlberto Gonzalez Iniesta <agi@inittab.org>
Date2020-06-08 12:40 +0200
SubjectBug#962454: Link failures after upgrade to +deb10u1
Message-ID<Afnma-3vR-27@gated-at.bofh.it>
Source: corosync
Version: 3.0.1-2+deb10u1
Severity: important

Hi,

Some weeks ago I upgraded corosync (3.0.1-2 -> 3.0.1-2+deb10u1) and
started to notice these messages in my nodes (two node cluster):
Jun  2 01:10:13 patty corosync[2346]:   [KNET  ] link: host: 2 link: 0 is down
Jun  2 01:10:13 patty corosync[2346]:   [KNET  ] host: host: 2 (passive) best link: 1 (pri: 1)
Jun  2 01:10:14 patty corosync[2346]:   [KNET  ] rx: host: 2 link: 0 is up
Jun  2 01:10:14 patty corosync[2346]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
Jun  3 03:11:07 patty corosync[2346]:   [KNET  ] link: host: 2 link: 1 is down
Jun  3 03:11:07 patty corosync[2346]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
Jun  3 03:11:08 patty corosync[2346]:   [KNET  ] rx: host: 2 link: 1 is up
Jun  3 03:11:08 patty corosync[2346]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)

Notice the failure happens on with both links.  One of the links is a
cross-over cable. The other uses a bond with two interfaces.

These errors are more common on one of the nodes that on the other.

Some times they match (both nodes log the link failure), but most of the
time only one node complains:

Jun  4 01:16:23 selma corosync[52890]:   [KNET  ] link: host: 1 link: 0 is down
Jun  4 01:16:23 selma corosync[52890]:   [KNET  ] host: host: 1 (passive) best link: 1 (pri: 1)
Jun  4 01:16:24 selma corosync[52890]:   [KNET  ] rx: host: 1 link: 0 is up
Jun  4 01:16:24 selma corosync[52890]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1)
Jun  4 01:16:55 patty corosync[2346]:   [KNET  ] link: host: 2 link: 0 is down
Jun  4 01:16:55 patty corosync[2346]:   [KNET  ] host: host: 2 (passive) best link: 1 (pri: 1)
Jun  4 01:16:56 patty corosync[2346]:   [KNET  ] rx: host: 2 link: 0 is up
Jun  4 01:16:56 patty corosync[2346]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)

Here's my config:
totem {
        version: 2
        cluster_name: web
        crypto_cipher: none
        crypto_hash: none
        interface {
                linknumber: 0
        }
        interface {
                linknumber: 1
        }
}
logging {
        fileline: off
        to_stderr: yes
        to_logfile: yes
        logfile: /var/log/corosync/corosync.log
        to_syslog: yes
        debug: off
        logger_subsys {
                subsys: QUORUM
                debug: off
        }
}
quorum {
        provider: corosync_votequorum
        expected_votes: 2
        two_node: 1
}
nodelist {
        node {
                name: patty
                nodeid: 1
                ring0_addr: 192.168.144.1
                ring1_addr: 10.10.1.5
        }
        node {
                name: selma
                nodeid: 2
                ring0_addr: 192.168.144.2
                ring1_addr: 10.10.1.6
        }
}


Any help is appreciated. Thanks,

Alberto


-- System Information:
Debian Release: bullseye/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (500, 'testing'), (500, 'stable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 5.6.0-1-amd64 (SMP w/4 CPU cores)
Kernel taint flags: TAINT_FIRMWARE_WORKAROUND
Locale: LANG=C.UTF-8, LC_CTYPE=C.UTF-8 (charmap=UTF-8), LANGUAGE= (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

[toc] | [next] | [standalone]

#1013488

FromValentin Vidic <vvidic@debian.org>
Date2020-06-11 11:10 +0200
Message-ID<AgrnI-1TC-7@gated-at.bofh.it>
In reply to#1013105
On Mon, Jun 08, 2020 at 12:29:35PM +0200, Alberto Gonzalez Iniesta wrote:
> Some weeks ago I upgraded corosync (3.0.1-2 -> 3.0.1-2+deb10u1) and
> started to notice these messages in my nodes (two node cluster):
> Jun  2 01:10:13 patty corosync[2346]:   [KNET  ] link: host: 2 link: 0 is down
> Jun  2 01:10:13 patty corosync[2346]:   [KNET  ] host: host: 2 (passive) best link: 1 (pri: 1)
> Jun  2 01:10:14 patty corosync[2346]:   [KNET  ] rx: host: 2 link: 0 is up
> Jun  2 01:10:14 patty corosync[2346]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
> Jun  3 03:11:07 patty corosync[2346]:   [KNET  ] link: host: 2 link: 1 is down
> Jun  3 03:11:07 patty corosync[2346]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
> Jun  3 03:11:08 patty corosync[2346]:   [KNET  ] rx: host: 2 link: 1 is up
> Jun  3 03:11:08 patty corosync[2346]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)

Hi, can you confirm that downgrading to the previous version solves the
link problem for you?

-- 
Valentin

[toc] | [prev] | [next] | [standalone]

#1013506

FromAlberto Gonzalez Iniesta <agi@inittab.org>
Date2020-06-11 14:30 +0200
Message-ID<Aguvf-3IW-11@gated-at.bofh.it>
In reply to#1013488
On Thu, Jun 11, 2020 at 10:59:06AM +0200, Valentin Vidic wrote:
> On Mon, Jun 08, 2020 at 12:29:35PM +0200, Alberto Gonzalez Iniesta wrote:
> > Some weeks ago I upgraded corosync (3.0.1-2 -> 3.0.1-2+deb10u1) and
> > started to notice these messages in my nodes (two node cluster):
> > Jun  2 01:10:13 patty corosync[2346]:   [KNET  ] link: host: 2 link: 0 is down
> > Jun  2 01:10:13 patty corosync[2346]:   [KNET  ] host: host: 2 (passive) best link: 1 (pri: 1)
> > Jun  2 01:10:14 patty corosync[2346]:   [KNET  ] rx: host: 2 link: 0 is up
> > Jun  2 01:10:14 patty corosync[2346]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
> > Jun  3 03:11:07 patty corosync[2346]:   [KNET  ] link: host: 2 link: 1 is down
> > Jun  3 03:11:07 patty corosync[2346]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
> > Jun  3 03:11:08 patty corosync[2346]:   [KNET  ] rx: host: 2 link: 1 is up
> > Jun  3 03:11:08 patty corosync[2346]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
> 
> Hi, can you confirm that downgrading to the previous version solves the
> link problem for you?
> 

Hi, I'll try that this weekend and keep you updated.

Thanks,
Alberto

-- 
Alberto Gonzalez Iniesta    | Formación, consultoría y soporte técnico
mailto/sip: agi@inittab.org | en GNU/Linux y software libre
Encrypted mail preferred    | http://inittab.com

Key fingerprint = 5347 CBD8 3E30 A9EB 4D7D  4BF2 009B 3375 6B9A AA55

[toc] | [prev] | [next] | [standalone]

#1016455

Fromwferi@niif.hu
Date2020-07-04 14:00 +0200
Message-ID<AoOZP-SI-5@gated-at.bofh.it>
In reply to#1013105
Alberto Gonzalez Iniesta <agi@inittab.org> writes:

> Hi, again. After some days with libknet1=1.16-2, I'm still getting
> errors from time to time (less than with buster's libknet1):
> Jun 22 01:16:54 selma corosync[28610]:   [KNET  ] link: host: 1 link: 0 is down
> Jun 22 01:16:54 selma corosync[28610]:   [KNET  ] host: host: 1 (passive) best link: 1 (pri: 1)
> Jun 22 01:16:55 selma corosync[28610]:   [KNET  ] rx: host: 1 link: 0 is up
> Jun 22 01:16:55 selma corosync[28610]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1)
> Jun 24 06:50:30 selma corosync[28610]:   [KNET  ] link: host: 1 link: 0 is down
> Jun 24 06:50:30 selma corosync[28610]:   [KNET  ] host: host: 1 (passive) best link: 1 (pri: 1)
> Jun 24 06:50:31 selma corosync[28610]:   [KNET  ] rx: host: 1 link: 0 is up
> Jun 24 06:50:31 selma corosync[28610]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1)

Hi,

Is this with corosync 3.0.1-2+deb10u1?
Please enable debug logging and collect full info for a link flap.
Do these correlate with some cluster or host activity?
Does your network link transfer big packets (up to 65536 bytes) reliably?
Can you capture the Kronosnet network traffic (preferably on both nodes)
around the time of a link flap?

I won't be able to interpret such detailed information, but I suggest
you open an issue at https://github.com/kronosnet/kronosnet/issues
complete with it.
-- 
Regards,
Feri

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.bugs.dist

csiph-web