Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #54768 > unrolled thread

Bug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian

Started byIngo Jürgensmann <ij@2013.bluespice.org>
First post2016-07-25 14:30 +0200
Last post2016-12-01 16:10 +0100
Articles 14 — 4 participants

Back to article view | Back to linux.debian.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Bug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian Ingo Jürgensmann <ij@2013.bluespice.org> - 2016-07-25 14:30 +0200
    Bug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian Wei Liu <wei.liu2@citrix.com> - 2016-07-25 15:30 +0200
      Bug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian Andreas Ziegler <ml@andreas-ziegler.de> - 2016-07-29 14:00 +0200
        Bug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian Wei Liu <wei.liu2@citrix.com> - 2016-07-29 15:10 +0200
        Bug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian Wei Liu <wei.liu2@citrix.com> - 2016-08-02 11:30 +0200
          Bug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian Ingo Jürgensmann <ij@2013.bluespice.org> - 2016-08-02 12:40 +0200
            Bug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian Wei Liu <wei.liu2@citrix.com> - 2016-08-02 14:50 +0200
              Bug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian Andreas Ziegler <ml@andreas-ziegler.de> - 2016-11-14 17:00 +0100
                Bug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian Wei Liu <wei.liu2@citrix.com> - 2016-11-29 10:30 +0100
                  Bug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian Ingo Jürgensmann <ij@2013.bluespice.org> - 2016-11-29 22:10 +0100
                    Bug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian Wei Liu <wei.liu2@citrix.com> - 2016-12-01 14:40 +0100
                      Bug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian Wei Liu <wei.liu2@citrix.com> - 2016-12-01 15:10 +0100
                      Bug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian Ingo Jürgensmann <ij@2013.bluespice.org> - 2016-12-01 15:40 +0100
                        Bug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian Sander Eikelenboom <linux@eikelenboom.it> - 2016-12-01 16:10 +0100

#54768 — Bug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian

FromIngo Jürgensmann <ij@2013.bluespice.org>
Date2016-07-25 14:30 +0200
SubjectBug#804079: [Xen-devel] Kernel panic on Xen virtualisation in Debian
Message-ID<rYNbb-4vR-1@gated-at.bofh.it>
On 25.07.2016 12:23, Wei Liu wrote:

First, thank you for replying! Very much appreciated! :)

> I did skim your emails. But the oops was happening in memcpy+0x6 which
> indicated it came back to the origin question why would it got an
> exception there.
> 
> Just by staring at the code doesn't get me anywhere. Without a concrete
> reproduction of the issue, I'm afraid I can't provide more input here.

Well, from my point of view, it happens quite often when accessing the 
server via SSH. For example today it crashed when I wanted to add 
something and after I clicked into putty and typed the first char. In 
another putty, where I have my netconsole log open, I instantly saw the 
oops.

But what exactly causing these kinds of reboots, I'm clueless as you 
too. Only that I do experience far more frequent crashes when accessing 
the server from workplace via putty on Windows than via SSH on OSX or 
Debian Linux.

> There are several moving parts:
> 0. Hardware
> 1. Xen hypervisor
> 2. Dom0 kernel
> Your report and the debian report suggested that Dom0 kernel is less
> likely to be the culprit because you've tried different Dom0 kernels.

As just written in the other mail, I already tried kernel 4.5 from 
backports. Still crashing.

> As for Xen, not sure if you would be up for trying a debug build from
> source tree. That would help provide information on whether this is a
> bug in Xen or not.

Will try to build from Debian source, but how to enable debug build?

-- 
Ciao...          //        http://blog.windfluechter.net
       Ingo     \X/     XMPP: ij@jabber.windfluechter.net


gpg pubkey:  http://www.juergensmann.de/ij_public_key.asc

[toc] | [next] | [standalone]

#54769

FromWei Liu <wei.liu2@citrix.com>
Date2016-07-25 15:30 +0200
Message-ID<rYO7f-54B-1@gated-at.bofh.it>
In reply to#54768
On Mon, Jul 25, 2016 at 01:41:41PM +0200, Ingo Jürgensmann wrote:
> On 25.07.2016 12:23, Wei Liu wrote:
> 
> First, thank you for replying! Very much appreciated! :)
> 
> >I did skim your emails. But the oops was happening in memcpy+0x6 which
> >indicated it came back to the origin question why would it got an
> >exception there.
> >
> >Just by staring at the code doesn't get me anywhere. Without a concrete
> >reproduction of the issue, I'm afraid I can't provide more input here.
> 
> Well, from my point of view, it happens quite often when accessing the
> server via SSH. For example today it crashed when I wanted to add something
> and after I clicked into putty and typed the first char. In another putty,
> where I have my netconsole log open, I instantly saw the oops.
> 
> But what exactly causing these kinds of reboots, I'm clueless as you too.
> Only that I do experience far more frequent crashes when accessing the
> server from workplace via putty on Windows than via SSH on OSX or Debian
> Linux.
> 
> >There are several moving parts:
> >0. Hardware
> >1. Xen hypervisor
> >2. Dom0 kernel
> >Your report and the debian report suggested that Dom0 kernel is less
> >likely to be the culprit because you've tried different Dom0 kernels.
> 
> As just written in the other mail, I already tried kernel 4.5 from
> backports. Still crashing.
> 
> >As for Xen, not sure if you would be up for trying a debug build from
> >source tree. That would help provide information on whether this is a
> >bug in Xen or not.
> 
> Will try to build from Debian source, but how to enable debug build?
> 

I was thinking about building directly from xen.git.

http://wiki.xenproject.org/wiki/Compiling_Xen_From_Source

Probably try the Xen 4.7 release.

Wei.

> -- 
> Ciao...          //        http://blog.windfluechter.net
>       Ingo     \X/     XMPP: ij@jabber.windfluechter.net
> 
> 
> gpg pubkey:  http://www.juergensmann.de/ij_public_key.asc

[toc] | [prev] | [next] | [standalone]

#54784

FromAndreas Ziegler <ml@andreas-ziegler.de>
Date2016-07-29 14:00 +0200
Message-ID<s0eCl-495-13@gated-at.bofh.it>
In reply to#54769
Hello Wei,

we tried with kernel 4.6 now, the crashed happened again, though.

next we want to try the Xen debug build, but we couldn't find any
information on how to enable debug for the build, perhaps you could give
us a hint.

- Andreas


-------- Original-Nachricht --------
Betreff: Re: [Xen-devel] Kernel panic on Xen virtualisation in Debian
Von: Wei Liu <wei.liu2@citrix.com>
An: Ingo Jürgensmann <ij@2013.bluespice.org>
Datum: 25.7.2016, 15:13:06

> On Mon, Jul 25, 2016 at 01:41:41PM +0200, Ingo Jürgensmann wrote:
>> On 25.07.2016 12:23, Wei Liu wrote:
>>
>> First, thank you for replying! Very much appreciated! :)
>>
>>> I did skim your emails. But the oops was happening in memcpy+0x6 which
>>> indicated it came back to the origin question why would it got an
>>> exception there.
>>>
>>> Just by staring at the code doesn't get me anywhere. Without a concrete
>>> reproduction of the issue, I'm afraid I can't provide more input here.
>>
>> Well, from my point of view, it happens quite often when accessing the
>> server via SSH. For example today it crashed when I wanted to add something
>> and after I clicked into putty and typed the first char. In another putty,
>> where I have my netconsole log open, I instantly saw the oops.
>>
>> But what exactly causing these kinds of reboots, I'm clueless as you too.
>> Only that I do experience far more frequent crashes when accessing the
>> server from workplace via putty on Windows than via SSH on OSX or Debian
>> Linux.
>>
>>> There are several moving parts:
>>> 0. Hardware
>>> 1. Xen hypervisor
>>> 2. Dom0 kernel
>>> Your report and the debian report suggested that Dom0 kernel is less
>>> likely to be the culprit because you've tried different Dom0 kernels.
>>
>> As just written in the other mail, I already tried kernel 4.5 from
>> backports. Still crashing.
>>
>>> As for Xen, not sure if you would be up for trying a debug build from
>>> source tree. That would help provide information on whether this is a
>>> bug in Xen or not.
>>
>> Will try to build from Debian source, but how to enable debug build?
>>
> 
> I was thinking about building directly from xen.git.
> 
> http://wiki.xenproject.org/wiki/Compiling_Xen_From_Source
> 
> Probably try the Xen 4.7 release.
> 
> Wei.
> 
>> -- 
>> Ciao...          //        http://blog.windfluechter.net
>>       Ingo     \X/     XMPP: ij@jabber.windfluechter.net
>>
>>
>> gpg pubkey:  http://www.juergensmann.de/ij_public_key.asc

[toc] | [prev] | [next] | [standalone]

#54785

FromWei Liu <wei.liu2@citrix.com>
Date2016-07-29 15:10 +0200
Message-ID<s0fI5-51U-1@gated-at.bofh.it>
In reply to#54784
On Fri, Jul 29, 2016 at 01:45:55PM +0200, Andreas Ziegler wrote:
> Hello Wei,
> 
> we tried with kernel 4.6 now, the crashed happened again, though.
> 
> next we want to try the Xen debug build, but we couldn't find any
> information on how to enable debug for the build, perhaps you could give
> us a hint.
> 

First please read:

http://wiki.xenproject.org/wiki/Compiling_Xen_From_Source

for general information.

Then in the cloned xen.git tree -- assuming you use Xen 4.7 (maybe 4.6 as
well)

  make -C xen menuconfig

to get a Linux style menuconfig experience

If you're using earlier version of Xen, you need to modify
xen.git/Config.mk to set the debug variable.

Feel free to ask if you have more questions.

Wei.

[toc] | [prev] | [next] | [standalone]

#54813

FromWei Liu <wei.liu2@citrix.com>
Date2016-08-02 11:30 +0200
Message-ID<s1Ebn-2xS-13@gated-at.bofh.it>
In reply to#54784
On Fri, Jul 29, 2016 at 10:17:22PM +0200, Ingo Jürgensmann wrote:
> Am 29.07.2016 um 13:45 schrieb Andreas Ziegler <ml@andreas-ziegler.de>:
> 
> > we tried with kernel 4.6 now, the crashed happened again, though.
> 
> Please find attached the netconsole log of 3 from 4 crashes so far today… 


What is also interesting is that you seem to be running some sort of
ip accounting software (pmacctd) which also segfault'ed.

Still not sure what to make of that though.

Wei.

> 
> -- 
> Ciao...          //        http://blog.windfluechter.net
>       Ingo     \X/     XMPP: ij@jabber.windfluechter.net
> 	
> gpg pubkey:  http://www.juergensmann.de/ij_public_key.asc
> 
> 
>

[toc] | [prev] | [next] | [standalone]

#54814

FromIngo Jürgensmann <ij@2013.bluespice.org>
Date2016-08-02 12:40 +0200
Message-ID<s1Fhc-3ao-3@gated-at.bofh.it>
In reply to#54813
On 02.08.2016 11:20, Wei Liu wrote:
> On Fri, Jul 29, 2016 at 10:17:22PM +0200, Ingo Jürgensmann wrote:
> What is also interesting is that you seem to be running some sort of
> ip accounting software (pmacctd) which also segfault'ed.

Yeah, it is segfaulting, because the database (in a domU VM) where it is 
storing the accounting is not yet available after the crash. When 
database is up&running, those segfaults go away.

> Still not sure what to make of that though.

Me neither. ;-)

I already tried to get a core dump by setting ulimit -c unlimited, but 
that didn't work as well, which makes me believe that the crash happens 
in hypervisor not in dom0 kernel. When it's dom0 kernel I would expect 
dumping a core file should work.

-- 
Ciao...          //        http://blog.windfluechter.net
       Ingo     \X/     XMPP: ij@jabber.windfluechter.net


gpg pubkey:  http://www.juergensmann.de/ij_public_key.asc

[toc] | [prev] | [next] | [standalone]

#54815

FromWei Liu <wei.liu2@citrix.com>
Date2016-08-02 14:50 +0200
Message-ID<s1HiV-4uK-5@gated-at.bofh.it>
In reply to#54814
On Tue, Aug 02, 2016 at 12:30:30PM +0200, Ingo Jürgensmann wrote:
> On 02.08.2016 11:20, Wei Liu wrote:
> >On Fri, Jul 29, 2016 at 10:17:22PM +0200, Ingo Jürgensmann wrote:
> >What is also interesting is that you seem to be running some sort of
> >ip accounting software (pmacctd) which also segfault'ed.
> 
> Yeah, it is segfaulting, because the database (in a domU VM) where it is
> storing the accounting is not yet available after the crash. When database
> is up&running, those segfaults go away.
> 

At least we can now rule out that it is not related to the issue you
reported.

> >Still not sure what to make of that though.
> 
> Me neither. ;-)
> 
> I already tried to get a core dump by setting ulimit -c unlimited, but that
> didn't work as well, which makes me believe that the crash happens in
> hypervisor not in dom0 kernel. When it's dom0 kernel I would expect dumping
> a core file should work.
> 

We can't draw the conclusion that the crash is in hypervisor yet. If
your dom0 crash, hypervisor would normally decide to reboot the machine.

Wei.

> -- 
> Ciao...          //        http://blog.windfluechter.net
>       Ingo     \X/     XMPP: ij@jabber.windfluechter.net
> 
> 
> gpg pubkey:  http://www.juergensmann.de/ij_public_key.asc

[toc] | [prev] | [next] | [standalone]

#55732

FromAndreas Ziegler <ml@andreas-ziegler.de>
Date2016-11-14 17:00 +0100
Message-ID<sDrPQ-sM-5@gated-at.bofh.it>
In reply to#54815
Hi,

few months later Ingo decided again to give it a try as he really
doesn't want to keep ipv6 disabled in 2016.
He tried Xen 4.8 - which didn't help, the crash reappeared.
He then managed to build Xen with debug=y and soon it crashed with the
following output, which looks a little bit longer than without debug:

http://paste.debian.net/895464/

If this still doesn't help, we would really appreciate more information
on how to do proper debugging, the information we found online is either
very old, confusing - or it's hidden very good?

Andreas.


-------- Original-Nachricht --------
Betreff: Re: [Xen-devel] Kernel panic on Xen virtualisation in Debian
Von: Wei Liu <wei.liu2@citrix.com>
An: Ingo Jürgensmann <ij@2013.bluespice.org>
Datum: 2.8.2016, 14:37:58

> On Tue, Aug 02, 2016 at 12:30:30PM +0200, Ingo Jürgensmann wrote:
>> On 02.08.2016 11:20, Wei Liu wrote:
>>> On Fri, Jul 29, 2016 at 10:17:22PM +0200, Ingo Jürgensmann wrote:
>>> What is also interesting is that you seem to be running some sort of
>>> ip accounting software (pmacctd) which also segfault'ed.
>>
>> Yeah, it is segfaulting, because the database (in a domU VM) where it is
>> storing the accounting is not yet available after the crash. When database
>> is up&running, those segfaults go away.
>>
> 
> At least we can now rule out that it is not related to the issue you
> reported.
> 
>>> Still not sure what to make of that though.
>>
>> Me neither. ;-)
>>
>> I already tried to get a core dump by setting ulimit -c unlimited, but that
>> didn't work as well, which makes me believe that the crash happens in
>> hypervisor not in dom0 kernel. When it's dom0 kernel I would expect dumping
>> a core file should work.
>>
> 
> We can't draw the conclusion that the crash is in hypervisor yet. If
> your dom0 crash, hypervisor would normally decide to reboot the machine.
> 
> Wei.
>

[toc] | [prev] | [next] | [standalone]

#55886

FromWei Liu <wei.liu2@citrix.com>
Date2016-11-29 10:30 +0100
Message-ID<sIMTD-6Co-17@gated-at.bofh.it>
In reply to#55732
On Mon, Nov 14, 2016 at 04:55:40PM +0100, Andreas Ziegler wrote:
> Hi,
> 
> few months later Ingo decided again to give it a try as he really
> doesn't want to keep ipv6 disabled in 2016.
> He tried Xen 4.8 - which didn't help, the crash reappeared.
> He then managed to build Xen with debug=y and soon it crashed with the
> following output, which looks a little bit longer than without debug:
> 
> http://paste.debian.net/895464/
> 

Entry not found -- maybe it expired... Sorry.

> If this still doesn't help, we would really appreciate more information
> on how to do proper debugging, the information we found online is either
> very old, confusing - or it's hidden very good?
> 

What sort of information did you find ? What do you need ?

Wei.

[toc] | [prev] | [next] | [standalone]

#55900

FromIngo Jürgensmann <ij@2013.bluespice.org>
Date2016-11-29 22:10 +0100
Message-ID<sIXP4-5gZ-27@gated-at.bofh.it>
In reply to#55886
Am 29.11.2016 um 10:08 schrieb Wei Liu <wei.liu2@citrix.com>:
>> http://paste.debian.net/895464/
> Entry not found -- maybe it expired... Sorry.

Here it is: 

Nov 14 09:19:52 31.172.31.251 [39677.027813] BUG: unable to handle kernel 
Nov 14 09:19:52 31.172.31.251  at ffff880002b4c06e
Nov 14 09:19:52 31.172.31.251 [39677.027868] IP:
Nov 14 09:19:52 31.172.31.251  [<ffffffff8158b3cc>] ndisc_send_redirect+0x44c/0x4d0
Nov 14 09:19:52 31.172.31.251 [39677.027902] PGD 1a07067 
Nov 14 09:19:52 31.172.31.251  
Nov 14 09:19:52 31.172.31.251 [39677.027934] Oops: 0000 [#1] 
Nov 14 09:19:52 31.172.31.251  
Nov 14 09:19:52 31.172.31.251 [39677.027956] Modules linked in:
Nov 14 09:19:52 31.172.31.251  authenc(E)
Nov 14 09:19:52 31.172.31.251  echainiv(E)
Nov 14 09:19:52 31.172.31.251  xfrm6_mode_tunnel(E)
Nov 14 09:19:52 31.172.31.251  xfrm4_mode_tunnel(E)
Nov 14 09:19:52 31.172.31.251  xt_physdev(E)
Nov 14 09:19:52 31.172.31.251  br_netfilter(E)
Nov 14 09:19:52 31.172.31.251  xen_netback(E)
Nov 14 09:19:52 31.172.31.251  tun(E)
Nov 14 09:19:52 31.172.31.251  xen_blkback(E)
Nov 14 09:19:52 31.172.31.251  xt_multiport(E)
Nov 14 09:19:52 31.172.31.251  twofish_generic(E)
Nov 14 09:19:52 31.172.31.251  twofish_avx_x86_64(E)
Nov 14 09:19:52 31.172.31.251  twofish_x86_64_3way(E)
Nov 14 09:19:52 31.172.31.251  twofish_x86_64(E)
Nov 14 09:19:52 31.172.31.251  twofish_common(E)
Nov 14 09:19:52 31.172.31.251  serpent_avx_x86_64(E)
Nov 14 09:19:52 31.172.31.251  serpent_sse2_x86_64(E)
Nov 14 09:19:52 31.172.31.251  serpent_generic(E)
Nov 14 09:19:52 31.172.31.251  blowfish_generic(E)
Nov 14 09:19:52 31.172.31.251  blowfish_x86_64(E)
Nov 14 09:19:52 31.172.31.251  blowfish_common(E)
Nov 14 09:19:52 31.172.31.251  cast5_avx_x86_64(E)
Nov 14 09:19:52 31.172.31.251  cast5_generic(E)
Nov 14 09:19:52 31.172.31.251  cast_common(E)
Nov 14 09:19:52 31.172.31.251  ctr(E)
Nov 14 09:19:52 31.172.31.251  des_generic(E)
Nov 14 09:19:52 31.172.31.251  cbc(E)
Nov 14 09:19:52 31.172.31.251  algif_skcipher(E)
Nov 14 09:19:52 31.172.31.251  camellia_generic(E)
Nov 14 09:19:52 31.172.31.251  camellia_aesni_avx_x86_64(E)
Nov 14 09:19:52 31.172.31.251  camellia_x86_64(E)
Nov 14 09:19:52 31.172.31.251  xts(E)
Nov 14 09:19:52 31.172.31.251  xcbc(E)
Nov 14 09:19:52 31.172.31.251  sha512_ssse3(E)
Nov 14 09:19:52 31.172.31.251  sha512_generic(E)
Nov 14 09:19:52 31.172.31.251  md4(E)
Nov 14 09:19:52 31.172.31.251  algif_hash(E)
Nov 14 09:19:52 31.172.31.251  af_alg(E)
Nov 14 09:19:52 31.172.31.251  xfrm_user(E)
Nov 14 09:19:52 31.172.31.251  xfrm4_tunnel(E)
Nov 14 09:19:52 31.172.31.251  tunnel4(E)
Nov 14 09:19:52 31.172.31.251  ipcomp(E)
Nov 14 09:19:52 31.172.31.251  xfrm_ipcomp(E)
Nov 14 09:19:52 31.172.31.251  esp4(E)
Nov 14 09:19:52 31.172.31.251  xen_gntdev(E)
Nov 14 09:19:52 31.172.31.251  xen_evtchn(E)
Nov 14 09:19:52 31.172.31.251  ah4(E)
Nov 14 09:19:52 31.172.31.251  xenfs(E)
Nov 14 09:19:52 31.172.31.251  af_key(E)
Nov 14 09:19:52 31.172.31.251  xfrm_algo(E)
Nov 14 09:19:52 31.172.31.251  xen_privcmd(E)
Nov 14 09:19:52 31.172.31.251  ipmi_poweroff(E)
Nov 14 09:19:52 31.172.31.251  video(E)
Nov 14 09:19:52 31.172.31.251  thermal(E)
Nov 14 09:19:52 31.172.31.251  fan(E)
Nov 14 09:19:52 31.172.31.251  ac(E)
Nov 14 09:19:52 31.172.31.251  battery(E)
Nov 14 09:19:52 31.172.31.251  ip6t_REJECT(E)
Nov 14 09:19:52 31.172.31.251  nf_reject_ipv6(E)
Nov 14 09:19:52 31.172.31.251  ip6table_filter(E)
Nov 14 09:19:52 31.172.31.251  ip6_tables(E)
Nov 14 09:19:52 31.172.31.251  ipt_REJECT(E)
Nov 14 09:19:52 31.172.31.251  nf_reject_ipv4(E)
Nov 14 09:19:52 31.172.31.251  xt_tcpudp(E)
Nov 14 09:19:52 31.172.31.251  iptable_filter(E)
Nov 14 09:19:52 31.172.31.251  ip_tables(E)
Nov 14 09:19:52 31.172.31.251  x_tables(E)
Nov 14 09:19:52 31.172.31.251  bridge(E)
Nov 14 09:19:52 31.172.31.251  stp(E)
Nov 14 09:19:52 31.172.31.251  llc(E)
Nov 14 09:19:52 31.172.31.251  ext4(E)
Nov 14 09:19:52 31.172.31.251  ecb(E)
Nov 14 09:19:52 31.172.31.251  crc16(E)
Nov 14 09:19:52 31.172.31.251  jbd2(E)
Nov 14 09:19:52 31.172.31.251  crc32c_generic(E)
Nov 14 09:19:52 31.172.31.251  mbcache(E)
Nov 14 09:19:52 31.172.31.251  fuse(E)
Nov 14 09:19:52 31.172.31.251  ipmi_devintf(E)
Nov 14 09:19:52 31.172.31.251  ipmi_watchdog(E)
Nov 14 09:19:52 31.172.31.251  w83627ehf(E)
Nov 14 09:19:52 31.172.31.251  hwmon_vid(E)
Nov 14 09:19:52 31.172.31.251  nf_conntrack_ipv4(E)
Nov 14 09:19:52 31.172.31.251  nf_defrag_ipv4(E)
Nov 14 09:19:52 31.172.31.251  nf_conntrack(E)
Nov 14 09:19:52 31.172.31.251  loop(E)
Nov 14 09:19:52 31.172.31.251  x86_pkg_temp_thermal(E)
Nov 14 09:19:52 31.172.31.251  intel_powerclamp(E)
Nov 14 09:19:52 31.172.31.251  coretemp(E)
Nov 14 09:19:52 31.172.31.251  crct10dif_pclmul(E)
Nov 14 09:19:52 31.172.31.251  crc32_pclmul(E)
Nov 14 09:19:52 31.172.31.251  ghash_clmulni_intel(E)
Nov 14 09:19:52 31.172.31.251  hmac(E)
Nov 14 09:19:52 31.172.31.251  drbg(E)
Nov 14 09:19:52 31.172.31.251  ansi_cprng(E)
Nov 14 09:19:52 31.172.31.251  aesni_intel(E)
Nov 14 09:19:52 31.172.31.251  aes_x86_64(E)
Nov 14 09:19:52 31.172.31.251  iTCO_wdt(E)
Nov 14 09:19:52 31.172.31.251  iTCO_vendor_support(E)
Nov 14 09:19:52 31.172.31.251  joydev(E)
Nov 14 09:19:52 31.172.31.251  evdev(E)
Nov 14 09:19:52 31.172.31.251  lrw(E)
Nov 14 09:19:52 31.172.31.251  pcspkr(E)
Nov 14 09:19:52 31.172.31.251  serio_raw(E)
Nov 14 09:19:52 31.172.31.251  gf128mul(E)
Nov 14 09:19:52 31.172.31.251  glue_helper(E)
Nov 14 09:19:52 31.172.31.251  mgag200(E)
Nov 14 09:19:52 31.172.31.251  ttm(E)
Nov 14 09:19:52 31.172.31.251  drm_kms_helper(E)
Nov 14 09:19:52 31.172.31.251  drm(E)
Nov 14 09:19:52 31.172.31.251  ablk_helper(E)
Nov 14 09:19:52 31.172.31.251  cryptd(E)
Nov 14 09:19:52 31.172.31.251  sb_edac(E)
Nov 14 09:19:52 31.172.31.251  8250_fintek(E)
Nov 14 09:19:52 31.172.31.251  edac_core(E)
Nov 14 09:19:52 31.172.31.251  wmi(E)
Nov 14 09:19:52 31.172.31.251  ipmi_si(E)
Nov 14 09:19:52 31.172.31.251  ipmi_msghandler(E)
Nov 14 09:19:52 31.172.31.251  tpm_tis(E)
Nov 14 09:19:52 31.172.31.251  tpm(E)
Nov 14 09:19:52 31.172.31.251  shpchp(E)
Nov 14 09:19:52 31.172.31.251  i2c_i801(E)
Nov 14 09:19:52 31.172.31.251  ioatdma(E)
Nov 14 09:19:52 31.172.31.251  mei_me(E)
Nov 14 09:19:52 31.172.31.251  mei(E)
Nov 14 09:19:52 31.172.31.251  lpc_ich(E)
Nov 14 09:19:52 31.172.31.251  mfd_core(E)
Nov 14 09:19:52 31.172.31.251  processor(E)
Nov 14 09:19:52 31.172.31.251  button(E)
Nov 14 09:19:52 31.172.31.251  xfs(E)
Nov 14 09:19:52 31.172.31.251  libcrc32c(E)
Nov 14 09:19:52 31.172.31.251  dm_mod(E)
Nov 14 09:19:52 31.172.31.251  raid10(E)
Nov 14 09:19:52 31.172.31.251  raid1(E)
Nov 14 09:19:52 31.172.31.251  md_mod(E)
Nov 14 09:19:52 31.172.31.251  uas(E)
Nov 14 09:19:52 31.172.31.251  usb_storage(E)
Nov 14 09:19:52 31.172.31.251  hid_generic(E)
Nov 14 09:19:52 31.172.31.251  usbhid(E)
Nov 14 09:19:52 31.172.31.251  hid(E)
Nov 14 09:19:52 31.172.31.251  netconsole(E)
Nov 14 09:19:52 31.172.31.251  configfs(E)
Nov 14 09:19:52 31.172.31.251  sg(E)
Nov 14 09:19:52 31.172.31.251  sd_mod(E)
Nov 14 09:19:52 31.172.31.251  crc32c_intel(E)
Nov 14 09:19:52 31.172.31.251  ahci(E)
Nov 14 09:19:52 31.172.31.251  psmouse(E)
Nov 14 09:19:52 31.172.31.251  libahci(E)
Nov 14 09:19:52 31.172.31.251  ehci_pci(E)
Nov 14 09:19:52 31.172.31.251  ehci_hcd(E)
Nov 14 09:19:52 31.172.31.251  libata(E)
Nov 14 09:19:52 31.172.31.251  scsi_mod(E)
Nov 14 09:19:52 31.172.31.251  usbcore(E)
Nov 14 09:19:52 31.172.31.251  igb(E)
Nov 14 09:19:52 31.172.31.251  usb_common(E)
Nov 14 09:19:52 31.172.31.251  i2c_algo_bit(E)
Nov 14 09:19:52 31.172.31.251  dca(E)
Nov 14 09:19:52 31.172.31.251  ptp(E)
Nov 14 09:19:52 31.172.31.251  pps_core(E)
Nov 14 09:19:52 31.172.31.251  fjes(E)
Nov 14 09:19:52 31.172.31.251  
Nov 14 09:19:52 31.172.31.251 [39677.028967] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W   E   4.6.0-0.bpo.1-amd64 #1 Debian 4.6.4-1~bpo8+1
Nov 14 09:19:52 31.172.31.251 [39677.029009] Hardware name: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 3.0a 01/03/2014
Nov 14 09:19:52 31.172.31.251 [39677.029050] task: ffffffff81a0d540 ti: ffffffff81a00000 task.ti: ffffffff81a00000
Nov 14 09:19:52 31.172.31.251 [39677.029159] RIP: e030:[<ffffffff8158b3cc>] 
Nov 14 09:19:52 31.172.31.251  [<ffffffff8158b3cc>] ndisc_send_redirect+0x44c/0x4d0
Nov 14 09:19:52 31.172.31.251 [39677.029196] RSP: e02b:ffff880281e038a0  EFLAGS: 00010202
Nov 14 09:19:52 31.172.31.251 [39677.029217] RAX: ffff88027b335c68 RBX: ffff880019e4c500 RCX: ffff88027b335c70
Nov 14 09:19:52 31.172.31.251 [39677.029246] RDX: 0000000000000228 RSI: ffff880002b4be4e RDI: 4006000200000060
Nov 14 09:19:52 31.172.31.251 [39677.029273] RBP: ffff880281e03990 R08: 00000000000002c0 R09: 0802294600a7012a
Nov 14 09:19:52 31.172.31.251 [39677.029300] R10: 0100000000000000 R11: 00000000e89ec242 R12: ffff88021da85800
Nov 14 09:19:52 31.172.31.251 [39677.029326] R13: ffff88027a66d240 R14: 0000000000000230 R15: ffff880002b4be66
Nov 14 09:19:52 31.172.31.251 [39677.029359] FS:  0000000000000000(0000) GS:ffff880281e00000(0000) knlGS:ffff880281e00000
Nov 14 09:19:52 31.172.31.251 [39677.029389] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 14 09:19:52 31.172.31.251 [39677.029411] CR2: ffff880002b4c06e CR3: 000000027af49000 CR4: 0000000000042660
Nov 14 09:19:52 31.172.31.251 [39677.029438] Stack:
Nov 14 09:19:52 31.172.31.251 [39677.029448]  000000000000000a
Nov 14 09:19:52 31.172.31.251  ffff88019d84e600
Nov 14 09:19:52 31.172.31.251  ffffffff00000000
Nov 14 09:19:52 31.172.31.251  ffff880281e03938
Nov 14 09:19:52 31.172.31.251  
Nov 14 09:19:52 31.172.31.251 [39677.029483]  00000000000080fe
Nov 14 09:19:52 31.172.31.251  1ad902feff7ac40e
Nov 14 09:19:52 31.172.31.251  9501294600a7012a
Nov 14 09:19:52 31.172.31.251  0200000000000000
Nov 14 09:19:52 31.172.31.251  
Nov 14 09:19:52 31.172.31.251 [39677.029518]  ffff88027af3000a
Nov 14 09:19:52 31.172.31.251  ffff880002a90b40
Nov 14 09:19:52 31.172.31.251  0000000100000004
Nov 14 09:19:52 31.172.31.251  003a000000000000
Nov 14 09:19:52 31.172.31.251  
Nov 14 09:19:52 31.172.31.251 [39677.029553] Call Trace:
Nov 14 09:19:52 31.172.31.251 [39677.029564]  <IRQ> 
Nov 14 09:19:52 31.172.31.251  
Nov 14 09:19:52 31.172.31.251 [39677.029580]  [<ffffffff81571d5c>] ? ip6_forward+0x8dc/0x8f0
Nov 14 09:19:52 31.172.31.251 [39677.029604]  [<ffffffff81581bc0>] ? ip6_pol_route+0x720/0x720
Nov 14 09:19:52 31.172.31.251 [39677.029628]  [<ffffffff81573950>] ? ipv6_rcv+0x350/0x510
Nov 14 09:19:52 31.172.31.251 [39677.029650]  [<ffffffff81573160>] ? ip6_make_skb+0x1e0/0x1e0
Nov 14 09:19:52 31.172.31.251 [39677.029674]  [<ffffffff814c886e>] ? __netif_receive_skb_core+0x2be/0xa50
Nov 14 09:19:52 31.172.31.251 [39677.030866]  [<ffffffffc0685d05>] ? br_dev_queue_push_xmit+0x75/0x130 [bridge]
Nov 14 09:19:52 31.172.31.251 [39677.032058]  [<ffffffffc0685dfd>] ? br_forward_finish+0x3d/0xc0 [bridge]
Nov 14 09:19:52 31.172.31.251 [39677.033248]  [<ffffffff814ffe07>] ? nf_register_hooks+0x27/0x80
Nov 14 09:19:52 31.172.31.251 [39677.034426]  [<ffffffff814c908f>] ? netif_receive_skb_internal+0x2f/0xa0
Nov 14 09:19:52 31.172.31.251 [39677.035594]  [<ffffffffc06873f6>] ? br_pass_frame_up+0xa6/0x150 [bridge]
Nov 14 09:19:52 31.172.31.251 [39677.036761]  [<ffffffffc0687340>] ? br_port_flags_change+0x20/0x20 [bridge]
Nov 14 09:19:52 31.172.31.251 [39677.037977]  [<ffffffffc0687687>] ? br_handle_frame_finish+0x1e7/0x4f0 [bridge]
Nov 14 09:19:52 31.172.31.251 [39677.039122]  [<ffffffff814ffeb4>] ? nf_iterate+0x54/0x70
Nov 14 09:19:52 31.172.31.251 [39677.040244]  [<ffffffffc0687bb3>] ? br_handle_frame+0x173/0x2d0 [bridge]
Nov 14 09:19:52 31.172.31.251 [39677.041350]  [<ffffffffc06874a0>] ? br_pass_frame_up+0x150/0x150 [bridge]
Nov 14 09:19:52 31.172.31.251 [39677.042424]  [<ffffffff814c88fb>] ? __netif_receive_skb_core+0x34b/0xa50
Nov 14 09:19:52 31.172.31.251 [39677.043486]  [<ffffffff81023e44>] ? set_phys_to_machine+0x14/0x40
Nov 14 09:19:52 31.172.31.251 [39677.044520]  [<ffffffff814c908f>] ? netif_receive_skb_internal+0x2f/0xa0
Nov 14 09:19:52 31.172.31.251 [39677.045536]  [<ffffffffc084ac5d>] ? xenvif_tx_action+0x6cd/0x890 [xen_netback]
Nov 14 09:19:52 31.172.31.251 [39677.046543]  [<ffffffffc004ea65>] ? igb_poll+0x865/0xe80 [igb]
Nov 14 09:19:52 31.172.31.251 [39677.047523]  [<ffffffffc084d9f8>] ? xenvif_poll+0x28/0x70 [xen_netback]
Nov 14 09:19:52 31.172.31.251 [39677.048524]  [<ffffffff814c9828>] ? net_rx_action+0x238/0x370
Nov 14 09:19:52 31.172.31.251 [39677.049486]  [<ffffffff815cb046>] ? __do_softirq+0x106/0x294
Nov 14 09:19:52 31.172.31.251 [39677.050394]  [<ffffffff810809b6>] ? irq_exit+0x86/0x90
Nov 14 09:19:52 31.172.31.251 [39677.051269]  [<ffffffff813dc1d1>] ? xen_evtchn_do_upcall+0x31/0x40
Nov 14 09:19:52 31.172.31.251 [39677.052125]  [<ffffffff815ca16e>] ? xen_do_hypervisor_callback+0x1e/0x40
Nov 14 09:19:52 31.172.31.251 [39677.052955]  <EOI> 
Nov 14 09:19:52 31.172.31.251  
Nov 14 09:19:52 31.172.31.251 [39677.052969]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
Nov 14 09:19:52 31.172.31.251 [39677.054544]  [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
Nov 14 09:19:52 31.172.31.251 [39677.055307]  [<ffffffff8102202c>] ? xen_safe_halt+0xc/0x20
Nov 14 09:19:52 31.172.31.251 [39677.056044]  [<ffffffff81036ad8>] ? default_idle+0x18/0xc0
Nov 14 09:19:52 31.172.31.251 [39677.056761]  [<ffffffff810bc0b7>] ? cpu_startup_entry+0x307/0x360
Nov 14 09:19:52 31.172.31.251 [39677.057464]  [<ffffffff81b3cf8d>] ? start_kernel+0x459/0x479
Nov 14 09:19:52 31.172.31.251 [39677.058210]  [<ffffffff81b42be0>] ? xen_start_kernel+0x522/0x52c
Nov 14 09:19:52 31.172.31.251 [39677.058926] Code: 
Nov 14 09:19:52 31.172.31.251  
Nov 14 09:19:52 31.172.31.251  
Nov 14 09:19:52 31.172.31.251 [39677.060558] RIP 
Nov 14 09:19:52 31.172.31.251  [<ffffffff8158b3cc>] ndisc_send_redirect+0x44c/0x4d0
Nov 14 09:19:52 31.172.31.251 [39677.061286]  RSP <ffff880281e038a0>
Nov 14 09:19:52 31.172.31.251 [39677.062003] CR2: ffff880002b4c06e
Nov 14 09:19:52 31.172.31.251 [39677.066328] ---[ end trace 98db42bf1a114a3d ]---
Nov 14 09:19:52 31.172.31.251 [39677.070151] Kernel panic - not syncing: Fatal exception in interrupt
Nov 14 09:19:52 31.172.31.251 [39677.070892] Kernel Offset: disabled

If the log entries of the starting-up server is of interest as well, please shout! ;-) 

-- 
Ciao...          //        http://blog.windfluechter.net
      Ingo     \X/     XMPP: ij@jabber.windfluechter.net
	
gpg pubkey:  http://www.juergensmann.de/ij_public_key.asc

[toc] | [prev] | [next] | [standalone]

#55914

FromWei Liu <wei.liu2@citrix.com>
Date2016-12-01 14:40 +0100
Message-ID<sJzKF-4qw-3@gated-at.bofh.it>
In reply to#55900
On Tue, Nov 29, 2016 at 09:20:55PM +0100, Ingo Jürgensmann wrote:
> Am 29.11.2016 um 10:08 schrieb Wei Liu <wei.liu2@citrix.com>:
> >> http://paste.debian.net/895464/
> > Entry not found -- maybe it expired... Sorry.
> 
> Here it is: 
> 

This is still the same kernel log that was sent some time ago.

So, if you have built Xen with debug=y, could you try to set Xen log
level to the highest and capture "xl dmesg" when guest crashes?

But I think this is increasingly likely to be a Linux kernel issue
because you've tried multiple versions of xen. Maybe it is time to try
different versions of Dom0 kernels (sorry if you've tried that, I can't
remember all the details over so many moons).

Wei.

[toc] | [prev] | [next] | [standalone]

#55916

FromWei Liu <wei.liu2@citrix.com>
Date2016-12-01 15:10 +0100
Message-ID<sJAdI-4Rh-15@gated-at.bofh.it>
In reply to#55914
On Thu, Dec 01, 2016 at 02:59:36PM +0100, Ingo Jürgensmann wrote:
> On 01.12.2016 14:26, Wei Liu wrote:
> 
> >This is still the same kernel log that was sent some time ago.
> >So, if you have built Xen with debug=y, could you try to set Xen log
> >level to the highest and capture "xl dmesg" when guest crashes?
> 
> It's not the guest that crashes, it's dom0. So when the host crashes, I'm
> not able to issue any commands anymore.
> 

Oh, sorry for speaking non-sense. In that case you will need to setup a
serial console to capture output on the fly.

Wei.

[toc] | [prev] | [next] | [standalone]

#55919

FromIngo Jürgensmann <ij@2013.bluespice.org>
Date2016-12-01 15:40 +0100
Message-ID<sJAdI-4Rh-17@gated-at.bofh.it>
In reply to#55914
On 01.12.2016 14:26, Wei Liu wrote:

> This is still the same kernel log that was sent some time ago.
> So, if you have built Xen with debug=y, could you try to set Xen log
> level to the highest and capture "xl dmesg" when guest crashes?

It's not the guest that crashes, it's dom0. So when the host crashes, 
I'm not able to issue any commands anymore.

> But I think this is increasingly likely to be a Linux kernel issue
> because you've tried multiple versions of xen. Maybe it is time to try
> different versions of Dom0 kernels (sorry if you've tried that, I can't
> remember all the details over so many moons).

Yes, indeed I have tried different kernels, but I can't remember details 
as well... ;/

-- 
Ciao...          //        http://blog.windfluechter.net
       Ingo     \X/     XMPP: ij@jabber.windfluechter.net


gpg pubkey:  http://www.juergensmann.de/ij_public_key.asc

[toc] | [prev] | [next] | [standalone]

#55920

FromSander Eikelenboom <linux@eikelenboom.it>
Date2016-12-01 16:10 +0100
Message-ID<sJB9M-5sv-21@gated-at.bofh.it>
In reply to#55919
Thursday, December 1, 2016, 2:59:36 PM, you wrote:

> On 01.12.2016 14:26, Wei Liu wrote:

>> This is still the same kernel log that was sent some time ago.
>> So, if you have built Xen with debug=y, could you try to set Xen log
>> level to the highest and capture "xl dmesg" when guest crashes?

> It's not the guest that crashes, it's dom0. So when the host crashes, 
> I'm not able to issue any commands anymore.

>> But I think this is increasingly likely to be a Linux kernel issue
>> because you've tried multiple versions of xen. Maybe it is time to try
>> different versions of Dom0 kernels (sorry if you've tried that, I can't
>> remember all the details over so many moons).

> Yes, indeed I have tried different kernels, but I can't remember details 
> as well... ;/


Hi Ingo,

Have you tried without enabling "ndisc" (QoS) and "ipv6" ?
They are both present in your log and i assume you are using a bridged network config ?
You wouldn't be the first to stumble over a more generic kernel network bug
while using Xen, due to less well tested combinations.
So it's worth testing if plain ipv4 and no QoS works.

--
Sander

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web