Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #59152 > unrolled thread

Bug#878242: linux-image-4.12.0-2-marvell: flood of messages "mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment"

Started byBen Hutchings <ben@decadent.org.uk>
First post2017-10-13 00:10 +0200
Last post2017-10-13 00:30 +0200
Articles 4 — 2 participants

Back to article view | Back to linux.debian.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Bug#878242: linux-image-4.12.0-2-marvell: flood of messages "mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment" Ben Hutchings <ben@decadent.org.uk> - 2017-10-13 00:10 +0200
    Bug#878242: linux-image-4.12.0-2-marvell: flood of messages "mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment" Ben Hutchings <ben@decadent.org.uk> - 2017-10-13 00:20 +0200
      Bug#878242: linux-image-4.12.0-2-marvell: flood of messages "mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment" Rogério Brito <rbrito@ime.usp.br> - 2017-10-19 20:50 +0200
    Bug#878242: linux-image-4.12.0-2-marvell: flood of messages "mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment" Rogério Brito <rbrito@ime.usp.br> - 2017-10-13 00:30 +0200

#59152 — Bug#878242: linux-image-4.12.0-2-marvell: flood of messages "mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment"

FromBen Hutchings <ben@decadent.org.uk>
Date2017-10-13 00:10 +0200
SubjectBug#878242: linux-image-4.12.0-2-marvell: flood of messages "mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment"
Message-ID<uzTPZ-2RZ-43@gated-at.bofh.it>

[Multipart message — attachments visible in raw view] — view raw

On Wed, 2017-10-11 at 11:48 -0300, rbrito@ime.usp.br wrote:
> Package: src:linux
> Version: 4.12.13-1
> Severity: normal
> 
> 
> Hi.
> 
> I am using an armel system with only 128MB of RAM (a KuroBox Pro) running
> Debian testing and, if my memory serves me well, I started seeing lots and
> lots of messages like:
> 
>     mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment
> 
> on my dmesg after I upgraded to kernel 4.12. I don't know if this is a
> serious problem or only a cosmetic issue.
[...]

This message and the code that generates it have been present in the
mv643xx_eth driver for a long time.  The network interface hardware is
generally capable of transmitting a packet that comes from multiple
buffers in memory, but in some cases the driver copies all the data
into a single buffer, probably to avoid a hardware bug.  If there is
not enough memory to make that copy then it logs this message.

I think the driver should drop the packet if this happens, but instead
it just reports that the hardware is busy.  That will cause the kernel
to try sending the packet again repeatedly, resulting in repetition of
this log message until there is more memory available.

If you haven't seen this error message before, it might be triggered by
an application change.  But I think it's more likely to be caused by
higher memory usage or fragmentation of memory.

Ben.

-- 
Ben Hutchings
Man invented language to satisfy his deep need to complain. - Lily
Tomlin

[toc] | [next] | [standalone]

#59158

FromBen Hutchings <ben@decadent.org.uk>
Date2017-10-13 00:20 +0200
Message-ID<uzTZT-34a-443@gated-at.bofh.it>
In reply to#59152

[Multipart message — attachments visible in raw view] — view raw

Control: tag -1 moreinfo

On Wed, 2017-10-11 at 18:09 -0300, Rogério Brito wrote:
> Hi, Ben.
> 
> On Oct 11 2017, Ben Hutchings wrote:
> > On Wed, 2017-10-11 at 11:48 -0300, rbrito@ime.usp.br wrote:
> > > I am using an armel system with only 128MB of RAM (a KuroBox Pro) running
> > > Debian testing and, if my memory serves me well, I started seeing lots and
> > > lots of messages like:
> > > 
> > >     mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment
> > > 
> > > on my dmesg after I upgraded to kernel 4.12. I don't know if this is a
> > > serious problem or only a cosmetic issue.
> > 
> > [...]
> > 
> > This message and the code that generates it have been present in the
> > mv643xx_eth driver for a long time.  The network interface hardware is
> > generally capable of transmitting a packet that comes from multiple
> > buffers in memory, but in some cases the driver copies all the data
> > into a single buffer, probably to avoid a hardware bug.  If there is
> > not enough memory to make that copy then it logs this message.
> 
> Oh, thanks for the explanation.
> 
> > I think the driver should drop the packet if this happens, but instead
> > it just reports that the hardware is busy.  That will cause the kernel
> > to try sending the packet again repeatedly, resulting in repetition of
> > this log message until there is more memory available.
> 
> Hummm, that may be related to some lower performance that I have felt (but
> not measured).
> 
> I see. Discarding the packet would flag congestion with TCP, if I understand
> it correctly.
> 
> > If you haven't seen this error message before, it might be triggered by an
> > application change.  But I think it's more likely to be caused by higher
> > memory usage or fragmentation of memory.
> 
> Yes, I think that the problem is that the programs are getting bigger and
> bigger.  Would setting vm.min_free_kbytes to a higher value worth it?

It might help, depending on how large the packets get.  This driver
supports TCP Segmentation Offload which means it can be asked to send
'packets' of up to 64K (which are split into multiple packets on the
wire).  It is very unlikely that it will be able to allocate a large
enough buffer to do that.  Whether it will ever get a packet that large
depends on many factors.

Now that I look at it, though, that hardware bug already has to be
handled separately for TSO so maybe the full packet copy should only be
done for normal packets.

> Anyway, I just upgraded to kernel 4.13 and I will report what I see.

OK.  I think we can take this upstream after that.

Ben.

-- 
Ben Hutchings
Man invented language to satisfy his deep need to complain. - Lily
Tomlin

[toc] | [prev] | [next] | [standalone]

#59237

FromRogério Brito <rbrito@ime.usp.br>
Date2017-10-19 20:50 +0200
Message-ID<uCo3f-1MQ-21@gated-at.bofh.it>
In reply to#59158
Hi, Ben.

On Oct 12 2017, Ben Hutchings wrote:
> On Wed, 2017-10-11 at 18:09 -0300, Rogério Brito wrote:
> > Yes, I think that the problem is that the programs are getting bigger and
> > bigger.  Would setting vm.min_free_kbytes to a higher value worth it?
> 
> It might help, depending on how large the packets get.  This driver
> supports TCP Segmentation Offload which means it can be asked to send
> 'packets' of up to 64K (which are split into multiple packets on the
> wire).  It is very unlikely that it will be able to allocate a large
> enough buffer to do that.  Whether it will ever get a packet that large
> depends on many factors.

OK, I have upped the value there and it, apparently, didn't make any difference.

> Now that I look at it, though, that hardware bug already has to be
> handled separately for TSO so maybe the full packet copy should only be
> done for normal packets.

Right, I tried to make sure to, indeed, have more memory than 64KB for sure
(or for copies of packets of that size and still allowing for some
overhead).

> > Anyway, I just upgraded to kernel 4.13 and I will report what I see.
> 
> OK.  I think we can take this upstream after that.

Well, I tried with kernel 4.13 and it still sends the same messages.  The
funny thing is that it stopped doing that after something happened, which I
don't quite remember, but I have, now:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(...)
[93296.373362] mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment
[93296.373403] mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment
[93296.373443] mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment
[93296.373483] mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment
[93296.373524] mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment
[93296.373564] mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment
[93296.373604] mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment
[93296.373645] mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment
[93296.373685] mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment
[93296.373725] mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment
[93296.373770] mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment
[93296.373810] mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment
[93296.373893] mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment
[93296.373935] mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment
[93296.383150] mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment
[93296.383336] mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment
[93296.385329] mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment
[239021.470850] NFSD: client 192.168.15.10 testing state ID with incorrect client ID
[239499.067232] NFSD: client 192.168.15.10 testing state ID with incorrect client ID
[248613.887573] NFSD: client 192.168.15.10 testing state ID with incorrect client ID
[249576.969259] NFSD: client 192.168.15.10 testing state ID with incorrect client ID
[253176.329239] NFSD: client 192.168.15.10 testing state ID with incorrect client ID
[399416.742201] systemd[1]: Stopping Journal Service...
[399416.853859] systemd-journald[129]: Received SIGTERM from PID 1 (systemd).
[399417.268457] systemd[1]: Stopped Journal Service.
[399417.306314] systemd[1]: Starting Journal Service...
[399417.636328] systemd[1]: Started Journal Service.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

The changes there regarding systemd were done by a package upgrade of
systemd to the version that migrated recently to testing (but the messages
stopped earlier than that). Those are the last messages that I have on my
dmesg log, BTW.

The version readable with human timestamps is:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(...)
[Thu Oct 12 19:43:38 2017] mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment
[Thu Oct 12 19:43:38 2017] mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment
[Thu Oct 12 19:43:38 2017] mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment
[Thu Oct 12 19:43:38 2017] mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment
[Thu Oct 12 19:43:38 2017] mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment
[Thu Oct 12 19:43:38 2017] mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment
[Thu Oct 12 19:43:38 2017] mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment
[Thu Oct 12 19:43:38 2017] mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment
[Thu Oct 12 19:43:38 2017] mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment
[Sat Oct 14 12:12:23 2017] NFSD: client 192.168.15.10 testing state ID with incorrect client ID
[Sat Oct 14 12:20:21 2017] NFSD: client 192.168.15.10 testing state ID with incorrect client ID
[Sat Oct 14 14:52:15 2017] NFSD: client 192.168.15.10 testing state ID with incorrect client ID
[Sat Oct 14 15:08:18 2017] NFSD: client 192.168.15.10 testing state ID with incorrect client ID
[Sat Oct 14 16:08:18 2017] NFSD: client 192.168.15.10 testing state ID with incorrect client ID
[Mon Oct 16 09:45:38 2017] systemd[1]: Stopping Journal Service...
[Mon Oct 16 09:45:38 2017] systemd-journald[129]: Received SIGTERM from PID 1 (systemd).
[Mon Oct 16 09:45:39 2017] systemd[1]: Stopped Journal Service.
[Mon Oct 16 09:45:39 2017] systemd[1]: Starting Journal Service...
[Mon Oct 16 09:45:39 2017] systemd[1]: Started Journal Service.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

I will try to disable the use of zswap.enabled=1 on the kernel command line
(even though I don't remember these problems happening before, while still
using that option with older kernels).

Anyway, it won't hurt to debug things a little more.


Thanks for your attention,

Rogério.

-- 
Rogério Brito : rbrito@{ime.usp.br,gmail.com} : GPG key 4096R/BCFCAAAA
http://cynic.cc/blog/ : github.com/rbrito : profiles.google.com/rbrito
DebianQA: http://qa.debian.org/developer.php?login=rbrito%40ime.usp.br

[toc] | [prev] | [next] | [standalone]

#59167

FromRogério Brito <rbrito@ime.usp.br>
Date2017-10-13 00:30 +0200
Message-ID<uzU01-34a-581@gated-at.bofh.it>
In reply to#59152
Hi, Ben.

On Oct 11 2017, Ben Hutchings wrote:
> On Wed, 2017-10-11 at 11:48 -0300, rbrito@ime.usp.br wrote:
> > I am using an armel system with only 128MB of RAM (a KuroBox Pro) running
> > Debian testing and, if my memory serves me well, I started seeing lots and
> > lots of messages like:
> > 
> >     mv643xx_eth_port mv643xx_eth_port.0 eth0: failed to linearize skb with tiny unaligned fragment
> > 
> > on my dmesg after I upgraded to kernel 4.12. I don't know if this is a
> > serious problem or only a cosmetic issue.
> [...]
> 
> This message and the code that generates it have been present in the
> mv643xx_eth driver for a long time.  The network interface hardware is
> generally capable of transmitting a packet that comes from multiple
> buffers in memory, but in some cases the driver copies all the data
> into a single buffer, probably to avoid a hardware bug.  If there is
> not enough memory to make that copy then it logs this message.

Oh, thanks for the explanation.

> I think the driver should drop the packet if this happens, but instead
> it just reports that the hardware is busy.  That will cause the kernel
> to try sending the packet again repeatedly, resulting in repetition of
> this log message until there is more memory available.

Hummm, that may be related to some lower performance that I have felt (but
not measured).

I see. Discarding the packet would flag congestion with TCP, if I understand
it correctly.

> If you haven't seen this error message before, it might be triggered by an
> application change.  But I think it's more likely to be caused by higher
> memory usage or fragmentation of memory.

Yes, I think that the problem is that the programs are getting bigger and
bigger.  Would setting vm.min_free_kbytes to a higher value worth it?

Anyway, I just upgraded to kernel 4.13 and I will report what I see.


Thanks,

-- 
Rogério Brito : rbrito@{ime.usp.br,gmail.com} : GPG key 4096R/BCFCAAAA
http://cynic.cc/blog/ : github.com/rbrito : profiles.google.com/rbrito
DebianQA: http://qa.debian.org/developer.php?login=rbrito%40ime.usp.br

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web