Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1598814 > unrolled thread
| Started by | Matt Turner <mattst88@gmail.com> |
|---|---|
| First post | 2017-03-13 02:50 +0100 |
| Last post | 2017-03-15 11:00 +0100 |
| Articles | 4 — 3 participants |
Back to article view | Back to linux.kernel
NFS corruption, fixed by echo 1 > /proc/sys/vm/drop_caches -- next debugging steps? Matt Turner <mattst88@gmail.com> - 2017-03-13 02:50 +0100
Re: NFS corruption, fixed by echo 1 > /proc/sys/vm/drop_caches -- next debugging steps? James Hogan <james.hogan@imgtec.com> - 2017-03-13 10:50 +0100
Re: NFS corruption, fixed by echo 1 > /proc/sys/vm/drop_caches -- next debugging steps? Matt Turner <mattst88@gmail.com> - 2017-03-13 18:20 +0100
Re: NFS corruption, fixed by echo 1 > /proc/sys/vm/drop_caches -- next debugging steps? Ralf Baechle <ralf@linux-mips.org> - 2017-03-15 11:00 +0100
| From | Matt Turner <mattst88@gmail.com> |
|---|---|
| Date | 2017-03-13 02:50 +0100 |
| Subject | NFS corruption, fixed by echo 1 > /proc/sys/vm/drop_caches -- next debugging steps? |
| Message-ID | <tknhv-1pV-9@gated-at.bofh.it> |
On a Broadcom BCM91250a MIPS system I can reliably trigger NFS corruption on the first file read. To demonstrate, I downloaded five identical copies of the gcc-5.4.0 source tarball. On the NFS server, they hash to the same value: server distfiles # md5sum gcc-5.4.0.tar.bz2* 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.1 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.2 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.3 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.4 On the MIPS system (the NFS client): bcm91250a-le distfiles # md5sum gcc-5.4.0.tar.bz2.2 35346975989954df8a8db2b034da610d gcc-5.4.0.tar.bz2.2 bcm91250a-le distfiles # md5sum gcc-5.4.0.tar.bz2* 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.1 35346975989954df8a8db2b034da610d gcc-5.4.0.tar.bz2.2 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.3 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.4 The first file read will contain some corruption, and it is persistent until... bcm91250a-le distfiles # echo 1 > /proc/sys/vm/drop_caches bcm91250a-le distfiles # md5sum gcc-5.4.0.tar.bz2* 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.1 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.2 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.3 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.4 the caches are dropped, at which point it reads back properly. Note that the corruption is different across reboots, both in the size of the corruption and the location. I saw 1900~ and 1400~ byte sequences corrupted on separate occasions, which don't correspond to the system's 16kB page size. I've tested kernels from v3.19 to 4.11-rc1+ (master branch from today). All exhibit this behavior with differing frequencies. Earlier kernels seem to reproduce the issue less often, while more recent kernels reliably exhibit the problem every boot. How can I further debug this?
[toc] | [next] | [standalone]
| From | James Hogan <james.hogan@imgtec.com> |
|---|---|
| Date | 2017-03-13 10:50 +0100 |
| Subject | Re: NFS corruption, fixed by echo 1 > /proc/sys/vm/drop_caches -- next debugging steps? |
| Message-ID | <tkuM2-717-15@gated-at.bofh.it> |
| In reply to | #1598814 |
[Multipart message — attachments visible in raw view] — view raw
On Sun, Mar 12, 2017 at 06:43:47PM -0700, Matt Turner wrote: > On a Broadcom BCM91250a MIPS system I can reliably trigger NFS > corruption on the first file read. > > To demonstrate, I downloaded five identical copies of the gcc-5.4.0 > source tarball. On the NFS server, they hash to the same value: > > server distfiles # md5sum gcc-5.4.0.tar.bz2* > 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2 > 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.1 > 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.2 > 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.3 > 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.4 > > On the MIPS system (the NFS client): > > bcm91250a-le distfiles # md5sum gcc-5.4.0.tar.bz2.2 > 35346975989954df8a8db2b034da610d gcc-5.4.0.tar.bz2.2 > bcm91250a-le distfiles # md5sum gcc-5.4.0.tar.bz2* > 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2 > 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.1 > 35346975989954df8a8db2b034da610d gcc-5.4.0.tar.bz2.2 > 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.3 > 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.4 > > The first file read will contain some corruption, and it is persistent until... > > bcm91250a-le distfiles # echo 1 > /proc/sys/vm/drop_caches > bcm91250a-le distfiles # md5sum gcc-5.4.0.tar.bz2* > 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2 > 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.1 > 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.2 > 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.3 > 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.4 > > the caches are dropped, at which point it reads back properly. > > Note that the corruption is different across reboots, both in the size > of the corruption and the location. I saw 1900~ and 1400~ byte > sequences corrupted on separate occasions, which don't correspond to > the system's 16kB page size. > > I've tested kernels from v3.19 to 4.11-rc1+ (master branch from > today). All exhibit this behavior with differing frequencies. Earlier > kernels seem to reproduce the issue less often, while more recent > kernels reliably exhibit the problem every boot. > > How can I further debug this? It smells a bit like a DMA / caching issue. Can you provide a full kernel log. That might provide some information about caching that might be relevant (e.g. does dcache have aliases?). Cheers James
[toc] | [prev] | [next] | [standalone]
| From | Matt Turner <mattst88@gmail.com> |
|---|---|
| Date | 2017-03-13 18:20 +0100 |
| Subject | Re: NFS corruption, fixed by echo 1 > /proc/sys/vm/drop_caches -- next debugging steps? |
| Message-ID | <tkBNv-3I1-15@gated-at.bofh.it> |
| In reply to | #1599151 |
[Multipart message — attachments visible in raw view] — view raw
On Mon, Mar 13, 2017 at 2:47 AM, James Hogan <james.hogan@imgtec.com> wrote: > On Sun, Mar 12, 2017 at 06:43:47PM -0700, Matt Turner wrote: >> On a Broadcom BCM91250a MIPS system I can reliably trigger NFS >> corruption on the first file read. >> >> To demonstrate, I downloaded five identical copies of the gcc-5.4.0 >> source tarball. On the NFS server, they hash to the same value: >> >> server distfiles # md5sum gcc-5.4.0.tar.bz2* >> 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2 >> 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.1 >> 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.2 >> 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.3 >> 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.4 >> >> On the MIPS system (the NFS client): >> >> bcm91250a-le distfiles # md5sum gcc-5.4.0.tar.bz2.2 >> 35346975989954df8a8db2b034da610d gcc-5.4.0.tar.bz2.2 >> bcm91250a-le distfiles # md5sum gcc-5.4.0.tar.bz2* >> 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2 >> 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.1 >> 35346975989954df8a8db2b034da610d gcc-5.4.0.tar.bz2.2 >> 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.3 >> 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.4 >> >> The first file read will contain some corruption, and it is persistent until... >> >> bcm91250a-le distfiles # echo 1 > /proc/sys/vm/drop_caches >> bcm91250a-le distfiles # md5sum gcc-5.4.0.tar.bz2* >> 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2 >> 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.1 >> 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.2 >> 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.3 >> 4c626ac2a83ef30dfb9260e6f59c2b30 gcc-5.4.0.tar.bz2.4 >> >> the caches are dropped, at which point it reads back properly. >> >> Note that the corruption is different across reboots, both in the size >> of the corruption and the location. I saw 1900~ and 1400~ byte >> sequences corrupted on separate occasions, which don't correspond to >> the system's 16kB page size. >> >> I've tested kernels from v3.19 to 4.11-rc1+ (master branch from >> today). All exhibit this behavior with differing frequencies. Earlier >> kernels seem to reproduce the issue less often, while more recent >> kernels reliably exhibit the problem every boot. >> >> How can I further debug this? > > It smells a bit like a DMA / caching issue. > > Can you provide a full kernel log. That might provide some information > about caching that might be relevant (e.g. does dcache have aliases?). Thanks for the reply. Please find attached dmesg from a clean boot (which reproduced the problem).
[toc] | [prev] | [next] | [standalone]
| From | Ralf Baechle <ralf@linux-mips.org> |
|---|---|
| Date | 2017-03-15 11:00 +0100 |
| Subject | Re: NFS corruption, fixed by echo 1 > /proc/sys/vm/drop_caches -- next debugging steps? |
| Message-ID | <tldSP-5Pm-39@gated-at.bofh.it> |
| In reply to | #1599151 |
On Mon, Mar 13, 2017 at 09:47:57AM +0000, James Hogan wrote: > > > > Note that the corruption is different across reboots, both in the size > > of the corruption and the location. I saw 1900~ and 1400~ byte > > sequences corrupted on separate occasions, which don't correspond to > > the system's 16kB page size. > > > > I've tested kernels from v3.19 to 4.11-rc1+ (master branch from > > today). All exhibit this behavior with differing frequencies. Earlier > > kernels seem to reproduce the issue less often, while more recent > > kernels reliably exhibit the problem every boot. > > > > How can I further debug this? > > It smells a bit like a DMA / caching issue. > > Can you provide a full kernel log. That might provide some information > about caching that might be relevant (e.g. does dcache have aliases?). The architecture of the BCM1250 SOC used for the BCM91250 boards are fully coherent, S-cache and D-cache are physically indexed and tagged. Only the VIVT (plus the usual ASID tagging) I-cache leaves space for software to screw up cache management but that shouldn't matter for this case, so I suggest to start looking into this from the NFS side. Ralf
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web