Bug#927184: linux-image-4.19.0-4-amd64: Linux 4.19 reports wrong numbers for disks in /proc/diskstats

From	jaroslav@thinline.cz
Newsgroups	linux.debian.bugs.dist, linux.debian.kernel
Subject	Bug#927184: linux-image-4.19.0-4-amd64: Linux 4.19 reports wrong numbers for disks in /proc/diskstats
Date	2020-12-23 11:40 +0100
Message-ID	<Bp9IJ-10Z-1@gated-at.bofh.it> (permalink)
References	<xNlou-2Ss-3@gated-at.bofh.it>
Organization	linux.* mail to news gateway

Cross-posted to 2 groups.

Show all headers | View raw

Package: linux-source-4.19
Version: 4.19.160-2
Followup-For: Bug #927184

We got hit by this bug after updating to most recent kernel version. In 
our case it affected all disk drives, regardless of their type (rotating 
or SSD.) After some research, we found out that the issue was triggered 
by change of default I/O scheduler done by us before the update (and 
taking effect after restart.) See the following console output:

root@server:~# cat /sys/block/sde/queue/scheduler
[mq-deadline] none
root@server:~# for a in $( seq 1 5 ) ; do sleep 1 ;  cat 
/sys/block/sde/stat ; done
11452700   133667 706138701  5139372 168800292 66117118 3870603434 
78079852        0 1156352620 1170874630  1819485        0 349746784  
1614457
11452700   133667 706138701  5139372 168800309 66117137 3870603746 
78079862        0 1156352630 1170874640  1819485        0 349746784  
1614457
11452700   133667 706138701  5139372 168800417 66117189 3870604914 
78079911        0 1156352640 1170874650  1819485        0 349746784  
1614457
11452700   133667 706138701  5139372 168800431 66117205 3870605154 
78080014        0 1156352740 1170874750  1819485        0 349746784  
1614457
11452700   133667 706138701  5139372 168800491 66117250 3870606090 
78080037        0 1156352750 1170874760  1819486        0 349746792  
1614457
root@server:~# echo "none"> /sys/block/sde/queue/scheduler
root@server:~# for a in $( seq 1 5 ) ; do sleep 1 ;  cat 
/sys/block/sde/stat ; done
11452709   133667 706138965  5139376 168803680 66118376 3870643746 
78081646        0 1156354860 1170878030  1819488        0 349746808  
1614459
11452710   133667 706138997  5139377 168804363 66118550 3870650634 
78081962        0 1156355800 1170879210  1819489        0 349746816  
1614459
11452710   133667 706138997  5139377 168804439 66118594 3870651642 
78081993        0 1156356740 1170880150  1819489        0 349746816  
1614459
11452710   133667 706138997  5139377 168804498 66118653 3870652762 
78082022        0 1156357680 1170881090  1819489        0 349746816  
1614459
11452710   133667 706138997  5139377 168804569 66118700 3870653834 
78082053        0 1156358630 1170882040  1819489        0 349746816  
1614459
root@server:~# echo "mq-deadline"> /sys/block/sde/queue/scheduler
root@server:~# for a in $( seq 1 5 ) ; do sleep 1 ;  cat 
/sys/block/sde/stat ; done
11452710   133667 706138997  5139377 168804885 66118930 3870658594 
78082208        0 1156361540 1170884950  1819491        0 349779592  
1614461
11452710   133667 706138997  5139377 168804942 66118957 3870659346 
78082236        0 1156361570 1170884980  1819491        0 349779592  
1614461
11452711   133667 706139005  5139377 168805194 66119062 3870673810 
78082346        0 1156361630 1170885050  1819491        0 349779592  
1614461
11452711   133667 706139005  5139377 168805240 66119088 3870674394 
78082368        0 1156361630 1170885050  1819491        0 349779592  
1614461
11452711   133667 706139005  5139377 168805464 66119186 3870688410 
78082457        0 1156361670 1170885130  1819491        0 349779592  
1614461

As long as mq-deadline scheduler is active, all works well, value in 
10th column is increasing by realistic steps. After switching to none 
scheduler, the steps increase to almost a second whenever there is at 
least some activity on the drive. That in turn shows as almost 100% disk 
utilization in tools that use these values. Note that the value in 10th 
column is supposed to be increasing when value in 9th column is non-zero 
(according to https://www.kernel.org/doc/Documentation/iostats.txt ), 
yet I saw no occurence of non-zero number in that column in my testing, 
which does not correspond to device under 100% load.

Using mq-deadline in virtual machines and for SSDs is most likely not 
optimal, but as a workaround, switching to mq-deadline works.

Back to linux.debian.kernel | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

Bug#927184: linux-image-4.19.0-4-amd64: Linux 4.19 reports wrong numbers for NVME disks in /proc/diskstats Yury Vidineev <adeptg@gmail.com> - 2019-04-16 03:50 +0200
  Bug#927184: SATA SSD affected as well Yury Vidineev <adeptg@gmail.com> - 2019-04-16 23:00 +0200
  Bug#927184: Looks like RedHat/CentOS has a possible solution Yury Vidineev <adeptg@gmail.com> - 2019-06-18 00:40 +0200
  Bug#927184: linux-image-4.19.0-4-amd64: Linux 4.19 reports wrong numbers for disks in /proc/diskstats jaroslav@thinline.cz - 2020-12-23 11:40 +0100

csiph-web