Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1700377 > unrolled thread
| Started by | Marcin Wojtas <mw@semihalf.com> |
|---|---|
| First post | 2017-07-31 22:30 +0200 |
| Last post | 2017-08-03 14:30 +0200 |
| Articles | 4 — 3 participants |
Back to article view | Back to linux.kernel
[PATCH] arm64: defconfig: enable fine-grained task level IRQ time accounting Marcin Wojtas <mw@semihalf.com> - 2017-07-31 22:30 +0200
Re: [PATCH] arm64: defconfig: enable fine-grained task level IRQ time accounting Gregory CLEMENT <gregory.clement@free-electrons.com> - 2017-08-02 15:20 +0200
Re: [PATCH] arm64: defconfig: enable fine-grained task level IRQ time accounting Catalin Marinas <catalin.marinas@arm.com> - 2017-08-02 16:40 +0200
Re: [PATCH] arm64: defconfig: enable fine-grained task level IRQ time accounting Gregory CLEMENT <gregory.clement@free-electrons.com> - 2017-08-03 14:30 +0200
| From | Marcin Wojtas <mw@semihalf.com> |
|---|---|
| Date | 2017-07-31 22:30 +0200 |
| Subject | [PATCH] arm64: defconfig: enable fine-grained task level IRQ time accounting |
| Message-ID | <u9pua-2mM-7@gated-at.bofh.it> |
Tests showed, that under certain conditions, the summary number of jiffies spent on softirq/idle, which are counted by system statistics can be even below 10% of expected value, resulting in false load presentation. The issue was observed on the quad-core Marvell Armada 8k SoC, whose two 10G ports were bound into L2 bridge. Load was controlled by bidirectional UDP traffic, produced by a packet generator. Under such condition, the dominant load is softirq. With 100% single CPU occupation or without any activity (all CPUs 100% idle), total number of jiffies is 10000 (2500 per each core) in 10s interval. Also with other kind of load this was true. However below a saturation threshold it was observed, that with CPU which was occupied almost by softirqs only, the statistic were awkward. See the mpstat output: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle all 0.00 0.00 0.13 0.00 0.00 0.55 0.00 0.00 0.00 99.32 0 0.00 0.00 0.00 0.00 0.00 23.08 0.00 0.00 0.00 76.92 1 0.00 0.00 0.40 0.00 0.00 0.00 0.00 0.00 0.00 99.60 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Above would mean basically no total load, debug CPU0 occupied in 25%. Raw statistics, printed every 10s from /proc/stat unveiled a root cause - summary idle/softirq jiffies on loaded CPU were below 200, i.e. over 90% samples lost. All problems were gone after enabling fine granulity IRQ time accounting. This patch fixes possible wrong statistics processing by enabling CONFIG_IRQ_TIME_ACCOUNTING for arm64 platfroms, which is by default done on other architectures, e.g. x86 and arm. Tests showed no noticeable performance penalty, nor stability impact. Signed-off-by: Marcin Wojtas <mw@semihalf.com> --- arch/arm64/configs/defconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig index 44423e6..ed51ac6 100644 --- a/arch/arm64/configs/defconfig +++ b/arch/arm64/configs/defconfig @@ -3,6 +3,7 @@ CONFIG_POSIX_MQUEUE=y CONFIG_AUDIT=y CONFIG_NO_HZ_IDLE=y CONFIG_HIGH_RES_TIMERS=y +CONFIG_IRQ_TIME_ACCOUNTING=y CONFIG_BSD_PROCESS_ACCT=y CONFIG_BSD_PROCESS_ACCT_V3=y CONFIG_TASKSTATS=y -- 1.8.3.1
[toc] | [next] | [standalone]
| From | Gregory CLEMENT <gregory.clement@free-electrons.com> |
|---|---|
| Date | 2017-08-02 15:20 +0200 |
| Message-ID | <ua1J7-1UY-7@gated-at.bofh.it> |
| In reply to | #1700377 |
Hi, (Adding Arnd and Olof) On lun., juil. 31 2017, Marcin Wojtas <mw@semihalf.com> wrote: > Tests showed, that under certain conditions, the summary number of jiffies > spent on softirq/idle, which are counted by system statistics can be even > below 10% of expected value, resulting in false load presentation. > > The issue was observed on the quad-core Marvell Armada 8k SoC, whose two > 10G ports were bound into L2 bridge. Load was controlled by bidirectional > UDP traffic, produced by a packet generator. Under such condition, > the dominant load is softirq. With 100% single CPU occupation or without > any activity (all CPUs 100% idle), total number of jiffies is 10000 (2500 > per each core) in 10s interval. Also with other kind of load this was > true. > > However below a saturation threshold it was observed, that with CPU which > was occupied almost by softirqs only, the statistic were awkward. See > the mpstat output: > > CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle > all 0.00 0.00 0.13 0.00 0.00 0.55 0.00 0.00 0.00 99.32 > 0 0.00 0.00 0.00 0.00 0.00 23.08 0.00 0.00 0.00 76.92 > 1 0.00 0.00 0.40 0.00 0.00 0.00 0.00 0.00 0.00 99.60 > 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 > 3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 > > Above would mean basically no total load, debug CPU0 occupied in 25%. > Raw statistics, printed every 10s from /proc/stat unveiled a root > cause - summary idle/softirq jiffies on loaded CPU were below 200, > i.e. over 90% samples lost. All problems were gone after enabling > fine granulity IRQ time accounting. > > This patch fixes possible wrong statistics processing by enabling > CONFIG_IRQ_TIME_ACCOUNTING for arm64 platfroms, which is by > default done on other architectures, e.g. x86 and arm. Tests > showed no noticeable performance penalty, nor stability impact. Who should take this patch? I think that all the defconfig under arm64 are merged through the arm-soc subsystem, but this one is not really specific to a SoC. However, as it was experimented on an mvebu SoC, if you agree I can take it. Thanks, Gregory > > Signed-off-by: Marcin Wojtas <mw@semihalf.com> > --- > arch/arm64/configs/defconfig | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig > index 44423e6..ed51ac6 100644 > --- a/arch/arm64/configs/defconfig > +++ b/arch/arm64/configs/defconfig > @@ -3,6 +3,7 @@ CONFIG_POSIX_MQUEUE=y > CONFIG_AUDIT=y > CONFIG_NO_HZ_IDLE=y > CONFIG_HIGH_RES_TIMERS=y > +CONFIG_IRQ_TIME_ACCOUNTING=y > CONFIG_BSD_PROCESS_ACCT=y > CONFIG_BSD_PROCESS_ACCT_V3=y > CONFIG_TASKSTATS=y > -- > 1.8.3.1 > -- Gregory Clement, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
[toc] | [prev] | [next] | [standalone]
| From | Catalin Marinas <catalin.marinas@arm.com> |
|---|---|
| Date | 2017-08-02 16:40 +0200 |
| Subject | Re: [PATCH] arm64: defconfig: enable fine-grained task level IRQ time accounting |
| Message-ID | <ua2Yx-2AD-1@gated-at.bofh.it> |
| In reply to | #1702102 |
On Wed, Aug 02, 2017 at 03:11:43PM +0200, Gregory CLEMENT wrote: > On lun., juil. 31 2017, Marcin Wojtas <mw@semihalf.com> wrote: > > Tests showed, that under certain conditions, the summary number of jiffies > > spent on softirq/idle, which are counted by system statistics can be even > > below 10% of expected value, resulting in false load presentation. > > > > The issue was observed on the quad-core Marvell Armada 8k SoC, whose two > > 10G ports were bound into L2 bridge. Load was controlled by bidirectional > > UDP traffic, produced by a packet generator. Under such condition, > > the dominant load is softirq. With 100% single CPU occupation or without > > any activity (all CPUs 100% idle), total number of jiffies is 10000 (2500 > > per each core) in 10s interval. Also with other kind of load this was > > true. > > > > However below a saturation threshold it was observed, that with CPU which > > was occupied almost by softirqs only, the statistic were awkward. See > > the mpstat output: > > > > CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle > > all 0.00 0.00 0.13 0.00 0.00 0.55 0.00 0.00 0.00 99.32 > > 0 0.00 0.00 0.00 0.00 0.00 23.08 0.00 0.00 0.00 76.92 > > 1 0.00 0.00 0.40 0.00 0.00 0.00 0.00 0.00 0.00 99.60 > > 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 > > 3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 > > > > Above would mean basically no total load, debug CPU0 occupied in 25%. > > Raw statistics, printed every 10s from /proc/stat unveiled a root > > cause - summary idle/softirq jiffies on loaded CPU were below 200, > > i.e. over 90% samples lost. All problems were gone after enabling > > fine granulity IRQ time accounting. > > > > This patch fixes possible wrong statistics processing by enabling > > CONFIG_IRQ_TIME_ACCOUNTING for arm64 platfroms, which is by > > default done on other architectures, e.g. x86 and arm. Tests > > showed no noticeable performance penalty, nor stability impact. > > Who should take this patch? > > I think that all the defconfig under arm64 are merged through the > arm-soc subsystem, but this one is not really specific to a > SoC. However, as it was experimented on an mvebu SoC, if you agree I > can take it. It's fine by me to go via arm-soc. -- Catalin
[toc] | [prev] | [next] | [standalone]
| From | Gregory CLEMENT <gregory.clement@free-electrons.com> |
|---|---|
| Date | 2017-08-03 14:30 +0200 |
| Message-ID | <uanqi-8kg-11@gated-at.bofh.it> |
| In reply to | #1700377 |
Hi Marcin, On lun., juil. 31 2017, Marcin Wojtas <mw@semihalf.com> wrote: > Tests showed, that under certain conditions, the summary number of jiffies > spent on softirq/idle, which are counted by system statistics can be even > below 10% of expected value, resulting in false load presentation. > > The issue was observed on the quad-core Marvell Armada 8k SoC, whose two > 10G ports were bound into L2 bridge. Load was controlled by bidirectional > UDP traffic, produced by a packet generator. Under such condition, > the dominant load is softirq. With 100% single CPU occupation or without > any activity (all CPUs 100% idle), total number of jiffies is 10000 (2500 > per each core) in 10s interval. Also with other kind of load this was > true. > > However below a saturation threshold it was observed, that with CPU which > was occupied almost by softirqs only, the statistic were awkward. See > the mpstat output: > > CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle > all 0.00 0.00 0.13 0.00 0.00 0.55 0.00 0.00 0.00 99.32 > 0 0.00 0.00 0.00 0.00 0.00 23.08 0.00 0.00 0.00 76.92 > 1 0.00 0.00 0.40 0.00 0.00 0.00 0.00 0.00 0.00 99.60 > 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 > 3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 > > Above would mean basically no total load, debug CPU0 occupied in 25%. > Raw statistics, printed every 10s from /proc/stat unveiled a root > cause - summary idle/softirq jiffies on loaded CPU were below 200, > i.e. over 90% samples lost. All problems were gone after enabling > fine granulity IRQ time accounting. > > This patch fixes possible wrong statistics processing by enabling > CONFIG_IRQ_TIME_ACCOUNTING for arm64 platfroms, which is by > default done on other architectures, e.g. x86 and arm. Tests > showed no noticeable performance penalty, nor stability impact. > > Signed-off-by: Marcin Wojtas <mw@semihalf.com> Applied on mvebu/arm64 Thanks, Gregory > --- > arch/arm64/configs/defconfig | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig > index 44423e6..ed51ac6 100644 > --- a/arch/arm64/configs/defconfig > +++ b/arch/arm64/configs/defconfig > @@ -3,6 +3,7 @@ CONFIG_POSIX_MQUEUE=y > CONFIG_AUDIT=y > CONFIG_NO_HZ_IDLE=y > CONFIG_HIGH_RES_TIMERS=y > +CONFIG_IRQ_TIME_ACCOUNTING=y > CONFIG_BSD_PROCESS_ACCT=y > CONFIG_BSD_PROCESS_ACCT_V3=y > CONFIG_TASKSTATS=y > -- > 1.8.3.1 > -- Gregory Clement, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web