Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1640425 > unrolled thread

Re: [patch 2/2] MM: allow per-cpu vmstat_threshold and vmstat_worker configuration

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [patch 2/2] MM: allow per-cpu vmstat_threshold and vmstat_worker  configuration Marcelo Tosatti <mtosatti@redhat.com> - 2017-05-12 14:30 +0200
    Re: [patch 2/2] MM: allow per-cpu vmstat_threshold and vmstat_worker  configuration Christoph Lameter <cl@linux.com> - 2017-05-12 17:20 +0200
      Re: [patch 2/2] MM: allow per-cpu vmstat_threshold and vmstat_worker  configuration Marcelo Tosatti <mtosatti@redhat.com> - 2017-05-12 17:50 +0200
        Re: [patch 2/2] MM: allow per-cpu vmstat_threshold and vmstat_worker  configuration Christoph Lameter <cl@linux.com> - 2017-05-12 18:10 +0200
        Re: [patch 2/2] MM: allow per-cpu vmstat_threshold and vmstat_worker  configuration Christoph Lameter <cl@linux.com> - 2017-05-12 18:10 +0200
          Re: [patch 2/2] MM: allow per-cpu vmstat_threshold and vmstat_worker  configuration Marcelo Tosatti <mtosatti@redhat.com> - 2017-05-12 18:30 +0200
            Re: [patch 2/2] MM: allow per-cpu vmstat_threshold and vmstat_worker  configuration Christoph Lameter <cl@linux.com> - 2017-05-12 19:00 +0200
              Re: [patch 2/2] MM: allow per-cpu vmstat_threshold and vmstat_worker  configuration Marcelo Tosatti <mtosatti@redhat.com> - 2017-05-15 21:20 +0200
                Re: [patch 2/2] MM: allow per-cpu vmstat_threshold and vmstat_worker  configuration Christoph Lameter <cl@linux.com> - 2017-05-16 15:40 +0200
                  Re: [patch 2/2] MM: allow per-cpu vmstat_threshold and vmstat_worker  configuration Marcelo Tosatti <mtosatti@redhat.com> - 2017-05-19 16:40 +0200
                    Re: [patch 2/2] MM: allow per-cpu vmstat_threshold and vmstat_worker  configuration Christoph Lameter <cl@linux.com> - 2017-05-19 19:20 +0200
                      Re: [patch 2/2] MM: allow per-cpu vmstat_threshold and  vmstat_worker configuration Luiz Capitulino <lcapitulino@redhat.com> - 2017-05-19 20:00 +0200
                        Re: [patch 2/2] MM: allow per-cpu vmstat_threshold and vmstat_worker  configuration Christoph Lameter <cl@linux.com> - 2017-05-22 18:40 +0200
                      Re: [patch 2/2] MM: allow per-cpu vmstat_threshold and vmstat_worker  configuration Marcelo Tosatti <mtosatti@redhat.com> - 2017-05-20 10:30 +0200
                        Re: [patch 2/2] MM: allow per-cpu vmstat_threshold and vmstat_worker  configuration Christoph Lameter <cl@linux.com> - 2017-05-22 18:40 +0200
                          Re: [patch 2/2] MM: allow per-cpu vmstat_threshold and vmstat_worker  configuration Marcelo Tosatti <mtosatti@redhat.com> - 2017-05-22 23:20 +0200

#1640425 — Re: [patch 2/2] MM: allow per-cpu vmstat_threshold and vmstat_worker configuration

On Thu, May 11, 2017 at 10:37:07AM -0500, Christoph Lameter wrote:
> On Tue, 2 May 2017, Luiz Capitulino wrote:
> 
> > Ah, OK. Got this now. I'll give this patch a try. But I think we want
> > to hear from Christoph (who worked on reducing the vmstat interruptions
> > in the past).
> 
> A bit confused by this one. The vmstat worker is already disabled if there
> are no updates. Also the patches by Chris Metcalf on data plane mode add a
> prctl to quiet the vmstat workers.
> 
> Why do we need more than this?

If there are vmstat statistic updates on a given CPU, and you don't
want intervention from the vmstat worker, you change the behaviour of
stat data collection to directly write to the global structures (which
disables the performance optimization of collecting data in per-cpu
counters).

This way you can disable vmstat worker (because it causes undesired
latencies), while allowing vmstatistics to function properly.

The prctl from Chris Metcalf patchset allows one to disable vmstat
worker per CPU? If so, they replace the functionality of the patch
"[patch 3/3] MM: allow per-cpu vmstat_worker configuration" 
of the -v2 series of my patchset, and we can use it instead.

Is it integrated already?

[toc] | [next] | [standalone]

#1640531

On Fri, 12 May 2017, Marcelo Tosatti wrote:

> > A bit confused by this one. The vmstat worker is already disabled if there
> > are no updates. Also the patches by Chris Metcalf on data plane mode add a
> > prctl to quiet the vmstat workers.
> >
> > Why do we need more than this?
>
> If there are vmstat statistic updates on a given CPU, and you don't
> want intervention from the vmstat worker, you change the behaviour of
> stat data collection to directly write to the global structures (which
> disables the performance optimization of collecting data in per-cpu
> counters).

Hmmm.... Ok. That is going to be expensive if you do this for each
individual vmstat update.

> This way you can disable vmstat worker (because it causes undesired
> latencies), while allowing vmstatistics to function properly.

Best then to run the vmstat update mechanism when you leave kernel mode to
get all the updates in one go.


> The prctl from Chris Metcalf patchset allows one to disable vmstat
> worker per CPU? If so, they replace the functionality of the patch
> "[patch 3/3] MM: allow per-cpu vmstat_worker configuration"
> of the -v2 series of my patchset, and we can use it instead.
>
> Is it integrated already?

The data plane mode patches disables vmstat processing  by updating the
vmstats immediately if necessary and switching off the kworker thread.

So the kworker wont be running until the next time statistics are checked
by the shepherd task from a remote cpu. If the counters have been updated
then the shepherd task will reenable the kworker. This is already merged
and has been working for a long time. Data plan mode has not been merged
yet but the infrastructure in vmstat.c is there because NOHZ needs it too.

See linux/vmstat.c:quiet_vmstat()

It would be easy to add a /proc file that allows the quieting of the
vmstat workers for a certain cpu. Just make it call the quiet_vmstat() on
the right cpu.

This will quiet vmstat down. The shepherd task will check the stats in 2
second intervals and will then reenable when necessasry.

Note that we already are updating the global structures directly if the
differential gets too high. Reducing the differential may get you what you
want.

[toc] | [prev] | [next] | [standalone]

#1640546

On Fri, May 12, 2017 at 10:11:14AM -0500, Christoph Lameter wrote:
> On Fri, 12 May 2017, Marcelo Tosatti wrote:
> 
> > > A bit confused by this one. The vmstat worker is already disabled if there
> > > are no updates. Also the patches by Chris Metcalf on data plane mode add a
> > > prctl to quiet the vmstat workers.
> > >
> > > Why do we need more than this?
> >
> > If there are vmstat statistic updates on a given CPU, and you don't
> > want intervention from the vmstat worker, you change the behaviour of
> > stat data collection to directly write to the global structures (which
> > disables the performance optimization of collecting data in per-cpu
> > counters).
> 
> Hmmm.... Ok. That is going to be expensive if you do this for each
> individual vmstat update.

In our case, vmstat updates are very rare (CPU is dominated by DPDK).

> > This way you can disable vmstat worker (because it causes undesired
> > latencies), while allowing vmstatistics to function properly.
> 
> Best then to run the vmstat update mechanism when you leave kernel mode to
> get all the updates in one go.

Again, vmstat updates are very rare (CPU is dominated by DPDK).

> > The prctl from Chris Metcalf patchset allows one to disable vmstat
> > worker per CPU? If so, they replace the functionality of the patch
> > "[patch 3/3] MM: allow per-cpu vmstat_worker configuration"
> > of the -v2 series of my patchset, and we can use it instead.
> >
> > Is it integrated already?
> 
> The data plane mode patches disables vmstat processing  by updating the
> vmstats immediately if necessary and switching off the kworker thread.

OK this is what my patch set is doing.

> So the kworker wont be running until the next time statistics are checked
> by the shepherd task from a remote cpu.

We don't want kworker thread to ever run.

>  If the counters have been updated
> then the shepherd task will reenable the kworker. This is already merged
> and has been working for a long time. Data plan mode has not been merged
> yet but the infrastructure in vmstat.c is there because NOHZ needs it too.

OK.

> 
> See linux/vmstat.c:quiet_vmstat()
> 
> It would be easy to add a /proc file that allows the quieting of the
> vmstat workers for a certain cpu. Just make it call the quiet_vmstat() on
> the right cpu.
> 
> This will quiet vmstat down. The shepherd task will check the stats in 2
> second intervals and will then reenable when necessasry.
> 
> Note that we already are updating the global structures directly if the
> differential gets too high. Reducing the differential may get you what you
> want.

Yes, we reduce the differential to 1 (== direct updates to global
structures).

OK, i'll check if the patches from Chris work for us and then add
Tested-by on that.

Thanks.

[toc] | [prev] | [next] | [standalone]

#1640559

On Fri, 12 May 2017, Marcelo Tosatti wrote:

> OK, i'll check if the patches from Chris work for us and then add
> Tested-by on that.

You may not need those if the quiet_vmstat() function is enough for you.

[toc] | [prev] | [next] | [standalone]

#1640561

On Fri, 12 May 2017, Marcelo Tosatti wrote:

> In our case, vmstat updates are very rare (CPU is dominated by DPDK).

What is the OS doing on the cores that DPDK runs on? I mean we here can
clean a processor of all activities and are able to run for a long time
without any interruptions.

Why would you still let the OS do things on that processor? If activities
by the OS are required then the existing NOHZ setup already minimizes
latency to a short burst (and Chris Metcalf's work improves on that).


What exactly is the issue you are seeing and want to address? I think we
have similar aims and as far as I know the current situation is already
good enough for what you may need. You may just not be aware of how to
configure this.

I doubt that doing inline updates will do much good compared to what we
already have and what the dataplan mode can do.

[toc] | [prev] | [next] | [standalone]

#1640569

On Fri, May 12, 2017 at 11:07:48AM -0500, Christoph Lameter wrote:
> On Fri, 12 May 2017, Marcelo Tosatti wrote:
> 
> > In our case, vmstat updates are very rare (CPU is dominated by DPDK).
> 
> What is the OS doing on the cores that DPDK runs on? I mean we here can
> clean a processor of all activities and are able to run for a long time
> without any interruptions.
> 
> Why would you still let the OS do things on that processor? If activities
> by the OS are required then the existing NOHZ setup already minimizes
> latency to a short burst (and Chris Metcalf's work improves on that).
> 
> 
> What exactly is the issue you are seeing and want to address? I think we
> have similar aims and as far as I know the current situation is already
> good enough for what you may need. You may just not be aware of how to
> configure this.

I want to disable vmstat worker thread completly from an isolated CPU.
Because it adds overhead to a latency target, target which 
the lower the better.

> I doubt that doing inline updates will do much good compared to what we
> already have and what the dataplan mode can do.

Can the dataplan mode disable vmstat worker thread completly on a given
CPU?

[toc] | [prev] | [next] | [standalone]

#1640600

On Fri, 12 May 2017, Marcelo Tosatti wrote:

> > What exactly is the issue you are seeing and want to address? I think we
> > have similar aims and as far as I know the current situation is already
> > good enough for what you may need. You may just not be aware of how to
> > configure this.
>
> I want to disable vmstat worker thread completly from an isolated CPU.
> Because it adds overhead to a latency target, target which
> the lower the better.

NOHZ already does that. I wanted to know what your problem is that you
see. The latency issue has already been solved as far as I can tell .
Please tell me why the existing solutions are not sufficient for you.

> > I doubt that doing inline updates will do much good compared to what we
> > already have and what the dataplan mode can do.
>
> Can the dataplan mode disable vmstat worker thread completly on a given
> CPU?

That already occurs when you call quiet_vmstat() and is used by the NOHZ
logic. Configure that correctly and you should be fine.

[toc] | [prev] | [next] | [standalone]

#1641994

On Fri, May 12, 2017 at 11:57:15AM -0500, Christoph Lameter wrote:
> On Fri, 12 May 2017, Marcelo Tosatti wrote:
> 
> > > What exactly is the issue you are seeing and want to address? I think we
> > > have similar aims and as far as I know the current situation is already
> > > good enough for what you may need. You may just not be aware of how to
> > > configure this.
> >
> > I want to disable vmstat worker thread completly from an isolated CPU.
> > Because it adds overhead to a latency target, target which
> > the lower the better.
> 
> NOHZ already does that. I wanted to know what your problem is that you
> see. The latency issue has already been solved as far as I can tell .
> Please tell me why the existing solutions are not sufficient for you.

We don't want vmstat_worker to execute on a given CPU, even if the local
CPU updates vm-statistics. 

Because:

    vmstat_worker increases latency of the application
       (i can measure it if you want on a given CPU,
        how many ns's the following takes:

            schedule_out(qemu-kvm-vcpu)
            schedule_in(kworker_thread)
            execute function to drain local vmstat counters to
                global counters
            schedule_out(kworker_thread)
            schedule_in(qemu-kvm-vcpu)
            x86 instruction to enter guest.
                                                (*)

But you can see right away without numbers that the sequence
above is not desired.

Why the existing solutions are not sufficient:

1) task-isolation patchset seems too heavy for our usecase (we do 
want IPIs, signals, etc).

2) With upstream linux-2.6.git, if dpdk running inside a guest happens
to trigger any vmstat update (say for example migration), we want the
statistics transferred directly from the point where they are generated,
and not the sequence (*).

> > > I doubt that doing inline updates will do much good compared to what we
> > > already have and what the dataplan mode can do.
> >
> > Can the dataplan mode disable vmstat worker thread completly on a given
> > CPU?
> 
> That already occurs when you call quiet_vmstat() and is used by the NOHZ
> logic. Configure that correctly and you should be fine.

quiet_vmstat() is not called by anyone today (upstream code). Are you
talking about task isolation patches?

Those seem a little heavy to me, for example:

1)
"Each time through the loop of TIF work to do, if TIF_TASK_ISOLATION
is set, we call the new task_isolation_enter() routine.  This
takes any actions that might avoid a future interrupt to the core,
such as a worker thread being scheduled that could be quiesced now
(e.g. the vmstat worker) or a future IPI to the core to clean up some
state that could be cleaned up now (e.g. the mm lru per-cpu cache).
In addition, it reqeusts rescheduling if the scheduler dyntick is
still running."

For example, what about

     static void do_sync_core(void *data)
             on_each_cpu(do_sync_core, NULL, 1);

You can't enable tracing with this feature?

"Prior to returning to userspace,
isolated tasks will arrange that no future kernel
activity will interrupt the task while the task is running
in userspace.  By default, attempting to re-enter the kernel
while in this mode will cause the task to be terminated
with a signal; you must explicitly use prctl() to disable
task isolation before resuming normal use of the kernel."

2)

A qemu-kvm-vcpu thread, process which runs on the host system,
executes guest code through

    ioctl(KVM_RUN) --> vcpu_enter_guest --> x86 instruction to execute
                                            guest code.

So the "isolation period where task does not want to be interrupted"
contains kernel code.

3) Before using any service of the operating system, through a
syscall, the application has to clear the TIF_TASK_ISOLATION flag,
then do the syscall, and when returning to userspace, setting it again.

Now what guarantees regarding low amount of interrupts do you provide
while this task is in kernel mode?

4)

"We also support a new "task_isolation_debug" flag which forces
the console stack to be dumped out regardless. We try to catch the
original source of the interrupt, e.g. if an IPI is dispatched to a
task-isolation task, we dump the backtrace of the remote core that is
sending the IPI, rather than just dumping out a trace showing the core
received an IPI from somewhere."

KVM uses IPI's to for example send virtual interrupts and update the
guest clock at certain conditions (for example after VM migration).

So this seems a little heavy for our usecase.

[toc] | [prev] | [next] | [standalone]

#1642564

On Mon, 15 May 2017, Marcelo Tosatti wrote:

> > NOHZ already does that. I wanted to know what your problem is that you
> > see. The latency issue has already been solved as far as I can tell .
> > Please tell me why the existing solutions are not sufficient for you.
>
> We don't want vmstat_worker to execute on a given CPU, even if the local
> CPU updates vm-statistics.

Instead of responding you repeat describing what you want.

> Because:
>
>     vmstat_worker increases latency of the application
>        (i can measure it if you want on a given CPU,
>         how many ns's the following takes:

That still is no use case. Just a measurement of vmstat_worker. Pointless.

If you move the latency from the vmstat worker into the code thats
updating the counters then you will require increased use of atomics
which will increase contention which in turn will significantly
increase the overall latency.

> Why the existing solutions are not sufficient:
>
> 1) task-isolation patchset seems too heavy for our usecase (we do
> want IPIs, signals, etc).

Ok then minor delays from remote random events are tolerable?
Then you can also have a vmstat update.

> So this seems a little heavy for our usecase.

Sorry all of this does not make sense to me. Maybe get some numbers of of
an app with intensive OS access running with atomics vs vmstat worker?

NOHZ currently disables the vmstat worker when no updates occur. This is
applicable to DPDK and will provide a quiet vmstat worker free environment
if no statistics activity is occurring.

[toc] | [prev] | [next] | [standalone]

#1645656

Hi Christoph,

On Tue, May 16, 2017 at 08:37:11AM -0500, Christoph Lameter wrote:
> On Mon, 15 May 2017, Marcelo Tosatti wrote:
> 
> > > NOHZ already does that. I wanted to know what your problem is that you
> > > see. The latency issue has already been solved as far as I can tell .
> > > Please tell me why the existing solutions are not sufficient for you.
> >
> > We don't want vmstat_worker to execute on a given CPU, even if the local
> > CPU updates vm-statistics.
> 
> Instead of responding you repeat describing what you want.
> 
> > Because:
> >
> >     vmstat_worker increases latency of the application
> >        (i can measure it if you want on a given CPU,
> >         how many ns's the following takes:
> 
> That still is no use case. 

Use-case: realtime application on an isolated core which for some reason
updates vmstatistics.

> Just a measurement of vmstat_worker. Pointless.

Shouldnt the focus be on general scenarios rather than particular
usecases, so that the solution covers a wider range of usecases?

The situation as i see is as follows:

Your point of view is: an "isolated CPU" with a set of applications
cannot update vm statistics, otherwise they pay the vmstat_update cost:

     kworker/5:1-245   [005] ....1..   673.454295: workqueue_execute_start: work struct ffffa0cf6e493e20: function vmstat_update
     kworker/5:1-245   [005] ....1..   673.454305: workqueue_execute_end: work struct ffffa0cf6e493e20

Thats 10us for example.

So if want to customize a realtime setup whose code updates vmstatistic, 
you are dead. You have to avoid any systemcall which possibly updates
vmstatistics (now and in the future kernel versions).

> If you move the latency from the vmstat worker into the code thats
> updating the counters then you will require increased use of atomics
> which will increase contention which in turn will significantly
> increase the overall latency.

The point is that these vmstat updates are rare. From 
http://www.7-cpu.com/cpu/Haswell.html:

RAM Latency = 36 cycles + 57 ns (3.4 GHz i7-4770)
RAM Latency = 62 cycles + 100 ns (3.6 GHz E5-2699 dual)

Lets round to 100ns = 0.1us.

You need 100 vmstat updates (all misses to RAM, the worst possible case)
to have equivalent amount of time of the batching version.

With more than 100 vmstat updates, then the batching is more efficient
(as in total amount of time to transfer to global counters).

But thats not the point. The point is the 10us interruption 
to execution of the realtime app (which can either mean 
your current deadline requirements are not met, or that 
another application with lowest latency requirement can't 
be used).

So i'd rather spend more time updating the aggregate of vmstatistics
(with the local->global transfer taking a small amount of time,
therefore not interrupting the realtime application for a long period),
than to batch the updates (which increases overall performance beyond 
a certain number of updates, but which is _ONE_ large interruption).

So lets assume i go and count the vmstat updates on the DPDK case 
(or any other realtime app), batching is more efficient 
for that case.

Still, the one-time interruption of batching is worse than less
efficient one bean at a time vmstatistics accounting.

No?

Also, you could reply that: "oh, there are no vmstat updates 
in fact in this setup, but the logic of disabling vmstat_update 
is broken". Lets assume thats the case.

Even if its fixed (vmstat_update properly shut down) the proposed patch
deals with both cases: no vmstat updates on isolated cpus, and vmstat
updates on isolated cpus.

So why are you against integrating this simple, isolated patch which 
does not affect how current logic works?

[toc] | [prev] | [next] | [standalone]

#1645779

On Fri, 19 May 2017, Marcelo Tosatti wrote:

> Use-case: realtime application on an isolated core which for some reason
> updates vmstatistics.

Ok that is already only happening every 2 seconds by default and that
interval is configurable via the vmstat_interval proc setting.

> > Just a measurement of vmstat_worker. Pointless.
>
> Shouldnt the focus be on general scenarios rather than particular
> usecases, so that the solution covers a wider range of usecases?

Yes indeed and as far as I can tell the wider usecases are covered. Not
sure that there is anything required here.

> The situation as i see is as follows:
>
> Your point of view is: an "isolated CPU" with a set of applications
> cannot update vm statistics, otherwise they pay the vmstat_update cost:
>
>      kworker/5:1-245   [005] ....1..   673.454295: workqueue_execute_start: work struct ffffa0cf6e493e20: function vmstat_update
>      kworker/5:1-245   [005] ....1..   673.454305: workqueue_execute_end: work struct ffffa0cf6e493e20
>
> Thats 10us for example.

Well with a decent cpu that is 3 usec and it occurs infrequently on the
order of once per multiple seconds.

> So if want to customize a realtime setup whose code updates vmstatistic,
> you are dead. You have to avoid any systemcall which possibly updates
> vmstatistics (now and in the future kernel versions).

You are already dead because you allow IPIs and other kernel processing
which creates far more overhead. Still fail to see the point.

> The point is that these vmstat updates are rare. From
> http://www.7-cpu.com/cpu/Haswell.html:
>
> RAM Latency = 36 cycles + 57 ns (3.4 GHz i7-4770)
> RAM Latency = 62 cycles + 100 ns (3.6 GHz E5-2699 dual)
>
> Lets round to 100ns = 0.1us.

That depends on the kernel functionality used.

> You need 100 vmstat updates (all misses to RAM, the worst possible case)
> to have equivalent amount of time of the batching version.

The batching version occurs every couple of seconds if at all.

> But thats not the point. The point is the 10us interruption
> to execution of the realtime app (which can either mean
> your current deadline requirements are not met, or that
> another application with lowest latency requirement can't
> be used).

Ok then you need to get rid of the IPIs and the other stuff that you have
going on with the OS first I think.

> So why are you against integrating this simple, isolated patch which
> does not affect how current logic works?

Frankly the argument does not make sense. Vmstat updates occur very
infrequently (probably even less than you IPIs and the other OS stuff that
also causes additional latencies that you seem to be willing to tolerate).

And you can configure the interval of vmstat updates freely.... Set
the vmstat_interval to 60 seconds instead of 2 for a try? Is that rare
enough?

[toc] | [prev] | [next] | [standalone]

#1645806 — Re: [patch 2/2] MM: allow per-cpu vmstat_threshold and vmstat_worker configuration

On Fri, 19 May 2017 12:13:26 -0500 (CDT)
Christoph Lameter <cl@linux.com> wrote:

> > So why are you against integrating this simple, isolated patch which
> > does not affect how current logic works?  
> 
> Frankly the argument does not make sense. Vmstat updates occur very
> infrequently (probably even less than you IPIs and the other OS stuff that
> also causes additional latencies that you seem to be willing to tolerate).

Infrequently is not good enough. It only has to happen once to
cause a problem.

Also, IPIs take a few us, usually less. That's not a problem. In our
testing we see the preemption caused by the kworker take 10us or
even more. I've never seeing it take 3us. I'm not saying this is not
true, I'm saying if this is causing a problem to us it will cause
a problem to other people too.

> And you can configure the interval of vmstat updates freely.... Set
> the vmstat_interval to 60 seconds instead of 2 for a try? Is that rare
> enough?

No, we'd have to set it high enough to disable it and this will
affect all CPUs.

Something that crossed my mind was to add a new tunable to set
the vmstat_interval for each CPU, this way we could essentially
disable it to the CPUs where DPDK is running. What's the implications
of doing this besides not getting up to date stats in /proc/vmstat
(which I still have to confirm would be OK)? Can this break anything
in the kernel for example?

[toc] | [prev] | [next] | [standalone]

#1647127

On Fri, 19 May 2017, Luiz Capitulino wrote:

> Something that crossed my mind was to add a new tunable to set
> the vmstat_interval for each CPU, this way we could essentially
> disable it to the CPUs where DPDK is running. What's the implications
> of doing this besides not getting up to date stats in /proc/vmstat
> (which I still have to confirm would be OK)? Can this break anything
> in the kernel for example?

The data is still going to be updated when the differential gets to big.

Increasing the vmstat interval and reducing the differential threshold
would get your there....

[toc] | [prev] | [next] | [standalone]

#1646082

On Fri, May 19, 2017 at 12:13:26PM -0500, Christoph Lameter wrote:
> On Fri, 19 May 2017, Marcelo Tosatti wrote:
> 
> > Use-case: realtime application on an isolated core which for some reason
> > updates vmstatistics.
> 
> Ok that is already only happening every 2 seconds by default and that
> interval is configurable via the vmstat_interval proc setting.
> 
> > > Just a measurement of vmstat_worker. Pointless.
> >
> > Shouldnt the focus be on general scenarios rather than particular
> > usecases, so that the solution covers a wider range of usecases?
> 
> Yes indeed and as far as I can tell the wider usecases are covered. Not
> sure that there is anything required here.
> 
> > The situation as i see is as follows:
> >
> > Your point of view is: an "isolated CPU" with a set of applications
> > cannot update vm statistics, otherwise they pay the vmstat_update cost:
> >
> >      kworker/5:1-245   [005] ....1..   673.454295: workqueue_execute_start: work struct ffffa0cf6e493e20: function vmstat_update
> >      kworker/5:1-245   [005] ....1..   673.454305: workqueue_execute_end: work struct ffffa0cf6e493e20
> >
> > Thats 10us for example.
> 
> Well with a decent cpu that is 3 usec and it occurs infrequently on the
> order of once per multiple seconds.
> 
> > So if want to customize a realtime setup whose code updates vmstatistic,
> > you are dead. You have to avoid any systemcall which possibly updates
> > vmstatistics (now and in the future kernel versions).
> 
> You are already dead because you allow IPIs and other kernel processing
> which creates far more overhead. Still fail to see the point.
> 
> > The point is that these vmstat updates are rare. From
> > http://www.7-cpu.com/cpu/Haswell.html:
> >
> > RAM Latency = 36 cycles + 57 ns (3.4 GHz i7-4770)
> > RAM Latency = 62 cycles + 100 ns (3.6 GHz E5-2699 dual)
> >
> > Lets round to 100ns = 0.1us.
> 
> That depends on the kernel functionality used.
> 
> > You need 100 vmstat updates (all misses to RAM, the worst possible case)
> > to have equivalent amount of time of the batching version.
> 
> The batching version occurs every couple of seconds if at all.
> 
> > But thats not the point. The point is the 10us interruption
> > to execution of the realtime app (which can either mean
> > your current deadline requirements are not met, or that
> > another application with lowest latency requirement can't
> > be used).
> 
> Ok then you need to get rid of the IPIs and the other stuff that you have
> going on with the OS first I think.

I'll measure the cost of all IPIs in the system to confirm
vmstat_update's costs is larger than the cost of any IPI.

> > So why are you against integrating this simple, isolated patch which
> > does not affect how current logic works?
> 
> Frankly the argument does not make sense. Vmstat updates occur very
> infrequently (probably even less than you IPIs and the other OS stuff that
> also causes additional latencies that you seem to be willing to tolerate).
> 
> And you can configure the interval of vmstat updates freely.... Set
> the vmstat_interval to 60 seconds instead of 2 for a try? Is that rare
> enough?

Not rare enough. Never is rare enough.

[toc] | [prev] | [next] | [standalone]

#1647130

On Sat, 20 May 2017, Marcelo Tosatti wrote:

> > And you can configure the interval of vmstat updates freely.... Set
> > the vmstat_interval to 60 seconds instead of 2 for a try? Is that rare
> > enough?
>
> Not rare enough. Never is rare enough.

Ok what about the other stuff that must be going on if you allow OS
activity like f.e. the tick, scheduler etc etc.

[toc] | [prev] | [next] | [standalone]

#1647409

On Mon, May 22, 2017 at 11:38:02AM -0500, Christoph Lameter wrote:
> On Sat, 20 May 2017, Marcelo Tosatti wrote:
> 
> > > And you can configure the interval of vmstat updates freely.... Set
> > > the vmstat_interval to 60 seconds instead of 2 for a try? Is that rare
> > > enough?
> >
> > Not rare enough. Never is rare enough.
> 
> Ok what about the other stuff that must be going on if you allow OS
> activity like f.e. the tick, scheduler etc etc.

Yes these are also problems... but we're either getting rid of them or
reducing their impact as much as possible.

vmstat_update is one member of the problematic set.

I'll get you the detailed IPI measures, hold on...

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web