Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1491233 > unrolled thread

Re: [PATCH 0/2 v3] cpu hotplug: Preserve topology directory after soft remove event

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH 0/2 v3] cpu hotplug: Preserve topology directory after  soft remove event Prarit Bhargava <prarit@redhat.com> - 2016-09-26 13:50 +0200
    Re: [PATCH 0/2 v3] cpu hotplug: Preserve topology directory after  soft remove event Borislav Petkov <bp@suse.de> - 2016-09-26 14:00 +0200
      Re: [PATCH 0/2 v3] cpu hotplug: Preserve topology directory after  soft remove event Prarit Bhargava <prarit@redhat.com> - 2016-09-27 13:50 +0200
        Re: [PATCH 0/2 v3] cpu hotplug: Preserve topology directory after  soft remove event Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2016-09-27 16:00 +0200
          Re: [PATCH 0/2 v3] cpu hotplug: Preserve topology directory after  soft remove event Prarit Bhargava <prarit@redhat.com> - 2016-09-27 17:30 +0200
            Re: [PATCH 0/2 v3] cpu hotplug: Preserve topology directory after  soft remove event Borislav Petkov <bp@suse.de> - 2016-09-28 07:10 +0200
            Re: [PATCH 0/2 v3] cpu hotplug: Preserve topology directory after  soft remove event Peter Zijlstra <peterz@infradead.org> - 2016-09-28 08:50 +0200
              Re: [PATCH 0/2 v3] cpu hotplug: Preserve topology directory after  soft remove event Prarit Bhargava <prarit@redhat.com> - 2016-09-28 12:10 +0200
        Re: [PATCH 0/2 v3] cpu hotplug: Preserve topology directory after  soft remove event Borislav Petkov <bp@suse.de> - 2016-09-28 07:10 +0200
    Re: [PATCH 0/2 v3] cpu hotplug: Preserve topology directory after  soft remove event Peter Zijlstra <peterz@infradead.org> - 2016-09-26 14:10 +0200
      Re: [PATCH 0/2 v3] cpu hotplug: Preserve topology directory after  soft remove event Prarit Bhargava <prarit@redhat.com> - 2016-09-27 13:50 +0200
        Re: [PATCH 0/2 v3] cpu hotplug: Preserve topology directory after  soft remove event Peter Zijlstra <peterz@infradead.org> - 2016-09-27 14:00 +0200

#1491233 — Re: [PATCH 0/2 v3] cpu hotplug: Preserve topology directory after soft remove event

On 09/22/2016 08:10 AM, Borislav Petkov wrote:
> On Thu, Sep 22, 2016 at 07:59:08AM -0400, Prarit Bhargava wrote:
>> System boots with (usually) with 2 threads/core.  Some performance users want
>> one thread per core.  Since there is no "noht" option anymore, users use /sys to
>> disable a thread on each core.
> 
> I see.
> 
>> core_siblings and thread_siblings are the online thread's sibling cores and
>> threads that are available to the scheduler
> 
> Hmm, I see something else:
> 
> <Documentation/cputopology.txt>:
> 7) /sys/devices/system/cpu/cpuX/topology/core_siblings:
> 
>         internal kernel map of cpuX's hardware threads within the same
>         physical_package_id.

Thanks.  I'll send a patch to modify the Documentation which is out-of-date.

> 
>> and should be 0 when the thread is offline. That comes directly from
>> reading the code.
> 
> But then code which reads those will have to *know* that those cores are
> offline - otherwise it would be confused by what it is reading there.

When offline, /sys/devices/system/cpuX/cpu/online is 0.  The problem is that
when online is 0, topology disappears so there is no way to determine _the
location_ of the offline'd thread.

> 
> For example, the core siblings of an offlined core are still the same,
> they don't change. It is just the core that is offline.

Please see (in latest linux.git)

arch/x86/kernel/smpboot.c:1493 function remove_siblinginfo()

Specifically,

        cpumask_clear(topology_sibling_cpumask(cpu));
        cpumask_clear(topology_core_cpumask(cpu));

> 
>> See commit 20102ac5bee3 ("cpupower: cpupower monitor reports uninitialized
>> values for offline cpus").  That patch papers over the bug of not being able to
>> find core_id and physical_package_id for an offline thread.
> 
> Right, and this is *exactly* the *right* thing to do - tools should
> handle the case gracefully when cores are offline.

cpupower should still print out all asterisks for down'd threads.  It does not
because the topology directory is incorrectly removed.

IOW how does userspace know the _location_ of the thread?  The topology
directory no longer exists when the thread is downed, so core_id and
physical_package_id (both of which would be effectively static) do not exist.
The whole point of this patchset is to know where the offline'd thread actually is.

P.

[toc] | [next] | [standalone]

#1491244

On Mon, Sep 26, 2016 at 07:45:37AM -0400, Prarit Bhargava wrote:
> When offline, /sys/devices/system/cpuX/cpu/online is 0.  The problem is that
> when online is 0, topology disappears so there is no way to determine _the
> location_ of the offline'd thread.

What does "the location" mean exactly?

> cpupower should still print out all asterisks for down'd threads.  It does not
> because the topology directory is incorrectly removed.
> 
> IOW how does userspace know the _location_ of the thread?  The topology
> directory no longer exists when the thread is downed, so core_id and
> physical_package_id (both of which would be effectively static) do not exist.
> The whole point of this patchset is to know where the offline'd thread actually is.

What do you mean "where"?

$ echo 0 > /sys/devices/system/cpu/cpu2/online
$ cat /sys/devices/system/cpu/online
0-1,3-7

So core 2 is right between 1 and 3.

If you need to show the package id, you still iterate over the core
numbers in an increasing order and show '*' for the offlined ones.

-- 
Regards/Gruss,
    Boris.

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
-- 

[toc] | [prev] | [next] | [standalone]

#1491841

On 09/26/2016 07:57 AM, Borislav Petkov wrote:
> On Mon, Sep 26, 2016 at 07:45:37AM -0400, Prarit Bhargava wrote:
>> When offline, /sys/devices/system/cpuX/cpu/online is 0.  The problem is that
>> when online is 0, topology disappears so there is no way to determine _the
>> location_ of the offline'd thread.
> 
> What does "the location" mean exactly?
> 
>> cpupower should still print out all asterisks for down'd threads.  It does not
>> because the topology directory is incorrectly removed.
>>
>> IOW how does userspace know the _location_ of the thread?  The topology
>> directory no longer exists when the thread is downed, so core_id and
>> physical_package_id (both of which would be effectively static) do not exist.
>> The whole point of this patchset is to know where the offline'd thread actually is.
> 
> What do you mean "where"?

The socket and core location.

Look at it this way (and let's get the terminology straight at the same time).

You have a socket CPU.  That socket has cores on it.  Each core (at least on
Intel) has two threads.

I down a thread (as you did):

> 
> $ echo 0 > /sys/devices/system/cpu/cpu2/online

This results in the topology directory being destroyed.  It shouldn't be -- the
socket and core are still there.  If you could open up your computer you could
touch them.  This is similar to downing a PCI device, or removing !kernel memory
DIMM from a system.  The device is still physically there.

> $ cat /sys/devices/system/cpu/online
> 0-1,3-7
> 
> So core 2 is right between 1 and 3.

Yes.  But *where* is it relative to the cores and socket(s)?

> 
> If you need to show the package id, you still iterate over the core
> numbers in an increasing order and show '*' for the offlined ones.
> 

Explain this in more detail please?

P.

[toc] | [prev] | [next] | [standalone]

#1491921

On Tue, Sep 27, 2016 at 07:45:56AM -0400, Prarit Bhargava wrote:
> On 09/26/2016 07:57 AM, Borislav Petkov wrote:
> > $ echo 0 > /sys/devices/system/cpu/cpu2/online
> 
> This results in the topology directory being destroyed.  It shouldn't be -- the
> socket and core are still there.  If you could open up your computer you could
> touch them.  This is similar to downing a PCI device, or removing !kernel memory
> DIMM from a system.  The device is still physically there.

If you remove a PCI device from the system, by turning it off from the
PCI hotplug interface, it will go away from sysfs, just like this CPU
device is going away, no matter if you physically remove the device or
not.

Thanks for validating that everything is working the same and correctly
:)

greg k-h

[toc] | [prev] | [next] | [standalone]

#1491984

On 09/27/2016 09:49 AM, Greg Kroah-Hartman wrote:
> On Tue, Sep 27, 2016 at 07:45:56AM -0400, Prarit Bhargava wrote:
>> On 09/26/2016 07:57 AM, Borislav Petkov wrote:
>>> $ echo 0 > /sys/devices/system/cpu/cpu2/online
>>
>> This results in the topology directory being destroyed.  It shouldn't be -- the
>> socket and core are still there.  If you could open up your computer you could
>> touch them.  This is similar to downing a PCI device, or removing !kernel memory
>> DIMM from a system.  The device is still physically there.
> 
> If you remove a PCI device from the system, by turning it off from the
> PCI hotplug interface, it will go away from sysfs, just like this CPU
> device is going away, no matter if you physically remove the device or
> not.
> 

While similar, the thread's device struct, link to structures, and masks in the
kernel are kept in place.  In the PCI case these links are all destroyed which
results in requiring a bus rescan to find the devcie.  The sysfs (soft) removal
of a thread has a very different outcome from PCI.

I see now that the issue is not understanding the difference between physical
and soft thread removal.  I will write that up and get back to everyone.

P.

> Thanks for validating that everything is working the same and correctly
> :)
> 
> greg k-h
> 

[toc] | [prev] | [next] | [standalone]

#1492265

On Tue, Sep 27, 2016 at 11:26:14AM -0400, Prarit Bhargava wrote:
> I see now that the issue is not understanding the difference between physical
> and soft thread removal.  I will write that up and get back to everyone.

No need - we understand the issue.

What I don't understand is what information you need *exactly* in sysfs
from the offlined cores and why. Something like "I need to know the id
of the thread on core X on socket Y because..."

I've been trying to get it out of you but I've failed so far at it.

-- 
Regards/Gruss,
    Boris.

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
-- 

[toc] | [prev] | [next] | [standalone]

#1492287

On Tue, Sep 27, 2016 at 11:26:14AM -0400, Prarit Bhargava wrote:
> I see now that the issue is not understanding the difference between physical
> and soft thread removal.  I will write that up and get back to everyone.

You don't seem to understand that from the kernels POV there is no such
distinction.

There is only one unplug operation that caters to both cases. Therefore
unplug needs to assume the most stringent (ie. physical) for things to
work right.

There is no 'soft thread removal', except maybe in the userspace API,
you're the one that is confused.

[toc] | [prev] | [next] | [standalone]

#1492508

On 09/28/2016 02:48 AM, Peter Zijlstra wrote:
> On Tue, Sep 27, 2016 at 11:26:14AM -0400, Prarit Bhargava wrote:
>> I see now that the issue is not understanding the difference between physical
>> and soft thread removal.  I will write that up and get back to everyone.
> 
> You don't seem to understand that from the kernels POV there is no such
> distinction.
> 

And I'm saying you're wrong.  Unlike PCI, CPU sysfs unplug does not completely
remove the device from the kernel's knowledge.  If that were the case then

/sys/devices/system/cpu/cpuX/online (and other files left after the unplug)
wouldn't exist.

If you do a physical removal (signaled via ACPI or interrupt) the entire device
is removed from the kernel's POV.

Yes, the sysfs removal is a subset of the physical removal.  But the end result
is very different.

> There is only one unplug operation that caters to both cases. Therefore
> unplug needs to assume the most stringent (ie. physical) for things to
> work right.

See last comment above.

P.

[toc] | [prev] | [next] | [standalone]

#1492261

On Tue, Sep 27, 2016 at 07:45:56AM -0400, Prarit Bhargava wrote:
> Yes.  But *where* is it relative to the cores and socket(s)?

And you need that information because...

> > If you need to show the package id, you still iterate over the core
> > numbers in an increasing order and show '*' for the offlined ones.
> > 
> 
> Explain this in more detail please?

Just s/package id/core id/ and then it makes sense - I meant "core id".

-- 
Regards/Gruss,
    Boris.

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
-- 

[toc] | [prev] | [next] | [standalone]

#1491247

On Mon, Sep 26, 2016 at 07:45:37AM -0400, Prarit Bhargava wrote:
> > But then code which reads those will have to *know* that those cores are
> > offline - otherwise it would be confused by what it is reading there.
> 
> When offline, /sys/devices/system/cpuX/cpu/online is 0.  The problem is that
> when online is 0, topology disappears so there is no way to determine _the
> location_ of the offline'd thread.

As far as all that code is concerned, that CPU doesn't even have a
location anymore.

While there might be some distinction between hotplug and physical
hotplug on the user API side (I really wouldn't know), there isn't on
the kernel side.

Once you unplug a CPU, its _gone_. There isn't another hotplug operation
once you really take the CPU out.

Offline means out gone, vamoosh.

And it doesn't make sense to talk about the location of a resource
that's not there.

[toc] | [prev] | [next] | [standalone]

#1491840

On 09/26/2016 07:59 AM, Peter Zijlstra wrote:
> On Mon, Sep 26, 2016 at 07:45:37AM -0400, Prarit Bhargava wrote:
>>> But then code which reads those will have to *know* that those cores are
>>> offline - otherwise it would be confused by what it is reading there.
>>
>> When offline, /sys/devices/system/cpuX/cpu/online is 0.  The problem is that
>> when online is 0, topology disappears so there is no way to determine _the
>> location_ of the offline'd thread.
> 
> As far as all that code is concerned, that CPU doesn't even have a
> location anymore.
> 
> While there might be some distinction between hotplug and physical
> hotplug on the user API side (I really wouldn't know), there isn't on
> the kernel side.
> 
> Once you unplug a CPU, its _gone_. There isn't another hotplug operation
> once you really take the CPU out.

There's a difference between soft remove (via sysfs) and a true hot remove
operation (where the whole thing is physically removed).  Soft remove only
results in the processor being made "not available" to the scheduler.

> 
> Offline means out gone, vamoosh.

No, that is incorrect.  The socket that contains the cores (and threads) is
still plugged in.

> 
> And it doesn't make sense to talk about the location of a resource
> that's not there.
> 

Again, it is _physically_ there.

P.

[toc] | [prev] | [next] | [standalone]

#1491852

On Tue, Sep 27, 2016 at 07:47:56AM -0400, Prarit Bhargava wrote:

> There's a difference between soft remove (via sysfs) and a true hot remove
> operation (where the whole thing is physically removed).  Soft remove only
> results in the processor being made "not available" to the scheduler.

How is it different? We do _ONE_ CPU unplug operation. We do not touch
the thing anymore after that. It _can_ be taken out after that.

Therefore hotplug wipes the topology information and clears the CPU from
relevant bitmasks.

From the kernel's POV there really is no distinction.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web