Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1386271 > unrolled thread

Re: [PATCH v2 0/5] Cavium ThunderX uncore PMU support

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH v2 0/5] Cavium ThunderX uncore PMU support Will Deacon <will.deacon@arm.com> - 2016-04-25 13:30 +0200
    Re: [PATCH v2 0/5] Cavium ThunderX uncore PMU support Jan Glauber <jan.glauber@caviumnetworks.com> - 2016-04-25 14:10 +0200
      Re: [PATCH v2 0/5] Cavium ThunderX uncore PMU support Will Deacon <will.deacon@arm.com> - 2016-04-25 15:20 +0200
        Re: [PATCH v2 0/5] Cavium ThunderX uncore PMU support Jan Glauber <jan.glauber@caviumnetworks.com> - 2016-04-26 14:10 +0200
          Re: [PATCH v2 0/5] Cavium ThunderX uncore PMU support Will Deacon <will.deacon@arm.com> - 2016-04-26 16:00 +0200
            Re: [PATCH v2 0/5] Cavium ThunderX uncore PMU support Jan Glauber <jan.glauber@caviumnetworks.com> - 2016-04-27 13:00 +0200
              Re: [PATCH v2 0/5] Cavium ThunderX uncore PMU support Mark Rutland <mark.rutland@arm.com> - 2016-04-27 13:20 +0200

#1386271 — Re: [PATCH v2 0/5] Cavium ThunderX uncore PMU support

Hi Jan,

On Mon, Apr 04, 2016 at 02:19:54PM +0200, Jan Glauber wrote:
> Hi Mark,
> 
> can you have a look at these patches?

Looks like Mark reviewed this last week -- are you planning to respin?

Will

[toc] | [next] | [standalone]

#1386317

Hi Will,

On Mon, Apr 25, 2016 at 12:22:07PM +0100, Will Deacon wrote:
> Hi Jan,
> 
> On Mon, Apr 04, 2016 at 02:19:54PM +0200, Jan Glauber wrote:
> > Hi Mark,
> > 
> > can you have a look at these patches?
> 
> Looks like Mark reviewed this last week -- are you planning to respin?
> 
> Will

Yes, of course. I just had no time yet and I'm a bit lost on how to
proceed without using the NUMA node information which Mark did not like
to be used.

The only way to know which device is on which node would be to look
at the PCI topology (which is also the source of the NUMA node_id).
We could do this manually in order to not depend on CONFIG_NUMA,
but I would like to know if that is acceptable before respinning the
patches.

Thanks!
Jan

[toc] | [prev] | [next] | [standalone]

#1386408

On Mon, Apr 25, 2016 at 02:02:22PM +0200, Jan Glauber wrote:
> On Mon, Apr 25, 2016 at 12:22:07PM +0100, Will Deacon wrote:
> > On Mon, Apr 04, 2016 at 02:19:54PM +0200, Jan Glauber wrote:
> > > can you have a look at these patches?
> > 
> > Looks like Mark reviewed this last week -- are you planning to respin?
> 
> Yes, of course. I just had no time yet and I'm a bit lost on how to
> proceed without using the NUMA node information which Mark did not like
> to be used.
> 
> The only way to know which device is on which node would be to look
> at the PCI topology (which is also the source of the NUMA node_id).
> We could do this manually in order to not depend on CONFIG_NUMA,
> but I would like to know if that is acceptable before respinning the
> patches.

That doesn't feel like it really addresses Mark's concerns -- it's just
another way to get the information that isn't a first-class PMU topology
description from firmware.

Now, I don't actually mind using the NUMA topology so much in the cases
where it genuinely correlates with the PMU topology. My objection is more
that we end up sticking everything on node 0 if !CONFIG_NUMA, which could
result in working with an incorrect PMU topology and passing all of that
through to userspace.

So I'd prefer either making the driver depend on NUMA, or at the very least
failing to probe  the PMU if we discover a socketed system and NUMA is not
selected. Do either of those work as a compromise?

Will

[toc] | [prev] | [next] | [standalone]

#1387412

On Mon, Apr 25, 2016 at 02:19:07PM +0100, Will Deacon wrote:
> On Mon, Apr 25, 2016 at 02:02:22PM +0200, Jan Glauber wrote:
> > On Mon, Apr 25, 2016 at 12:22:07PM +0100, Will Deacon wrote:
> > > On Mon, Apr 04, 2016 at 02:19:54PM +0200, Jan Glauber wrote:
> > > > can you have a look at these patches?
> > > 
> > > Looks like Mark reviewed this last week -- are you planning to respin?
> > 
> > Yes, of course. I just had no time yet and I'm a bit lost on how to
> > proceed without using the NUMA node information which Mark did not like
> > to be used.
> > 
> > The only way to know which device is on which node would be to look
> > at the PCI topology (which is also the source of the NUMA node_id).
> > We could do this manually in order to not depend on CONFIG_NUMA,
> > but I would like to know if that is acceptable before respinning the
> > patches.
> 
> That doesn't feel like it really addresses Mark's concerns -- it's just
> another way to get the information that isn't a first-class PMU topology
> description from firmware.
> 
> Now, I don't actually mind using the NUMA topology so much in the cases
> where it genuinely correlates with the PMU topology. My objection is more
> that we end up sticking everything on node 0 if !CONFIG_NUMA, which could
> result in working with an incorrect PMU topology and passing all of that
> through to userspace.
> 
> So I'd prefer either making the driver depend on NUMA, or at the very least
> failing to probe  the PMU if we discover a socketed system and NUMA is not
> selected. Do either of those work as a compromise?
> 
> Will

That sounds like a good compromise.

So I could do the following:

1) In the uncore setup check for CONFIG_NUMA, if set use the NUMA
   information to determine the device node

2) If CONFIG_NUMA is not set we check if we run on a socketed system

   a) In that case we return an error and give a message that CONFIG_NUMA needs
      to be enabled
   b) Otherwise we have a single node system and use node_id = 0

David noted that it would also be possible to extract the node id from
the physical address of the device, but I'm not sure that classifies as
'first-class' topology description...

--Jan

[toc] | [prev] | [next] | [standalone]

#1387507

On Tue, Apr 26, 2016 at 02:08:09PM +0200, Jan Glauber wrote:
> On Mon, Apr 25, 2016 at 02:19:07PM +0100, Will Deacon wrote:
> > On Mon, Apr 25, 2016 at 02:02:22PM +0200, Jan Glauber wrote:
> > > On Mon, Apr 25, 2016 at 12:22:07PM +0100, Will Deacon wrote:
> > > > On Mon, Apr 04, 2016 at 02:19:54PM +0200, Jan Glauber wrote:
> > > > > can you have a look at these patches?
> > > > 
> > > > Looks like Mark reviewed this last week -- are you planning to respin?
> > > 
> > > Yes, of course. I just had no time yet and I'm a bit lost on how to
> > > proceed without using the NUMA node information which Mark did not like
> > > to be used.
> > > 
> > > The only way to know which device is on which node would be to look
> > > at the PCI topology (which is also the source of the NUMA node_id).
> > > We could do this manually in order to not depend on CONFIG_NUMA,
> > > but I would like to know if that is acceptable before respinning the
> > > patches.
> > 
> > That doesn't feel like it really addresses Mark's concerns -- it's just
> > another way to get the information that isn't a first-class PMU topology
> > description from firmware.
> > 
> > Now, I don't actually mind using the NUMA topology so much in the cases
> > where it genuinely correlates with the PMU topology. My objection is more
> > that we end up sticking everything on node 0 if !CONFIG_NUMA, which could
> > result in working with an incorrect PMU topology and passing all of that
> > through to userspace.
> > 
> > So I'd prefer either making the driver depend on NUMA, or at the very least
> > failing to probe  the PMU if we discover a socketed system and NUMA is not
> > selected. Do either of those work as a compromise?
> > 
> > Will
> 
> That sounds like a good compromise.
> 
> So I could do the following:
> 
> 1) In the uncore setup check for CONFIG_NUMA, if set use the NUMA
>    information to determine the device node
> 
> 2) If CONFIG_NUMA is not set we check if we run on a socketed system
> 
>    a) In that case we return an error and give a message that CONFIG_NUMA needs
>       to be enabled
>    b) Otherwise we have a single node system and use node_id = 0

That sounds sensible to me. How do you "check if we run on a socketed
system"? My assumption would be that you could figure this out from the
firmware tables?

> David noted that it would also be possible to extract the node id from
> the physical address of the device, but I'm not sure that classifies as
> 'first-class' topology description...

I'd rather avoid this sort of probing, as it inevitably breaks when it
sees new hardware that doesn't follow the unwritten assumptions of the
old hardware.

Will

[toc] | [prev] | [next] | [standalone]

#1388893

On Tue, Apr 26, 2016 at 02:53:54PM +0100, Will Deacon wrote:

[...]

> > 
> > That sounds like a good compromise.
> > 
> > So I could do the following:
> > 
> > 1) In the uncore setup check for CONFIG_NUMA, if set use the NUMA
> >    information to determine the device node
> > 
> > 2) If CONFIG_NUMA is not set we check if we run on a socketed system
> > 
> >    a) In that case we return an error and give a message that CONFIG_NUMA needs
> >       to be enabled
> >    b) Otherwise we have a single node system and use node_id = 0
> 
> That sounds sensible to me. How do you "check if we run on a socketed
> system"? My assumption would be that you could figure this out from the
> firmware tables?

There are probably multiple ways to detect a socketed system, with some quite
hardware specific. I would like to avoid parsing DT (and ACPI) though,
if possible.

A generic approach would be to do a query of the multiprocessor affinity
register (MPIDR_EL1) on all CPUs. The AFF2 part (bits 23:16) contains the 
socket number on ThunderX. If this is non-zero on any CPU I would assume a
socketed system.

Would that be feasible?

thanks,
Jan

[toc] | [prev] | [next] | [standalone]

#1388911

On Wed, Apr 27, 2016 at 12:51:56PM +0200, Jan Glauber wrote:
> On Tue, Apr 26, 2016 at 02:53:54PM +0100, Will Deacon wrote:
> 
> [...]
> 
> > > 
> > > That sounds like a good compromise.
> > > 
> > > So I could do the following:
> > > 
> > > 1) In the uncore setup check for CONFIG_NUMA, if set use the NUMA
> > >    information to determine the device node
> > > 
> > > 2) If CONFIG_NUMA is not set we check if we run on a socketed system
> > > 
> > >    a) In that case we return an error and give a message that CONFIG_NUMA needs
> > >       to be enabled
> > >    b) Otherwise we have a single node system and use node_id = 0
> > 
> > That sounds sensible to me. How do you "check if we run on a socketed
> > system"? My assumption would be that you could figure this out from the
> > firmware tables?
> 
> There are probably multiple ways to detect a socketed system, with some quite
> hardware specific. I would like to avoid parsing DT (and ACPI) though,
> if possible.
> 
> A generic approach would be to do a query of the multiprocessor affinity
> register (MPIDR_EL1) on all CPUs. The AFF2 part (bits 23:16) contains the 
> socket number on ThunderX. If this is non-zero on any CPU I would assume a
> socketed system.
> 
> Would that be feasible?

As with checking the physical address of a peripheral, this is an
unwritten assumption, and I suspect that similarly, it will inevitably
break (e.g. if Aff3 becomes used).

If you expect kernels relevant to your platform to have NUMA support,
you can simply depend on NUMA to determine whether or not you have NUMA
nodes.

Regarding relying on NUMA nodes, I have two concerns:

In general a NUMA node is not necessarily a socket, as you can have NUMA
properties even within a socket. If you can guarantee that for your
platform NUMA nodes will always be sockets, then I guess using NUMA
nodes is ok, though I imagine that as with the physical address map and
organisation of CPU IDs, that's difficult to have set in stone.

Linux NUMA node IDs are arbitrary tokens, and may not necessarily idmap
to documented socket IDs for your platform (even if they happen to
today). If you're happy to have users figure out how those IDs map to
clusters, that's fine, but otherwise you need to expose additional
information such that users get what they expect (at which point, if you
have said information we probably don't need NUMA information).

Thanks,
Mark.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web