Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1301308 > unrolled thread

[RFC PATCH 0/3] Implement getcpu_cache system call

Back to article view | Back to linux.kernel

Contents

  [RFC PATCH 0/3] Implement getcpu_cache system call Mathieu Desnoyers <mathieu.desnoyers@efficios.com> - 2016-01-05 08:10 +0100
    RE: [RFC PATCH 0/3] Implement getcpu_cache system call "Seymour, Shane M" <shane.seymour@hpe.com> - 2016-01-11 23:40 +0100
      Re: [RFC PATCH 0/3] Implement getcpu_cache system call Josh Triplett <josh@joshtriplett.org> - 2016-01-12 00:10 +0100
        Re: [RFC PATCH 0/3] Implement getcpu_cache system call Mathieu Desnoyers <mathieu.desnoyers@efficios.com> - 2016-01-12 01:50 +0100
          Re: [RFC PATCH 0/3] Implement getcpu_cache system call Josh Triplett <josh@joshtriplett.org> - 2016-01-12 03:50 +0100
            Re: [RFC PATCH 0/3] Implement getcpu_cache system call Ben Maurer <bmaurer@fb.com> - 2016-01-12 05:30 +0100
              RE: [RFC PATCH 0/3] Implement getcpu_cache system call "Seymour, Shane M" <shane.seymour@hpe.com> - 2016-01-12 07:50 +0100
              Re: [RFC PATCH 0/3] Implement getcpu_cache system call Mathieu Desnoyers <mathieu.desnoyers@efficios.com> - 2016-01-12 14:20 +0100
                RE: [RFC PATCH 0/3] Implement getcpu_cache system call Ben Maurer <bmaurer@fb.com> - 2016-01-12 22:10 +0100
                  Re: [RFC PATCH 0/3] Implement getcpu_cache system call Mathieu Desnoyers <mathieu.desnoyers@efficios.com> - 2016-01-13 01:30 +0100
                    Re: [RFC PATCH 0/3] Implement getcpu_cache system call Josh Triplett <josh@joshtriplett.org> - 2016-01-13 02:50 +0100
                      Re: [RFC PATCH 0/3] Implement getcpu_cache system call Mathieu Desnoyers <mathieu.desnoyers@efficios.com> - 2016-01-14 17:00 +0100
      RE: [RFC PATCH 0/3] Implement getcpu_cache system call "Seymour, Shane M" <shane.seymour@hpe.com> - 2016-01-12 00:20 +0100

#1301308 — [RFC PATCH 0/3] Implement getcpu_cache system call

Hi,

Here is a patchset implementing a cache for the CPU number of the
currently running thread in user-space.

Benchmarks comparing this approach to a system call-based getcpu on ARM
show a 44x speedup. They show a 14x speedup on x86-64 compared to
executing lsl from a vDSO through glibc.

I'm added a man page in the changelog of patch 1/3, which shows an
example usage of this new system call.

This patchset is sent as RFC. It applies on Linux 4.3.

Feedback is welcome,

Thanks!

Mathieu

Mathieu Desnoyers (3):
  getcpu_cache system call: cache CPU number of running thread
  getcpu_cache: wire up ARM system call
  getcpu_cache: wire up x86 32/64 system call

 arch/arm/include/uapi/asm/unistd.h     |   1 +
 arch/arm/kernel/calls.S                |   1 +
 arch/arm/kernel/signal.c               |   1 +
 arch/x86/entry/common.c                |   1 +
 arch/x86/entry/syscalls/syscall_32.tbl |   1 +
 arch/x86/entry/syscalls/syscall_64.tbl |   1 +
 fs/exec.c                              |   1 +
 include/linux/init_task.h              |   8 ++
 include/linux/sched.h                  |  43 +++++++++
 include/uapi/linux/Kbuild              |   1 +
 include/uapi/linux/getcpu_cache.h      |  44 +++++++++
 init/Kconfig                           |  10 ++
 kernel/Makefile                        |   1 +
 kernel/fork.c                          |   7 ++
 kernel/getcpu_cache.c                  | 170 +++++++++++++++++++++++++++++++++
 kernel/sched/core.c                    |   3 +
 kernel/sched/sched.h                   |   1 +
 kernel/sys_ni.c                        |   3 +
 18 files changed, 298 insertions(+)
 create mode 100644 include/uapi/linux/getcpu_cache.h
 create mode 100644 kernel/getcpu_cache.c

-- 
2.1.4

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1306780

Hi Mathieu,

I have some concerns and suggestions for you about this.

What's to stop someone in user space from requesting an arbitrarily large number of CPU # cache locations that the kernel needs to allocate memory to track and each time the task migrates to a new CPU it needs to update them all? Could you use it to dramatically slow down a system/task switching? Should there be a ulimit type value or a sysctl setting to limit the number that you're allowed to register per-task?

If you can just register consecutive addresses each 4 bytes apart the size of the structure required to keep track of it in the kernel looks to be 20 or 24 bytes long depending on kernel bitness (using kmalloc it should be 32 bytes allocated either way) so if you do something like tie up 1GiB of memory in user space registered with CPU cache locations it will tie up >8GiB of memory in the kernel and there will be a huge linked list that will take significant amounts of time to traverse. You could use it as a local denial of service attack to try and soak up memory and cause a kernel OOM because the kernel needs more memory to keep track of the request compared to the size of the memory used by user space to create it. There doesn't currently appear to be any upper bounds on the number that can be registered.

In terms of tracking what it's doing would you consider some sysfs attribute files (or something in debugfs) that tracked (these would all be in the add path so it shouldn't be performance sensitive):

1) The largest number of entries someone has created in the list in any task
2) The number of times (assuming you implement an upper bound on the number allowed) the upper bound is being hit (to allow someone to monitor for issues where the upper bound is being hit)

Assuming that something (e.g. glibc) is willing to register and make an entry available for the life of the task consider allowing one flag, for example, GETCPU_CACHE_PERSISTENT with GETCPU_CACHE_CMD_REGISTER and have a new command GETCPU_CACHE_CMD_GET_ PERSISTENT to allow someone to ask for the user space address of an entry that something has guaranteed will be there until the task ends. If none exist they can fall back and allocate a new one - it allows for better reuse of an existing resource but log a warning if you're forced to remove a persistent entry or someone attempts to unregister it (which should always fail) since someone in user space will have broken their promise that it will always be there until the task ends (that means persistent ones should be left to be torn down by the kernel not unregistered from user space). If you do this you might need to optimize the find process so it's more likely the first persistent one will be the first one found if you think someone is more likely to take the approach of asking for that first and falling back to creating a new one if there isn't already one present. Having this will also tend to limit the number of these that anyone will need to create if most libraries ask for a persistent entry first.

Thanks
Shane

[toc] | [prev] | [next] | [standalone]

#1306805

On Mon, Jan 11, 2016 at 10:38:28PM +0000, Seymour, Shane M wrote:
> I have some concerns and suggestions for you about this.
> 
> What's to stop someone in user space from requesting an arbitrarily large number of CPU # cache locations that the kernel needs to allocate memory to track and each time the task migrates to a new CPU it needs to update them all? Could you use it to dramatically slow down a system/task switching? Should there be a ulimit type value or a sysctl setting to limit the number that you're allowed to register per-task?

The documented behavior of the syscall allows only one location per
thread, so the kernel can track that one and only address rather easily
in the task_struct.  Allowing dynamic allocation definitely doesn't seem
like a good idea.

- Josh Triplett

[toc] | [prev] | [next] | [standalone]

#1306868

----- On Jan 11, 2016, at 6:03 PM, Josh Triplett josh@joshtriplett.org wrote:

> On Mon, Jan 11, 2016 at 10:38:28PM +0000, Seymour, Shane M wrote:
>> I have some concerns and suggestions for you about this.
>> 
>> What's to stop someone in user space from requesting an arbitrarily large number
>> of CPU # cache locations that the kernel needs to allocate memory to track and
>> each time the task migrates to a new CPU it needs to update them all? Could you
>> use it to dramatically slow down a system/task switching? Should there be a
>> ulimit type value or a sysctl setting to limit the number that you're allowed
>> to register per-task?
> 
> The documented behavior of the syscall allows only one location per
> thread, so the kernel can track that one and only address rather easily
> in the task_struct.  Allowing dynamic allocation definitely doesn't seem
> like a good idea.

The current implementation now allows more than one location per
thread. Which piece of documentation states that only one location
per thread is allowed ? This was indeed the case for the prior
implementations, but I moved to implementing a linked-list of
cpu_cache areas per thread to allow the getcpu_cache system call to
be used by more than a single shared object within a given program.

Without the linked list, as soon as more than one shared object try
to register their cache, the first one will prohibit all others from
doing so.

We could perhaps try to document that this system call should only
ever be used by *libc, and all libraries and applications should
then use the libc TLS cache variable, but it seems rather fragile,
and any app/lib could try to register its own cache.

Thoughts ?

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

[toc] | [prev] | [next] | [standalone]

#1306948

On Tue, Jan 12, 2016 at 12:49:18AM +0000, Mathieu Desnoyers wrote:
> ----- On Jan 11, 2016, at 6:03 PM, Josh Triplett josh@joshtriplett.org wrote:
> 
> > On Mon, Jan 11, 2016 at 10:38:28PM +0000, Seymour, Shane M wrote:
> >> I have some concerns and suggestions for you about this.
> >> 
> >> What's to stop someone in user space from requesting an arbitrarily large number
> >> of CPU # cache locations that the kernel needs to allocate memory to track and
> >> each time the task migrates to a new CPU it needs to update them all? Could you
> >> use it to dramatically slow down a system/task switching? Should there be a
> >> ulimit type value or a sysctl setting to limit the number that you're allowed
> >> to register per-task?
> > 
> > The documented behavior of the syscall allows only one location per
> > thread, so the kernel can track that one and only address rather easily
> > in the task_struct.  Allowing dynamic allocation definitely doesn't seem
> > like a good idea.
> 
> The current implementation now allows more than one location per
> thread. Which piece of documentation states that only one location
> per thread is allowed ? This was indeed the case for the prior
> implementations, but I moved to implementing a linked-list of
> cpu_cache areas per thread to allow the getcpu_cache system call to
> be used by more than a single shared object within a given program.

Ah, I missed that change.

> Without the linked list, as soon as more than one shared object try
> to register their cache, the first one will prohibit all others from
> doing so.
> 
> We could perhaps try to document that this system call should only
> ever be used by *libc, and all libraries and applications should
> then use the libc TLS cache variable, but it seems rather fragile,
> and any app/lib could try to register its own cache.

That does seem a bit fragile, true; on the other hand, the linked-list
approach would allow userspace to allocate an unbounded amount of kernel
memory, without any particular control on it.  That doesn't seem
reasonable.  Introducing an rlimit or similar for this seems like
massive overkill, and hardcoding a fixed limit breaks the 0-1-infinity
rule.

Given that any registered location will always provide the same value,
allowing only a single registration doesn't seem *too* problematic;
libc-based programs can use the libc implementation, and non-libc-based
programs can register a location themselves.  And users of this API will
already likely want to use some TLS mechanism, which already interacts
heavily with libc (set_thread_area/clone).

Allowing only one registration at a time seems preferable to introducing
another way to allocate kernel resources on a process's behalf.

- Josh Triplett

[toc] | [prev] | [next] | [standalone]

#1306978

One disadvantage of only allowing one is that high performance server applications tend to statically link. It'd suck to have to go through what ever type of relocation we'd need to pull this out of glibc. But if there's only one registration allowed a statically linked app couldn't create its own if glibc might use it some day. 



Sent from my iPhone

> On Jan 11, 2016, at 6:46 PM, Josh Triplett <josh@joshtriplett.org> wrote:
> 
>> On Tue, Jan 12, 2016 at 12:49:18AM +0000, Mathieu Desnoyers wrote:
>> ----- On Jan 11, 2016, at 6:03 PM, Josh Triplett josh@joshtriplett.org wrote:
>> 
>>>> On Mon, Jan 11, 2016 at 10:38:28PM +0000, Seymour, Shane M wrote:
>>>> I have some concerns and suggestions for you about this.
>>>> 
>>>> What's to stop someone in user space from requesting an arbitrarily large number
>>>> of CPU # cache locations that the kernel needs to allocate memory to track and
>>>> each time the task migrates to a new CPU it needs to update them all? Could you
>>>> use it to dramatically slow down a system/task switching? Should there be a
>>>> ulimit type value or a sysctl setting to limit the number that you're allowed
>>>> to register per-task?
>>> 
>>> The documented behavior of the syscall allows only one location per
>>> thread, so the kernel can track that one and only address rather easily
>>> in the task_struct.  Allowing dynamic allocation definitely doesn't seem
>>> like a good idea.
>> 
>> The current implementation now allows more than one location per
>> thread. Which piece of documentation states that only one location
>> per thread is allowed ? This was indeed the case for the prior
>> implementations, but I moved to implementing a linked-list of
>> cpu_cache areas per thread to allow the getcpu_cache system call to
>> be used by more than a single shared object within a given program.
> 
> Ah, I missed that change.
> 
>> Without the linked list, as soon as more than one shared object try
>> to register their cache, the first one will prohibit all others from
>> doing so.
>> 
>> We could perhaps try to document that this system call should only
>> ever be used by *libc, and all libraries and applications should
>> then use the libc TLS cache variable, but it seems rather fragile,
>> and any app/lib could try to register its own cache.
> 
> That does seem a bit fragile, true; on the other hand, the linked-list
> approach would allow userspace to allocate an unbounded amount of kernel
> memory, without any particular control on it.  That doesn't seem
> reasonable.  Introducing an rlimit or similar for this seems like
> massive overkill, and hardcoding a fixed limit breaks the 0-1-infinity
> rule.
> 
> Given that any registered location will always provide the same value,
> allowing only a single registration doesn't seem *too* problematic;
> libc-based programs can use the libc implementation, and non-libc-based
> programs can register a location themselves.  And users of this API will
> already likely want to use some TLS mechanism, which already interacts
> heavily with libc (set_thread_area/clone).
> 
> Allowing only one registration at a time seems preferable to introducing
> another way to allocate kernel resources on a process's behalf.
> 
> - Josh Triplett

[toc] | [prev] | [next] | [standalone]

#1307011

> -----Original Message-----
> From: Ben Maurer [mailto:bmaurer@fb.com]
> Sent: Tuesday, January 12, 2016 3:28 PM
> 
> One disadvantage of only allowing one is that high performance server
> applications tend to statically link. It'd suck to have to go through what ever
> type of relocation we'd need to pull this out of glibc. But if there's only one
> registration allowed a statically linked app couldn't create its own if glibc
> might use it some day.
> 

If there was a new command like GETCPU_CACHE_CMD_ADDRESS that returned the address that is currently registered for that task that wouldn't be an issue (the kernel knows the address it's going to write to for that task there's no reason not to make it available back to user space on request). The main limitation of only allowing one address is that whomever registered that address would be providing an (unenforceable implicit) guarantee that it would always be there until the task ended (GETCPU_CACHE_CMD_UNREGISTER would have to go away with only one registerable address). It's highly likely that anyone registering an address would keep it for the life of the task but it's hard to guarantee it.

There only two impacts that I can think of quickly are:

1) if multiple shared libraries wanted to register an address that were dynamically loaded after a program starts using an explicit dlopen (in a process that didn't already have an address registered). They shouldn't register an address - they should only ask for an already existing one and have a fallback if there isn't one currently. If it did register something and was unloaded and the memory is freed/unmapped anyone else using the address isn't going to be happy since you could either have non-existent addresses or use after free happening. The library that registered it would need to leak the address so anyone using it can still do so or have some other method of doing cleanup when the task ends after it's been unloaded. The potential impact of that depends on if anyone thinks that is at all likely to happen.
2) There could be ordering issues for shared libraries with initializers and finalizers if an cpu cache address is registered in an initializer and used in a finalizer of another library that is ran after the finalizer of the library that registered it (if it's in memory that is no longer available or it's possible to unregister an address). 

> 
> 
> Sent from my iPhone
> 
> > On Jan 11, 2016, at 6:46 PM, Josh Triplett <josh@joshtriplett.org> wrote:
> >
> >> On Tue, Jan 12, 2016 at 12:49:18AM +0000, Mathieu Desnoyers wrote:
> >> ----- On Jan 11, 2016, at 6:03 PM, Josh Triplett josh@joshtriplett.org wrote:
> >>
> >>>> On Mon, Jan 11, 2016 at 10:38:28PM +0000, Seymour, Shane M wrote:
> >>>> I have some concerns and suggestions for you about this.
> >>>>
> >>>> What's to stop someone in user space from requesting an arbitrarily
> >>>> large number of CPU # cache locations that the kernel needs to
> >>>> allocate memory to track and each time the task migrates to a new
> >>>> CPU it needs to update them all? Could you use it to dramatically
> >>>> slow down a system/task switching? Should there be a ulimit type
> >>>> value or a sysctl setting to limit the number that you're allowed to
> register per-task?
> >>>
> >>> The documented behavior of the syscall allows only one location per
> >>> thread, so the kernel can track that one and only address rather
> >>> easily in the task_struct.  Allowing dynamic allocation definitely
> >>> doesn't seem like a good idea.
> >>
> >> The current implementation now allows more than one location per
> >> thread. Which piece of documentation states that only one location
> >> per thread is allowed ? This was indeed the case for the prior
> >> implementations, but I moved to implementing a linked-list of
> >> cpu_cache areas per thread to allow the getcpu_cache system call to
> >> be used by more than a single shared object within a given program.
> >
> > Ah, I missed that change.
> >
> >> Without the linked list, as soon as more than one shared object try
> >> to register their cache, the first one will prohibit all others from
> >> doing so.
> >>
> >> We could perhaps try to document that this system call should only
> >> ever be used by *libc, and all libraries and applications should then
> >> use the libc TLS cache variable, but it seems rather fragile, and any
> >> app/lib could try to register its own cache.
> >
> > That does seem a bit fragile, true; on the other hand, the linked-list
> > approach would allow userspace to allocate an unbounded amount of
> > kernel memory, without any particular control on it.  That doesn't
> > seem reasonable.  Introducing an rlimit or similar for this seems like
> > massive overkill, and hardcoding a fixed limit breaks the 0-1-infinity
> > rule.
> >
> > Given that any registered location will always provide the same value,
> > allowing only a single registration doesn't seem *too* problematic;
> > libc-based programs can use the libc implementation, and
> > non-libc-based programs can register a location themselves.  And users
> > of this API will already likely want to use some TLS mechanism, which
> > already interacts heavily with libc (set_thread_area/clone).
> >
> > Allowing only one registration at a time seems preferable to
> > introducing another way to allocate kernel resources on a process's behalf.
> >
> > - Josh Triplett

[toc] | [prev] | [next] | [standalone]

#1307391

----- On Jan 11, 2016, at 11:27 PM, Ben Maurer bmaurer@fb.com wrote:

> One disadvantage of only allowing one is that high performance server
> applications tend to statically link. It'd suck to have to go through what ever
> type of relocation we'd need to pull this out of glibc. But if there's only one
> registration allowed a statically linked app couldn't create its own if glibc
> might use it some day.

One idea I have would be to let the kernel reserve some space either after the
first stack address (for a stack growing down) or at the beginning of the
allocated TLS area for each thread in copy_thread_tls() by fiddling with
sp or the tls base address when creating a thread.

In theory, this would allow always returning the same address, and the memory
would exist as long as the thread exists.

Not sure whether it may have unforeseen impact though.

Thoughts ?

Thanks,

Mathieu

> 
> 
> 
> Sent from my iPhone
> 
>> On Jan 11, 2016, at 6:46 PM, Josh Triplett <josh@joshtriplett.org> wrote:
>> 
>>> On Tue, Jan 12, 2016 at 12:49:18AM +0000, Mathieu Desnoyers wrote:
>>> ----- On Jan 11, 2016, at 6:03 PM, Josh Triplett josh@joshtriplett.org wrote:
>>> 
>>>>> On Mon, Jan 11, 2016 at 10:38:28PM +0000, Seymour, Shane M wrote:
>>>>> I have some concerns and suggestions for you about this.
>>>>> 
>>>>> What's to stop someone in user space from requesting an arbitrarily large number
>>>>> of CPU # cache locations that the kernel needs to allocate memory to track and
>>>>> each time the task migrates to a new CPU it needs to update them all? Could you
>>>>> use it to dramatically slow down a system/task switching? Should there be a
>>>>> ulimit type value or a sysctl setting to limit the number that you're allowed
>>>>> to register per-task?
>>>> 
>>>> The documented behavior of the syscall allows only one location per
>>>> thread, so the kernel can track that one and only address rather easily
>>>> in the task_struct.  Allowing dynamic allocation definitely doesn't seem
>>>> like a good idea.
>>> 
>>> The current implementation now allows more than one location per
>>> thread. Which piece of documentation states that only one location
>>> per thread is allowed ? This was indeed the case for the prior
>>> implementations, but I moved to implementing a linked-list of
>>> cpu_cache areas per thread to allow the getcpu_cache system call to
>>> be used by more than a single shared object within a given program.
>> 
>> Ah, I missed that change.
>> 
>>> Without the linked list, as soon as more than one shared object try
>>> to register their cache, the first one will prohibit all others from
>>> doing so.
>>> 
>>> We could perhaps try to document that this system call should only
>>> ever be used by *libc, and all libraries and applications should
>>> then use the libc TLS cache variable, but it seems rather fragile,
>>> and any app/lib could try to register its own cache.
>> 
>> That does seem a bit fragile, true; on the other hand, the linked-list
>> approach would allow userspace to allocate an unbounded amount of kernel
>> memory, without any particular control on it.  That doesn't seem
>> reasonable.  Introducing an rlimit or similar for this seems like
>> massive overkill, and hardcoding a fixed limit breaks the 0-1-infinity
>> rule.
>> 
>> Given that any registered location will always provide the same value,
>> allowing only a single registration doesn't seem *too* problematic;
>> libc-based programs can use the libc implementation, and non-libc-based
>> programs can register a location themselves.  And users of this API will
>> already likely want to use some TLS mechanism, which already interacts
>> heavily with libc (set_thread_area/clone).
>> 
>> Allowing only one registration at a time seems preferable to introducing
>> another way to allocate kernel resources on a process's behalf.
>> 
> > - Josh Triplett

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

[toc] | [prev] | [next] | [standalone]

#1307836

> One idea I have would be to let the kernel reserve some space either after the
> first stack address (for a stack growing down) or at the beginning of the
> allocated TLS area for each thread in copy_thread_tls() by fiddling with
> sp or the tls base address when creating a thread.

Could this be implemented by having glibc use a well known symbol name to define the per-thread TLS area? If an high performance application wants to avoid any relocations in accessing this variable it would define it and that definition would override glibc's. This is how things work with malloc. glibc has a default malloc implementation but we link jemalloc directly into our binaries. in addition to changing the malloc implementation this means that calls to malloc don't go through the PLT.

-b

[toc] | [prev] | [next] | [standalone]

#1307937

----- On Jan 12, 2016, at 4:02 PM, Ben Maurer bmaurer@fb.com wrote:

>> One idea I have would be to let the kernel reserve some space either after the
>> first stack address (for a stack growing down) or at the beginning of the
>> allocated TLS area for each thread in copy_thread_tls() by fiddling with
>> sp or the tls base address when creating a thread.
> 
> Could this be implemented by having glibc use a well known symbol name to define
> the per-thread TLS area? If an high performance application wants to avoid any
> relocations in accessing this variable it would define it and that definition
> would override glibc's. This is how things work with malloc. glibc has a
> default malloc implementation but we link jemalloc directly into our binaries.
> in addition to changing the malloc implementation this means that calls to
> malloc don't go through the PLT.

Just to make sure I understand your proposal: defining a well known symbol
with a weak attribute in glibc (or bionic...), e.g.:

int32_t __thread __attribute__((weak)) __getcpu_cache;

so that applications which care about bypassing the PLT can override it with:

int32_t __thread __getcpu_cache;

glibc/bionic would be responsible for calling the getcpu_cache() system call
to register/unregister this TLS variable for each thread.

One thing I would like to figure out is whether we can use this in a way that
would allow introducing getcpu_cache() into applications and libraries
(e.g. lttng-ust tracer) before it gets implemented into glibc, in a way that
would keep forward compatibility for whenever it gets introduced in glibc.

We can declare __getcpu_cache as a weak symbol in arbitrary libraries, and
make them register/unregister the cache through the getcpu_cache syscall.
The main thing that I would need to tweak at the kernel level within the
system call would be to keep a refcount of the number of times the
__getcpu_cache is registered per thread. This would allow multiple registrations,
one per library (e.g. lttng-ust) and one for glibc, but we would validate
that they all register the exact same address for a given thread.

The reference counting trick should also work for cases where applications
define a non-weak __getcpu_cache, and want to call the getcpu_cache
system call to register it themselves (before glibc adds support for it).

Thoughts ?

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

[toc] | [prev] | [next] | [standalone]

#1307964

On January 12, 2016 4:22:29 PM PST, Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:
>----- On Jan 12, 2016, at 4:02 PM, Ben Maurer bmaurer@fb.com wrote:
>
>>> One idea I have would be to let the kernel reserve some space either
>after the
>>> first stack address (for a stack growing down) or at the beginning
>of the
>>> allocated TLS area for each thread in copy_thread_tls() by fiddling
>with
>>> sp or the tls base address when creating a thread.
>> 
>> Could this be implemented by having glibc use a well known symbol
>name to define
>> the per-thread TLS area? If an high performance application wants to
>avoid any
>> relocations in accessing this variable it would define it and that
>definition
>> would override glibc's. This is how things work with malloc. glibc
>has a
>> default malloc implementation but we link jemalloc directly into our
>binaries.
>> in addition to changing the malloc implementation this means that
>calls to
>> malloc don't go through the PLT.
>
>Just to make sure I understand your proposal: defining a well known
>symbol
>with a weak attribute in glibc (or bionic...), e.g.:
>
>int32_t __thread __attribute__((weak)) __getcpu_cache;
>
>so that applications which care about bypassing the PLT can override it
>with:
>
>int32_t __thread __getcpu_cache;
>
>glibc/bionic would be responsible for calling the getcpu_cache() system
>call
>to register/unregister this TLS variable for each thread.
>
>One thing I would like to figure out is whether we can use this in a
>way that
>would allow introducing getcpu_cache() into applications and libraries
>(e.g. lttng-ust tracer) before it gets implemented into glibc, in a way
>that
>would keep forward compatibility for whenever it gets introduced in
>glibc.
>
>We can declare __getcpu_cache as a weak symbol in arbitrary libraries,
>and
>make them register/unregister the cache through the getcpu_cache
>syscall.
>The main thing that I would need to tweak at the kernel level within
>the
>system call would be to keep a refcount of the number of times the
>__getcpu_cache is registered per thread. This would allow multiple
>registrations,
>one per library (e.g. lttng-ust) and one for glibc, but we would
>validate
>that they all register the exact same address for a given thread.
>
>The reference counting trick should also work for cases where
>applications
>define a non-weak __getcpu_cache, and want to call the getcpu_cache
>system call to register it themselves (before glibc adds support for
>it).

This seems like something better done in a tiny common library, rather than the kernel or by playing symbol resolution games.

[toc] | [prev] | [next] | [standalone]

#1309446

----- On Jan 12, 2016, at 7:51 PM, Josh Triplett josh@joshtriplett.org wrote:

> On January 12, 2016 4:22:29 PM PST, Mathieu Desnoyers
> <mathieu.desnoyers@efficios.com> wrote:
>>----- On Jan 12, 2016, at 4:02 PM, Ben Maurer bmaurer@fb.com wrote:
>>
>>>> One idea I have would be to let the kernel reserve some space either
>>after the
>>>> first stack address (for a stack growing down) or at the beginning
>>of the
>>>> allocated TLS area for each thread in copy_thread_tls() by fiddling
>>with
>>>> sp or the tls base address when creating a thread.
>>> 
>>> Could this be implemented by having glibc use a well known symbol
>>name to define
>>> the per-thread TLS area? If an high performance application wants to
>>avoid any
>>> relocations in accessing this variable it would define it and that
>>definition
>>> would override glibc's. This is how things work with malloc. glibc
>>has a
>>> default malloc implementation but we link jemalloc directly into our
>>binaries.
>>> in addition to changing the malloc implementation this means that
>>calls to
>>> malloc don't go through the PLT.
>>
>>Just to make sure I understand your proposal: defining a well known
>>symbol
>>with a weak attribute in glibc (or bionic...), e.g.:
>>
>>int32_t __thread __attribute__((weak)) __getcpu_cache;
>>
>>so that applications which care about bypassing the PLT can override it
>>with:
>>
>>int32_t __thread __getcpu_cache;
>>
>>glibc/bionic would be responsible for calling the getcpu_cache() system
>>call
>>to register/unregister this TLS variable for each thread.
>>
>>One thing I would like to figure out is whether we can use this in a
>>way that
>>would allow introducing getcpu_cache() into applications and libraries
>>(e.g. lttng-ust tracer) before it gets implemented into glibc, in a way
>>that
>>would keep forward compatibility for whenever it gets introduced in
>>glibc.
>>
>>We can declare __getcpu_cache as a weak symbol in arbitrary libraries,
>>and
>>make them register/unregister the cache through the getcpu_cache
>>syscall.
>>The main thing that I would need to tweak at the kernel level within
>>the
>>system call would be to keep a refcount of the number of times the
>>__getcpu_cache is registered per thread. This would allow multiple
>>registrations,
>>one per library (e.g. lttng-ust) and one for glibc, but we would
>>validate
>>that they all register the exact same address for a given thread.
>>
>>The reference counting trick should also work for cases where
>>applications
>>define a non-weak __getcpu_cache, and want to call the getcpu_cache
>>system call to register it themselves (before glibc adds support for
>>it).
> 
> This seems like something better done in a tiny common library, rather than the
> kernel or by playing symbol resolution games.

It does not cost much to recommend a specific symbol name and marking the
symbol as weak in shared libraries. We could then also remove the "unregister"
command, which then means any library registering its cache cannot be unloaded.
This would remove the need to keep track of registration/unregistration with
a reference count within the kernel.

We should then document that a registered cpu_cache should not be freed before
its associated thread exits.

Would it be simple enough, or too simplistic ?

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

[toc] | [prev] | [next] | [standalone]

#1306811

Ignore my email I'd overlooked one bit of code misunderstood how it worked.

-----Original Message-----
From: linux-api-owner@vger.kernel.org [mailto:linux-api-owner@vger.kernel.org] On Behalf Of Seymour, Shane M
Sent: Tuesday, January 12, 2016 9:38 AM
To: Mathieu Desnoyers; Thomas Gleixner; Paul Turner; Andrew Hunter; Peter Zijlstra
Cc: linux-kernel@vger.kernel.org; linux-api@vger.kernel.org; Andy Lutomirski; Andi Kleen; Dave Watson; Chris Lameter; Ingo Molnar; Ben Maurer; Steven Rostedt; Paul E. McKenney; Josh Triplett; Linus Torvalds; Andrew Morton; Russell King; Catalin Marinas; Will Deacon; Michael Kerrisk
Subject: RE: [RFC PATCH 0/3] Implement getcpu_cache system call

Hi Mathieu,

I have some concerns and suggestions for you about this.

What's to stop someone in user space from requesting an arbitrarily large number of CPU # cache locations that the kernel needs to allocate memory to track and each time the task migrates to a new CPU it needs to update them all? Could you use it to dramatically slow down a system/task switching? Should there be a ulimit type value or a sysctl setting to limit the number that you're allowed to register per-task?

If you can just register consecutive addresses each 4 bytes apart the size of the structure required to keep track of it in the kernel looks to be 20 or 24 bytes long depending on kernel bitness (using kmalloc it should be 32 bytes allocated either way) so if you do something like tie up 1GiB of memory in user space registered with CPU cache locations it will tie up >8GiB of memory in the kernel and there will be a huge linked list that will take significant amounts of time to traverse. You could use it as a local denial of service attack to try and soak up memory and cause a kernel OOM because the kernel needs more memory to keep track of the request compared to the size of the memory used by user space to create it. There doesn't currently appear to be any upper bounds on the number that can be registered.

In terms of tracking what it's doing would you consider some sysfs attribute files (or something in debugfs) that tracked (these would all be in the add path so it shouldn't be performance sensitive):

1) The largest number of entries someone has created in the list in any task
2) The number of times (assuming you implement an upper bound on the number allowed) the upper bound is being hit (to allow someone to monitor for issues where the upper bound is being hit)

Assuming that something (e.g. glibc) is willing to register and make an entry available for the life of the task consider allowing one flag, for example, GETCPU_CACHE_PERSISTENT with GETCPU_CACHE_CMD_REGISTER and have a new command GETCPU_CACHE_CMD_GET_ PERSISTENT to allow someone to ask for the user space address of an entry that something has guaranteed will be there until the task ends. If none exist they can fall back and allocate a new one - it allows for better reuse of an existing resource but log a warning if you're forced to remove a persistent entry or someone attempts to unregister it (which should always fail) since someone in user space will have broken their promise that it will always be there until the task ends (that means persistent ones should be left to be torn down by the kernel not unregistered from user space). If you do this you might need to optimize the find process so it's more likely the first persistent one will be the first one found if you think someone is more likely to take the approach of asking for that first and falling back to creating a new one if there isn't already one present. Having this will also tend to limit the number of these that anyone will need to create if most libraries ask for a persistent entry first.

Thanks
Shane
�{.n�+�������+%��lzwm��b�맲��r��zX����)���w*jg��������ݢj/���z�ޖ��2�ޙ���&�)ߡ�a�����G���h��j:+v���w�٥

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web