Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1705213 > unrolled thread

Re: FSGSBASE ABI considerations

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: FSGSBASE ABI considerations Stas Sergeev <stsp@list.ru> - 2017-08-07 10:10 +0200
    Re: FSGSBASE ABI considerations Andy Lutomirski <luto@kernel.org> - 2017-08-07 18:30 +0200
      Re: FSGSBASE ABI considerations Christopher Lameter <cl@linux.com> - 2017-08-07 19:00 +0200
      Re: FSGSBASE ABI considerations Linus Torvalds <torvalds@linux-foundation.org> - 2017-08-07 19:40 +0200
        Re: FSGSBASE ABI considerations Andy Lutomirski <luto@kernel.org> - 2017-08-07 21:10 +0200
      Re: FSGSBASE ABI considerations Stas Sergeev <stsp@list.ru> - 2017-08-08 00:20 +0200

#1705213 — Re: FSGSBASE ABI considerations

Hello.

31.07.2017 06:05, Andy Lutomirski пишет:
>   - User code can use the new RD/WR FS/GS BASE instructions.
> Apparently some users really want this for, umm, userspace threading.
> Think Java.
I wonder how java avoids the lack of the user-space
continuations support while getting the userspace threading.
(swapcontext() calls to kernel for sigprocmask())

> The major disadvantage is that user code can use the new instructions.
> Now userspace is going to do totally stupid shite like writing some
> nonzero value to GS and then doing WRGSBASE or like linking some
> idiotic library that uses WRGSBASE into a perfectly innocent program
> like dosemu2 and resulting in utterly nonsensical descriptor state.
I don't think this can represent the problem, at least not
for dosemu1/2. dosemu2 does the full context switch via
a sighandler, dosemu1 uses iret with manually changing
all registers before jumping to compatibility mode. I don't
think any state changes done in long mode, can affect the
state after jump to compatibility mode.

> ----- interaction with modify_ldt() -----
>
> The first sticking point we'll hit is modify_ldt() and, in particular,
> what happens if you call modify_ldt() to change the base of a segment
> that is ioaded into gs by another thread in the same mm.
>
> Our current behavior here is nonsensical: on 32-bit kernels, FS would
> be fully refreshed on other threads and GS might be depending on
> compiler options.  On 64-bit kernels, neither FS nor GS is immediately
> refreshed.  Historically, we didn't refresh anything reliably.  On the
> bright side, this means that existing modify_ldt() users are (AFAIK)
> tolerant of somewhat crazy behavior.
>
> On an FSGSBASE-enabled system, I think we need to provide
> deterministic, documented, tested behavior.  I can think of three
> plausible choices:
>
> 1a. modify_ldt() immediately updates FSBASE and GSBASE all threads
> that reference the modified selector.
>
> 1b. modify_ldt() immediatley updates FSBASE and GSBASE on all threads
> that reference the LDT.
Does 1b mean that any call to modify_ldt(), even the
read call, will reset all bases to the ones of LDT? I think
this is the half-step. It clearly shows that you don't want
such state to ever exist, but why not to go a step further
and just make the bases to be reset not only by any
unrelated modify_ldt() call, but always on schedule?
You can state that using wrgsbase on non-zero selector
is invalid, reset it to LDT state and maybe send a signal
to the program so that it knows it did something wrong.
This may sound too rough, but I really don't see how it
differs from resetting all LDT bases on some unrelated
modify_ldt() that was done for read, not write.
Or you may want to reset selector to 0 rather than
base to LDT.

> 2. modify_ldt() leaves FSBASE and GSBASE alone on all threads.
>
> (2) is trivial to implement, whereas (1a) and (1b) are a bit nasty to
> implement when FSGSBASE is on.
>
> The tricky bit is that 32-bit kernels can't do (2), so, if we want
But do we have fsgsbase on 32bit kernels at all?
I think it works only in long mode, no?
I really tried to google some extensive description
on this feature, but failed.

> modify_ldt() to behave the same on 32-bit and 64-bit kernels, we're
> stuck with (1).
If you mean 1a, then to me it looks like a lot of efforts
for something no one ever needs.

> Thoughts?
I am far from the kernel development so my thoughts
may be naive, but IMHO you should just disallow this
by some means (like by doing a fixup on schedule() and
sending a signal). No one will suffer, people will just
write 0 to segreg first. Note that such a problem can
be provoked by the fact that the sighandler does not
reset the segregs to their default values, and someone
may simply forget to reset it to 0. You need to remind
him to do so rather than to invent the tricky code to
do something theoretically correct.

[toc] | [next] | [standalone]

#1705700

On Mon, Aug 7, 2017 at 1:06 AM, Stas Sergeev <stsp@list.ru> wrote:
> Hello.
>
> 31.07.2017 06:05, Andy Lutomirski пишет:
>>
>>   - User code can use the new RD/WR FS/GS BASE instructions.
>> Apparently some users really want this for, umm, userspace threading.
>> Think Java.
>
> I wonder how java avoids the lack of the user-space
> continuations support while getting the userspace threading.
> (swapcontext() calls to kernel for sigprocmask())
>
>> The major disadvantage is that user code can use the new instructions.
>> Now userspace is going to do totally stupid shite like writing some
>> nonzero value to GS and then doing WRGSBASE or like linking some
>> idiotic library that uses WRGSBASE into a perfectly innocent program
>> like dosemu2 and resulting in utterly nonsensical descriptor state.
>
> I don't think this can represent the problem, at least not
> for dosemu1/2. dosemu2 does the full context switch via
> a sighandler, dosemu1 uses iret with manually changing
> all registers before jumping to compatibility mode. I don't
> think any state changes done in long mode, can affect the
> state after jump to compatibility mode.

Hmm, right.  DOSEMU could get tripped up on the way back to long mode,
and we've discussed this a little bit before, but this is certainly
manageable.

>
>> ----- interaction with modify_ldt() -----
>>
>> The first sticking point we'll hit is modify_ldt() and, in particular,
>> what happens if you call modify_ldt() to change the base of a segment
>> that is ioaded into gs by another thread in the same mm.
>>
>> Our current behavior here is nonsensical: on 32-bit kernels, FS would
>> be fully refreshed on other threads and GS might be depending on
>> compiler options.  On 64-bit kernels, neither FS nor GS is immediately
>> refreshed.  Historically, we didn't refresh anything reliably.  On the
>> bright side, this means that existing modify_ldt() users are (AFAIK)
>> tolerant of somewhat crazy behavior.
>>
>> On an FSGSBASE-enabled system, I think we need to provide
>> deterministic, documented, tested behavior.  I can think of three
>> plausible choices:
>>
>> 1a. modify_ldt() immediately updates FSBASE and GSBASE all threads
>> that reference the modified selector.
>>
>> 1b. modify_ldt() immediatley updates FSBASE and GSBASE on all threads
>> that reference the LDT.
>
> Does 1b mean that any call to modify_ldt(), even the
> read call, will reset all bases to the ones of LDT?

Nah, just writes.  Doing it this way makes the tracking easier, since
we don't need to keep track of which selectors have been changed.

Note that 1a and 1b are indistinguishable to any user program that
doesn't use WRFSBASE or WRGSBASE, though.

> I think
> this is the half-step. It clearly shows that you don't want
> such state to ever exist, but why not to go a step further
> and just make the bases to be reset not only by any
> unrelated modify_ldt() call, but always on schedule?
> You can state that using wrgsbase on non-zero selector
> is invalid, reset it to LDT state and maybe send a signal
> to the program so that it knows it did something wrong.
> This may sound too rough, but I really don't see how it
> differs from resetting all LDT bases on some unrelated
> modify_ldt() that was done for read, not write.
> Or you may want to reset selector to 0 rather than
> base to LDT.

Windows does something sort of like this (I think), but I don't like
this solution.  I fully expect that someone will write a program that
does:

old = rdgsbase();
wrgsbase(new);
call_very_fast_function();
wrgsbase(old);

This will work if GS == 0, which is fine.  The problem is that it will
*also* work if GS != 0 with very high probability, especially if this
code sequence is right after some operation that sleeps.  And then
we'll get random crashes with very low probability, depending on where
the scheduler hits.

>
>> 2. modify_ldt() leaves FSBASE and GSBASE alone on all threads.
>>
>> (2) is trivial to implement, whereas (1a) and (1b) are a bit nasty to
>> implement when FSGSBASE is on.
>>
>> The tricky bit is that 32-bit kernels can't do (2), so, if we want
>
> But do we have fsgsbase on 32bit kernels at all?

No, and we don't have MSR_FS_BASE, etc either, so the scheduler
basically can't preserve the base across a context switch.

> I think it works only in long mode, no?
> I really tried to google some extensive description
> on this feature, but failed.
>
>> modify_ldt() to behave the same on 32-bit and 64-bit kernels, we're
>> stuck with (1).
>
> If you mean 1a, then to me it looks like a lot of efforts
> for something no one ever needs.
>
>> Thoughts?
>
> I am far from the kernel development so my thoughts
> may be naive, but IMHO you should just disallow this
> by some means (like by doing a fixup on schedule() and
> sending a signal). No one will suffer, people will just
> write 0 to segreg first. Note that such a problem can
> be provoked by the fact that the sighandler does not
> reset the segregs to their default values, and someone
> may simply forget to reset it to 0. You need to remind
> him to do so rather than to invent the tricky code to
> do something theoretically correct.

I would *love* to disallow it.  The problem is that I don't believe it
to be possible in a way that doesn't cause more problems than it
solves.

--Andy

[toc] | [prev] | [next] | [standalone]

#1705715

I hope this will finally enable thread local support to work in a sane way
in gcc so that we can actually use it in kernel space and get rid of all
the this_cpu_xxx() macros?

And thread local RMVs primitives may actually be provided by gcc and
be usable in user space so that we can write user space code with
effective cpu local variable access?

[toc] | [prev] | [next] | [standalone]

#1705732

On Mon, Aug 7, 2017 at 9:20 AM, Andy Lutomirski <luto@kernel.org> wrote:
>
> Windows does something sort of like this (I think), but I don't like
> this solution.  I fully expect that someone will write a program that
> does:
>
> old = rdgsbase();
> wrgsbase(new);
> call_very_fast_function();
> wrgsbase(old);
>
> This will work if GS == 0, which is fine.  The problem is that it will
> *also* work if GS != 0 with very high probability, especially if this
> code sequence is right after some operation that sleeps.  And then
> we'll get random crashes with very low probability, depending on where
> the scheduler hits.

It will work reliably if you just make the scheduler save/restore the
base rather than the selector.

I really think you need to walk away from the "selector is meaningful"
model. Yes, yes, it's the legacy model, but it's the *insane* model.

So screw the selector. It doesn't matter. We'll need to save/restore
the value, but that's it. What we *really* save and restore is just
the base pointer.

Why do you care so much about the selector? If people *don't* use the
fsgsbase, then the selector and the base of the segment will always
match anyway (modulo the system calls that actually change the
gdt/ldt, and we can just sat that *then* selectors matter).

And if people *do* use fsgsbase, then the selector is by definition
not important.

So just make the scheduler save the base first, and restore it last.
End of problem. Your user-space code above just works. There is no
race, i doesn't matter one whit whether GS is 0 ir not, there simply
is no problem.

So just what is the problem you're trying to solve?

               Linus

[toc] | [prev] | [next] | [standalone]

#1705810

On Mon, Aug 7, 2017 at 10:35 AM, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
> On Mon, Aug 7, 2017 at 9:20 AM, Andy Lutomirski <luto@kernel.org> wrote:
>>
>> Windows does something sort of like this (I think), but I don't like
>> this solution.  I fully expect that someone will write a program that
>> does:
>>
>> old = rdgsbase();
>> wrgsbase(new);
>> call_very_fast_function();
>> wrgsbase(old);
>>
>> This will work if GS == 0, which is fine.  The problem is that it will
>> *also* work if GS != 0 with very high probability, especially if this
>> code sequence is right after some operation that sleeps.  And then
>> we'll get random crashes with very low probability, depending on where
>> the scheduler hits.
>
> It will work reliably if you just make the scheduler save/restore the
> base rather than the selector.
>
> I really think you need to walk away from the "selector is meaningful"
> model. Yes, yes, it's the legacy model, but it's the *insane* model.
>
> So screw the selector. It doesn't matter. We'll need to save/restore
> the value, but that's it. What we *really* save and restore is just
> the base pointer.
>
> Why do you care so much about the selector? If people *don't* use the
> fsgsbase, then the selector and the base of the segment will always
> match anyway (modulo the system calls that actually change the
> gdt/ldt, and we can just sat that *then* selectors matter).
>
> And if people *do* use fsgsbase, then the selector is by definition
> not important.
>
> So just make the scheduler save the base first, and restore it last.
> End of problem. Your user-space code above just works. There is no
> race, i doesn't matter one whit whether GS is 0 ir not, there simply
> is no problem.

I agree completely.  The scheduler should do exactly this and, with my
patches applied, it does.

>
> So just what is the problem you're trying to solve?
>

I'm trying to avoid a situation where we implement that policy and the
interaction with modify_ldt() becomes very strange.  Linux has a long
history of having ill-defined semantics x86_64, and I don't want to
make it worse.

If we *just* change the way the scheduler works, then we end up with
modify_ldt() behaving determinstically on IVB+ and behaving
deterministically on 32-bit kernels, but having that deterministic
behavior be *different*.  This makes me rather unhappy about the whole
situation.

Also, I don't want to break gdb, and even telling whether a change
breaks gdb is an incredible PITA.  Whern GDB saves and restores a
context, it currently restores the base first and the selector second,
and I have no idea whether gdb expects restoring the selector to
update the base.

[toc] | [prev] | [next] | [standalone]

#1705893

07.08.2017 19:20, Andy Lutomirski пишет:
>> I think
>> this is the half-step. It clearly shows that you don't want
>> such state to ever exist, but why not to go a step further
>> and just make the bases to be reset not only by any
>> unrelated modify_ldt() call, but always on schedule?
>> You can state that using wrgsbase on non-zero selector
>> is invalid, reset it to LDT state and maybe send a signal
>> to the program so that it knows it did something wrong.
>> This may sound too rough, but I really don't see how it
>> differs from resetting all LDT bases on some unrelated
>> modify_ldt() that was done for read, not write.
>> Or you may want to reset selector to 0 rather than
>> base to LDT.
> Windows does something sort of like this (I think), but I don't like
> this solution.  I fully expect that someone will write a program that
> does:
>
> old = rdgsbase();
> wrgsbase(new);
> call_very_fast_function();
> wrgsbase(old);
>
> This will work if GS == 0, which is fine.  The problem is that it will
> *also* work if GS != 0 with very high probability, especially if this
> code sequence is right after some operation that sleeps.  And then
> we'll get random crashes with very low probability, depending on where
> the scheduler hits.
So, as Linus already pointed, if the fixup is to
zero out the selector, then this will still work fine.


>> I am far from the kernel development so my thoughts
>> may be naive, but IMHO you should just disallow this
>> by some means (like by doing a fixup on schedule() and
>> sending a signal). No one will suffer, people will just
>> write 0 to segreg first. Note that such a problem can
>> be provoked by the fact that the sighandler does not
>> reset the segregs to their default values, and someone
>> may simply forget to reset it to 0. You need to remind
>> him to do so rather than to invent the tricky code to
>> do something theoretically correct.
> I would *love* to disallow it.  The problem is that I don't believe it
> to be possible in a way that doesn't cause more problems than it
> solves.
I wonder if sending a signal (after doing a fixup)
is too much of a punishment?

> I'm trying to avoid a situation where we implement that policy and the
> interaction with modify_ldt() becomes very strange.
IMHO if you do the fixup on schedule (like setting
the selector to zero), then the interaction with
modify_ldt() is completely avoided, i.e. modify_ldt()
should then never special-case the threads that
did wrgsbase. So if something inconsistent comes
out, then it was likely there already without wrgsbase.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web