Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1551446 > unrolled thread

[RFC] x86/mm/KASLR: Remap GDTs at fixed location

Back to article view | Back to linux.kernel

Contents

  [RFC] x86/mm/KASLR: Remap GDTs at fixed location Thomas Garnier <thgarnie@google.com> - 2017-01-04 23:20 +0100
    Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Ingo Molnar <mingo@kernel.org> - 2017-01-05 09:30 +0100
      Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Arjan van de Ven <arjan@linux.intel.com> - 2017-01-05 16:10 +0100
        Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Thomas Garnier <thgarnie@google.com> - 2017-01-05 17:50 +0100
          Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Arjan van de Ven <arjan@linux.intel.com> - 2017-01-05 20:10 +0100
            Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Thomas Garnier <thgarnie@google.com> - 2017-01-05 20:10 +0100
          Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Borislav Petkov <bp@alien8.de> - 2017-01-06 18:50 +0100
            Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Thomas Garnier <thgarnie@google.com> - 2017-01-06 19:10 +0100
      Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Thomas Garnier <thgarnie@google.com> - 2017-01-05 17:40 +0100
        Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Ingo Molnar <mingo@kernel.org> - 2017-01-06 07:40 +0100
    Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Thomas Garnier <thgarnie@google.com> - 2017-01-05 19:20 +0100
      Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Andy Lutomirski <luto@amacapital.net> - 2017-01-05 19:40 +0100
        Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Thomas Garnier <thgarnie@google.com> - 2017-01-05 19:40 +0100
      Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Arjan van de Ven <arjan@linux.intel.com> - 2017-01-05 20:10 +0100
        Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Thomas Garnier <thgarnie@google.com> - 2017-01-05 20:20 +0100
          Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Andy Lutomirski <luto@kernel.org> - 2017-01-05 21:20 +0100
            Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Thomas Garnier <thgarnie@google.com> - 2017-01-05 22:10 +0100
              Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Andy Lutomirski <luto@amacapital.net> - 2017-01-05 22:30 +0100
                Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Thomas Garnier <thgarnie@google.com> - 2017-01-05 23:00 +0100
                  Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Ingo Molnar <mingo@kernel.org> - 2017-01-06 08:00 +0100
                    Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Thomas Garnier <thgarnie@google.com> - 2017-01-06 19:10 +0100
                      Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Andy Lutomirski <luto@kernel.org> - 2017-01-06 23:30 +0100
                        Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Thomas Garnier <thgarnie@google.com> - 2017-01-07 00:00 +0100
                          Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Andy Lutomirski <luto@kernel.org> - 2017-01-07 00:40 +0100
                            Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Ingo Molnar <mingo@kernel.org> - 2017-01-07 08:50 +0100
                              Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Andy Lutomirski <luto@amacapital.net> - 2017-01-07 17:00 +0100
                          Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Ingo Molnar <mingo@kernel.org> - 2017-01-07 08:40 +0100
                            Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Andy Lutomirski <luto@amacapital.net> - 2017-01-07 17:10 +0100
                            Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Thomas Garnier <thgarnie@google.com> - 2017-01-09 23:40 +0100
                              Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Ingo Molnar <mingo@kernel.org> - 2017-01-10 11:30 +0100
                                Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Thomas Garnier <thgarnie@google.com> - 2017-01-10 18:20 +0100
            Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Linus Torvalds <torvalds@linux-foundation.org> - 2017-01-06 00:10 +0100
              Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Thomas Garnier <thgarnie@google.com> - 2017-01-06 00:30 +0100
              Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Andy Lutomirski <luto@amacapital.net> - 2017-01-06 03:40 +0100
                Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Thomas Garnier <thgarnie@google.com> - 2017-01-06 19:10 +0100
                  Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Andy Lutomirski <luto@kernel.org> - 2017-01-06 23:00 +0100
                    Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Ingo Molnar <mingo@kernel.org> - 2017-01-07 08:50 +0100
        Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Ingo Molnar <mingo@kernel.org> - 2017-01-06 07:50 +0100
    Re: [RFC] x86/mm/KASLR: Remap GDTs at fixed location Andy Lutomirski <luto@amacapital.net> - 2017-01-05 19:30 +0100

Page 2 of 2 — ← Prev page 1 [2]

#1553002

On Thu, Jan 5, 2017 at 10:49 PM, Ingo Molnar <mingo@kernel.org> wrote:
>
> * Thomas Garnier <thgarnie@google.com> wrote:
>
>> >> Not sure I fully understood and I don't want to miss an important point. Do
>> >> you mean making GDT (remapping and per-cpu) read-only and switch the
>> >> writeable flag only when we write to the per-cpu entry?
>> >
>> > What I mean is: write to the GDT through normal percpu access (or whatever the
>> > normal mapping is) but load a read-only alias into the GDT register.  As long
>> > as nothing ever tries to write through the GDTR alias, no page faults will be
>> > generated.  So we just need to make sure that nothing ever writes to it
>> > through GDTR.  AFAIK the only reason the CPU ever writes to the address in
>> > GDTR is to set an accessed bit.
>>
>> A write is made when we use load_TR_desc (ltr). I didn't see any other yet.
>
> Is this write to the GDT, generated by the LTR instruction, done unconditionally
> by the hardware?
>

That was my experience. I didn't look into details. Do you think we
could change something so that ltr never writes to the GDT? (just mark
the TSS entry busy).

> Thanks,
>
>         Ingo



-- 
Thomas

[toc] | [prev] | [next] | [standalone]

#1553331

On Fri, Jan 6, 2017 at 10:03 AM, Thomas Garnier <thgarnie@google.com> wrote:
> On Thu, Jan 5, 2017 at 10:49 PM, Ingo Molnar <mingo@kernel.org> wrote:
>>
>> * Thomas Garnier <thgarnie@google.com> wrote:
>>
>>> >> Not sure I fully understood and I don't want to miss an important point. Do
>>> >> you mean making GDT (remapping and per-cpu) read-only and switch the
>>> >> writeable flag only when we write to the per-cpu entry?
>>> >
>>> > What I mean is: write to the GDT through normal percpu access (or whatever the
>>> > normal mapping is) but load a read-only alias into the GDT register.  As long
>>> > as nothing ever tries to write through the GDTR alias, no page faults will be
>>> > generated.  So we just need to make sure that nothing ever writes to it
>>> > through GDTR.  AFAIK the only reason the CPU ever writes to the address in
>>> > GDTR is to set an accessed bit.
>>>
>>> A write is made when we use load_TR_desc (ltr). I didn't see any other yet.
>>
>> Is this write to the GDT, generated by the LTR instruction, done unconditionally
>> by the hardware?
>>
>
> That was my experience. I didn't look into details. Do you think we
> could change something so that ltr never writes to the GDT? (just mark
> the TSS entry busy).

No, and I had the way this worked on 64-bit wrong.  LTR requires an
available TSS and changes it to busy.  So here are my thoughts on how
this should work:

Let's get rid of any connection between this code and KASLR.  Every
time KASLR makes something work differently, a kitten turns all
Schrödinger on us.  This is moving the GDT to the fixmap, plain and
simple.  For now, make it one page per CPU and don't worry about the
GDT limit.

On 32-bit, we're going to have to make the fixmap GDT be read-write
because making it read-only will break double-fault handling.

On 64-bit, we can use your trick of temporarily mapping the GDT
read-write every time we load TR, which should happen very rarely.
Alternatively, we can reload the *GDT* every time we reload TR, which
should be comparably slow.  This is going to regress performance in
the extremely rare case where KVM exits to a process that uses
ioperm() (I think), but I doubt anyone cares.  Or maybe we could
arrange to never reload TR when GDT points at the fixmap by having KVM
set the host GDT to the direct version and letting KVM's code to
reload the GDT switch to the fixmap copy.

If we need a quirk to keep the fixmap copy read-write, so be it.

None of this should depend on KASLR.  IMO it should happen unconditionally.

Once all if it works, then we can build on it to allocate four pages
per CPU (with the extra three pointing to the zero page) and speeding
up KVM.

--Andy

[toc] | [prev] | [next] | [standalone]

#1553455

On Fri, Jan 6, 2017 at 1:59 PM, Andy Lutomirski <luto@kernel.org> wrote:
> On Fri, Jan 6, 2017 at 10:03 AM, Thomas Garnier <thgarnie@google.com> wrote:
>> On Thu, Jan 5, 2017 at 10:49 PM, Ingo Molnar <mingo@kernel.org> wrote:
>>>
>>> * Thomas Garnier <thgarnie@google.com> wrote:
>>>
>>>> >> Not sure I fully understood and I don't want to miss an important point. Do
>>>> >> you mean making GDT (remapping and per-cpu) read-only and switch the
>>>> >> writeable flag only when we write to the per-cpu entry?
>>>> >
>>>> > What I mean is: write to the GDT through normal percpu access (or whatever the
>>>> > normal mapping is) but load a read-only alias into the GDT register.  As long
>>>> > as nothing ever tries to write through the GDTR alias, no page faults will be
>>>> > generated.  So we just need to make sure that nothing ever writes to it
>>>> > through GDTR.  AFAIK the only reason the CPU ever writes to the address in
>>>> > GDTR is to set an accessed bit.
>>>>
>>>> A write is made when we use load_TR_desc (ltr). I didn't see any other yet.
>>>
>>> Is this write to the GDT, generated by the LTR instruction, done unconditionally
>>> by the hardware?
>>>
>>
>> That was my experience. I didn't look into details. Do you think we
>> could change something so that ltr never writes to the GDT? (just mark
>> the TSS entry busy).
>
> No, and I had the way this worked on 64-bit wrong.  LTR requires an
> available TSS and changes it to busy.  So here are my thoughts on how
> this should work:
>
> Let's get rid of any connection between this code and KASLR.  Every
> time KASLR makes something work differently, a kitten turns all
> Schrödinger on us.  This is moving the GDT to the fixmap, plain and
> simple.  For now, make it one page per CPU and don't worry about the
> GDT limit.

I am all for this change but that's more significant.

Ingo: What do you think about that?

>
> On 32-bit, we're going to have to make the fixmap GDT be read-write
> because making it read-only will break double-fault handling.
>
> On 64-bit, we can use your trick of temporarily mapping the GDT
> read-write every time we load TR, which should happen very rarely.
> Alternatively, we can reload the *GDT* every time we reload TR, which
> should be comparably slow.  This is going to regress performance in
> the extremely rare case where KVM exits to a process that uses
> ioperm() (I think), but I doubt anyone cares.  Or maybe we could
> arrange to never reload TR when GDT points at the fixmap by having KVM
> set the host GDT to the direct version and letting KVM's code to
> reload the GDT switch to the fixmap copy.
>
> If we need a quirk to keep the fixmap copy read-write, so be it.
>
> None of this should depend on KASLR.  IMO it should happen unconditionally.
>

I looked back at the fixmap, and I can see a way it could be done
(using NR_CPUS) like the other fixmap ranges. It would limit the
number of cpus to 512 (there is 2M memory left on fixmap on the
default configuration). That's if we never add any other fixmap on
x64. I don't know if it is an acceptable number and if the fixmap
region could be increased. (128 if we do your kvm trick, of course).

Ingo: What do you think?

> Once all if it works, then we can build on it to allocate four pages
> per CPU (with the extra three pointing to the zero page) and speeding
> up KVM.
>
> --Andy



-- 
Thomas

[toc] | [prev] | [next] | [standalone]

#1553503

On Fri, Jan 6, 2017 at 2:54 PM, Thomas Garnier <thgarnie@google.com> wrote:
> On Fri, Jan 6, 2017 at 1:59 PM, Andy Lutomirski <luto@kernel.org> wrote:
>> On Fri, Jan 6, 2017 at 10:03 AM, Thomas Garnier <thgarnie@google.com> wrote:
>>> On Thu, Jan 5, 2017 at 10:49 PM, Ingo Molnar <mingo@kernel.org> wrote:
>>>>
>>>> * Thomas Garnier <thgarnie@google.com> wrote:
>>>>
>>>>> >> Not sure I fully understood and I don't want to miss an important point. Do
>>>>> >> you mean making GDT (remapping and per-cpu) read-only and switch the
>>>>> >> writeable flag only when we write to the per-cpu entry?
>>>>> >
>>>>> > What I mean is: write to the GDT through normal percpu access (or whatever the
>>>>> > normal mapping is) but load a read-only alias into the GDT register.  As long
>>>>> > as nothing ever tries to write through the GDTR alias, no page faults will be
>>>>> > generated.  So we just need to make sure that nothing ever writes to it
>>>>> > through GDTR.  AFAIK the only reason the CPU ever writes to the address in
>>>>> > GDTR is to set an accessed bit.
>>>>>
>>>>> A write is made when we use load_TR_desc (ltr). I didn't see any other yet.
>>>>
>>>> Is this write to the GDT, generated by the LTR instruction, done unconditionally
>>>> by the hardware?
>>>>
>>>
>>> That was my experience. I didn't look into details. Do you think we
>>> could change something so that ltr never writes to the GDT? (just mark
>>> the TSS entry busy).
>>
>> No, and I had the way this worked on 64-bit wrong.  LTR requires an
>> available TSS and changes it to busy.  So here are my thoughts on how
>> this should work:
>>
>> Let's get rid of any connection between this code and KASLR.  Every
>> time KASLR makes something work differently, a kitten turns all
>> Schrödinger on us.  This is moving the GDT to the fixmap, plain and
>> simple.  For now, make it one page per CPU and don't worry about the
>> GDT limit.
>
> I am all for this change but that's more significant.
>
> Ingo: What do you think about that?
>
>>
>> On 32-bit, we're going to have to make the fixmap GDT be read-write
>> because making it read-only will break double-fault handling.
>>
>> On 64-bit, we can use your trick of temporarily mapping the GDT
>> read-write every time we load TR, which should happen very rarely.
>> Alternatively, we can reload the *GDT* every time we reload TR, which
>> should be comparably slow.  This is going to regress performance in
>> the extremely rare case where KVM exits to a process that uses
>> ioperm() (I think), but I doubt anyone cares.  Or maybe we could
>> arrange to never reload TR when GDT points at the fixmap by having KVM
>> set the host GDT to the direct version and letting KVM's code to
>> reload the GDT switch to the fixmap copy.
>>
>> If we need a quirk to keep the fixmap copy read-write, so be it.
>>
>> None of this should depend on KASLR.  IMO it should happen unconditionally.
>>
>
> I looked back at the fixmap, and I can see a way it could be done
> (using NR_CPUS) like the other fixmap ranges. It would limit the
> number of cpus to 512 (there is 2M memory left on fixmap on the
> default configuration). That's if we never add any other fixmap on
> x64. I don't know if it is an acceptable number and if the fixmap
> region could be increased. (128 if we do your kvm trick, of course).
>

IIRC we need 4096 CPUs.  But that 2M limit seems eminently fixable.  I
just tried sticking 4096 pages of nothing right near the top of the
fixmap and the only problem I saw was that I had to move MODULES_END
down a little bit.

--Andy

P.S. Let's do the move to the fixmap, read/write as a separate patch.
That will make bisecting much easier.

[toc] | [prev] | [next] | [standalone]

#1553608

* Andy Lutomirski <luto@kernel.org> wrote:

> > I looked back at the fixmap, and I can see a way it could be done (using 
> > NR_CPUS) like the other fixmap ranges. It would limit the number of cpus to 
> > 512 (there is 2M memory left on fixmap on the default configuration). That's 
> > if we never add any other fixmap on x64. I don't know if it is an acceptable 
> > number and if the fixmap region could be increased. (128 if we do your kvm 
> > trick, of course).
> 
> IIRC we need 4096 CPUs.

On 64-bit the limit is 8192 CPUs, and the SGI guys are relying on that up to the 
tune of 6144 cores already, and I'd say the 64-bit CPU count is likely to go up 
further with 5-level paging.

On 32-bit the reasonable CPU limit is the number that the Intel 32-bit cluster 
computing nodes use. The latest public numbers are I think 36 'tiles' with each 
tile being a 2-CPU SMT core - i.e. a limit of 72 CPUs has to be maintained. 
(They'll obviously go to 64-bit as well so this problem will go away in a hardware 
generation or two.)

So I'd say 128 CPUs on 32-bit should be a reasonable practical limit going 
forward. Right now our 32-bit limit is 512 CPUs IIRC, but I don't think any real 
hardware in production is reaching that.

> P.S. Let's do the move to the fixmap, read/write as a separate patch. That will 
> make bisecting much easier.

Absolutely, but this has to be within the same series, as the interim fixmap-only 
step is less secure in some circumstances: we are moving the writable GDT from a 
previously randomized location to a fixed location.

Thanks,

	Ingo

[toc] | [prev] | [next] | [standalone]

#1553683

On Fri, Jan 6, 2017 at 11:45 PM, Ingo Molnar <mingo@kernel.org> wrote:
>
> * Andy Lutomirski <luto@kernel.org> wrote:

>> P.S. Let's do the move to the fixmap, read/write as a separate patch. That will
>> make bisecting much easier.
>
> Absolutely, but this has to be within the same series, as the interim fixmap-only
> step is less secure in some circumstances: we are moving the writable GDT from a
> previously randomized location to a fixed location.

True, but despite being randomized its location was never even
remotely secret.  (Except on Kaby Lake or Foobar Lake or whatever CPU
that is.)

--Andy

[toc] | [prev] | [next] | [standalone]

#1553601

* Thomas Garnier <thgarnie@google.com> wrote:

> > No, and I had the way this worked on 64-bit wrong.  LTR requires an
> > available TSS and changes it to busy.  So here are my thoughts on how
> > this should work:
> >
> > Let's get rid of any connection between this code and KASLR.  Every
> > time KASLR makes something work differently, a kitten turns all
> > Schrödinger on us.  This is moving the GDT to the fixmap, plain and
> > simple.  For now, make it one page per CPU and don't worry about the
> > GDT limit.
> 
> I am all for this change but that's more significant.
> 
> Ingo: What do you think about that?

I agree with Andy: as I alluded to earlier as well this should be an unconditional 
change (tested properly, etc.) that robustifies the GDT mapping for everyone. That 
KASLR kernels improve too is a happy side effect!

> > On 32-bit, we're going to have to make the fixmap GDT be read-write because 
> > making it read-only will break double-fault handling.
> >
> > On 64-bit, we can use your trick of temporarily mapping the GDT read-write 
> > every time we load TR, which should happen very rarely. Alternatively, we can 
> > reload the *GDT* every time we reload TR, which should be comparably slow.  
> > This is going to regress performance in the extremely rare case where KVM 
> > exits to a process that uses ioperm() (I think), but I doubt anyone cares.  Or 
> > maybe we could arrange to never reload TR when GDT points at the fixmap by 
> > having KVM set the host GDT to the direct version and letting KVM's code to 
> > reload the GDT switch to the fixmap copy.

Please check whether the LTR write generates a page fault to a RO PTE even if the 
busy bit is already set. LTR is pretty slow which suggests that it's microcode, 
and microcode is usually not sloppy about such things: i.e. LTR would only 
generate an unconditional write if there's a compatibility dependency on it. But I 
could easily be wrong ...

> > If we need a quirk to keep the fixmap copy read-write, so be it.
> >
> > None of this should depend on KASLR.  IMO it should happen unconditionally.
> 
> I looked back at the fixmap, and I can see a way it could be done
> (using NR_CPUS) like the other fixmap ranges. It would limit the
> number of cpus to 512 (there is 2M memory left on fixmap on the
> default configuration). That's if we never add any other fixmap on
> x64. I don't know if it is an acceptable number and if the fixmap
> region could be increased. (128 if we do your kvm trick, of course).
> 
> Ingo: What do you think?

I think we should scale the fixmap size flexibly with NR_CPUs on 64-bit, and we 
should limit CPUs on 32-bit to a reasonable value.

I.e. let's just do it, if we run into problems it's all solvable AFAICS.

Thanks,

	Ingo

[toc] | [prev] | [next] | [standalone]

#1553684

On Fri, Jan 6, 2017 at 11:35 PM, Ingo Molnar <mingo@kernel.org> wrote:
>
> * Thomas Garnier <thgarnie@google.com> wrote:
>
>> > No, and I had the way this worked on 64-bit wrong.  LTR requires an
>> > available TSS and changes it to busy.  So here are my thoughts on how
>> > this should work:
>> >
>> > Let's get rid of any connection between this code and KASLR.  Every
>> > time KASLR makes something work differently, a kitten turns all
>> > Schrödinger on us.  This is moving the GDT to the fixmap, plain and
>> > simple.  For now, make it one page per CPU and don't worry about the
>> > GDT limit.
>>
>> I am all for this change but that's more significant.
>>
>> Ingo: What do you think about that?
>
> I agree with Andy: as I alluded to earlier as well this should be an unconditional
> change (tested properly, etc.) that robustifies the GDT mapping for everyone. That
> KASLR kernels improve too is a happy side effect!
>
>> > On 32-bit, we're going to have to make the fixmap GDT be read-write because
>> > making it read-only will break double-fault handling.
>> >
>> > On 64-bit, we can use your trick of temporarily mapping the GDT read-write
>> > every time we load TR, which should happen very rarely. Alternatively, we can
>> > reload the *GDT* every time we reload TR, which should be comparably slow.
>> > This is going to regress performance in the extremely rare case where KVM
>> > exits to a process that uses ioperm() (I think), but I doubt anyone cares.  Or
>> > maybe we could arrange to never reload TR when GDT points at the fixmap by
>> > having KVM set the host GDT to the direct version and letting KVM's code to
>> > reload the GDT switch to the fixmap copy.
>
> Please check whether the LTR write generates a page fault to a RO PTE even if the
> busy bit is already set. LTR is pretty slow which suggests that it's microcode,
> and microcode is usually not sloppy about such things: i.e. LTR would only
> generate an unconditional write if there's a compatibility dependency on it. But I
> could easily be wrong ...

The SDM says:

IF segment descriptor is not for an available TSS
THEN #GP(segment selector); FI;

so I think it's #GP not #PF.

>
>> > If we need a quirk to keep the fixmap copy read-write, so be it.
>> >
>> > None of this should depend on KASLR.  IMO it should happen unconditionally.
>>
>> I looked back at the fixmap, and I can see a way it could be done
>> (using NR_CPUS) like the other fixmap ranges. It would limit the
>> number of cpus to 512 (there is 2M memory left on fixmap on the
>> default configuration). That's if we never add any other fixmap on
>> x64. I don't know if it is an acceptable number and if the fixmap
>> region could be increased. (128 if we do your kvm trick, of course).
>>
>> Ingo: What do you think?
>
> I think we should scale the fixmap size flexibly with NR_CPUs on 64-bit, and we
> should limit CPUs on 32-bit to a reasonable value.

Unless the headers are even more tangled than usual, I think we could
just handle this exactly.  The top and bottom of the fixmap are known
exactly at compile time, so we should just be able to make the next mm
range adjust its start point accordingly.  The main issue right now
seems to be that MODULES_END is hard-coded.

For 32-bit, the task gate in #DF is going to be a show-stopper for an
RO fixmap, I think.  I'm still slightly in favor of enabling the code
but with an RW mapping on 32-bit, though.  And anyone running hundreds
to thousands of CPUs on 32-bit is nuts anyway.

--Andy

[toc] | [prev] | [next] | [standalone]

#1554768

On Fri, Jan 6, 2017 at 11:35 PM, Ingo Molnar <mingo@kernel.org> wrote:
>
> * Thomas Garnier <thgarnie@google.com> wrote:
>
>> > No, and I had the way this worked on 64-bit wrong.  LTR requires an
>> > available TSS and changes it to busy.  So here are my thoughts on how
>> > this should work:
>> >
>> > Let's get rid of any connection between this code and KASLR.  Every
>> > time KASLR makes something work differently, a kitten turns all
>> > Schrödinger on us.  This is moving the GDT to the fixmap, plain and
>> > simple.  For now, make it one page per CPU and don't worry about the
>> > GDT limit.
>>
>> I am all for this change but that's more significant.
>>
>> Ingo: What do you think about that?
>
> I agree with Andy: as I alluded to earlier as well this should be an unconditional
> change (tested properly, etc.) that robustifies the GDT mapping for everyone. That
> KASLR kernels improve too is a happy side effect!
>
>> > On 32-bit, we're going to have to make the fixmap GDT be read-write because
>> > making it read-only will break double-fault handling.
>> >
>> > On 64-bit, we can use your trick of temporarily mapping the GDT read-write
>> > every time we load TR, which should happen very rarely. Alternatively, we can
>> > reload the *GDT* every time we reload TR, which should be comparably slow.
>> > This is going to regress performance in the extremely rare case where KVM
>> > exits to a process that uses ioperm() (I think), but I doubt anyone cares.  Or
>> > maybe we could arrange to never reload TR when GDT points at the fixmap by
>> > having KVM set the host GDT to the direct version and letting KVM's code to
>> > reload the GDT switch to the fixmap copy.
>
> Please check whether the LTR write generates a page fault to a RO PTE even if the
> busy bit is already set. LTR is pretty slow which suggests that it's microcode,
> and microcode is usually not sloppy about such things: i.e. LTR would only
> generate an unconditional write if there's a compatibility dependency on it. But I
> could easily be wrong ...
>

Coming back on that after a bit more testing. The LTR instruction
check if the busy bit is already set, if already set then it will just
issue a #GP given a bad selector:

[    0.000000] general protection fault: 0040 [#1] SMP
...
[    0.000000] RIP: 0010:native_load_tr_desc+0x9/0x10
...
[    0.000000] Call Trace:
[    0.000000]  cpu_init+0x2d0/0x3c0
[    0.000000]  trap_init+0x2a2/0x312
[    0.000000]  start_kernel+0x1fb/0x43b
[    0.000000]  ? set_init_arg+0x55/0x55
[    0.000000]  ? early_idt_handler_array+0x120/0x120
[    0.000000]  x86_64_start_reservations+0x2a/0x2c
[    0.000000]  x86_64_start_kernel+0x13d/0x14c
[    0.000000]  start_cpu+0x14/0x14

I assume that's in this part of the pseudo-code:

if(!IsWithinDescriptorTableLimit(Source.Offset) || Source.Type !=
TypeGlobal) Exception(GP(SegmentSelector));
SegmentDescriptor = ReadSegmentDescriptor();
if(!IsForAnAvailableTSS(SegmentDescriptor))
Exception(GP(SegmentSelector)); <---- That's where I got the GP
TSSSegmentDescriptor.Busy = 1;
<------------------------------------------------------------------
That's the pagefault I get otherwise
//Locked read-modify-write operation on the entire descriptor when
setting busy flag
TaskRegister.SegmentSelector = Source;
TaskRegister.SegmentDescriptor.TSSSegmentDescriptor;

I assume the best option would be to make the remap read-write for the
LTR instruction. What do you think?

>> > If we need a quirk to keep the fixmap copy read-write, so be it.
>> >
>> > None of this should depend on KASLR.  IMO it should happen unconditionally.
>>
>> I looked back at the fixmap, and I can see a way it could be done
>> (using NR_CPUS) like the other fixmap ranges. It would limit the
>> number of cpus to 512 (there is 2M memory left on fixmap on the
>> default configuration). That's if we never add any other fixmap on
>> x64. I don't know if it is an acceptable number and if the fixmap
>> region could be increased. (128 if we do your kvm trick, of course).
>>
>> Ingo: What do you think?
>
> I think we should scale the fixmap size flexibly with NR_CPUs on 64-bit, and we
> should limit CPUs on 32-bit to a reasonable value.
>
> I.e. let's just do it, if we run into problems it's all solvable AFAICS.
>
> Thanks,
>
>         Ingo



-- 
Thomas

[toc] | [prev] | [next] | [standalone]

#1555115

* Thomas Garnier <thgarnie@google.com> wrote:

> Coming back on that after a bit more testing. The LTR instruction
> check if the busy bit is already set, if already set then it will just
> issue a #GP given a bad selector:
> 
> [    0.000000] general protection fault: 0040 [#1] SMP
> ...
> [    0.000000] RIP: 0010:native_load_tr_desc+0x9/0x10
> ...
> [    0.000000] Call Trace:
> [    0.000000]  cpu_init+0x2d0/0x3c0
> [    0.000000]  trap_init+0x2a2/0x312
> [    0.000000]  start_kernel+0x1fb/0x43b
> [    0.000000]  ? set_init_arg+0x55/0x55
> [    0.000000]  ? early_idt_handler_array+0x120/0x120
> [    0.000000]  x86_64_start_reservations+0x2a/0x2c
> [    0.000000]  x86_64_start_kernel+0x13d/0x14c
> [    0.000000]  start_cpu+0x14/0x14
> 
> I assume that's in this part of the pseudo-code:
> 
> if(!IsWithinDescriptorTableLimit(Source.Offset) || Source.Type !=
> TypeGlobal) Exception(GP(SegmentSelector));
> SegmentDescriptor = ReadSegmentDescriptor();
> if(!IsForAnAvailableTSS(SegmentDescriptor))
> Exception(GP(SegmentSelector)); <---- That's where I got the GP
> TSSSegmentDescriptor.Busy = 1;
> <------------------------------------------------------------------
> That's the pagefault I get otherwise
> //Locked read-modify-write operation on the entire descriptor when
> setting busy flag
> TaskRegister.SegmentSelector = Source;
> TaskRegister.SegmentDescriptor.TSSSegmentDescriptor;
> 
> I assume the best option would be to make the remap read-write for the
> LTR instruction. What do you think?

So if LTR does not modify the GDT if the busy bit is already set, why don't we set 
the busy bit in the descriptor (via the linear mapping rw alias).

Then the remapped GDT can stay read-only all the time and LTR won't fault.

Am I missing something here?

Thanks,

	Ingo

[toc] | [prev] | [next] | [standalone]

#1555728

On Tue, Jan 10, 2017 at 2:27 AM, Ingo Molnar <mingo@kernel.org> wrote:
>
> * Thomas Garnier <thgarnie@google.com> wrote:
>
>> Coming back on that after a bit more testing. The LTR instruction
>> check if the busy bit is already set, if already set then it will just
>> issue a #GP given a bad selector:
>>
>> [    0.000000] general protection fault: 0040 [#1] SMP
>> ...
>> [    0.000000] RIP: 0010:native_load_tr_desc+0x9/0x10
>> ...
>> [    0.000000] Call Trace:
>> [    0.000000]  cpu_init+0x2d0/0x3c0
>> [    0.000000]  trap_init+0x2a2/0x312
>> [    0.000000]  start_kernel+0x1fb/0x43b
>> [    0.000000]  ? set_init_arg+0x55/0x55
>> [    0.000000]  ? early_idt_handler_array+0x120/0x120
>> [    0.000000]  x86_64_start_reservations+0x2a/0x2c
>> [    0.000000]  x86_64_start_kernel+0x13d/0x14c
>> [    0.000000]  start_cpu+0x14/0x14
>>
>> I assume that's in this part of the pseudo-code:
>>
>> if(!IsWithinDescriptorTableLimit(Source.Offset) || Source.Type !=
>> TypeGlobal) Exception(GP(SegmentSelector));
>> SegmentDescriptor = ReadSegmentDescriptor();
>> if(!IsForAnAvailableTSS(SegmentDescriptor))
>> Exception(GP(SegmentSelector)); <---- That's where I got the GP
>> TSSSegmentDescriptor.Busy = 1;
>> <------------------------------------------------------------------
>> That's the pagefault I get otherwise
>> //Locked read-modify-write operation on the entire descriptor when
>> setting busy flag
>> TaskRegister.SegmentSelector = Source;
>> TaskRegister.SegmentDescriptor.TSSSegmentDescriptor;
>>
>> I assume the best option would be to make the remap read-write for the
>> LTR instruction. What do you think?
>
> So if LTR does not modify the GDT if the busy bit is already set, why don't we set
> the busy bit in the descriptor (via the linear mapping rw alias).
>
> Then the remapped GDT can stay read-only all the time and LTR won't fault.
>
> Am I missing something here?
>

Sorry, I may not have explained myself well.

If you set the busy bit on the GDT TSS entry, you get a #GP on LTR.

When we use the LTR instruction, we need the GDT to be writeable.

I think I can handle it by switching to the remap GDT after
load_TR_desc. I will try this approach.

> Thanks,
>
>         Ingo



-- 
Thomas

[toc] | [prev] | [next] | [standalone]

#1552391

On Thu, Jan 5, 2017 at 12:18 PM, Andy Lutomirski <luto@kernel.org> wrote:
>
> Hmm.  I bet that if we preset the accessed bits in all the segments
> then we don't need it to be writable in general.

I'm not sure that this is architecturally safe.

IIRC, we do mark the IDT read-only - but that one we started doing due
to the f00f bug, so we knew it was ok. I'm not sure you can do the
same with the GDT/LDT.

                   Linus

[toc] | [prev] | [next] | [standalone]

#1552406

On Thu, Jan 5, 2017 at 3:05 PM, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
> On Thu, Jan 5, 2017 at 12:18 PM, Andy Lutomirski <luto@kernel.org> wrote:
>>
>> Hmm.  I bet that if we preset the accessed bits in all the segments
>> then we don't need it to be writable in general.
>
> I'm not sure that this is architecturally safe.
>
> IIRC, we do mark the IDT read-only - but that one we started doing due
> to the f00f bug, so we knew it was ok. I'm not sure you can do the
> same with the GDT/LDT.
>

I started testing a variant that make the GDT remapping read-only by
default and writeable only for LTR. Everything works fine, even
hibernation. I need to do more testing though on different
architectures.

To be on the safe side, I could separate the read-only part in a
separate patch so we can easily remove it if extended testing show
something.

>                    Linus



-- 
Thomas

[toc] | [prev] | [next] | [standalone]

#1552475

On Thu, Jan 5, 2017 at 3:05 PM, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
> On Thu, Jan 5, 2017 at 12:18 PM, Andy Lutomirski <luto@kernel.org> wrote:
>>
>> Hmm.  I bet that if we preset the accessed bits in all the segments
>> then we don't need it to be writable in general.
>
> I'm not sure that this is architecturally safe.
>

Hmm.  Last time I looked, I couldn't find *anything* in the SDM
explaining what happened if a GDT access resulted in a page fault.  I
did discover that Xen intentionally (!) lazily populates and maps LDT
pages.  An attempt to access a not-present page results in #PF with
the error cod e indicating kernel access even if the access came from
user mode.

SDM volume 3 7.2.2 says "Pages corresponding to the previous task’s
TSS, the current task’s TSS, and the descriptor table entries for
each all should be marked as read/write."  But I don't see how a CPU
implementation could possibly care what the page table for the TSS
descriptor table entries says after LTR is done because the CPU isn't
even supposed to *read* that memory.

OTOH a valid implementation could easily require that the page table
says that the page is writable merely to load a segment, especially in
weird cases (IRET?).  That being said, this is all quite easy to test.

Also, Thomas, why are you creating a new memory region?  I don't see
any benefit to randomizing the GDT address.  How about just putting it
in the fixmap?  This  would be NR_CPUS * 4 pages if do my limit=0xffff
idea.  I'm not sure if the fixmap code knows how to handle this much
space.

[toc] | [prev] | [next] | [standalone]

#1553001

On Thu, Jan 5, 2017 at 6:34 PM, Andy Lutomirski <luto@amacapital.net> wrote:
> On Thu, Jan 5, 2017 at 3:05 PM, Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
>> On Thu, Jan 5, 2017 at 12:18 PM, Andy Lutomirski <luto@kernel.org> wrote:
>>>
>>> Hmm.  I bet that if we preset the accessed bits in all the segments
>>> then we don't need it to be writable in general.
>>
>> I'm not sure that this is architecturally safe.
>>
>
> Hmm.  Last time I looked, I couldn't find *anything* in the SDM
> explaining what happened if a GDT access resulted in a page fault.  I
> did discover that Xen intentionally (!) lazily populates and maps LDT
> pages.  An attempt to access a not-present page results in #PF with
> the error cod e indicating kernel access even if the access came from
> user mode.
>
> SDM volume 3 7.2.2 says "Pages corresponding to the previous task’s
> TSS, the current task’s TSS, and the descriptor table entries for
> each all should be marked as read/write."  But I don't see how a CPU
> implementation could possibly care what the page table for the TSS
> descriptor table entries says after LTR is done because the CPU isn't
> even supposed to *read* that memory.
>
> OTOH a valid implementation could easily require that the page table
> says that the page is writable merely to load a segment, especially in
> weird cases (IRET?).  That being said, this is all quite easy to test.
>
> Also, Thomas, why are you creating a new memory region?  I don't see
> any benefit to randomizing the GDT address.  How about just putting it
> in the fixmap?  This  would be NR_CPUS * 4 pages if do my limit=0xffff
> idea.  I'm not sure if the fixmap code knows how to handle this much
> space.

When I looked at the fixmap, you had to define the space you need
ahead of time and I am not sure there was enough space as you said.

-- 
Thomas

[toc] | [prev] | [next] | [standalone]

#1553212

On Fri, Jan 6, 2017 at 10:02 AM, Thomas Garnier <thgarnie@google.com> wrote:
> On Thu, Jan 5, 2017 at 6:34 PM, Andy Lutomirski <luto@amacapital.net> wrote:
>> On Thu, Jan 5, 2017 at 3:05 PM, Linus Torvalds
>> <torvalds@linux-foundation.org> wrote:
>>> On Thu, Jan 5, 2017 at 12:18 PM, Andy Lutomirski <luto@kernel.org> wrote:
>>>>
>>>> Hmm.  I bet that if we preset the accessed bits in all the segments
>>>> then we don't need it to be writable in general.
>>>
>>> I'm not sure that this is architecturally safe.
>>>
>>
>> Hmm.  Last time I looked, I couldn't find *anything* in the SDM
>> explaining what happened if a GDT access resulted in a page fault.  I
>> did discover that Xen intentionally (!) lazily populates and maps LDT
>> pages.  An attempt to access a not-present page results in #PF with
>> the error cod e indicating kernel access even if the access came from
>> user mode.
>>
>> SDM volume 3 7.2.2 says "Pages corresponding to the previous task’s
>> TSS, the current task’s TSS, and the descriptor table entries for
>> each all should be marked as read/write."  But I don't see how a CPU
>> implementation could possibly care what the page table for the TSS
>> descriptor table entries says after LTR is done because the CPU isn't
>> even supposed to *read* that memory.
>>
>> OTOH a valid implementation could easily require that the page table
>> says that the page is writable merely to load a segment, especially in
>> weird cases (IRET?).  That being said, this is all quite easy to test.
>>
>> Also, Thomas, why are you creating a new memory region?  I don't see
>> any benefit to randomizing the GDT address.  How about just putting it
>> in the fixmap?  This  would be NR_CPUS * 4 pages if do my limit=0xffff
>> idea.  I'm not sure if the fixmap code knows how to handle this much
>> space.
>
> When I looked at the fixmap, you had to define the space you need
> ahead of time and I am not sure there was enough space as you said.

Can you try it and see if anything goes wrong?  Even if something does
go wrong, I think we should fix *that* rather than making the memory
layout more complicated.

[toc] | [prev] | [next] | [standalone]

#1553606

* Andy Lutomirski <luto@kernel.org> wrote:

> > When I looked at the fixmap, you had to define the space you need ahead of 
> > time and I am not sure there was enough space as you said.
> 
> Can you try it and see if anything goes wrong?  Even if something does go wrong, 
> I think we should fix *that* rather than making the memory layout more 
> complicated.

Absolutely! This should always be the driving principle when complicating the 
kernel's memory layout.

Thanks,

	Ingo

[toc] | [prev] | [next] | [standalone]

#1552555

* Arjan van de Ven <arjan@linux.intel.com> wrote:

> On 1/5/2017 9:54 AM, Thomas Garnier wrote:
> 
> > That's my goal too. I started by doing a RO remap and got couple problems with 
> > hibernation. I can try again for the next iteration or delay it for another 
> > patch. I also need to look at KVM GDT usage, I am not familiar with it yet.
> 
> don't we write to the GDT as part of the TLS segment stuff for glibc ?

Yes - but the idea would be to have two virtual memory aliases:

 1) A world-readable RO mapping of the GDT put into the fixmap area (randomization
    is not really required as the SGDT instruction exposes it anyway).

    This read-only GDT mapping has two advantages:

     - Because the GDT address is read-only it cannot be used by exploits as an 
       easy trampoline for 100% safe, deterministic rooting of the system from a 
       local process.

     - Even on older CPUs the SGDT instruction won't expose the absolute address 
       of critical kernel data structures.

 2) A kernel-only RW mapping in the linear kernel memory map where the original
    page is - randomized (as a natural part of kernel physical and virtual memory
    randomization) and only known and accessible to the kernel, SGDT doesn't 
    expose it even on older CPUs.

This way we IMHO have the best of both worlds.

Assuming there are no strange CPU errata surfacing if we use a RO GDT - OTOH that 
should be fairly easy to test for, and worst-case we could quirk-off the feature 
on CPUs where this cannot be done safely.

Thanks,

	Ingo

[toc] | [prev] | [next] | [standalone]

#1552199

On Wed, Jan 4, 2017 at 2:16 PM, Thomas Garnier <thgarnie@google.com> wrote:
> Each processor holds a GDT in its per-cpu structure. The sgdt
> instruction gives the base address of the current GDT. This address can
> be used to bypass KASLR memory randomization. With another bug, an
> attacker could target other per-cpu structures or deduce the base of the
> main memory section (PAGE_OFFSET).
>
> In this change, a space is reserved at the end of the memory range
> available for KASLR memory randomization. The space is big enough to hold
> the maximum number of CPUs (as defined by setup_max_cpus). Each GDT is
> mapped at specific offset based on the target CPU. Note that if there is
> not enough space available, the GDTs are not remapped.

Can we remap it read-only?  I.e. use PAGE_KERNEL_RO instead of
PAGE_KERNEL.  After all, the ability to modify the GDT is instant
root.

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.kernel

csiph-web