Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1219932 > unrolled thread

Re: Dealing with the NMI mess

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Dealing with the NMI mess "Maciej W. Rozycki" <macro@linux-mips.org> - 2015-09-07 07:40 +0200
    Re: Dealing with the NMI mess Ingo Molnar <mingo@kernel.org> - 2015-09-07 09:50 +0200
      Re: Dealing with the NMI mess "Maciej W. Rozycki" <macro@linux-mips.org> - 2015-09-07 10:20 +0200
        Re: Dealing with the NMI mess Paolo Bonzini <pbonzini@redhat.com> - 2015-09-07 12:30 +0200
          Re: Dealing with the NMI mess "Maciej W. Rozycki" <macro@linux-mips.org> - 2015-09-07 19:10 +0200
            Re: Dealing with the NMI mess Andy Lutomirski <luto@amacapital.net> - 2015-09-07 19:30 +0200
              Re: Dealing with the NMI mess "Maciej W. Rozycki" <macro@linux-mips.org> - 2015-09-07 21:40 +0200
                Re: Dealing with the NMI mess Andy Lutomirski <luto@amacapital.net> - 2015-09-08 00:00 +0200
                  Re: Dealing with the NMI mess "Maciej W. Rozycki" <macro@linux-mips.org> - 2015-09-08 18:30 +0200

#1219932 — Re: Dealing with the NMI mess

On Fri, 31 Jul 2015, Borislav Petkov wrote:

> Yeah, INT 1. I wonder whether INT 1, i.e. CD imm8 does the same thing.
> 
> But why do you say it is special - it simply raises #DB, i.e. vector 1.
> Web page seems to say so when interrupt redirection is disabled. It
> sounds like a nice and quick way to generate a breakpoint. You can do
> that with INT 01, i.e., the CD opcode, too.
> 
> If I'd had to guess, it isn't documented because of the proprietary ICE
> aspect. And no one uses ICEs anymore so it is going to be forgotten with
> people popping off and on and asking about the undocumented opcode.

 FYI, it's actually still in use with modern hardware, as a software 
breakpoint (and hence it has to be a single byte INT1 instruction rather 
than a multiple-byte regular INT 1 encoding) with JTAG probe hardware used 
for bare-metal debugging.  E.g. Intel Atom supports it and boards have 
been available with a JTAG connector, which Intel calls XDP aka Extended 
Debug Port, e.g. the D945GCLF board (aka Crown Beach IIRC) had one.

 By fiddling with some bits in the CPU, which are only accessible through 
JTAG, probe firmware takes control over #DB making it trap into the debug 
mode rather than into the kernel.  As noted above INT1 is used rather than 
INT3 (which still traps into the kernel with #BP as usually) for software 
breakpoints, but all the other DR0-7 resources are also available to the 
probe and the General Detect fault is used to prevent the kernel from 
fiddling with them.  Similarly single-stepping traps into probe firmware.  
Debug mode transitions are completely transparent to any kernel-mode 
software run.

 I did some work on this a few years ago, including emulating DR0-7 
accesses in software down the JTAG handler upon a General Detect fault to 
keep the kernel both happy and away from real debug registers. ;)  Yes, 
you can debug any software with this stuff, including the Linux kernel: 
set instruction and data breakpoints, single-step it, poke at all hardware 
registers, including descriptor registers not otherwise accessible (you 
can set funny modes for segments, also in the 64-bit mode), etc.  One 
complication though is you operate on physical addresses when poking at 
memory, you can't ask the CPU's MMU to remap them for you (you can walk 
page tables manually of course, just as the MMU would).

 I hope this clears things a bit around this stuff. :)  You might be able 
to find some more by issuing a query for "Extended Debug Port" with your 
favourite Internet search engine.

 It's been a while since this discussion, but I thought I'd chime in as 
you might find it interesting.  I'm actually a bit surprised the knowledge 
about this is so poor among x86 experts.

  Maciej
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1219970

* Maciej W. Rozycki <macro@linux-mips.org> wrote:

>  I did some work on this a few years ago, including emulating DR0-7 accesses in 
> software down the JTAG handler upon a General Detect fault to keep the kernel 
> both happy and away from real debug registers. ;) Yes, you can debug any 
> software with this stuff, including the Linux kernel: set instruction and data 
> breakpoints, single-step it, poke at all hardware registers, including 
> descriptor registers not otherwise accessible (you can set funny modes for 
> segments, also in the 64-bit mode), etc.  One complication though is you operate 
> on physical addresses when poking at memory, you can't ask the CPU's MMU to 
> remap them for you (you can walk page tables manually of course, just as the MMU 
> would).

Essentially the ICE breakpoint instruction enters SMM mode?

Thanks,

	Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1219980

On Mon, 7 Sep 2015, Ingo Molnar wrote:

> >  I did some work on this a few years ago, including emulating DR0-7 accesses in 
> > software down the JTAG handler upon a General Detect fault to keep the kernel 
> > both happy and away from real debug registers. ;) Yes, you can debug any 
> > software with this stuff, including the Linux kernel: set instruction and data 
> > breakpoints, single-step it, poke at all hardware registers, including 
> > descriptor registers not otherwise accessible (you can set funny modes for 
> > segments, also in the 64-bit mode), etc.  One complication though is you operate 
> > on physical addresses when poking at memory, you can't ask the CPU's MMU to 
> > remap them for you (you can walk page tables manually of course, just as the MMU 
> > would).
> 
> Essentially the ICE breakpoint instruction enters SMM mode?

 I didn't do stuff at the probe firmware level so I can't say for sure, 
but my gut feeling is the debug mode is indeed very close if not the same 
as SMM.  I think duplicating the logic would be an unnecessary waste of 
silicon.

 And obviously it's any cause of #DB that enters this mode.  The probe can
also request it right at the exit from the reset state, so that you can 
debug software (e.g BIOS startup) right from the reset vector.  You don't 
need working RAM for that.

  Maciej
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1220042

On 07/09/2015 10:19, Maciej W. Rozycki wrote:
>> > Essentially the ICE breakpoint instruction enters SMM mode?
>  I didn't do stuff at the probe firmware level so I can't say for sure, 
> but my gut feeling is the debug mode is indeed very close if not the same 
> as SMM.  I think duplicating the logic would be an unnecessary waste of 
> silicon.

I researched SMM a bit recently in order to implement it in KVM, and the
best source of folklore seems to be http://www.rcollins.org/ddj (which I
also have on paper :)).

The author there says that SMM design was roughly based on the 386's
probe/ICE mode design, but it's actually separate.  Most notably, on the
386 the state save areas almost mirror each other, but when I say
mirror... I do mean mirror: directions are reversed, and what is on top
for probe mode is on bottom for SMM. :)

In addition, AMD tried reusing ICE mode for SMM, and was sued by Intel
who actually won the lawsuit.  I couldn't find more information about
the lawsuit.

It's probably diverged more and more over time, for example because SMM
is now considered security-sensitive while probe mode isn't.  In
addition, the same DDJ article says that Pentium JTAG probe mode
"doesn't resemble SMM at all, doesn't use a state save map, or even
execute any code of its own", whatever that means.

Paolo

>  And obviously it's any cause of #DB that enters this mode.  The probe can
> also request it right at the exit from the reset state, so that you can 
> debug software (e.g BIOS startup) right from the reset vector.  You don't 
> need working RAM for that.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1220343

On Mon, 7 Sep 2015, Paolo Bonzini wrote:

> >  I didn't do stuff at the probe firmware level so I can't say for sure, 
> > but my gut feeling is the debug mode is indeed very close if not the same 
> > as SMM.  I think duplicating the logic would be an unnecessary waste of 
> > silicon.
> 
> I researched SMM a bit recently in order to implement it in KVM, and the
> best source of folklore seems to be http://www.rcollins.org/ddj (which I
> also have on paper :)).

 Robert did an excellent job figuring it all, but his stuff is a bit 
dated, things may have changed since, especially as JTAG debugging has 
since become ubiquitous in the embedded world and consequently better 
developed.

> The author there says that SMM design was roughly based on the 386's
> probe/ICE mode design, but it's actually separate.  Most notably, on the
> 386 the state save areas almost mirror each other, but when I say
> mirror... I do mean mirror: directions are reversed, and what is on top
> for probe mode is on bottom for SMM. :)

 That might be a minor implementation detail, needed for whatever reason. 

> In addition, AMD tried reusing ICE mode for SMM, and was sued by Intel
> who actually won the lawsuit.  I couldn't find more information about
> the lawsuit.
> 
> It's probably diverged more and more over time, for example because SMM
> is now considered security-sensitive while probe mode isn't.  In
> addition, the same DDJ article says that Pentium JTAG probe mode
> "doesn't resemble SMM at all, doesn't use a state save map, or even
> execute any code of its own", whatever that means.

 At least I am fairly sure the RSM instruction is used to quit the debug 
mode just like with SMM, so I'd be surprised if they bothered implementing 
separate state save area structures for the two modes.  Some control bits 
in the CPU may well be set differently between the two modes though, 
addressing issues like security sensitivity you mentioned.

 A state save/restore approach is definitely used (unlike with some other 
processors that expose internal registers through JTAG directly) as you 
cannot switch between operation modes (e.g. real vs protected) on the fly 
while in the debug mode.  You actually need to return to the regular mode 
(e.g. ask to single-step a NOP) for a mode change to take effect.  Ditto 
about other registers -- any read-only bits are only masked out in the 
register state once a regular-mode instruction has executed.

 The use of RSM also prompts a question whether you can nest debug mode in 
SMM (to debug SMM code) -- this is actually similar to the NMI vs IRET 
issue considered in this thread -- or nest debug mode in debug mode, e.g. 
by taking a #DB exception from an INT1 instruction while in either mode.  
I don't know.  Some other processors (MIPS) that implement a JTAG debug 
mode allow such nesting and care has to be taken in probe firmware to 
handle it correctly and ensure the context to return to is not clobbered 
if such a situation is to be arranged.  And also -- as you may have 
expected -- the debug mode return instruction has to be avoided in the 
nested handler.

 These are all implementation-specific details, including the INT1 
instruction, which is why I am not at all surprised that they are omitted 
from architecture manuals.  The JTAG debug mode itself is no rocket 
science though, everybody seems to have it these days.  Though for cost 
and power consumption saving reasons the RTL block implementing the debug 
module may obviously be omitted from production silicon known, perhaps by 
definition, to never require one.

  Maciej
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1220347

On Mon, Sep 7, 2015 at 10:01 AM, Maciej W. Rozycki <macro@linux-mips.org> wrote:
>  These are all implementation-specific details, including the INT1
> instruction, which is why I am not at all surprised that they are omitted
> from architecture manuals.

That bit is BS, though.  The INT1 instruction, executed in user mode
(CPL3) with no hardware debugger attached, will enter the kernel
through a gate at vector 1, *even if that gate has DPL == 0*.

If there's an instruction that bypasses hardware protection
mechanisms, then Intel should document it rather than relying on OS
writers to know enough folklore to get it right.

Heck, SDM Volume 3 6.12.1.1 says "The processor checks the DPL of the
interrupt or trap gate only if an exception or interrupt is generated
with an INT n, INT 3, or INTO instruction."  It does not say "the
processor does not check the DPL of the interrupt or trap gate if the
exception or interrupt is generated with the undocumented ICEBP
instruction."

--Andy
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1220365

On Mon, 7 Sep 2015, Andy Lutomirski wrote:

> >  These are all implementation-specific details, including the INT1
> > instruction, which is why I am not at all surprised that they are omitted
> > from architecture manuals.
> 
> That bit is BS, though.  The INT1 instruction, executed in user mode
> (CPL3) with no hardware debugger attached, will enter the kernel
> through a gate at vector 1, *even if that gate has DPL == 0*.
> 
> If there's an instruction that bypasses hardware protection
> mechanisms, then Intel should document it rather than relying on OS
> writers to know enough folklore to get it right.
> 
> Heck, SDM Volume 3 6.12.1.1 says "The processor checks the DPL of the
> interrupt or trap gate only if an exception or interrupt is generated
> with an INT n, INT 3, or INTO instruction."  It does not say "the
> processor does not check the DPL of the interrupt or trap gate if the
> exception or interrupt is generated with the undocumented ICEBP
> instruction."

 It does not have to be mentioned, because it's implied by how the #DB 
exception is propagated: regardless of its origin it never checks the DPL.  
And user-mode software may well use POPF at any time to set the TF bit in 
the flags register to the same effect, so the OS needs to be prepared for 
a #DB exception it hasn't scheduled itself anyway.

  Maciej
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1220412

On Mon, Sep 7, 2015 at 12:30 PM, Maciej W. Rozycki <macro@linux-mips.org> wrote:
> On Mon, 7 Sep 2015, Andy Lutomirski wrote:
>
>> >  These are all implementation-specific details, including the INT1
>> > instruction, which is why I am not at all surprised that they are omitted
>> > from architecture manuals.
>>
>> That bit is BS, though.  The INT1 instruction, executed in user mode
>> (CPL3) with no hardware debugger attached, will enter the kernel
>> through a gate at vector 1, *even if that gate has DPL == 0*.
>>
>> If there's an instruction that bypasses hardware protection
>> mechanisms, then Intel should document it rather than relying on OS
>> writers to know enough folklore to get it right.
>>
>> Heck, SDM Volume 3 6.12.1.1 says "The processor checks the DPL of the
>> interrupt or trap gate only if an exception or interrupt is generated
>> with an INT n, INT 3, or INTO instruction."  It does not say "the
>> processor does not check the DPL of the interrupt or trap gate if the
>> exception or interrupt is generated with the undocumented ICEBP
>> instruction."
>
>  It does not have to be mentioned, because it's implied by how the #DB
> exception is propagated: regardless of its origin it never checks the DPL.
> And user-mode software may well use POPF at any time to set the TF bit in
> the flags register to the same effect, so the OS needs to be prepared for
> a #DB exception it hasn't scheduled itself anyway.

Not really.

int $1 checks DPL.  Setting TF results in saved TF set and the
corresponding bit in DR6 set as well.  Triggering a #DB using the
debug registers requires active OS help.

So operating systems need to handle a #DB without no indicated cause
without spewing warnings or crashing, and there is no indication
whatsoever in the SDM or APM that this is the case.

--Andy
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1220963

On Mon, 7 Sep 2015, Andy Lutomirski wrote:

> >  It does not have to be mentioned, because it's implied by how the #DB
> > exception is propagated: regardless of its origin it never checks the DPL.
> > And user-mode software may well use POPF at any time to set the TF bit in
> > the flags register to the same effect, so the OS needs to be prepared for
> > a #DB exception it hasn't scheduled itself anyway.
> 
> Not really.
> 
> int $1 checks DPL.  Setting TF results in saved TF set and the
> corresponding bit in DR6 set as well.  Triggering a #DB using the
> debug registers requires active OS help.

 INT $1 is a software interrupt instruction, it does not trigger a #DB.  
Similarly INT $13 checks DPL while #GP does not.  Or maybe INT $6 vs UD2 
is a better analogy; the latter is as much INT6 as the 0xf1 encoding is 
INT1.

 Yes, you'll get a DR6 status with no new bits set.  So what?  You can 
ignore it and IRET with no adverse effects.  You can print diagnostics if 
you're pedantic.  You can kill the offending user program, but that's no 
harm, because it already did the undefined.  None of these is an issue, 
and certainly not one for security.

 Panicking OTOH would be, but that would IMHO be a silly choice and a bad 
OS design.  You never need to crash due to a user-mode exception, even an 
unknown one.  What if you run on a new CPU which has a new user-mode 
exception unknown at the time the OS binary was compiled?  That's an 
analogous situation for an architecture like x86 where strict backwards 
compatibility is maintained.

 A reasonable #DB handler will do something like:

{
	int dr6 = read_dr6();

	write_dr6(0);
	if (dr6 & DR6_MASK_X)
		handle_dr6_x();
	if (dr6 & DR6_MASK_Y)
		handle_dr6_y();
	/* Etc... */

	return;
}

and will work just fine where invoked with no bits set in DR6.

> So operating systems need to handle a #DB without no indicated cause
> without spewing warnings or crashing, and there is no indication
> whatsoever in the SDM or APM that this is the case.

 Strictly speaking the SDM does not state that at least one status bit 
shall be set in DR6 either.

 FAOD I'm not saying of course that documenting INT1 as a model-specific 
instruction encoding reserved for #DB generation or stating something to 
the effect that the OS is required to handle (e.g. discard) a #DB 
exception seen with no status bits set in DR6 would be bad.  No, it would 
certainly be nice.  But I maintain that I don't see it as strictly 
necessary.

 Pester Intel if you disagree, I'm not the right person to complain about 
it anyway. ;)

  Maciej
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web