Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1510291 > unrolled thread

Re: Getting interrupt every million cache misses

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Getting interrupt every million cache misses Pavel Machek <pavel@ucw.cz> - 2016-10-27 16:40 +0200
    Re: Getting interrupt every million cache misses Peter Zijlstra <peterz@infradead.org> - 2016-10-27 16:40 +0200
      Re: Getting interrupt every million cache misses Kees Cook <keescook@chromium.org> - 2016-10-27 22:50 +0200
        rowhammer protection [was Re: Getting interrupt every million cache  misses] Pavel Machek <pavel@ucw.cz> - 2016-10-27 23:30 +0200
          Re: rowhammer protection [was Re: Getting interrupt every million  cache misses] Ingo Molnar <mingo@kernel.org> - 2016-10-28 09:10 +0200
            Re: rowhammer protection [was Re: Getting interrupt every million  cache misses] Pavel Machek <pavel@ucw.cz> - 2016-10-28 11:00 +0200
              Re: rowhammer protection [was Re: Getting interrupt every million  cache misses] Ingo Molnar <mingo@kernel.org> - 2016-10-28 11:00 +0200
                Re: rowhammer protection [was Re: Getting interrupt every million  cache misses] Pavel Machek <pavel@ucw.cz> - 2016-10-28 14:00 +0200
              Re: rowhammer protection [was Re: Getting interrupt every million  cache misses] Peter Zijlstra <peterz@infradead.org> - 2016-10-28 11:10 +0200
                Re: rowhammer protection [was Re: Getting interrupt every million  cache misses] Vegard Nossum <vegard.nossum@gmail.com> - 2016-10-28 11:30 +0200
                  Re: rowhammer protection [was Re: Getting interrupt every million  cache misses] Ingo Molnar <mingo@kernel.org> - 2016-10-28 11:40 +0200
                    Re: rowhammer protection [was Re: Getting interrupt every million  cache misses] Vegard Nossum <vegard.nossum@gmail.com> - 2016-10-28 11:50 +0200
                    Re: [kernel-hardening] Re: rowhammer protection [was Re: Getting  interrupt every million cache misses] Mark Rutland <mark.rutland@arm.com> - 2016-10-28 12:00 +0200
                Re: rowhammer protection [was Re: Getting interrupt every million  cache misses] Pavel Machek <pavel@ucw.cz> - 2016-10-28 13:30 +0200
          Re: [kernel-hardening] rowhammer protection [was Re: Getting  interrupt every million cache misses] Mark Rutland <mark.rutland@arm.com> - 2016-10-28 12:00 +0200
            Re: [kernel-hardening] rowhammer protection [was Re: Getting  interrupt every million cache misses] Pavel Machek <pavel@ucw.cz> - 2016-10-28 13:30 +0200
              Re: [kernel-hardening] rowhammer protection [was Re: Getting  interrupt every million cache misses] Mark Rutland <mark.rutland@arm.com> - 2016-10-28 16:10 +0200
                Re: [kernel-hardening] rowhammer protection [was Re: Getting  interrupt every million cache misses] Peter Zijlstra <peterz@infradead.org> - 2016-10-28 16:20 +0200
                  Re: [kernel-hardening] rowhammer protection [was Re: Getting  interrupt every million cache misses] Pavel Machek <pavel@ucw.cz> - 2016-10-28 20:40 +0200
                    Re: [kernel-hardening] rowhammer protection [was Re: Getting  interrupt every million cache misses] Peter Zijlstra <peterz@infradead.org> - 2016-10-28 20:50 +0200
                  Re: [kernel-hardening] rowhammer protection [was Re: Getting  interrupt every million cache misses] Pavel Machek <pavel@ucw.cz> - 2016-11-02 19:20 +0100
                Re: [kernel-hardening] rowhammer protection [was Re: Getting  interrupt every million cache misses] Pavel Machek <pavel@ucw.cz> - 2016-10-28 19:30 +0200
                  Re: Re: [kernel-hardening] rowhammer protection [was Re: Getting  interrupt every million cache misses] Daniel Gruss <daniel@gruss.cc> - 2016-10-29 15:20 +0200
                    Re: Re: [kernel-hardening] rowhammer protection [was Re: Getting  interrupt every million cache misses] Pavel Machek <pavel@ucw.cz> - 2016-10-29 21:50 +0200
                      Re: [kernel-hardening] rowhammer protection [was Re: Getting  interrupt every million cache misses] Daniel Gruss <daniel@gruss.cc> - 2016-10-29 22:10 +0200
                        Re: [kernel-hardening] rowhammer protection [was Re: Getting  interrupt every million cache misses] Pavel Machek <pavel@ucw.cz> - 2016-10-29 23:10 +0200
                          Re: [kernel-hardening] rowhammer protection [was Re: Getting  interrupt every million cache misses] Daniel Gruss <daniel@gruss.cc> - 2016-10-29 23:10 +0200
                            Re: [kernel-hardening] rowhammer protection [was Re: Getting  interrupt every million cache misses] Pavel Machek <pavel@ucw.cz> - 2016-10-29 23:50 +0200
                              Re: [kernel-hardening] rowhammer protection [was Re: Getting  interrupt every million cache misses] Daniel Gruss <daniel@gruss.cc> - 2016-10-30 00:00 +0200
                                Re: [kernel-hardening] rowhammer protection [was Re: Getting  interrupt every million cache misses] Pavel Machek <pavel@ucw.cz> - 2016-10-30 00:10 +0200
                                  Re: [kernel-hardening] rowhammer protection [was Re: Getting  interrupt every million cache misses] Daniel Gruss <daniel@gruss.cc> - 2016-10-30 00:10 +0200
                Re: [kernel-hardening] rowhammer protection [was Re: Getting  interrupt every million cache misses] Pavel Machek <pavel@ucw.cz> - 2016-10-31 09:30 +0100
                  Re: [kernel-hardening] rowhammer protection [was Re: Getting  interrupt every million cache misses] Mark Rutland <mark.rutland@arm.com> - 2016-10-31 15:50 +0100
                    Re: [kernel-hardening] rowhammer protection [was Re: Getting  interrupt every million cache misses] Pavel Machek <pavel@ucw.cz> - 2016-10-31 22:20 +0100
                      Re: [kernel-hardening] rowhammer protection [was Re: Getting  interrupt every million cache misses] Mark Rutland <mark.rutland@arm.com> - 2016-10-31 23:10 +0100
                  Re: [kernel-hardening] rowhammer protection [was Re: Getting  interrupt every million cache misses] Ingo Molnar <mingo@kernel.org> - 2016-11-01 07:40 +0100
                    Re: [kernel-hardening] rowhammer protection [was Re: Getting  interrupt every million cache misses] Daniel Micay <danielmicay@gmail.com> - 2016-11-01 08:30 +0100
                    Re: [kernel-hardening] rowhammer protection [was Re: Getting  interrupt every million cache misses] Daniel Gruss <daniel@gruss.cc> - 2016-11-01 09:00 +0100
                    Re: [kernel-hardening] rowhammer protection [was Re: Getting  interrupt every million cache misses] Daniel Gruss <daniel@gruss.cc> - 2016-11-01 09:20 +0100
                    Re: [kernel-hardening] rowhammer protection [was Re: Getting  interrupt every million cache misses] Pavel Machek <pavel@ucw.cz> - 2016-11-01 09:20 +0100

Page 2 of 2 — ← Prev page 1 [2]

#1514054 — Re: [kernel-hardening] rowhammer protection [was Re: Getting interrupt every million cache misses]

[Multipart message — attachments visible in raw view] — view raw

Hi!

> On Fri, Oct 28, 2016 at 03:05:22PM +0100, Mark Rutland wrote:
> > 
> > > > * the precise semantics of performance counter events varies drastically
> > > >   across implementations. PERF_COUNT_HW_CACHE_MISSES, might only map to
> > > >   one particular level of cache, and/or may not be implemented on all
> > > >   cores.
> > > 
> > > If it maps to one particular cache level, we are fine (or maybe will
> > > trigger protection too often). If some cores are not counted, that's bad.
> > 
> > Perhaps, but that depends on a number of implementation details. If "too
> > often" means "all the time", people will turn this off when they could
> > otherwise have been protected (e.g. if we can accurately monitor the
> > last level of cache).
> 
> Right, so one of the things mentioned in the paper is x86 NT stores.
> Those are not cached and I'm not at all sure they're accounted in the
> event we use for cache misses.

Well, I tried this... and the movnti is as fast as plain mov. Clearly
it is being cached here.

I guess we could switch to different performance counter, such as

+       [PERF_COUNT_HW_BUS_CYCLES]              = 0xc06f, /* Non
halted bus cycles: 0x013c */

if NT stores are indeed a problem. But so far I don't have any
indication they are, so I'd like to have an working example to test
against. (It does not have to produce bitflips, it would be enough to
produce enough memory traffic bypassing cache.)

Best regards,
								Pavel

/*
 * gcc -O2 rowhammer.c -o rowhammer
 */

char pad[1024];
long long foo;
char pad2[1024];

void main(void)
{
	long long i;
	asm volatile(
		"mov $foo, %%edi \n\
			clflush (%%edi)" ::: "%edi");
	
	for (i=0; i<1000000000; i++) {
#if 1
		asm volatile(
			"mov $foo, %%edi \n\
			movnti %%eax, (%%edi)" ::: "%edi");
#endif

		// asm volatile( "" );
	}
}


-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[toc] | [prev] | [next] | [standalone]

#1511286 — Re: [kernel-hardening] rowhammer protection [was Re: Getting interrupt every million cache misses]

[Multipart message — attachments visible in raw view] — view raw

Hi!

> On Fri, Oct 28, 2016 at 01:21:36PM +0200, Pavel Machek wrote:
> > > Has this been tested on a system vulnerable to rowhammer, and if so, was
> > > it reliable in mitigating the issue?
> > > 
> > > Which particular attack codebase was it tested against?
> > 
> > I have rowhammer-test here,
> > 
> > commit 9824453fff76e0a3f5d1ac8200bc6c447c4fff57
> > Author: Mark Seaborn <mseaborn@chromium.org>
> 
> ... from which repo?

https://github.com/mseaborn/rowhammer-test.git

> > I do not have vulnerable machine near me, so no "real" tests, but
> > I'm pretty sure it will make the error no longer reproducible with the
> > newer version. [Help welcome ;-)]
> 
> Even if we hope this works, I think we have to be very careful with that
> kind of assertion. Until we have data is to its efficacy, I don't think
> we should claim that this is an effective mitigation.

On my hardware, rowhammer errors are not trivial to reproduce. It
takes time (minutes). I'm pretty sure this will be enough to stop the
exploit. If you have machines where rowhammer errors are really easy
to reproduce, testing on it would be welcome.

> > Well, I'd like to postpone debate 'where does it live' to the later
> > stage. The problem is not arch-specific, the solution is not too
> > arch-specific either. I believe we can use Kconfig to hide it from
> > users where it does not apply. Anyway, lets decide if it works and
> > where, first.
> 
> You seem to have forgotten the drammer case here, which this would not
> have protected against. I'm not sure, but I suspect that we could have
> similar issues with mappings using other attributes (e.g write-through),
> as these would cause the memory traffic without cache miss events.

Can you get me example code for x86 or x86-64? If this is trivial to
workaround using movnt or something like that, it would be good to
know.

I did not go through the drammer paper in too great detail. They have
some kind of DMA-able memory, and they abuse it to do direct writes?
So you can "simply" stop providing DMA-able memory to the userland,
right? [Ok, bye bye accelerated graphics, I guess. But living w/o
graphics acceleration is preferable to remote root...]

OTOH... the exploit that scares me most is javascript sandbox
escape. I should be able to stop that... and other JIT escape cases
where untrusted code does not have access to special instructions.

On x86, there seems to be "DATA_MEM_REFS" performance counter, if
cache misses do not account movnt, this one should. Will need checking.

> Perhaps, but that depends on a number of implementation details. If "too
> often" means "all the time", people will turn this off when they could
> otherwise have been protected (e.g. if we can accurately monitor the
> last level of cache).

Yup. Doing it well is preferable to doing it badly.

> > > * On some implementations, it may be that the counters are not
> > >   interchangeable, and for those this would take away
> > >   PERF_COUNT_HW_CACHE_MISSES from existing users.
> > 
> > Yup. Note that with this kind of protection, one missing performance
> > counter is likely to be small problem.
> 
> That depends. Who chooses when to turn this on? If it's down to the
> distro, this can adversely affect users with perfectly safe DRAM.

You don't want this enabled on machines with working DRAM, there will
be performance impact.

> > > > +	/* FIXME msec per usec, reverse logic? */
> > > > +	if (delta < 64 * NSEC_PER_MSEC)
> > > > +		mdelay(56);
> > > > +}
> > > 
> > > If I round-robin my attack across CPUs, how much does this help?
> > 
> > See below for new explanation. With 2 CPUs, we are fine. On monster
> > big-little 8-core machines, we'd probably trigger protection too
> > often.
> 
> We see larger core counts in mobile devices these days. In China,
> octa-core phones are popular, for example. Servers go much larger.

Well, I can't help everyone :-(. On servers, there's ECC. On phones,
well, don't buy broken machines. This will work, but performance
impact will not be nice.

> > +static struct perf_event_attr rh_attr = {
...
> > +	.sample_period = 10000,
> > +};
> 
> What kind of overhead (just from taking the interrupt) will this come
> with?

This is not used, see below.

> > +/*
> > + * How often is the DRAM refreshed. Setting it too high is safe.
> > + */
> 
> Stale comment? Given the check against delta below, this doesn't look to
> be true.

Thinko, actually. Too low is safe, AFAICT.

> > +/*
> > + * DRAM is shared between CPUs, but these performance counters are per-CPU.
> > + */
> > +	int max_attacking_cpus = 2;
> 
> As above, many systems today have more than two CPUs. In the drammmer
> paper, it looks like the majority had four.

We can do set this automatically, and we should also take cpu hotplug
into account. But lets get it working first.

Actually in the ARM case (Drammer), it may be better to stop exploit
some other way. Turning off/redesigning GPU acceleration should work
there, right?

Best regards,
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[toc] | [prev] | [next] | [standalone]

#1511648 — Re: Re: [kernel-hardening] rowhammer protection [was Re: Getting interrupt every million cache misses]

I think that this idea to mitigate Rowhammer is not a good approach.

I wrote Rowhammer.js (we published a paper on that) and I had the first 
reproducible bit flips on DDR4 at both, increased and default refresh 
rates (published in our DRAMA paper).

We have researched the number of cache misses induced from different 
applications in the past and there are many applications that cause more 
cache misses than Rowhammer (published in our Flush+Flush paper) they 
just cause them on different rows.
Slowing down a system surely works, but you could also, as a mitigation 
just make this CPU core run at the lowest possible frequency. That would 
likely be more effective than the solution you suggest.

Now, every Rowhammer attack exploits not only the DRAM effects but also 
the way the operating system organizes memory.

Some papers exploit page deduplication and disabling page deduplication 
should be the default also for other reasons, such as information 
disclosure attacks. If page deduplication is disabled, attacks like 
Dedup est Machina and Flip Feng Shui are inherently not possible anymore.

Most other attacks target page tables (the Google exploit, Rowhammer.js, 
Drammer). Now in Rowhammer.js we suggested a very simple fix, that is 
just an extension of what Linux already does.
Unless out of memory page tables and user pages are not placed in the 
same 2MB region. We suggested that this behavior should be more strict 
even in memory pressure situations. If the OS can only find a page table 
that resides in the same 2MB region as a user page, the request should 
fail instead and the process requesting it should go out of memory. More 
generally, the attack surface is gone if the OS never places a page 
table in proximity of less than 2MB to a user page.
That is a simple fix that does not cost any runtime performance. It 
mitigates all these scary attacks and won't even incur a memory cost in 
most situation.

[toc] | [prev] | [next] | [standalone]

#1511872 — Re: Re: [kernel-hardening] rowhammer protection [was Re: Getting interrupt every million cache misses]

[Multipart message — attachments visible in raw view] — view raw

Hi!

> I think that this idea to mitigate Rowhammer is not a good approach.

Well.. it does not have to be good if it is the best we have.

> I wrote Rowhammer.js (we published a paper on that) and I had the first
> reproducible bit flips on DDR4 at both, increased and default refresh rates
> (published in our DRAMA paper).

Congratulations. Now I'd like to take away your toys :-).

> We have researched the number of cache misses induced from different
> applications in the past and there are many applications that cause more
> cache misses than Rowhammer (published in our Flush+Flush paper) they just
> cause them on different rows.
> Slowing down a system surely works, but you could also, as a mitigation just
> make this CPU core run at the lowest possible frequency. That would likely
> be more effective than the solution you suggest.

Not in my testing. First, I'm not at all sure lowest CPU speed would
make any difference at all (even CPU at lowest clock is way faster
than DRAM). Second, going to lowest clock speed will reduce
performance

[But if you can test it and it works... it would be nice to know. It
is very simple to implement w/o kernel changes.] 

> Now, every Rowhammer attack exploits not only the DRAM effects but also the
> way the operating system organizes memory.
> 
> Some papers exploit page deduplication and disabling page deduplication
> should be the default also for other reasons, such as information disclosure
> attacks. If page deduplication is disabled, attacks like Dedup est Machina
> and Flip Feng Shui are inherently not possible anymore.

No, sorry, not going to play this particular whack-a-mole game. Linux
is designed for working hardware, and with bit flips, something is
going to break. (Does Flip Feng Shui really depend on dedup?) 

> Most other attacks target page tables (the Google exploit, Rowhammer.js,
> Drammer). Now in Rowhammer.js we suggested a very simple fix, that is just
> an extension of what Linux already does.
> Unless out of memory page tables and user pages are not placed in the same
> 2MB region. We suggested that this behavior should be more strict even in
> memory pressure situations. If the OS can only find a page table that
> resides in the same 2MB region as a user page, the request should fail
> instead and the process requesting it should go out of memory. More
> generally, the attack surface is gone if the OS never places a page table in
> proximity of less than 2MB to a user page.

But it will be nowhere near complete fix, right?

Will fix user attacking kernel, but not user1 attacking user2. You
could put each "user" into separate 2MB region, but then you'd have to
track who needs go go where. (Same uid is not enough, probably "can
ptrace"?)

But more importantly....

That'll still let remote server gain permissons of local user running
web server... using javascript exploit right? And that's actually
attack that I find most scary. Local user to root exploit is bad, but
getting permissions of web browser from remote web server is very,
very, very bad.

> That is a simple fix that does not cost any runtime performance.

Simple? Not really, I'm afraid. Feel free to try to implement it.

Best regards,

									Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[toc] | [prev] | [next] | [standalone]

#1511890 — Re: [kernel-hardening] rowhammer protection [was Re: Getting interrupt every million cache misses]

On 29.10.2016 21:42, Pavel Machek wrote:
> Congratulations. Now I'd like to take away your toys :-).

I'm would like you to do that, but I'm very confident you're not 
successful the way your starting ;)

> Not in my testing.

Have you tried music/video reencoding? Games? Anything that works with a 
decent amount of memory but not too much hard disk i/o?
Numbers are very clear there...

> First, I'm not at all sure lowest CPU speed would
> make any difference at all

It would. I've seen many bitflips but none where the CPU operated in the 
lower frequency range.

> Second, going to lowest clock speed will reduce performance

As does the countermeasure you propose...

> No, sorry, not going to play this particular whack-a-mole game.

But you are already with the countermeasure you propose...

> Linux is designed for working hardware, and with bit flips, something is
> going to break. (Does Flip Feng Shui really depend on dedup?)

Deduplication should be disabled not because of bit flips but because of 
information leakage (deduplication attacks, cache side-channel attacks, ...)

Yes, Flip Feng Shui requires deduplication and does not work without.
Disabling deduplication is what the authors recommend as a countermeasure.

> But it will be nowhere near complete fix, right?
>
> Will fix user attacking kernel, but not user1 attacking user2. You
> could put each "user" into separate 2MB region, but then you'd have to
> track who needs go go where. (Same uid is not enough, probably "can
> ptrace"?)

Exactly. But preventing user2kernel is already a good start, and you 
would prevent that without any doubt and without any cost.

user2user is something else to think about and more complicated because 
you have shared libraries + copy on write --> same problems as 
deduplication. I think it might make sense to discuss whether separating 
by uids or even pids would be viable.

> That'll still let remote server gain permissons of local user running
> web server... using javascript exploit right?  And that's actually
> attack that I find most scary. Local user to root exploit is bad, but
> getting permissions of web browser from remote web server is very,
> very, very bad.

Rowhammer.js skips the browser... it goes JS to full phys. memory 
access. Anyway, preventing Rowhammer from JS should be easy because even 
the slightest slow down should be enough to prevent any Rowhammer attack 
from JS.

>> That is a simple fix that does not cost any runtime performance.
>
> Simple? Not really, I'm afraid. Feel free to try to implement it.

I had a student who already implemented this in another OS, I'm 
confident it can be done in Linux as well...


Cheers,
Daniel

[toc] | [prev] | [next] | [standalone]

#1511904 — Re: [kernel-hardening] rowhammer protection [was Re: Getting interrupt every million cache misses]

[Multipart message — attachments visible in raw view] — view raw

Hi!

On Sat 2016-10-29 22:05:16, Daniel Gruss wrote:
> On 29.10.2016 21:42, Pavel Machek wrote:
> >Congratulations. Now I'd like to take away your toys :-).
> 
> I'm would like you to do that, but I'm very confident you're not successful
> the way your starting ;)

:-). Lets see.

> >Not in my testing.
> 
> Have you tried music/video reencoding? Games? Anything that works with a
> decent amount of memory but not too much hard disk i/o?
> Numbers are very clear there...

So far I did bzip2 and kernel compilation. I believe I can prevent
flips in rowhammer-test with bzip2 going from 4 seconds to 5
seconds... let me see.

If you have simple test that you'd like me to try, speak up. Best if
it takes cca 10 seconds to run.

> >First, I'm not at all sure lowest CPU speed would
> >make any difference at all
> 
> It would. I've seen many bitflips but none where the CPU operated in the
> lower frequency range.

Ok, let me try that. Problem is that the machine I'm testing on takes
20 minutes to produce bit flip...

> >Second, going to lowest clock speed will reduce performance
> 
> As does the countermeasure you propose...

Yes. But hopefully not quite _as_ drastically. (going to lowest clock
would make bzip2 go from 4 to 12 seconds or so, right?)

> Yes, Flip Feng Shui requires deduplication and does not work without.
> Disabling deduplication is what the authors recommend as a
> countermeasure.

Ok, Flip Feng Shui is easy, then. :-).

> >But it will be nowhere near complete fix, right?
> >
> >Will fix user attacking kernel, but not user1 attacking user2. You
> >could put each "user" into separate 2MB region, but then you'd have to
> >track who needs go go where. (Same uid is not enough, probably "can
> >ptrace"?)
> 
> Exactly. But preventing user2kernel is already a good start, and you would
> prevent that without any doubt and without any cost.

Well, it is only good start if the result is mergeable, and can be
used to prevent all attacks we care about.

> >That'll still let remote server gain permissons of local user running
> >web server... using javascript exploit right?  And that's actually
> >attack that I find most scary. Local user to root exploit is bad, but
> >getting permissions of web browser from remote web server is very,
> >very, very bad.
> 
> Rowhammer.js skips the browser... it goes JS to full phys. memory access.
> Anyway, preventing Rowhammer from JS should be easy because even the
> slightest slow down should be enough to prevent any Rowhammer attack from
> JS.

Are you sure? How much slowdown is enough to prevent the attack? (And
can I get patched chromium? Patched JVM? Patched qemu?) Dunno.. are
only just in time compilers affected? Or can I get for example pdf
document that does all the wrong memory accesses during rendering,
triggering buffer overrun in xpdf and arbitrary code execution?

Running userland on non-working machine is scary :-(.

Shall we introduce new syscall "get_mandatory_jit_slowdown()"?

I'd like kernel patch that works around rowhammer problem... in
kernel. I'm willing to accept some slowdown (say from 4 to 6 seconds
for common tasks). I'd prefer solution to be contained in kernel, and
present working (but slower) machine to userspace. I believe I can do
that.

> >>That is a simple fix that does not cost any runtime performance.
> >
> >Simple? Not really, I'm afraid. Feel free to try to implement it.
> 
> I had a student who already implemented this in another OS, I'm confident it
> can be done in Linux as well...

Well, I'm not saying its impossible. But I'd like to see the
implementation. Its definitely more work than nohammer.c. Order of
magnitude more, at least.

But yes, it will help with side channel attacks, etc. So yes, I'd like
to see the patch.

Best regards,

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[toc] | [prev] | [next] | [standalone]

#1511905 — Re: [kernel-hardening] rowhammer protection [was Re: Getting interrupt every million cache misses]

On 29.10.2016 23:05, Pavel Machek wrote:
> So far I did bzip2 and kernel compilation. I believe I can prevent
> flips in rowhammer-test with bzip2 going from 4 seconds to 5
> seconds... let me see.

can you prevent bitflips in this one? 
https://github.com/IAIK/rowhammerjs/tree/master/native

> Ok, let me try that. Problem is that the machine I'm testing on takes
> 20 minutes to produce bit flip...

will be lots faster with my code above ;)

[toc] | [prev] | [next] | [standalone]

#1511918 — Re: [kernel-hardening] rowhammer protection [was Re: Getting interrupt every million cache misses]

[Multipart message — attachments visible in raw view] — view raw

On Sat 2016-10-29 23:07:59, Daniel Gruss wrote:
> On 29.10.2016 23:05, Pavel Machek wrote:
> >So far I did bzip2 and kernel compilation. I believe I can prevent
> >flips in rowhammer-test with bzip2 going from 4 seconds to 5
> >seconds... let me see.
> 
> can you prevent bitflips in this one?
> https://github.com/IAIK/rowhammerjs/tree/master/native

Thanks for the pointer. Unfortunately, my test machine has 64-bit
kernel, but 32-bit userland, so I can't compile it:

g++ -g -pthread -std=c++11 -O3 -o rowhammer rowhammer.cc
rowhammer.cc: In function ‘int main(int, char**)’:
rowhammer.cc:243:57: error: inconsistent operand constraints in an
‘asm’
   asm volatile ("rdtscp" : "=a" (a), "=d" (d) : : "rcx");

I tried g++ -m64, but that does not seem to work here at all. I'll try
to find some way to compile it during the week.

(BTW any idea which version would be right for this cpu?

processor     : 1
vendor_id     : GenuineIntel
cpu family    : 6
model	      	: 23
model name	: Intel(R) Core(TM)2 Duo CPU     E7400  @ 2.80GHz
stepping	: 10

Its Wolfdale-3M according to wikipedia... that seems older than
indy/sandy/haswell/skylake, so I'll just use the generic version...?)

> >Ok, let me try that. Problem is that the machine I'm testing on takes
> >20 minutes to produce bit flip...
> 
> will be lots faster with my code above ;)

Yes, that will help :-).
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[toc] | [prev] | [next] | [standalone]

#1511919 — Re: [kernel-hardening] rowhammer protection [was Re: Getting interrupt every million cache misses]

On 29.10.2016 23:45, Pavel Machek wrote:
> indy/sandy/haswell/skylake, so I'll just use the generic version...?)

yes, generic might work, but i never tested it on anything that old...

on my system i have >30 bit flips per second (ivy bridge i5-3xxx) with 
the rowhammer-ivy test... sometimes even more than 100 per second...

[toc] | [prev] | [next] | [standalone]

#1511922 — Re: [kernel-hardening] rowhammer protection [was Re: Getting interrupt every million cache misses]

[Multipart message — attachments visible in raw view] — view raw

On Sat 2016-10-29 23:49:57, Daniel Gruss wrote:
> On 29.10.2016 23:45, Pavel Machek wrote:
> >indy/sandy/haswell/skylake, so I'll just use the generic version...?)
> 
> yes, generic might work, but i never tested it on anything that old...
> 
> on my system i have >30 bit flips per second (ivy bridge i5-3xxx) with the
> rowhammer-ivy test... sometimes even more than 100 per second...

Hmm, maybe I'm glad I don't have a new machine :-).

I assume you still get _some_ bitflips with generic "rowhammer"?

Best regards,
									Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[toc] | [prev] | [next] | [standalone]

#1511923 — Re: [kernel-hardening] rowhammer protection [was Re: Getting interrupt every million cache misses]

On 30.10.2016 00:01, Pavel Machek wrote:
> Hmm, maybe I'm glad I don't have a new machine :-).
>
> I assume you still get _some_ bitflips with generic "rowhammer"?

1 or 2 every 20-30 minutes...

[toc] | [prev] | [next] | [standalone]

#1512400 — Re: [kernel-hardening] rowhammer protection [was Re: Getting interrupt every million cache misses]

[Multipart message — attachments visible in raw view] — view raw

Hi!

> On Fri, Oct 28, 2016 at 01:21:36PM +0200, Pavel Machek wrote:
> > > Has this been tested on a system vulnerable to rowhammer, and if so, was
> > > it reliable in mitigating the issue?
> > > 
> > > Which particular attack codebase was it tested against?
> > 
> > I have rowhammer-test here,
> > 
> > commit 9824453fff76e0a3f5d1ac8200bc6c447c4fff57
> > Author: Mark Seaborn <mseaborn@chromium.org>
> 
> ... from which repo?
> 
> > I do not have vulnerable machine near me, so no "real" tests, but
> > I'm pretty sure it will make the error no longer reproducible with the
> > newer version. [Help welcome ;-)]
> 
> Even if we hope this works, I think we have to be very careful with that
> kind of assertion. Until we have data is to its efficacy, I don't think
> we should claim that this is an effective mitigation.

Ok, so it turns out I was right. On my vulnerable machine, normally
bug is reproducible in less than 500 iterations:

Iteration 432 (after 1013.31s)
  error at 0xda7cf280: got 0xffffffffffffffef
Iteration 446 (after 1102.56s)
  error at 0xec21ea00: got 0xffffffefffffffff
Iteration 206 (after 497.50s)
  error at 0xd07d1438: got 0xffffffffffffffdf
Iteration 409 (after 1350.96s)
  error at 0xbd3b9108: got 0xefffffffffffffff
Iteration 120 (after 326.08s)
  error at 0xe398c438: got 0xffffffffffffffdf

With nohammer, I'm at 2300 iterations, and still no faults.

Daniel Gruss <daniel@gruss.cc> claims he has an attack that can do 30
flips a second on modern hardware. I'm not going to buy broken
hardware just for a test. Code is at
https://github.com/IAIK/rowhammerjs/tree/master/native . Would someone
be willing to get it running on vulnerable machine and test kernel
patches?

Thanks,

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[toc] | [prev] | [next] | [standalone]

#1512583 — Re: [kernel-hardening] rowhammer protection [was Re: Getting interrupt every million cache misses]

On Mon, Oct 31, 2016 at 09:27:05AM +0100, Pavel Machek wrote:
> > On Fri, Oct 28, 2016 at 01:21:36PM +0200, Pavel Machek wrote:
> > > > Has this been tested on a system vulnerable to rowhammer, and if so, was
> > > > it reliable in mitigating the issue?

> > > I do not have vulnerable machine near me, so no "real" tests, but
> > > I'm pretty sure it will make the error no longer reproducible with the
> > > newer version. [Help welcome ;-)]
> > 
> > Even if we hope this works, I think we have to be very careful with that
> > kind of assertion. Until we have data is to its efficacy, I don't think
> > we should claim that this is an effective mitigation.
> 
> Ok, so it turns out I was right. On my vulnerable machine, normally
> bug is reproducible in less than 500 iterations:

> With nohammer, I'm at 2300 iterations, and still no faults.

To be quite frank, this is anecdotal. It only shows one particular attack is
made slower (or perhaps defeated), and doesn't show that the mitigation is
reliable or generally applicable (to other machines or other variants of the
attack).

Even if this happens to work on some machines, I still do not think one can
sell this as a generally applicable and reliable mitigation. Especially given
that others working in this area seem to have evidence otherwise, e.g. [1] (as
noted by spender in the LWN comments).

Thanks,
Mark.

[1] https://twitter.com/halvarflake/status/792314613568311296

[toc] | [prev] | [next] | [standalone]

#1512910 — Re: [kernel-hardening] rowhammer protection [was Re: Getting interrupt every million cache misses]

[Multipart message — attachments visible in raw view] — view raw

On Mon 2016-10-31 14:47:39, Mark Rutland wrote:
> On Mon, Oct 31, 2016 at 09:27:05AM +0100, Pavel Machek wrote:
> > > On Fri, Oct 28, 2016 at 01:21:36PM +0200, Pavel Machek wrote:
> > > > > Has this been tested on a system vulnerable to rowhammer, and if so, was
> > > > > it reliable in mitigating the issue?
> 
> > > > I do not have vulnerable machine near me, so no "real" tests, but
> > > > I'm pretty sure it will make the error no longer reproducible with the
> > > > newer version. [Help welcome ;-)]
> > > 
> > > Even if we hope this works, I think we have to be very careful with that
> > > kind of assertion. Until we have data is to its efficacy, I don't think
> > > we should claim that this is an effective mitigation.
...
> 
> To be quite frank, this is anecdotal. It only shows one particular attack is
> made slower (or perhaps defeated), and doesn't show that the mitigation is
> reliable or generally applicable (to other machines or other variants of the
> attack).

So... I said that I'm pretty sure it will fix problem in my testing,
then you say that I should be careful with my words, I confirm it was
true, and now you complain that it is anecdotal?

Are you serious?

Of course I know that fixing rowhammer-test on my machine is quite a
low bar to ask. _And that's also why I said I'm pretty sure I'd pass
that bar_.

I'm still asking for help with testing, but all you do is claim that
"we can't be sure".

> Even if this happens to work on some machines, I still do not think one can
> sell this as a generally applicable and reliable mitigation. Especially given
> that others working in this area seem to have evidence otherwise, e.g. [1] (as
> noted by spender in the LWN comments).

Slowing this attack _is_ defeating it. It is enough to slow it 8
times, and it is gone, boom, not there any more.

Now.. I have to figure out what to do with movnt. No currently known
attack uses movnt. Still, that one should be solved.

Other than that... this is not magic. Attack is quite well
understood. All you have to do is prevent more than 8msec worth of
memory accesses. My patch can do that, and it will work,
everywhere... you just won't like the fact that your machine now works
on 10% of original performance.

Now, it is possible that researches will come up with attack that only
needs 2msec worth of accesses. So we change the constants. Performance
will be even worse. It is also possible that even more broken DRAM
comes out. Same solution. Plus someone certainly has a memory that
flips some bits even without help from funny access patterns. Too
bad. We can't help them.

Would it be less confusing if we redefined task description from
"prevent rowhammer" to "prevent more than X memory accesses in 64
msec"?

Best regards,
									Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[toc] | [prev] | [next] | [standalone]

#1512936 — Re: [kernel-hardening] rowhammer protection [was Re: Getting interrupt every million cache misses]

On Mon, Oct 31, 2016 at 10:13:03PM +0100, Pavel Machek wrote:
> On Mon 2016-10-31 14:47:39, Mark Rutland wrote:
> > On Mon, Oct 31, 2016 at 09:27:05AM +0100, Pavel Machek wrote:
> > > > On Fri, Oct 28, 2016 at 01:21:36PM +0200, Pavel Machek wrote:
> > > > > > Has this been tested on a system vulnerable to rowhammer, and if so, was
> > > > > > it reliable in mitigating the issue?
> > 
> > > > > I do not have vulnerable machine near me, so no "real" tests, but
> > > > > I'm pretty sure it will make the error no longer reproducible with the
> > > > > newer version. [Help welcome ;-)]
> > > > 
> > > > Even if we hope this works, I think we have to be very careful with that
> > > > kind of assertion. Until we have data is to its efficacy, I don't think
> > > > we should claim that this is an effective mitigation.
> ...
> > 
> > To be quite frank, this is anecdotal. It only shows one particular attack is
> > made slower (or perhaps defeated), and doesn't show that the mitigation is
> > reliable or generally applicable (to other machines or other variants of the
> > attack).
> 
> So... I said that I'm pretty sure it will fix problem in my testing,
> then you say that I should be careful with my words, I confirm it was
> true, and now you complain that it is anecdotal?

Clearly I have chosen my words poorly here. I believe that this may help
against some attacks on some machines and workloads, and I believe your results
for your machine.

My main concern was that this appears to be described as a general solution, as
in the Kconfig text:

	Enable rowhammer attack prevention. Will degrade system
	performance under attack so much that attack should not
	be feasible.

... yet there are a number of reasons why this may not be the case given varied
attack mechanisms (e.g. using non-cacheable mappings, movnt, etc), given some
hardware configurations (e.g. "large" SMP machines or where timing is
marginal), given some workloads may incidentally trip often enough to be
severely penalised, and given that performance counter support is sufficiently
varied (across architectures, CPU implementations, and even boards using the
same CPU if one considers things like interrupt routing).

Given that, I think that makes an overly-strong, and perhaps misleading claim
(i.e. people could turn the option on and believe that they are protected, when
they are not, leaving them worse off). It isn't really possible to fail
gracefully here, and even if this is suitable for some hardware, very few
people are in a position to determine whether their hardware falls in that
category.

Unfortunately, I do not believe that there is a simple and/or general software
mitigation.

> Would it be less confusing if we redefined task description from
> "prevent rowhammer" to "prevent more than X memory accesses in 64
> msec"?

Definitely. Quantifying exactly what you're trying to defend against (and
therefore what you are not) would help to address at least one of my concerns.

Thanks,
Mark.

[toc] | [prev] | [next] | [standalone]

#1513108 — Re: [kernel-hardening] rowhammer protection [was Re: Getting interrupt every million cache misses]

* Pavel Machek <pavel@ucw.cz> wrote:

> I'm not going to buy broken hardware just for a test.

Can you suggest a method to find heavily rowhammer affected hardware? Only by 
testing it, or are there some chipset IDs ranges or dmidecode info that will 
pinpoint potentially affected machines?

Thanks,

	Ingo

[toc] | [prev] | [next] | [standalone]

#1513139 — Re: [kernel-hardening] rowhammer protection [was Re: Getting interrupt every million cache misses]

[Multipart message — attachments visible in raw view] — view raw

On Tue, 2016-11-01 at 07:33 +0100, Ingo Molnar wrote:
> * Pavel Machek <pavel@ucw.cz> wrote:
> 
> > I'm not going to buy broken hardware just for a test.
> 
> Can you suggest a method to find heavily rowhammer affected hardware?
> Only by 
> testing it, or are there some chipset IDs ranges or dmidecode info
> that will 
> pinpoint potentially affected machines?
> 
> Thanks,
> 
> 	Ingo

You can read the memory timing values, but you can't know if they're
reasonable for that hardware. Higher quality memory can have better
timings without being broken. The only relevant information would be the
memory model, combined with an expensive / time consuming effort to
build a blacklist based on testing. It doesn't seem realistic, unless
it's done in a coarse way based on brand and the date information.

I don't know how to get this data on Linux. The CPU-Z tool for Windows
knows how to obtain it but it's based on a proprietary library.

You definitely don't need to buy broken hardware to test a broken
hardware setup though. You just need a custom computer build where
motherboards expose the memory timing configuration. You can make it
more vulnerable by raising the refresh period (tREF). I wanted to play
around with that but haven't gotten around to it.

[toc] | [prev] | [next] | [standalone]

#1513149 — Re: [kernel-hardening] rowhammer protection [was Re: Getting interrupt every million cache misses]

On 01.11.2016 07:33, Ingo Molnar wrote:
> Can you suggest a method to find heavily rowhammer affected hardware? Only by
> testing it, or are there some chipset IDs ranges or dmidecode info that will
> pinpoint potentially affected machines?

I have worked with many different systems both on running rowhammer 
attacks and testing defense mechanisms. So far, every Ivy Bridge i5 
(DDR3) that I had access to was susceptible to bit flips - you will have 
highest chances with Ivy Bridge i5...

[toc] | [prev] | [next] | [standalone]

#1513152 — Re: [kernel-hardening] rowhammer protection [was Re: Getting interrupt every million cache misses]

On 01.11.2016 09:10, Pavel Machek wrote:
> cpu family     : 6
> model	       	 : 23
> model name	 : Intel(R) Core(TM)2 Duo CPU     E7400  @ 2.80GHz
> stepping	 : 10
> microcode	 : 0xa07
>
> so rowhammerjs/native is not available for this system. Bit mapping
> for memory hash functions would need to be reverse engineered for more
> effective attack.

By coincidence, we wrote a tool to do that in software: 
https://github.com/IAIK/drama ;)

[toc] | [prev] | [next] | [standalone]

#1513155 — Re: [kernel-hardening] rowhammer protection [was Re: Getting interrupt every million cache misses]

[Multipart message — attachments visible in raw view] — view raw

Hi!

> * Pavel Machek <pavel@ucw.cz> wrote:
> 
> > I'm not going to buy broken hardware just for a test.
> 
> Can you suggest a method to find heavily rowhammer affected hardware? Only by 
> testing it, or are there some chipset IDs ranges or dmidecode info that will 
> pinpoint potentially affected machines?

Testing can be used. https://github.com/mseaborn/rowhammer-test.git
. It finds faults at 1 of 2 machines here (but takes half an
hour). Then, if your hardware is one of ivy/sandy/haswell/skylake,
https://github.com/IAIK/rowhammerjs.git can be used for much faster
attack (many flips a second).

Unfortunately, what I have here is:

cpu family     : 6
model	       	 : 23
model name	 : Intel(R) Core(TM)2 Duo CPU     E7400  @ 2.80GHz
stepping	 : 10
microcode	 : 0xa07

so rowhammerjs/native is not available for this system. Bit mapping
for memory hash functions would need to be reverse engineered for more
effective attack.

Best regards,
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.kernel

csiph-web