Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > linux.kernel > #1263645 > unrolled thread

Re: [PATCH 0/2] "big hammer" for DAX msync/fsync correctness

Started byDan Williams <dan.j.williams@intel.com>
First post2015-11-06 01:00 +0100
Last post2015-11-06 21:30 +0100
Articles 12 — 3 participants

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.


Contents

  Re: [PATCH 0/2] "big hammer" for DAX msync/fsync correctness Dan Williams <dan.j.williams@intel.com> - 2015-11-06 01:00 +0100
    Re: [PATCH 0/2] "big hammer" for DAX msync/fsync correctness Thomas Gleixner <tglx@linutronix.de> - 2015-11-06 09:10 +0100
      Re: [PATCH 0/2] "big hammer" for DAX msync/fsync correctness Dan Williams <dan.j.williams@intel.com> - 2015-11-06 17:10 +0100
        Re: [PATCH 0/2] "big hammer" for DAX msync/fsync correctness Thomas Gleixner <tglx@linutronix.de> - 2015-11-06 18:40 +0100
          Re: [PATCH 0/2] "big hammer" for DAX msync/fsync correctness Dan Williams <dan.j.williams@intel.com> - 2015-11-07 00:20 +0100
            Re: [PATCH 0/2] "big hammer" for DAX msync/fsync correctness "H. Peter Anvin" <hpa@zytor.com> - 2015-11-07 02:00 +0100
              Re: [PATCH 0/2] "big hammer" for DAX msync/fsync correctness Thomas Gleixner <tglx@linutronix.de> - 2015-11-07 08:00 +0100
                Re: [PATCH 0/2] "big hammer" for DAX msync/fsync correctness Dan Williams <dan.j.williams@intel.com> - 2015-11-07 09:20 +0100
                  Re: [PATCH 0/2] "big hammer" for DAX msync/fsync correctness Thomas Gleixner <tglx@linutronix.de> - 2015-11-07 09:50 +0100
                    Re: [PATCH 0/2] "big hammer" for DAX msync/fsync correctness Dan Williams <dan.j.williams@intel.com> - 2015-11-07 10:10 +0100
                      Re: [PATCH 0/2] "big hammer" for DAX msync/fsync correctness Thomas Gleixner <tglx@linutronix.de> - 2015-11-07 10:30 +0100
    Re: [PATCH 0/2] "big hammer" for DAX msync/fsync correctness "H. Peter Anvin" <hpa@zytor.com> - 2015-11-06 21:30 +0100

#1263645 — Re: [PATCH 0/2] "big hammer" for DAX msync/fsync correctness

FromDan Williams <dan.j.williams@intel.com>
Date2015-11-06 01:00 +0100
SubjectRe: [PATCH 0/2] "big hammer" for DAX msync/fsync correctness
Message-ID<qrCBI-5dP-13@gated-at.bofh.it>
On Wed, Oct 28, 2015 at 3:51 PM, Ross Zwisler
<ross.zwisler@linux.intel.com> wrote:
> On Wed, Oct 28, 2015 at 06:24:29PM -0400, Jeff Moyer wrote:
>> Ross Zwisler <ross.zwisler@linux.intel.com> writes:
>>
>> > This series implements the very slow but correct handling for
>> > blkdev_issue_flush() with DAX mappings, as discussed here:
>> >
>> > https://lkml.org/lkml/2015/10/26/116
>> >
>> > I don't think that we can actually do the
>> >
>> >     on_each_cpu(sync_cache, ...);
>> >
>> > ...where sync_cache is something like:
>> >
>> >     cache_disable();
>> >     wbinvd();
>> >     pcommit();
>> >     cache_enable();
>> >
>> > solution as proposed by Dan because WBINVD + PCOMMIT doesn't guarantee that
>> > your writes actually make it durably onto the DIMMs.  I believe you really do
>> > need to loop through the cache lines, flush them with CLWB, then fence and
>> > PCOMMIT.
>>
>> *blink*
>> *blink*
>>
>> So much for not violating the principal of least surprise.  I suppose
>> you've asked the hardware folks, and they've sent you down this path?
>
> Sadly, yes, this was the guidance from the hardware folks.

So it turns out we weren't asking the right question.  wbinvd may
indeed be viable... we're still working through the caveats.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]


#1263825

FromThomas Gleixner <tglx@linutronix.de>
Date2015-11-06 09:10 +0100
Message-ID<qrKfU-1Vl-19@gated-at.bofh.it>
In reply to#1263645
On Thu, 5 Nov 2015, Dan Williams wrote:
> On Wed, Oct 28, 2015 at 3:51 PM, Ross Zwisler
> <ross.zwisler@linux.intel.com> wrote:
> > On Wed, Oct 28, 2015 at 06:24:29PM -0400, Jeff Moyer wrote:
> >> Ross Zwisler <ross.zwisler@linux.intel.com> writes:
> >>
> >> > This series implements the very slow but correct handling for
> >> > blkdev_issue_flush() with DAX mappings, as discussed here:
> >> >
> >> > https://lkml.org/lkml/2015/10/26/116
> >> >
> >> > I don't think that we can actually do the
> >> >
> >> >     on_each_cpu(sync_cache, ...);
> >> >
> >> > ...where sync_cache is something like:
> >> >
> >> >     cache_disable();
> >> >     wbinvd();
> >> >     pcommit();
> >> >     cache_enable();
> >> >
> >> > solution as proposed by Dan because WBINVD + PCOMMIT doesn't guarantee that
> >> > your writes actually make it durably onto the DIMMs.  I believe you really do
> >> > need to loop through the cache lines, flush them with CLWB, then fence and
> >> > PCOMMIT.
> >>
> >> *blink*
> >> *blink*
> >>
> >> So much for not violating the principal of least surprise.  I suppose
> >> you've asked the hardware folks, and they've sent you down this path?
> >
> > Sadly, yes, this was the guidance from the hardware folks.
> 
> So it turns out we weren't asking the right question.  wbinvd may
> indeed be viable... we're still working through the caveats.

Just for the record. Such a flush mechanism with 

     on_each_cpu()
	wbinvd()
	...
 
will make that stuff completely unusable on Real-Time systems. We've
been there with the big hammer approach of the intel graphics
driver.

Thanks,

	tglx
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]


#1264125

FromDan Williams <dan.j.williams@intel.com>
Date2015-11-06 17:10 +0100
Message-ID<qrRKr-6IF-29@gated-at.bofh.it>
In reply to#1263825
On Fri, Nov 6, 2015 at 12:06 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
> On Thu, 5 Nov 2015, Dan Williams wrote:
>> On Wed, Oct 28, 2015 at 3:51 PM, Ross Zwisler
>> <ross.zwisler@linux.intel.com> wrote:
>> > On Wed, Oct 28, 2015 at 06:24:29PM -0400, Jeff Moyer wrote:
>> >> Ross Zwisler <ross.zwisler@linux.intel.com> writes:
>> >>
>> >> > This series implements the very slow but correct handling for
>> >> > blkdev_issue_flush() with DAX mappings, as discussed here:
>> >> >
>> >> > https://lkml.org/lkml/2015/10/26/116
>> >> >
>> >> > I don't think that we can actually do the
>> >> >
>> >> >     on_each_cpu(sync_cache, ...);
>> >> >
>> >> > ...where sync_cache is something like:
>> >> >
>> >> >     cache_disable();
>> >> >     wbinvd();
>> >> >     pcommit();
>> >> >     cache_enable();
>> >> >
>> >> > solution as proposed by Dan because WBINVD + PCOMMIT doesn't guarantee that
>> >> > your writes actually make it durably onto the DIMMs.  I believe you really do
>> >> > need to loop through the cache lines, flush them with CLWB, then fence and
>> >> > PCOMMIT.
>> >>
>> >> *blink*
>> >> *blink*
>> >>
>> >> So much for not violating the principal of least surprise.  I suppose
>> >> you've asked the hardware folks, and they've sent you down this path?
>> >
>> > Sadly, yes, this was the guidance from the hardware folks.
>>
>> So it turns out we weren't asking the right question.  wbinvd may
>> indeed be viable... we're still working through the caveats.
>
> Just for the record. Such a flush mechanism with
>
>      on_each_cpu()
>         wbinvd()
>         ...
>
> will make that stuff completely unusable on Real-Time systems. We've
> been there with the big hammer approach of the intel graphics
> driver.

Noted.  This means RT systems either need to disable DAX or avoid
fsync.  Yes, this is a wart, but not an unexpected one in a first
generation persistent memory platform.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]


#1264187

FromThomas Gleixner <tglx@linutronix.de>
Date2015-11-06 18:40 +0100
Message-ID<qrT9x-7vW-11@gated-at.bofh.it>
In reply to#1264125
On Fri, 6 Nov 2015, Dan Williams wrote:
> On Fri, Nov 6, 2015 at 12:06 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
> > Just for the record. Such a flush mechanism with
> >
> >      on_each_cpu()
> >         wbinvd()
> >         ...
> >
> > will make that stuff completely unusable on Real-Time systems. We've
> > been there with the big hammer approach of the intel graphics
> > driver.
> 
> Noted.  This means RT systems either need to disable DAX or avoid
> fsync.  Yes, this is a wart, but not an unexpected one in a first
> generation persistent memory platform.

And it's not just only RT. The folks who are aiming for 100%
undisturbed user space (NOHZ_FULL) will be massively unhappy about
that as well.

Is it really required to do that on all cpus?

Thanks,

	tglx
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]


#1264641

FromDan Williams <dan.j.williams@intel.com>
Date2015-11-07 00:20 +0100
Message-ID<qrYsx-2Hd-5@gated-at.bofh.it>
In reply to#1264187
On Fri, Nov 6, 2015 at 9:35 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
> On Fri, 6 Nov 2015, Dan Williams wrote:
>> On Fri, Nov 6, 2015 at 12:06 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
>> > Just for the record. Such a flush mechanism with
>> >
>> >      on_each_cpu()
>> >         wbinvd()
>> >         ...
>> >
>> > will make that stuff completely unusable on Real-Time systems. We've
>> > been there with the big hammer approach of the intel graphics
>> > driver.
>>
>> Noted.  This means RT systems either need to disable DAX or avoid
>> fsync.  Yes, this is a wart, but not an unexpected one in a first
>> generation persistent memory platform.
>
> And it's not just only RT. The folks who are aiming for 100%
> undisturbed user space (NOHZ_FULL) will be massively unhappy about
> that as well.
>
> Is it really required to do that on all cpus?
>

I believe it is, but I'll double check.

I assume the folks that want undisturbed userspace are ok with the
mitigation to modify their application to flush by individual cache
lines if they want to use DAX without fsync.  At least until the
platform can provide a cheaper fsync implementation.

The option to drive cache flushing from the radix is at least
interruptible, but it might be long running depending on how much
virtual address space is dirty.  Altogether, the options in the
current generation are:

1/ wbinvd driven: quick flush O(size of cache), but long interrupt-off latency

2/ radix driven: long flush O(size of dirty range), but at least preempt-able

3/ DAX without calling fsync: userspace takes direct responsibility
for cache management of DAX mappings

4/ DAX disabled: fsync is the standard page cache writeback latency

We could potentially argue about 1 vs 2 ad nauseum, but I wonder if
there is room to it punt it to a configuration option or make it
dynamic?  My stance is do 1 with the hope of riding options 3 and 4
until the platform happens to provide a better alternative.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]


#1264677

From"H. Peter Anvin" <hpa@zytor.com>
Date2015-11-07 02:00 +0100
Message-ID<qs01j-3vl-3@gated-at.bofh.it>
In reply to#1264641
On 11/06/15 15:17, Dan Williams wrote:
>>
>> Is it really required to do that on all cpus?
> 
> I believe it is, but I'll double check.
> 

It's required on all CPUs on which the DAX memory may have been dirtied.
 This is similar to the way we flush TLBs.

	-hpa


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]


#1264744

FromThomas Gleixner <tglx@linutronix.de>
Date2015-11-07 08:00 +0100
Message-ID<qs5DI-7fZ-5@gated-at.bofh.it>
In reply to#1264677
On Fri, 6 Nov 2015, H. Peter Anvin wrote:
> On 11/06/15 15:17, Dan Williams wrote:
> >>
> >> Is it really required to do that on all cpus?
> > 
> > I believe it is, but I'll double check.
> > 
> 
> It's required on all CPUs on which the DAX memory may have been dirtied.
>  This is similar to the way we flush TLBs.

Right. And that's exactly the problem: "may have been dirtied"

If DAX is used on 50% of the CPUs and the other 50% are plumming away
happily in user space or run low latency RT tasks w/o ever touching
it, then having an unconditional flush on ALL CPUs is just wrong
because you penalize the uninvolved cores with a completely pointless
SMP function call and drain their caches.

Thanks,

	tglx
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]


#1264767

FromDan Williams <dan.j.williams@intel.com>
Date2015-11-07 09:20 +0100
Message-ID<qs6T7-8cb-1@gated-at.bofh.it>
In reply to#1264744
On Fri, Nov 6, 2015 at 10:50 PM, Thomas Gleixner <tglx@linutronix.de> wrote:
> On Fri, 6 Nov 2015, H. Peter Anvin wrote:
>> On 11/06/15 15:17, Dan Williams wrote:
>> >>
>> >> Is it really required to do that on all cpus?
>> >
>> > I believe it is, but I'll double check.
>> >
>>
>> It's required on all CPUs on which the DAX memory may have been dirtied.
>>  This is similar to the way we flush TLBs.
>
> Right. And that's exactly the problem: "may have been dirtied"
>
> If DAX is used on 50% of the CPUs and the other 50% are plumming away
> happily in user space or run low latency RT tasks w/o ever touching
> it, then having an unconditional flush on ALL CPUs is just wrong
> because you penalize the uninvolved cores with a completely pointless
> SMP function call and drain their caches.
>

It's not wrong and pointless, it's all we have available outside of
having the kernel remember every virtual address that might have been
touched since the last fsync and sit in a loop flushing those virtual
address cache line by cache line.

There is a crossover point where wbinvd is better than a clwb loop
that needs to be determined.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]


#1264770

FromThomas Gleixner <tglx@linutronix.de>
Date2015-11-07 09:50 +0100
Message-ID<qs7m9-8o7-5@gated-at.bofh.it>
In reply to#1264767
On Sat, 7 Nov 2015, Dan Williams wrote:
> On Fri, Nov 6, 2015 at 10:50 PM, Thomas Gleixner <tglx@linutronix.de> wrote:
> > On Fri, 6 Nov 2015, H. Peter Anvin wrote:
> >> On 11/06/15 15:17, Dan Williams wrote:
> >> >>
> >> >> Is it really required to do that on all cpus?
> >> >
> >> > I believe it is, but I'll double check.
> >> >
> >>
> >> It's required on all CPUs on which the DAX memory may have been dirtied.
> >>  This is similar to the way we flush TLBs.
> >
> > Right. And that's exactly the problem: "may have been dirtied"
> >
> > If DAX is used on 50% of the CPUs and the other 50% are plumming away
> > happily in user space or run low latency RT tasks w/o ever touching
> > it, then having an unconditional flush on ALL CPUs is just wrong
> > because you penalize the uninvolved cores with a completely pointless
> > SMP function call and drain their caches.
> >
> 
> It's not wrong and pointless, it's all we have available outside of
> having the kernel remember every virtual address that might have been
> touched since the last fsync and sit in a loop flushing those virtual
> address cache line by cache line.
> 
> There is a crossover point where wbinvd is better than a clwb loop
> that needs to be determined.

This is a totally different issue and I'm well aware that there is a
tradeoff between wbinvd() and a clwb loop. wbinvd() might be more
efficient performance wise above some number of cache lines, but then
again it's draining all unrelated stuff as well, which can result in a
even larger performance hit.

Now what really concerns me more is that you just unconditionally
flush on all CPUs whether they were involved in that DAX stuff or not.

Assume that DAX using application on CPU 0-3 and some other unrelated
workload on CPU4-7. That flush will 

  - Interrupt CPU4-7 for no reason (whether you use clwb or wbinvd)

  - Drain the cache for CPU4-7 for no reason if done with wbinvd()
   
  - Render Cache Allocation useless if done with wbinvd()

And we are not talking about a few micro seconds here. Assume that
CPU4-7 have cache allocated and it's mostly dirty. We've measured the
wbinvd() impact on RT, back then when the graphic folks used it as a
big hammer. The maximum latency spike was way above one millisecond.

We have similar issues with TLB flushing, but there we 

  - are tracking where it was used and never flush on innocent cpus

  - one can design his application in a way that it uses different
    processes so cross CPU flushing does not happen

I know that this is not an easy problem to solve, but you should be
aware that various application scenarios are going to be massively
unhappy about that.

Thanks,

	tglx







--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]


#1264771

FromDan Williams <dan.j.williams@intel.com>
Date2015-11-07 10:10 +0100
Message-ID<qs7Fv-jH-1@gated-at.bofh.it>
In reply to#1264770
On Sat, Nov 7, 2015 at 12:38 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
> On Sat, 7 Nov 2015, Dan Williams wrote:
>> On Fri, Nov 6, 2015 at 10:50 PM, Thomas Gleixner <tglx@linutronix.de> wrote:
>> > On Fri, 6 Nov 2015, H. Peter Anvin wrote:
>> >> On 11/06/15 15:17, Dan Williams wrote:
>> >> >>
>> >> >> Is it really required to do that on all cpus?
>> >> >
>> >> > I believe it is, but I'll double check.
>> >> >
>> >>
>> >> It's required on all CPUs on which the DAX memory may have been dirtied.
>> >>  This is similar to the way we flush TLBs.
>> >
>> > Right. And that's exactly the problem: "may have been dirtied"
>> >
>> > If DAX is used on 50% of the CPUs and the other 50% are plumming away
>> > happily in user space or run low latency RT tasks w/o ever touching
>> > it, then having an unconditional flush on ALL CPUs is just wrong
>> > because you penalize the uninvolved cores with a completely pointless
>> > SMP function call and drain their caches.
>> >
>>
>> It's not wrong and pointless, it's all we have available outside of
>> having the kernel remember every virtual address that might have been
>> touched since the last fsync and sit in a loop flushing those virtual
>> address cache line by cache line.
>>
>> There is a crossover point where wbinvd is better than a clwb loop
>> that needs to be determined.
>
> This is a totally different issue and I'm well aware that there is a
> tradeoff between wbinvd() and a clwb loop. wbinvd() might be more
> efficient performance wise above some number of cache lines, but then
> again it's draining all unrelated stuff as well, which can result in a
> even larger performance hit.
>
> Now what really concerns me more is that you just unconditionally
> flush on all CPUs whether they were involved in that DAX stuff or not.
>
> Assume that DAX using application on CPU 0-3 and some other unrelated
> workload on CPU4-7. That flush will
>
>   - Interrupt CPU4-7 for no reason (whether you use clwb or wbinvd)
>
>   - Drain the cache for CPU4-7 for no reason if done with wbinvd()
>
>   - Render Cache Allocation useless if done with wbinvd()
>
> And we are not talking about a few micro seconds here. Assume that
> CPU4-7 have cache allocated and it's mostly dirty. We've measured the
> wbinvd() impact on RT, back then when the graphic folks used it as a
> big hammer. The maximum latency spike was way above one millisecond.
>
> We have similar issues with TLB flushing, but there we
>
>   - are tracking where it was used and never flush on innocent cpus
>
>   - one can design his application in a way that it uses different
>     processes so cross CPU flushing does not happen
>
> I know that this is not an easy problem to solve, but you should be
> aware that various application scenarios are going to be massively
> unhappy about that.
>

Thanks for that explanation.  Peter had alluded to it at KS, but I
indeed did not know that it was as horrible as milliseconds of
latency, hmm...

One other mitigation that follows on with Dave's plan of per-inode DAX
control, is to also track when an inode has a writable DAX mmap
established.  With that we could have a REQ_DAX flag to augment
REQ_FLUSH to potentially reduce committing violence on the cache.  In
an earlier thread I also recall an idea to have an mmap flag that an
app can use to say "yes, I'm doing a writable DAX mapping, but I'm
taking care of the cache myself".  We could track innocent cpus, but
I'm thinking that would be a core change to write-protect pages when a
thread migrates?  In general I feel there's a limit for how much
hardware workaround is reasonable to do in the core kernel vs waiting
for the platform to offer better options...

Sorry if I'm being a bit punchy, but I'm still feeling like I need to
defend the notion that DAX may just need to be turned off in some
situations.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]


#1264774

FromThomas Gleixner <tglx@linutronix.de>
Date2015-11-07 10:30 +0100
Message-ID<qs7YS-qx-11@gated-at.bofh.it>
In reply to#1264771
On Sat, 7 Nov 2015, Dan Williams wrote:
> Thanks for that explanation.  Peter had alluded to it at KS, but I
> indeed did not know that it was as horrible as milliseconds of
> latency, hmm...

Yes, I was pretty surprised as well. But even if it's just in the
hundreds of microseconds it can be too much for latency sensitive
applications.
 
> One other mitigation that follows on with Dave's plan of per-inode DAX
> control, is to also track when an inode has a writable DAX mmap
> established.  With that we could have a REQ_DAX flag to augment
> REQ_FLUSH to potentially reduce committing violence on the cache.  In
> an earlier thread I also recall an idea to have an mmap flag that an
> app can use to say "yes, I'm doing a writable DAX mapping, but I'm
> taking care of the cache myself".  We could track innocent cpus, but
> I'm thinking that would be a core change to write-protect pages when a
> thread migrates?  In general I feel there's a limit for how much
> hardware workaround is reasonable to do in the core kernel vs waiting
> for the platform to offer better options...

One thing vs. the mmaps: We exactly know which CPUs are involved in
that mapping. We know that from the TLB management. So we probably can
make use of that knowledge.

> Sorry if I'm being a bit punchy, but I'm still feeling like I need to
> defend the notion that DAX may just need to be turned off in some
> situations.

That's fine, if there is no reasonable way around it. It just needs to
be documented so people won't be surprised.

Thanks,

	tglx

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]


#1264444

From"H. Peter Anvin" <hpa@zytor.com>
Date2015-11-06 21:30 +0100
Message-ID<qrVO4-Q7-47@gated-at.bofh.it>
In reply to#1263645
On November 5, 2015 3:59:46 PM PST, Dan Williams <dan.j.williams@intel.com> wrote:
>On Wed, Oct 28, 2015 at 3:51 PM, Ross Zwisler
><ross.zwisler@linux.intel.com> wrote:
>> On Wed, Oct 28, 2015 at 06:24:29PM -0400, Jeff Moyer wrote:
>>> Ross Zwisler <ross.zwisler@linux.intel.com> writes:
>>>
>>> > This series implements the very slow but correct handling for
>>> > blkdev_issue_flush() with DAX mappings, as discussed here:
>>> >
>>> > https://lkml.org/lkml/2015/10/26/116
>>> >
>>> > I don't think that we can actually do the
>>> >
>>> >     on_each_cpu(sync_cache, ...);
>>> >
>>> > ...where sync_cache is something like:
>>> >
>>> >     cache_disable();
>>> >     wbinvd();
>>> >     pcommit();
>>> >     cache_enable();
>>> >
>>> > solution as proposed by Dan because WBINVD + PCOMMIT doesn't
>guarantee that
>>> > your writes actually make it durably onto the DIMMs.  I believe
>you really do
>>> > need to loop through the cache lines, flush them with CLWB, then
>fence and
>>> > PCOMMIT.
>>>
>>> *blink*
>>> *blink*
>>>
>>> So much for not violating the principal of least surprise.  I
>suppose
>>> you've asked the hardware folks, and they've sent you down this
>path?
>>
>> Sadly, yes, this was the guidance from the hardware folks.
>
>So it turns out we weren't asking the right question.  wbinvd may
>indeed be viable... we're still working through the caveats.

Do not disable the caches here.
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]


Back to top | Article view | linux.kernel


csiph-web