Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1690377 > unrolled thread

Re: [PATCH 0/6] Cache coherent device memory (CDM) with HMM v5

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH 0/6] Cache coherent device memory (CDM) with HMM v5 Jerome Glisse <jglisse@redhat.com> - 2017-07-18 17:40 +0200
    Re: [PATCH 0/6] Cache coherent device memory (CDM) with HMM v5 Bob Liu <liubo95@huawei.com> - 2017-07-19 03:50 +0200
      Re: [PATCH 0/6] Cache coherent device memory (CDM) with HMM v5 Jerome Glisse <jglisse@redhat.com> - 2017-07-19 04:30 +0200
        Re: [PATCH 0/6] Cache coherent device memory (CDM) with HMM v5 Bob Liu <liubo95@huawei.com> - 2017-07-19 11:10 +0200
          Re: [PATCH 0/6] Cache coherent device memory (CDM) with HMM v5 Jerome Glisse <jglisse@redhat.com> - 2017-07-20 17:10 +0200
            Re: [PATCH 0/6] Cache coherent device memory (CDM) with HMM v5 Bob Liu <liubo95@huawei.com> - 2017-07-21 03:20 +0200
              Re: [PATCH 0/6] Cache coherent device memory (CDM) with HMM v5 Jerome Glisse <jglisse@redhat.com> - 2017-07-21 03:50 +0200
                Re: [PATCH 0/6] Cache coherent device memory (CDM) with HMM v5 Bob Liu <liubo95@huawei.com> - 2017-07-21 04:20 +0200
                  Re: [PATCH 0/6] Cache coherent device memory (CDM) with HMM v5 Bob Liu <lliubbo@gmail.com> - 2017-07-21 14:10 +0200
                    Re: [PATCH 0/6] Cache coherent device memory (CDM) with HMM v5 Jerome Glisse <jglisse@redhat.com> - 2017-07-21 17:30 +0200
                Re: [PATCH 0/6] Cache coherent device memory (CDM) with HMM v5 Dan Williams <dan.j.williams@intel.com> - 2017-07-21 05:50 +0200
                  Re: [PATCH 0/6] Cache coherent device memory (CDM) with HMM v5 Jerome Glisse <jglisse@redhat.com> - 2017-07-21 17:30 +0200

#1690377 — Re: [PATCH 0/6] Cache coherent device memory (CDM) with HMM v5

On Tue, Jul 18, 2017 at 11:26:51AM +0800, Bob Liu wrote:
> On 2017/7/14 5:15, Jérôme Glisse wrote:
> > Sorry i made horrible mistake on names in v4, i completly miss-
> > understood the suggestion. So here i repost with proper naming.
> > This is the only change since v3. Again sorry about the noise
> > with v4.
> > 
> > Changes since v4:
> >   - s/DEVICE_HOST/DEVICE_PUBLIC
> > 
> > Git tree:
> > https://cgit.freedesktop.org/~glisse/linux/log/?h=hmm-cdm-v5
> > 
> > 
> > Cache coherent device memory apply to architecture with system bus
> > like CAPI or CCIX. Device connected to such system bus can expose
> > their memory to the system and allow cache coherent access to it
> > from the CPU.
> > 
> > Even if for all intent and purposes device memory behave like regular
> > memory, we still want to manage it in isolation from regular memory.
> > Several reasons for that, first and foremost this memory is less
> > reliable than regular memory if the device hangs because of invalid
> > commands we can loose access to device memory. Second CPU access to
> > this memory is expected to be slower than to regular memory. Third
> > having random memory into device means that some of the bus bandwith
> > wouldn't be available to the device but would be use by CPU access.
> > 
> > This is why we want to manage such memory in isolation from regular
> > memory. Kernel should not try to use this memory even as last resort
> > when running out of memory, at least for now.
> >
> 
> I think set a very large node distance for "Cache Coherent Device Memory"
> may be a easier way to address these concerns.

Such approach was discuss at length in the past see links below. Outcome
of discussion:
  - CPU less node are bad
  - device memory can be unreliable (device hang) no way for application
    to understand that
  - application and driver NUMA madvise/mbind/mempolicy ... can conflict
    with each other and no way the kernel can figure out which should
    apply
  - NUMA as it is now would not work as we need further isolation that
    what a large node distance would provide

Probably few others argument i forget.

https://lists.gt.net/linux/kernel/2551369
https://groups.google.com/forum/#!topic/linux.kernel/Za_e8C3XnRs%5B1-25%5D
https://lwn.net/Articles/720380/

Cheers,
Jérôme

[toc] | [next] | [standalone]

#1690884

On 2017/7/18 23:38, Jerome Glisse wrote:
> On Tue, Jul 18, 2017 at 11:26:51AM +0800, Bob Liu wrote:
>> On 2017/7/14 5:15, Jérôme Glisse wrote:
>>> Sorry i made horrible mistake on names in v4, i completly miss-
>>> understood the suggestion. So here i repost with proper naming.
>>> This is the only change since v3. Again sorry about the noise
>>> with v4.
>>>
>>> Changes since v4:
>>>   - s/DEVICE_HOST/DEVICE_PUBLIC
>>>
>>> Git tree:
>>> https://cgit.freedesktop.org/~glisse/linux/log/?h=hmm-cdm-v5
>>>
>>>
>>> Cache coherent device memory apply to architecture with system bus
>>> like CAPI or CCIX. Device connected to such system bus can expose
>>> their memory to the system and allow cache coherent access to it
>>> from the CPU.
>>>
>>> Even if for all intent and purposes device memory behave like regular
>>> memory, we still want to manage it in isolation from regular memory.
>>> Several reasons for that, first and foremost this memory is less
>>> reliable than regular memory if the device hangs because of invalid
>>> commands we can loose access to device memory. Second CPU access to
>>> this memory is expected to be slower than to regular memory. Third
>>> having random memory into device means that some of the bus bandwith
>>> wouldn't be available to the device but would be use by CPU access.
>>>
>>> This is why we want to manage such memory in isolation from regular
>>> memory. Kernel should not try to use this memory even as last resort
>>> when running out of memory, at least for now.
>>>
>>
>> I think set a very large node distance for "Cache Coherent Device Memory"
>> may be a easier way to address these concerns.
> 
> Such approach was discuss at length in the past see links below. Outcome
> of discussion:
>   - CPU less node are bad
>   - device memory can be unreliable (device hang) no way for application
>     to understand that

Device memory can also be more reliable if using high quality and expensive memory.

>   - application and driver NUMA madvise/mbind/mempolicy ... can conflict
>     with each other and no way the kernel can figure out which should
>     apply
>   - NUMA as it is now would not work as we need further isolation that
>     what a large node distance would provide
> 

Agree, that's where we need spend time on.

One drawback of HMM-CDM I'm worry about is one more extra copy.
In the cache coherent case, CPU can write data to device memory directly then start fpga/GPU/other accelerators.

Thanks,
Bob Liu

[toc] | [prev] | [next] | [standalone]

#1690913

On Wed, Jul 19, 2017 at 09:46:10AM +0800, Bob Liu wrote:
> On 2017/7/18 23:38, Jerome Glisse wrote:
> > On Tue, Jul 18, 2017 at 11:26:51AM +0800, Bob Liu wrote:
> >> On 2017/7/14 5:15, Jérôme Glisse wrote:
> >>> Sorry i made horrible mistake on names in v4, i completly miss-
> >>> understood the suggestion. So here i repost with proper naming.
> >>> This is the only change since v3. Again sorry about the noise
> >>> with v4.
> >>>
> >>> Changes since v4:
> >>>   - s/DEVICE_HOST/DEVICE_PUBLIC
> >>>
> >>> Git tree:
> >>> https://cgit.freedesktop.org/~glisse/linux/log/?h=hmm-cdm-v5
> >>>
> >>>
> >>> Cache coherent device memory apply to architecture with system bus
> >>> like CAPI or CCIX. Device connected to such system bus can expose
> >>> their memory to the system and allow cache coherent access to it
> >>> from the CPU.
> >>>
> >>> Even if for all intent and purposes device memory behave like regular
> >>> memory, we still want to manage it in isolation from regular memory.
> >>> Several reasons for that, first and foremost this memory is less
> >>> reliable than regular memory if the device hangs because of invalid
> >>> commands we can loose access to device memory. Second CPU access to
> >>> this memory is expected to be slower than to regular memory. Third
> >>> having random memory into device means that some of the bus bandwith
> >>> wouldn't be available to the device but would be use by CPU access.
> >>>
> >>> This is why we want to manage such memory in isolation from regular
> >>> memory. Kernel should not try to use this memory even as last resort
> >>> when running out of memory, at least for now.
> >>>
> >>
> >> I think set a very large node distance for "Cache Coherent Device Memory"
> >> may be a easier way to address these concerns.
> > 
> > Such approach was discuss at length in the past see links below. Outcome
> > of discussion:
> >   - CPU less node are bad
> >   - device memory can be unreliable (device hang) no way for application
> >     to understand that
> 
> Device memory can also be more reliable if using high quality and expensive memory.

Even ECC memory does not compensate for device hang. When your GPU lockups
you might need to re-init GPU from scratch after which the content of the
device memory is unreliable. During init the device memory might not get
proper clock or proper refresh cycle and thus is susceptible to corruption.

> 
> >   - application and driver NUMA madvise/mbind/mempolicy ... can conflict
> >     with each other and no way the kernel can figure out which should
> >     apply
> >   - NUMA as it is now would not work as we need further isolation that
> >     what a large node distance would provide
> > 
> 
> Agree, that's where we need spend time on.
> 
> One drawback of HMM-CDM I'm worry about is one more extra copy.
> In the cache coherent case, CPU can write data to device memory
> directly then start fpga/GPU/other accelerators.

There is not necessarily an extra copy. Device driver can pre-allocate
virtual address range of a process with device memory. Device page fault
can directly allocate device memory. Once allocated CPU access will use
the device memory.

There is plan to allow other allocation (CPU page fault, file cache, ...)
to also use device memory directly. We just don't know what kind of
userspace API will fit best for that so at first it might be hidden behind
device driver specific ioctl. 

Jérôme

[toc] | [prev] | [next] | [standalone]

#1691148

On 2017/7/19 10:25, Jerome Glisse wrote:
> On Wed, Jul 19, 2017 at 09:46:10AM +0800, Bob Liu wrote:
>> On 2017/7/18 23:38, Jerome Glisse wrote:
>>> On Tue, Jul 18, 2017 at 11:26:51AM +0800, Bob Liu wrote:
>>>> On 2017/7/14 5:15, Jérôme Glisse wrote:
>>>>> Sorry i made horrible mistake on names in v4, i completly miss-
>>>>> understood the suggestion. So here i repost with proper naming.
>>>>> This is the only change since v3. Again sorry about the noise
>>>>> with v4.
>>>>>
>>>>> Changes since v4:
>>>>>   - s/DEVICE_HOST/DEVICE_PUBLIC
>>>>>
>>>>> Git tree:
>>>>> https://cgit.freedesktop.org/~glisse/linux/log/?h=hmm-cdm-v5
>>>>>
>>>>>
>>>>> Cache coherent device memory apply to architecture with system bus
>>>>> like CAPI or CCIX. Device connected to such system bus can expose
>>>>> their memory to the system and allow cache coherent access to it
>>>>> from the CPU.
>>>>>
>>>>> Even if for all intent and purposes device memory behave like regular
>>>>> memory, we still want to manage it in isolation from regular memory.
>>>>> Several reasons for that, first and foremost this memory is less
>>>>> reliable than regular memory if the device hangs because of invalid
>>>>> commands we can loose access to device memory. Second CPU access to
>>>>> this memory is expected to be slower than to regular memory. Third
>>>>> having random memory into device means that some of the bus bandwith
>>>>> wouldn't be available to the device but would be use by CPU access.
>>>>>
>>>>> This is why we want to manage such memory in isolation from regular
>>>>> memory. Kernel should not try to use this memory even as last resort
>>>>> when running out of memory, at least for now.
>>>>>
>>>>
>>>> I think set a very large node distance for "Cache Coherent Device Memory"
>>>> may be a easier way to address these concerns.
>>>
>>> Such approach was discuss at length in the past see links below. Outcome
>>> of discussion:
>>>   - CPU less node are bad
>>>   - device memory can be unreliable (device hang) no way for application
>>>     to understand that
>>
>> Device memory can also be more reliable if using high quality and expensive memory.
> 
> Even ECC memory does not compensate for device hang. When your GPU lockups
> you might need to re-init GPU from scratch after which the content of the
> device memory is unreliable. During init the device memory might not get
> proper clock or proper refresh cycle and thus is susceptible to corruption.
> 
>>
>>>   - application and driver NUMA madvise/mbind/mempolicy ... can conflict
>>>     with each other and no way the kernel can figure out which should
>>>     apply
>>>   - NUMA as it is now would not work as we need further isolation that
>>>     what a large node distance would provide
>>>
>>
>> Agree, that's where we need spend time on.
>>
>> One drawback of HMM-CDM I'm worry about is one more extra copy.
>> In the cache coherent case, CPU can write data to device memory
>> directly then start fpga/GPU/other accelerators.
> 
> There is not necessarily an extra copy. Device driver can pre-allocate
> virtual address range of a process with device memory. Device page fault

Okay, I get your point.
But the typical use case is CPU allocate a memory and prepare/write data then launch GPU "cuda kernel".
How to control the allocation go to device memory e.g HBM or system DDR at the beginning without user explicit advise?
If goes to DDR by default, there is an extra copy. If goes to HBM by default, the HBM may be waste.

> can directly allocate device memory. Once allocated CPU access will use
> the device memory.
> 

Then it's more like replace the numa node solution(CDM) with ZONE_DEVICE(type MEMORY_DEVICE_PUBLIC).
But the problem is the same, e.g how to make sure the device memory say HBM won't be occupied by normal CPU allocation.
Things will be more complex if there are multi GPU connected by nvlink(also cache coherent) in a system, each GPU has their own HBM.
How to decide allocate physical memory from local HBM/DDR or remote HBM/DDR? 
If using numa(CDM) approach there are NUMA mempolicy and autonuma mechanism at least.

Thanks,
Bob

[toc] | [prev] | [next] | [standalone]

#1693016

On Wed, Jul 19, 2017 at 05:09:04PM +0800, Bob Liu wrote:
> On 2017/7/19 10:25, Jerome Glisse wrote:
> > On Wed, Jul 19, 2017 at 09:46:10AM +0800, Bob Liu wrote:
> >> On 2017/7/18 23:38, Jerome Glisse wrote:
> >>> On Tue, Jul 18, 2017 at 11:26:51AM +0800, Bob Liu wrote:
> >>>> On 2017/7/14 5:15, Jérôme Glisse wrote:
> >>>>> Sorry i made horrible mistake on names in v4, i completly miss-
> >>>>> understood the suggestion. So here i repost with proper naming.
> >>>>> This is the only change since v3. Again sorry about the noise
> >>>>> with v4.
> >>>>>
> >>>>> Changes since v4:
> >>>>>   - s/DEVICE_HOST/DEVICE_PUBLIC
> >>>>>
> >>>>> Git tree:
> >>>>> https://cgit.freedesktop.org/~glisse/linux/log/?h=hmm-cdm-v5
> >>>>>
> >>>>>
> >>>>> Cache coherent device memory apply to architecture with system bus
> >>>>> like CAPI or CCIX. Device connected to such system bus can expose
> >>>>> their memory to the system and allow cache coherent access to it
> >>>>> from the CPU.
> >>>>>
> >>>>> Even if for all intent and purposes device memory behave like regular
> >>>>> memory, we still want to manage it in isolation from regular memory.
> >>>>> Several reasons for that, first and foremost this memory is less
> >>>>> reliable than regular memory if the device hangs because of invalid
> >>>>> commands we can loose access to device memory. Second CPU access to
> >>>>> this memory is expected to be slower than to regular memory. Third
> >>>>> having random memory into device means that some of the bus bandwith
> >>>>> wouldn't be available to the device but would be use by CPU access.
> >>>>>
> >>>>> This is why we want to manage such memory in isolation from regular
> >>>>> memory. Kernel should not try to use this memory even as last resort
> >>>>> when running out of memory, at least for now.
> >>>>>
> >>>>
> >>>> I think set a very large node distance for "Cache Coherent Device Memory"
> >>>> may be a easier way to address these concerns.
> >>>
> >>> Such approach was discuss at length in the past see links below. Outcome
> >>> of discussion:
> >>>   - CPU less node are bad
> >>>   - device memory can be unreliable (device hang) no way for application
> >>>     to understand that
> >>
> >> Device memory can also be more reliable if using high quality and expensive memory.
> > 
> > Even ECC memory does not compensate for device hang. When your GPU lockups
> > you might need to re-init GPU from scratch after which the content of the
> > device memory is unreliable. During init the device memory might not get
> > proper clock or proper refresh cycle and thus is susceptible to corruption.
> > 
> >>
> >>>   - application and driver NUMA madvise/mbind/mempolicy ... can conflict
> >>>     with each other and no way the kernel can figure out which should
> >>>     apply
> >>>   - NUMA as it is now would not work as we need further isolation that
> >>>     what a large node distance would provide
> >>>
> >>
> >> Agree, that's where we need spend time on.
> >>
> >> One drawback of HMM-CDM I'm worry about is one more extra copy.
> >> In the cache coherent case, CPU can write data to device memory
> >> directly then start fpga/GPU/other accelerators.
> > 
> > There is not necessarily an extra copy. Device driver can pre-allocate
> > virtual address range of a process with device memory. Device page fault
> 
> Okay, I get your point. But the typical use case is CPU allocate a memory
> and prepare/write data then launch GPU "cuda kernel".

I don't think we should make to many assumption on what is typical case.
GPU compute is fast evolving and they are new domains where it is apply
for instance some folks use it to process network stream and the network
adapter directly write into GPU memory so there is never a CPU copy of
it. So i rather not make any restrictive assumption on how it will be use.

> How to control the allocation go to device memory e.g HBM or system
> DDR at the beginning without user explicit advise? If goes to DDR by
> default, there is an extra copy. If goes to HBM by default, the HBM
> may be waste.

Yes it is a hard problem to solve. We are working with NVidia and IBM
on this and there are several path. But as first solution we will rely
on hint/directive given by userspace program through existing GPGPU API
like CUDA or OpenCL. They are plan to have hardware monitor bus traffic
to gather statistics and do automatic memory placement from thos.


> > can directly allocate device memory. Once allocated CPU access will use
> > the device memory.
> > 
> 
> Then it's more like replace the numa node solution(CDM) with ZONE_DEVICE
> (type MEMORY_DEVICE_PUBLIC). But the problem is the same, e.g how to make
> sure the device memory say HBM won't be occupied by normal CPU allocation.
> Things will be more complex if there are multi GPU connected by nvlink
> (also cache coherent) in a system, each GPU has their own HBM.
>
> How to decide allocate physical memory from local HBM/DDR or remote HBM/
> DDR? 
>
> If using numa(CDM) approach there are NUMA mempolicy and autonuma mechanism
> at least.

NUMA is not as easy as you think. First like i said we want the device
memory to be isolated from most existing mm mechanism. Because memory
is unreliable and also because device might need to be able to evict
memory to make contiguous physical memory allocation for graphics.

Second device driver are not integrated that closely within mm and the
scheduler kernel code to allow to efficiently plug in device access
notification to page (ie to update struct page so that numa worker
thread can migrate memory base on accurate informations).

Third it can be hard to decide who win between CPU and device access
when it comes to updating thing like last CPU id.

Fourth there is no such thing like device id ie equivalent of CPU id.
If we were to add something the CPU id field in flags of struct page
would not be big enough so this can have repercusion on struct page
size. This is not an easy sell.

They are other issues i can't think of right now. I think for now it
is easier and better to take the HMM-CDM approach and latter down the
road once we have more existing user to start thinking about numa or
numa like solution.

Bottom line is we spend time thinking about this and yes numa make
sense from conceptual point of view but they are many things we do
not know to feel confident that we can make something good with numa
as it is.

Cheers,
Jérôme

[toc] | [prev] | [next] | [standalone]

#1693319

On 2017/7/20 23:03, Jerome Glisse wrote:
> On Wed, Jul 19, 2017 at 05:09:04PM +0800, Bob Liu wrote:
>> On 2017/7/19 10:25, Jerome Glisse wrote:
>>> On Wed, Jul 19, 2017 at 09:46:10AM +0800, Bob Liu wrote:
>>>> On 2017/7/18 23:38, Jerome Glisse wrote:
>>>>> On Tue, Jul 18, 2017 at 11:26:51AM +0800, Bob Liu wrote:
>>>>>> On 2017/7/14 5:15, Jérôme Glisse wrote:
>>>>>>> Sorry i made horrible mistake on names in v4, i completly miss-
>>>>>>> understood the suggestion. So here i repost with proper naming.
>>>>>>> This is the only change since v3. Again sorry about the noise
>>>>>>> with v4.
>>>>>>>
>>>>>>> Changes since v4:
>>>>>>>   - s/DEVICE_HOST/DEVICE_PUBLIC
>>>>>>>
>>>>>>> Git tree:
>>>>>>> https://cgit.freedesktop.org/~glisse/linux/log/?h=hmm-cdm-v5
>>>>>>>
>>>>>>>
>>>>>>> Cache coherent device memory apply to architecture with system bus
>>>>>>> like CAPI or CCIX. Device connected to such system bus can expose
>>>>>>> their memory to the system and allow cache coherent access to it
>>>>>>> from the CPU.
>>>>>>>
>>>>>>> Even if for all intent and purposes device memory behave like regular
>>>>>>> memory, we still want to manage it in isolation from regular memory.
>>>>>>> Several reasons for that, first and foremost this memory is less
>>>>>>> reliable than regular memory if the device hangs because of invalid
>>>>>>> commands we can loose access to device memory. Second CPU access to
>>>>>>> this memory is expected to be slower than to regular memory. Third
>>>>>>> having random memory into device means that some of the bus bandwith
>>>>>>> wouldn't be available to the device but would be use by CPU access.
>>>>>>>
>>>>>>> This is why we want to manage such memory in isolation from regular
>>>>>>> memory. Kernel should not try to use this memory even as last resort
>>>>>>> when running out of memory, at least for now.
>>>>>>>
>>>>>>
>>>>>> I think set a very large node distance for "Cache Coherent Device Memory"
>>>>>> may be a easier way to address these concerns.
>>>>>
>>>>> Such approach was discuss at length in the past see links below. Outcome
>>>>> of discussion:
>>>>>   - CPU less node are bad
>>>>>   - device memory can be unreliable (device hang) no way for application
>>>>>     to understand that
>>>>
>>>> Device memory can also be more reliable if using high quality and expensive memory.
>>>
>>> Even ECC memory does not compensate for device hang. When your GPU lockups
>>> you might need to re-init GPU from scratch after which the content of the
>>> device memory is unreliable. During init the device memory might not get
>>> proper clock or proper refresh cycle and thus is susceptible to corruption.
>>>
>>>>
>>>>>   - application and driver NUMA madvise/mbind/mempolicy ... can conflict
>>>>>     with each other and no way the kernel can figure out which should
>>>>>     apply
>>>>>   - NUMA as it is now would not work as we need further isolation that
>>>>>     what a large node distance would provide
>>>>>
>>>>
>>>> Agree, that's where we need spend time on.
>>>>
>>>> One drawback of HMM-CDM I'm worry about is one more extra copy.
>>>> In the cache coherent case, CPU can write data to device memory
>>>> directly then start fpga/GPU/other accelerators.
>>>
>>> There is not necessarily an extra copy. Device driver can pre-allocate
>>> virtual address range of a process with device memory. Device page fault
>>
>> Okay, I get your point. But the typical use case is CPU allocate a memory
>> and prepare/write data then launch GPU "cuda kernel".
> 
> I don't think we should make to many assumption on what is typical case.
> GPU compute is fast evolving and they are new domains where it is apply
> for instance some folks use it to process network stream and the network
> adapter directly write into GPU memory so there is never a CPU copy of
> it. So i rather not make any restrictive assumption on how it will be use.
> 
>> How to control the allocation go to device memory e.g HBM or system
>> DDR at the beginning without user explicit advise? If goes to DDR by
>> default, there is an extra copy. If goes to HBM by default, the HBM
>> may be waste.
> 
> Yes it is a hard problem to solve. We are working with NVidia and IBM
> on this and there are several path. But as first solution we will rely
> on hint/directive given by userspace program through existing GPGPU API
> like CUDA or OpenCL. They are plan to have hardware monitor bus traffic
> to gather statistics and do automatic memory placement from thos.
> 
> 
>>> can directly allocate device memory. Once allocated CPU access will use
>>> the device memory.
>>>
>>
>> Then it's more like replace the numa node solution(CDM) with ZONE_DEVICE
>> (type MEMORY_DEVICE_PUBLIC). But the problem is the same, e.g how to make
>> sure the device memory say HBM won't be occupied by normal CPU allocation.
>> Things will be more complex if there are multi GPU connected by nvlink
>> (also cache coherent) in a system, each GPU has their own HBM.
>>
>> How to decide allocate physical memory from local HBM/DDR or remote HBM/
>> DDR? 
>>
>> If using numa(CDM) approach there are NUMA mempolicy and autonuma mechanism
>> at least.
> 
> NUMA is not as easy as you think. First like i said we want the device
> memory to be isolated from most existing mm mechanism. Because memory
> is unreliable and also because device might need to be able to evict
> memory to make contiguous physical memory allocation for graphics.
> 

Right, but we need isolation any way.
For hmm-cdm, the isolation is not adding device memory to lru list, and many
if (is_device_public_page(page)) ...

But how to evict device memory?

> Second device driver are not integrated that closely within mm and the
> scheduler kernel code to allow to efficiently plug in device access
> notification to page (ie to update struct page so that numa worker
> thread can migrate memory base on accurate informations).
> 
> Third it can be hard to decide who win between CPU and device access
> when it comes to updating thing like last CPU id.
> 
> Fourth there is no such thing like device id ie equivalent of CPU id.
> If we were to add something the CPU id field in flags of struct page
> would not be big enough so this can have repercusion on struct page
> size. This is not an easy sell.
> 
> They are other issues i can't think of right now. I think for now it

My opinion is most of the issues are the same no matter use CDM or HMM-CDM.
I just care about a more complete solution no matter CDM,HMM-CDM or other ways.
HMM or HMM-CDM depends on device driver, but haven't see a public/full driver to 
demonstrate the whole solution works fine.

Cheers,
Bob

> is easier and better to take the HMM-CDM approach and latter down the
> road once we have more existing user to start thinking about numa or
> numa like solution.
> 
> Bottom line is we spend time thinking about this and yes numa make
> sense from conceptual point of view but they are many things we do
> not know to feel confident that we can make something good with numa
> as it is.

[toc] | [prev] | [next] | [standalone]

#1693330

On Fri, Jul 21, 2017 at 09:15:29AM +0800, Bob Liu wrote:
> On 2017/7/20 23:03, Jerome Glisse wrote:
> > On Wed, Jul 19, 2017 at 05:09:04PM +0800, Bob Liu wrote:
> >> On 2017/7/19 10:25, Jerome Glisse wrote:
> >>> On Wed, Jul 19, 2017 at 09:46:10AM +0800, Bob Liu wrote:
> >>>> On 2017/7/18 23:38, Jerome Glisse wrote:
> >>>>> On Tue, Jul 18, 2017 at 11:26:51AM +0800, Bob Liu wrote:
> >>>>>> On 2017/7/14 5:15, Jérôme Glisse wrote:

[...]

> >> Then it's more like replace the numa node solution(CDM) with ZONE_DEVICE
> >> (type MEMORY_DEVICE_PUBLIC). But the problem is the same, e.g how to make
> >> sure the device memory say HBM won't be occupied by normal CPU allocation.
> >> Things will be more complex if there are multi GPU connected by nvlink
> >> (also cache coherent) in a system, each GPU has their own HBM.
> >>
> >> How to decide allocate physical memory from local HBM/DDR or remote HBM/
> >> DDR? 
> >>
> >> If using numa(CDM) approach there are NUMA mempolicy and autonuma mechanism
> >> at least.
> > 
> > NUMA is not as easy as you think. First like i said we want the device
> > memory to be isolated from most existing mm mechanism. Because memory
> > is unreliable and also because device might need to be able to evict
> > memory to make contiguous physical memory allocation for graphics.
> > 
> 
> Right, but we need isolation any way.
> For hmm-cdm, the isolation is not adding device memory to lru list, and many
> if (is_device_public_page(page)) ...
> 
> But how to evict device memory?

What you mean by evict ? Device driver can evict whenever they see the need
to do so. CPU page fault will evict too. Process exit or munmap() will free
the device memory.

Are you refering to evict in the sense of memory reclaim under pressure ?

So the way it flows for memory pressure is that if device driver want to
make room it can evict stuff to system memory and if there is not enough
system memory than thing get reclaim as usual before device driver can
make progress on device memory reclaim.


> > Second device driver are not integrated that closely within mm and the
> > scheduler kernel code to allow to efficiently plug in device access
> > notification to page (ie to update struct page so that numa worker
> > thread can migrate memory base on accurate informations).
> > 
> > Third it can be hard to decide who win between CPU and device access
> > when it comes to updating thing like last CPU id.
> > 
> > Fourth there is no such thing like device id ie equivalent of CPU id.
> > If we were to add something the CPU id field in flags of struct page
> > would not be big enough so this can have repercusion on struct page
> > size. This is not an easy sell.
> > 
> > They are other issues i can't think of right now. I think for now it
> 
> My opinion is most of the issues are the same no matter use CDM or HMM-CDM.
> I just care about a more complete solution no matter CDM,HMM-CDM or other ways.
> HMM or HMM-CDM depends on device driver, but haven't see a public/full driver to 
> demonstrate the whole solution works fine.

I am working with NVidia close source driver team to make sure that it works
well for them. I am also working on nouveau open source driver for same NVidia
hardware thought it will be of less use as what is missing there is a solid
open source userspace to leverage this. Nonetheless open source driver are in
the work.

The way i see it is start with HMM-CDM which isolate most of the changes in
hmm code. Once we get more experience with real workload and not with device
driver test suite then we can start revisiting NUMA and deeper integration
with the linux kernel. I rather grow organicaly toward that than trying to
design something that would make major changes all over the kernel without
knowing for sure that we are going in the right direction. I hope that this
make sense to others too.

Cheers,
Jérôme

[toc] | [prev] | [next] | [standalone]

#1693344

On 2017/7/21 9:41, Jerome Glisse wrote:
> On Fri, Jul 21, 2017 at 09:15:29AM +0800, Bob Liu wrote:
>> On 2017/7/20 23:03, Jerome Glisse wrote:
>>> On Wed, Jul 19, 2017 at 05:09:04PM +0800, Bob Liu wrote:
>>>> On 2017/7/19 10:25, Jerome Glisse wrote:
>>>>> On Wed, Jul 19, 2017 at 09:46:10AM +0800, Bob Liu wrote:
>>>>>> On 2017/7/18 23:38, Jerome Glisse wrote:
>>>>>>> On Tue, Jul 18, 2017 at 11:26:51AM +0800, Bob Liu wrote:
>>>>>>>> On 2017/7/14 5:15, Jérôme Glisse wrote:
> 
> [...]
> 
>>>> Then it's more like replace the numa node solution(CDM) with ZONE_DEVICE
>>>> (type MEMORY_DEVICE_PUBLIC). But the problem is the same, e.g how to make
>>>> sure the device memory say HBM won't be occupied by normal CPU allocation.
>>>> Things will be more complex if there are multi GPU connected by nvlink
>>>> (also cache coherent) in a system, each GPU has their own HBM.
>>>>
>>>> How to decide allocate physical memory from local HBM/DDR or remote HBM/
>>>> DDR? 
>>>>
>>>> If using numa(CDM) approach there are NUMA mempolicy and autonuma mechanism
>>>> at least.
>>>
>>> NUMA is not as easy as you think. First like i said we want the device
>>> memory to be isolated from most existing mm mechanism. Because memory
>>> is unreliable and also because device might need to be able to evict
>>> memory to make contiguous physical memory allocation for graphics.
>>>
>>
>> Right, but we need isolation any way.
>> For hmm-cdm, the isolation is not adding device memory to lru list, and many
>> if (is_device_public_page(page)) ...
>>
>> But how to evict device memory?
> 
> What you mean by evict ? Device driver can evict whenever they see the need
> to do so. CPU page fault will evict too. Process exit or munmap() will free
> the device memory.
> 
> Are you refering to evict in the sense of memory reclaim under pressure ?
> 
> So the way it flows for memory pressure is that if device driver want to
> make room it can evict stuff to system memory and if there is not enough

Yes, I mean this. 
So every driver have to maintain their own LRU-similar list instead of reuse what already in linux kernel.

> system memory than thing get reclaim as usual before device driver can
> make progress on device memory reclaim.
> 
> 
>>> Second device driver are not integrated that closely within mm and the
>>> scheduler kernel code to allow to efficiently plug in device access
>>> notification to page (ie to update struct page so that numa worker
>>> thread can migrate memory base on accurate informations).
>>>
>>> Third it can be hard to decide who win between CPU and device access
>>> when it comes to updating thing like last CPU id.
>>>
>>> Fourth there is no such thing like device id ie equivalent of CPU id.
>>> If we were to add something the CPU id field in flags of struct page
>>> would not be big enough so this can have repercusion on struct page
>>> size. This is not an easy sell.
>>>
>>> They are other issues i can't think of right now. I think for now it
>>
>> My opinion is most of the issues are the same no matter use CDM or HMM-CDM.
>> I just care about a more complete solution no matter CDM,HMM-CDM or other ways.
>> HMM or HMM-CDM depends on device driver, but haven't see a public/full driver to 
>> demonstrate the whole solution works fine.
> 
> I am working with NVidia close source driver team to make sure that it works
> well for them. I am also working on nouveau open source driver for same NVidia
> hardware thought it will be of less use as what is missing there is a solid
> open source userspace to leverage this. Nonetheless open source driver are in
> the work.
> 

Looking forward to see these drivers be public.

> The way i see it is start with HMM-CDM which isolate most of the changes in
> hmm code. Once we get more experience with real workload and not with device
> driver test suite then we can start revisiting NUMA and deeper integration
> with the linux kernel. I rather grow organicaly toward that than trying to
> design something that would make major changes all over the kernel without
> knowing for sure that we are going in the right direction. I hope that this
> make sense to others too.
> 

Make sense.

Thanks,
Bob Liu

[toc] | [prev] | [next] | [standalone]

#1693651

On Fri, Jul 21, 2017 at 10:10 AM, Bob Liu <liubo95@huawei.com> wrote:
> On 2017/7/21 9:41, Jerome Glisse wrote:
>> On Fri, Jul 21, 2017 at 09:15:29AM +0800, Bob Liu wrote:
>>> On 2017/7/20 23:03, Jerome Glisse wrote:
>>>> On Wed, Jul 19, 2017 at 05:09:04PM +0800, Bob Liu wrote:
>>>>> On 2017/7/19 10:25, Jerome Glisse wrote:
>>>>>> On Wed, Jul 19, 2017 at 09:46:10AM +0800, Bob Liu wrote:
>>>>>>> On 2017/7/18 23:38, Jerome Glisse wrote:
>>>>>>>> On Tue, Jul 18, 2017 at 11:26:51AM +0800, Bob Liu wrote:
>>>>>>>>> On 2017/7/14 5:15, Jérôme Glisse wrote:
>>
>> [...]
>>
>>>>> Then it's more like replace the numa node solution(CDM) with ZONE_DEVICE
>>>>> (type MEMORY_DEVICE_PUBLIC). But the problem is the same, e.g how to make
>>>>> sure the device memory say HBM won't be occupied by normal CPU allocation.
>>>>> Things will be more complex if there are multi GPU connected by nvlink
>>>>> (also cache coherent) in a system, each GPU has their own HBM.
>>>>>
>>>>> How to decide allocate physical memory from local HBM/DDR or remote HBM/
>>>>> DDR?
>>>>>
>>>>> If using numa(CDM) approach there are NUMA mempolicy and autonuma mechanism
>>>>> at least.
>>>>
>>>> NUMA is not as easy as you think. First like i said we want the device
>>>> memory to be isolated from most existing mm mechanism. Because memory
>>>> is unreliable and also because device might need to be able to evict
>>>> memory to make contiguous physical memory allocation for graphics.
>>>>
>>>
>>> Right, but we need isolation any way.
>>> For hmm-cdm, the isolation is not adding device memory to lru list, and many
>>> if (is_device_public_page(page)) ...
>>>
>>> But how to evict device memory?
>>
>> What you mean by evict ? Device driver can evict whenever they see the need
>> to do so. CPU page fault will evict too. Process exit or munmap() will free
>> the device memory.
>>
>> Are you refering to evict in the sense of memory reclaim under pressure ?
>>
>> So the way it flows for memory pressure is that if device driver want to
>> make room it can evict stuff to system memory and if there is not enough
>
> Yes, I mean this.
> So every driver have to maintain their own LRU-similar list instead of reuse what already in linux kernel.
>

And how HMM-CDM can handle multiple devices or device with multiple
device memories(may with different properties also)?
This kind of hardware platform would be very common when CCIX is out soon.

Thanks,
Bob Liu



>> system memory than thing get reclaim as usual before device driver can
>> make progress on device memory reclaim.
>>
>>
>>>> Second device driver are not integrated that closely within mm and the
>>>> scheduler kernel code to allow to efficiently plug in device access
>>>> notification to page (ie to update struct page so that numa worker
>>>> thread can migrate memory base on accurate informations).
>>>>
>>>> Third it can be hard to decide who win between CPU and device access
>>>> when it comes to updating thing like last CPU id.
>>>>
>>>> Fourth there is no such thing like device id ie equivalent of CPU id.
>>>> If we were to add something the CPU id field in flags of struct page
>>>> would not be big enough so this can have repercusion on struct page
>>>> size. This is not an easy sell.
>>>>
>>>> They are other issues i can't think of right now. I think for now it
>>>
>>> My opinion is most of the issues are the same no matter use CDM or HMM-CDM.
>>> I just care about a more complete solution no matter CDM,HMM-CDM or other ways.
>>> HMM or HMM-CDM depends on device driver, but haven't see a public/full driver to
>>> demonstrate the whole solution works fine.
>>
>> I am working with NVidia close source driver team to make sure that it works
>> well for them. I am also working on nouveau open source driver for same NVidia
>> hardware thought it will be of less use as what is missing there is a solid
>> open source userspace to leverage this. Nonetheless open source driver are in
>> the work.
>>
>
> Looking forward to see these drivers be public.
>
>> The way i see it is start with HMM-CDM which isolate most of the changes in
>> hmm code. Once we get more experience with real workload and not with device
>> driver test suite then we can start revisiting NUMA and deeper integration
>> with the linux kernel. I rather grow organicaly toward that than trying to
>> design something that would make major changes all over the kernel without
>> knowing for sure that we are going in the right direction. I hope that this
>> make sense to others too.
>>
>
> Make sense.
>
> Thanks,
> Bob Liu
>
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>



-- 
Regards,
--Bob

[toc] | [prev] | [next] | [standalone]

#1693793

On Fri, Jul 21, 2017 at 08:01:07PM +0800, Bob Liu wrote:
> On Fri, Jul 21, 2017 at 10:10 AM, Bob Liu <liubo95@huawei.com> wrote:
> > On 2017/7/21 9:41, Jerome Glisse wrote:
> >> On Fri, Jul 21, 2017 at 09:15:29AM +0800, Bob Liu wrote:
> >>> On 2017/7/20 23:03, Jerome Glisse wrote:
> >>>> On Wed, Jul 19, 2017 at 05:09:04PM +0800, Bob Liu wrote:
> >>>>> On 2017/7/19 10:25, Jerome Glisse wrote:
> >>>>>> On Wed, Jul 19, 2017 at 09:46:10AM +0800, Bob Liu wrote:
> >>>>>>> On 2017/7/18 23:38, Jerome Glisse wrote:
> >>>>>>>> On Tue, Jul 18, 2017 at 11:26:51AM +0800, Bob Liu wrote:
> >>>>>>>>> On 2017/7/14 5:15, Jérôme Glisse wrote:
> >>
> >> [...]
> >>
> >>>>> Then it's more like replace the numa node solution(CDM) with ZONE_DEVICE
> >>>>> (type MEMORY_DEVICE_PUBLIC). But the problem is the same, e.g how to make
> >>>>> sure the device memory say HBM won't be occupied by normal CPU allocation.
> >>>>> Things will be more complex if there are multi GPU connected by nvlink
> >>>>> (also cache coherent) in a system, each GPU has their own HBM.
> >>>>>
> >>>>> How to decide allocate physical memory from local HBM/DDR or remote HBM/
> >>>>> DDR?
> >>>>>
> >>>>> If using numa(CDM) approach there are NUMA mempolicy and autonuma mechanism
> >>>>> at least.
> >>>>
> >>>> NUMA is not as easy as you think. First like i said we want the device
> >>>> memory to be isolated from most existing mm mechanism. Because memory
> >>>> is unreliable and also because device might need to be able to evict
> >>>> memory to make contiguous physical memory allocation for graphics.
> >>>>
> >>>
> >>> Right, but we need isolation any way.
> >>> For hmm-cdm, the isolation is not adding device memory to lru list, and many
> >>> if (is_device_public_page(page)) ...
> >>>
> >>> But how to evict device memory?
> >>
> >> What you mean by evict ? Device driver can evict whenever they see the need
> >> to do so. CPU page fault will evict too. Process exit or munmap() will free
> >> the device memory.
> >>
> >> Are you refering to evict in the sense of memory reclaim under pressure ?
> >>
> >> So the way it flows for memory pressure is that if device driver want to
> >> make room it can evict stuff to system memory and if there is not enough
> >
> > Yes, I mean this.
> > So every driver have to maintain their own LRU-similar list instead of
> > reuse what already in linux kernel.

Regarding LRU it is again not as easy. First we do necessarily have access
information like CPU page table for device page table. Second the mmu_notifier
callback on per page basis is costly. Finaly device are use differently than
CPU, usualy you schedule a job and once that job is done you can safely evict
memory it was using. Existing device driver already have quite large memory
management code of their own because of that different usage model.

LRU might make sense at one point but so far i doubt it is the right solution
for device memory.

> 
> And how HMM-CDM can handle multiple devices or device with multiple
> device memories(may with different properties also)?
> This kind of hardware platform would be very common when CCIX is out soon.

A) Multiple device is under control of device driver. Multiple devices link
to each other through dedicated link can have themself a complex topology and
remote access between device is highly tie to the device (how to program the
device mmu and device registers) and thus to the device driver.

If we identify common design pattern between different hardware then we might
start thinking about factoring out some common code to help those cases.


B) Multiple different device is an harder problem. Each device provide their
own userspace API and that is through that API that you will get memory
placement advise. If several device fight for placement of same chunk of
memory one can argue that the application is broken or device is broken.
But for now we assume that device and application will behave.

Rate limiting migration is hard, you need to keep migration statistics and
that need memory. So unless we really need to do that i would rather avoid
doing that. Again this is a thing for which we will have to wait and see
how thing panout.


Maybe i should stress that HMM is a set of helpers for device memory and it
is not intended to be a policy maker or to manage device memory. Intention
is that device driver will keep managing device memory as they already do
today.

A deeper integration with process memory management is probably bound to
happen but for now it is just about having toolbox for device driver.

Jérôme

[toc] | [prev] | [next] | [standalone]

#1693371

On Thu, Jul 20, 2017 at 6:41 PM, Jerome Glisse <jglisse@redhat.com> wrote:
> On Fri, Jul 21, 2017 at 09:15:29AM +0800, Bob Liu wrote:
>> On 2017/7/20 23:03, Jerome Glisse wrote:
>> > On Wed, Jul 19, 2017 at 05:09:04PM +0800, Bob Liu wrote:
>> >> On 2017/7/19 10:25, Jerome Glisse wrote:
>> >>> On Wed, Jul 19, 2017 at 09:46:10AM +0800, Bob Liu wrote:
>> >>>> On 2017/7/18 23:38, Jerome Glisse wrote:
>> >>>>> On Tue, Jul 18, 2017 at 11:26:51AM +0800, Bob Liu wrote:
>> >>>>>> On 2017/7/14 5:15, Jérôme Glisse wrote:
>
> [...]
>
>> >> Then it's more like replace the numa node solution(CDM) with ZONE_DEVICE
>> >> (type MEMORY_DEVICE_PUBLIC). But the problem is the same, e.g how to make
>> >> sure the device memory say HBM won't be occupied by normal CPU allocation.
>> >> Things will be more complex if there are multi GPU connected by nvlink
>> >> (also cache coherent) in a system, each GPU has their own HBM.
>> >>
>> >> How to decide allocate physical memory from local HBM/DDR or remote HBM/
>> >> DDR?
>> >>
>> >> If using numa(CDM) approach there are NUMA mempolicy and autonuma mechanism
>> >> at least.
>> >
>> > NUMA is not as easy as you think. First like i said we want the device
>> > memory to be isolated from most existing mm mechanism. Because memory
>> > is unreliable and also because device might need to be able to evict
>> > memory to make contiguous physical memory allocation for graphics.
>> >
>>
>> Right, but we need isolation any way.
>> For hmm-cdm, the isolation is not adding device memory to lru list, and many
>> if (is_device_public_page(page)) ...
>>
>> But how to evict device memory?
>
> What you mean by evict ? Device driver can evict whenever they see the need
> to do so. CPU page fault will evict too. Process exit or munmap() will free
> the device memory.
>
> Are you refering to evict in the sense of memory reclaim under pressure ?
>
> So the way it flows for memory pressure is that if device driver want to
> make room it can evict stuff to system memory and if there is not enough
> system memory than thing get reclaim as usual before device driver can
> make progress on device memory reclaim.
>
>
>> > Second device driver are not integrated that closely within mm and the
>> > scheduler kernel code to allow to efficiently plug in device access
>> > notification to page (ie to update struct page so that numa worker
>> > thread can migrate memory base on accurate informations).
>> >
>> > Third it can be hard to decide who win between CPU and device access
>> > when it comes to updating thing like last CPU id.
>> >
>> > Fourth there is no such thing like device id ie equivalent of CPU id.
>> > If we were to add something the CPU id field in flags of struct page
>> > would not be big enough so this can have repercusion on struct page
>> > size. This is not an easy sell.
>> >
>> > They are other issues i can't think of right now. I think for now it
>>
>> My opinion is most of the issues are the same no matter use CDM or HMM-CDM.
>> I just care about a more complete solution no matter CDM,HMM-CDM or other ways.
>> HMM or HMM-CDM depends on device driver, but haven't see a public/full driver to
>> demonstrate the whole solution works fine.
>
> I am working with NVidia close source driver team to make sure that it works
> well for them. I am also working on nouveau open source driver for same NVidia
> hardware thought it will be of less use as what is missing there is a solid
> open source userspace to leverage this. Nonetheless open source driver are in
> the work.

Can you point to the nouveau patches? I still find these HMM patches
un-reviewable without an upstream consumer.

[toc] | [prev] | [next] | [standalone]

#1693800

On Thu, Jul 20, 2017 at 08:48:20PM -0700, Dan Williams wrote:
> On Thu, Jul 20, 2017 at 6:41 PM, Jerome Glisse <jglisse@redhat.com> wrote:
> > On Fri, Jul 21, 2017 at 09:15:29AM +0800, Bob Liu wrote:
> >> On 2017/7/20 23:03, Jerome Glisse wrote:
> >> > On Wed, Jul 19, 2017 at 05:09:04PM +0800, Bob Liu wrote:
> >> >> On 2017/7/19 10:25, Jerome Glisse wrote:
> >> >>> On Wed, Jul 19, 2017 at 09:46:10AM +0800, Bob Liu wrote:
> >> >>>> On 2017/7/18 23:38, Jerome Glisse wrote:
> >> >>>>> On Tue, Jul 18, 2017 at 11:26:51AM +0800, Bob Liu wrote:
> >> >>>>>> On 2017/7/14 5:15, Jérôme Glisse wrote:
> >
> > [...]
> >
> >> >> Then it's more like replace the numa node solution(CDM) with ZONE_DEVICE
> >> >> (type MEMORY_DEVICE_PUBLIC). But the problem is the same, e.g how to make
> >> >> sure the device memory say HBM won't be occupied by normal CPU allocation.
> >> >> Things will be more complex if there are multi GPU connected by nvlink
> >> >> (also cache coherent) in a system, each GPU has their own HBM.
> >> >>
> >> >> How to decide allocate physical memory from local HBM/DDR or remote HBM/
> >> >> DDR?
> >> >>
> >> >> If using numa(CDM) approach there are NUMA mempolicy and autonuma mechanism
> >> >> at least.
> >> >
> >> > NUMA is not as easy as you think. First like i said we want the device
> >> > memory to be isolated from most existing mm mechanism. Because memory
> >> > is unreliable and also because device might need to be able to evict
> >> > memory to make contiguous physical memory allocation for graphics.
> >> >
> >>
> >> Right, but we need isolation any way.
> >> For hmm-cdm, the isolation is not adding device memory to lru list, and many
> >> if (is_device_public_page(page)) ...
> >>
> >> But how to evict device memory?
> >
> > What you mean by evict ? Device driver can evict whenever they see the need
> > to do so. CPU page fault will evict too. Process exit or munmap() will free
> > the device memory.
> >
> > Are you refering to evict in the sense of memory reclaim under pressure ?
> >
> > So the way it flows for memory pressure is that if device driver want to
> > make room it can evict stuff to system memory and if there is not enough
> > system memory than thing get reclaim as usual before device driver can
> > make progress on device memory reclaim.
> >
> >
> >> > Second device driver are not integrated that closely within mm and the
> >> > scheduler kernel code to allow to efficiently plug in device access
> >> > notification to page (ie to update struct page so that numa worker
> >> > thread can migrate memory base on accurate informations).
> >> >
> >> > Third it can be hard to decide who win between CPU and device access
> >> > when it comes to updating thing like last CPU id.
> >> >
> >> > Fourth there is no such thing like device id ie equivalent of CPU id.
> >> > If we were to add something the CPU id field in flags of struct page
> >> > would not be big enough so this can have repercusion on struct page
> >> > size. This is not an easy sell.
> >> >
> >> > They are other issues i can't think of right now. I think for now it
> >>
> >> My opinion is most of the issues are the same no matter use CDM or HMM-CDM.
> >> I just care about a more complete solution no matter CDM,HMM-CDM or other ways.
> >> HMM or HMM-CDM depends on device driver, but haven't see a public/full driver to
> >> demonstrate the whole solution works fine.
> >
> > I am working with NVidia close source driver team to make sure that it works
> > well for them. I am also working on nouveau open source driver for same NVidia
> > hardware thought it will be of less use as what is missing there is a solid
> > open source userspace to leverage this. Nonetheless open source driver are in
> > the work.
> 
> Can you point to the nouveau patches? I still find these HMM patches
> un-reviewable without an upstream consumer.

I am still working on those, i hope i will be able to post them in 3 weeks or so.

Cheers,
Jérôme

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web