Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1730582 > unrolled thread

Re: [PATCH 0/6] Cache coherent device memory (CDM) with HMM v5

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH 0/6] Cache coherent device memory (CDM) with HMM v5 Jerome Glisse <jglisse@redhat.com> - 2017-09-12 01:40 +0200
    Re: [PATCH 0/6] Cache coherent device memory (CDM) with HMM v5 Bob Liu <liubo95@huawei.com> - 2017-09-12 03:20 +0200
      Re: [PATCH 0/6] Cache coherent device memory (CDM) with HMM v5 Jerome Glisse <jglisse@redhat.com> - 2017-09-12 18:20 +0200
    Re: [PATCH 0/6] Cache coherent device memory (CDM) with HMM v5 Bob Liu <lliubbo@gmail.com> - 2017-09-26 12:00 +0200
      Re: [PATCH 0/6] Cache coherent device memory (CDM) with HMM v5 Jerome Glisse <jglisse@redhat.com> - 2017-09-26 18:20 +0200
        Re: [PATCH 0/6] Cache coherent device memory (CDM) with HMM v5 Bob Liu <liubo95@huawei.com> - 2017-09-30 05:00 +0200
          Re: [PATCH 0/6] Cache coherent device memory (CDM) with HMM v5 Jerome Glisse <jglisse@redhat.com> - 2017-10-01 00:50 +0200

#1730582 — Re: [PATCH 0/6] Cache coherent device memory (CDM) with HMM v5

On Sun, Sep 10, 2017 at 07:22:58AM +0800, Bob Liu wrote:
> On Wed, Sep 6, 2017 at 3:36 AM, Jerome Glisse <jglisse@redhat.com> wrote:
> > On Thu, Jul 20, 2017 at 08:48:20PM -0700, Dan Williams wrote:
> >> On Thu, Jul 20, 2017 at 6:41 PM, Jerome Glisse <jglisse@redhat.com> wrote:
> >> > On Fri, Jul 21, 2017 at 09:15:29AM +0800, Bob Liu wrote:
> >> >> On 2017/7/20 23:03, Jerome Glisse wrote:
> >> >> > On Wed, Jul 19, 2017 at 05:09:04PM +0800, Bob Liu wrote:
> >> >> >> On 2017/7/19 10:25, Jerome Glisse wrote:
> >> >> >>> On Wed, Jul 19, 2017 at 09:46:10AM +0800, Bob Liu wrote:
> >> >> >>>> On 2017/7/18 23:38, Jerome Glisse wrote:
> >> >> >>>>> On Tue, Jul 18, 2017 at 11:26:51AM +0800, Bob Liu wrote:
> >> >> >>>>>> On 2017/7/14 5:15, Jérôme Glisse wrote:
> >
> > [...]
> >
> >> >> > Second device driver are not integrated that closely within mm and the
> >> >> > scheduler kernel code to allow to efficiently plug in device access
> >> >> > notification to page (ie to update struct page so that numa worker
> >> >> > thread can migrate memory base on accurate informations).
> >> >> >
> >> >> > Third it can be hard to decide who win between CPU and device access
> >> >> > when it comes to updating thing like last CPU id.
> >> >> >
> >> >> > Fourth there is no such thing like device id ie equivalent of CPU id.
> >> >> > If we were to add something the CPU id field in flags of struct page
> >> >> > would not be big enough so this can have repercusion on struct page
> >> >> > size. This is not an easy sell.
> >> >> >
> >> >> > They are other issues i can't think of right now. I think for now it
> >> >>
> >> >> My opinion is most of the issues are the same no matter use CDM or HMM-CDM.
> >> >> I just care about a more complete solution no matter CDM,HMM-CDM or other ways.
> >> >> HMM or HMM-CDM depends on device driver, but haven't see a public/full driver to
> >> >> demonstrate the whole solution works fine.
> >> >
> >> > I am working with NVidia close source driver team to make sure that it works
> >> > well for them. I am also working on nouveau open source driver for same NVidia
> >> > hardware thought it will be of less use as what is missing there is a solid
> >> > open source userspace to leverage this. Nonetheless open source driver are in
> >> > the work.
> >>
> >> Can you point to the nouveau patches? I still find these HMM patches
> >> un-reviewable without an upstream consumer.
> >
> > So i pushed a branch with WIP for nouveau to use HMM:
> >
> > https://cgit.freedesktop.org/~glisse/linux/log/?h=hmm-nouveau
> >
> 
> Nice to see that.
> Btw, do you have any plan for a CDM-HMM driver? CPU can write to
> Device memory directly without extra copy.

Yes nouveau CDM support on PPC (which is the only CDM platform commercialy
available today) is on the TODO list. Note that the driver changes for CDM
are minimal (probably less than 100 lines of code). From the driver point
of view this is memory and it doesn't matter if it is CDM or not.

The real burden is on the application developpers who need to update their
code to leverage this.


Also as a data point you want to avoid CPU access to CDM device memory as
much as possible. The overhead for single cache line access are high (this
is PCIE or derivative protocol and it is a packet protocol).

Cheers,
Jérôme

[toc] | [next] | [standalone]

#1730600

On 2017/9/12 7:36, Jerome Glisse wrote:
> On Sun, Sep 10, 2017 at 07:22:58AM +0800, Bob Liu wrote:
>> On Wed, Sep 6, 2017 at 3:36 AM, Jerome Glisse <jglisse@redhat.com> wrote:
>>> On Thu, Jul 20, 2017 at 08:48:20PM -0700, Dan Williams wrote:
>>>> On Thu, Jul 20, 2017 at 6:41 PM, Jerome Glisse <jglisse@redhat.com> wrote:
>>>>> On Fri, Jul 21, 2017 at 09:15:29AM +0800, Bob Liu wrote:
>>>>>> On 2017/7/20 23:03, Jerome Glisse wrote:
>>>>>>> On Wed, Jul 19, 2017 at 05:09:04PM +0800, Bob Liu wrote:
>>>>>>>> On 2017/7/19 10:25, Jerome Glisse wrote:
>>>>>>>>> On Wed, Jul 19, 2017 at 09:46:10AM +0800, Bob Liu wrote:
>>>>>>>>>> On 2017/7/18 23:38, Jerome Glisse wrote:
>>>>>>>>>>> On Tue, Jul 18, 2017 at 11:26:51AM +0800, Bob Liu wrote:
>>>>>>>>>>>> On 2017/7/14 5:15, Jérôme Glisse wrote:
>>>
>>> [...]
>>>
>>>>>>> Second device driver are not integrated that closely within mm and the
>>>>>>> scheduler kernel code to allow to efficiently plug in device access
>>>>>>> notification to page (ie to update struct page so that numa worker
>>>>>>> thread can migrate memory base on accurate informations).
>>>>>>>
>>>>>>> Third it can be hard to decide who win between CPU and device access
>>>>>>> when it comes to updating thing like last CPU id.
>>>>>>>
>>>>>>> Fourth there is no such thing like device id ie equivalent of CPU id.
>>>>>>> If we were to add something the CPU id field in flags of struct page
>>>>>>> would not be big enough so this can have repercusion on struct page
>>>>>>> size. This is not an easy sell.
>>>>>>>
>>>>>>> They are other issues i can't think of right now. I think for now it
>>>>>>
>>>>>> My opinion is most of the issues are the same no matter use CDM or HMM-CDM.
>>>>>> I just care about a more complete solution no matter CDM,HMM-CDM or other ways.
>>>>>> HMM or HMM-CDM depends on device driver, but haven't see a public/full driver to
>>>>>> demonstrate the whole solution works fine.
>>>>>
>>>>> I am working with NVidia close source driver team to make sure that it works
>>>>> well for them. I am also working on nouveau open source driver for same NVidia
>>>>> hardware thought it will be of less use as what is missing there is a solid
>>>>> open source userspace to leverage this. Nonetheless open source driver are in
>>>>> the work.
>>>>
>>>> Can you point to the nouveau patches? I still find these HMM patches
>>>> un-reviewable without an upstream consumer.
>>>
>>> So i pushed a branch with WIP for nouveau to use HMM:
>>>
>>> https://cgit.freedesktop.org/~glisse/linux/log/?h=hmm-nouveau
>>>
>>
>> Nice to see that.
>> Btw, do you have any plan for a CDM-HMM driver? CPU can write to
>> Device memory directly without extra copy.
> 
> Yes nouveau CDM support on PPC (which is the only CDM platform commercialy
> available today) is on the TODO list. Note that the driver changes for CDM
> are minimal (probably less than 100 lines of code). From the driver point
> of view this is memory and it doesn't matter if it is CDM or not.
> 
> The real burden is on the application developpers who need to update their
> code to leverage this.
> 

Why it's not transparent to application?
Application just use system malloc() and don't care whether the data is copied or not.

> 
> Also as a data point you want to avoid CPU access to CDM device memory as
> much as possible. The overhead for single cache line access are high (this
> is PCIE or derivative protocol and it is a packet protocol).
> 

Thank you for the hint, we are going to follow cdm-hmm since HMM already merged into upstream.

--
Thanks,
Bob

[toc] | [prev] | [next] | [standalone]

#1730987

On Tue, Sep 12, 2017 at 09:02:19AM +0800, Bob Liu wrote:
> On 2017/9/12 7:36, Jerome Glisse wrote:
> > On Sun, Sep 10, 2017 at 07:22:58AM +0800, Bob Liu wrote:
> >> On Wed, Sep 6, 2017 at 3:36 AM, Jerome Glisse <jglisse@redhat.com> wrote:
> >>> On Thu, Jul 20, 2017 at 08:48:20PM -0700, Dan Williams wrote:
> >>>> On Thu, Jul 20, 2017 at 6:41 PM, Jerome Glisse <jglisse@redhat.com> wrote:
> >>>>> On Fri, Jul 21, 2017 at 09:15:29AM +0800, Bob Liu wrote:
> >>>>>> On 2017/7/20 23:03, Jerome Glisse wrote:
> >>>>>>> On Wed, Jul 19, 2017 at 05:09:04PM +0800, Bob Liu wrote:
> >>>>>>>> On 2017/7/19 10:25, Jerome Glisse wrote:
> >>>>>>>>> On Wed, Jul 19, 2017 at 09:46:10AM +0800, Bob Liu wrote:
> >>>>>>>>>> On 2017/7/18 23:38, Jerome Glisse wrote:
> >>>>>>>>>>> On Tue, Jul 18, 2017 at 11:26:51AM +0800, Bob Liu wrote:
> >>>>>>>>>>>> On 2017/7/14 5:15, Jérôme Glisse wrote:
> >>>
> >>> [...]
> >>>
> >>>>>>> Second device driver are not integrated that closely within mm and the
> >>>>>>> scheduler kernel code to allow to efficiently plug in device access
> >>>>>>> notification to page (ie to update struct page so that numa worker
> >>>>>>> thread can migrate memory base on accurate informations).
> >>>>>>>
> >>>>>>> Third it can be hard to decide who win between CPU and device access
> >>>>>>> when it comes to updating thing like last CPU id.
> >>>>>>>
> >>>>>>> Fourth there is no such thing like device id ie equivalent of CPU id.
> >>>>>>> If we were to add something the CPU id field in flags of struct page
> >>>>>>> would not be big enough so this can have repercusion on struct page
> >>>>>>> size. This is not an easy sell.
> >>>>>>>
> >>>>>>> They are other issues i can't think of right now. I think for now it
> >>>>>>
> >>>>>> My opinion is most of the issues are the same no matter use CDM or HMM-CDM.
> >>>>>> I just care about a more complete solution no matter CDM,HMM-CDM or other ways.
> >>>>>> HMM or HMM-CDM depends on device driver, but haven't see a public/full driver to
> >>>>>> demonstrate the whole solution works fine.
> >>>>>
> >>>>> I am working with NVidia close source driver team to make sure that it works
> >>>>> well for them. I am also working on nouveau open source driver for same NVidia
> >>>>> hardware thought it will be of less use as what is missing there is a solid
> >>>>> open source userspace to leverage this. Nonetheless open source driver are in
> >>>>> the work.
> >>>>
> >>>> Can you point to the nouveau patches? I still find these HMM patches
> >>>> un-reviewable without an upstream consumer.
> >>>
> >>> So i pushed a branch with WIP for nouveau to use HMM:
> >>>
> >>> https://cgit.freedesktop.org/~glisse/linux/log/?h=hmm-nouveau
> >>>
> >>
> >> Nice to see that.
> >> Btw, do you have any plan for a CDM-HMM driver? CPU can write to
> >> Device memory directly without extra copy.
> > 
> > Yes nouveau CDM support on PPC (which is the only CDM platform commercialy
> > available today) is on the TODO list. Note that the driver changes for CDM
> > are minimal (probably less than 100 lines of code). From the driver point
> > of view this is memory and it doesn't matter if it is CDM or not.
> > 
> > The real burden is on the application developpers who need to update their
> > code to leverage this.
> > 
> 
> Why it's not transparent to application?
> Application just use system malloc() and don't care whether the data is copied or not.

Porting today software to malloc/mmap is easy and apply to both non CDM and
CDM hardware.

So malloc/mmap is a given what i mean is that having CPU capable of doing
cache coherent access to device memory is a new thing. It never existed before
and thus no one ever though of how to take advantages of that ie there is no
existing program designed with that in mind.

Cheers,
Jérôme

[toc] | [prev] | [next] | [standalone]

#1739751

On Tue, Sep 12, 2017 at 7:36 AM, Jerome Glisse <jglisse@redhat.com> wrote:
> On Sun, Sep 10, 2017 at 07:22:58AM +0800, Bob Liu wrote:
>> On Wed, Sep 6, 2017 at 3:36 AM, Jerome Glisse <jglisse@redhat.com> wrote:
>> > On Thu, Jul 20, 2017 at 08:48:20PM -0700, Dan Williams wrote:
>> >> On Thu, Jul 20, 2017 at 6:41 PM, Jerome Glisse <jglisse@redhat.com> wrote:
>> >> > On Fri, Jul 21, 2017 at 09:15:29AM +0800, Bob Liu wrote:
>> >> >> On 2017/7/20 23:03, Jerome Glisse wrote:
>> >> >> > On Wed, Jul 19, 2017 at 05:09:04PM +0800, Bob Liu wrote:
>> >> >> >> On 2017/7/19 10:25, Jerome Glisse wrote:
>> >> >> >>> On Wed, Jul 19, 2017 at 09:46:10AM +0800, Bob Liu wrote:
>> >> >> >>>> On 2017/7/18 23:38, Jerome Glisse wrote:
>> >> >> >>>>> On Tue, Jul 18, 2017 at 11:26:51AM +0800, Bob Liu wrote:
>> >> >> >>>>>> On 2017/7/14 5:15, Jérôme Glisse wrote:
>> >
>> > [...]
>> >
>> >> >> > Second device driver are not integrated that closely within mm and the
>> >> >> > scheduler kernel code to allow to efficiently plug in device access
>> >> >> > notification to page (ie to update struct page so that numa worker
>> >> >> > thread can migrate memory base on accurate informations).
>> >> >> >
>> >> >> > Third it can be hard to decide who win between CPU and device access
>> >> >> > when it comes to updating thing like last CPU id.
>> >> >> >
>> >> >> > Fourth there is no such thing like device id ie equivalent of CPU id.
>> >> >> > If we were to add something the CPU id field in flags of struct page
>> >> >> > would not be big enough so this can have repercusion on struct page
>> >> >> > size. This is not an easy sell.
>> >> >> >
>> >> >> > They are other issues i can't think of right now. I think for now it
>> >> >>
>> >> >> My opinion is most of the issues are the same no matter use CDM or HMM-CDM.
>> >> >> I just care about a more complete solution no matter CDM,HMM-CDM or other ways.
>> >> >> HMM or HMM-CDM depends on device driver, but haven't see a public/full driver to
>> >> >> demonstrate the whole solution works fine.
>> >> >
>> >> > I am working with NVidia close source driver team to make sure that it works
>> >> > well for them. I am also working on nouveau open source driver for same NVidia
>> >> > hardware thought it will be of less use as what is missing there is a solid
>> >> > open source userspace to leverage this. Nonetheless open source driver are in
>> >> > the work.
>> >>
>> >> Can you point to the nouveau patches? I still find these HMM patches
>> >> un-reviewable without an upstream consumer.
>> >
>> > So i pushed a branch with WIP for nouveau to use HMM:
>> >
>> > https://cgit.freedesktop.org/~glisse/linux/log/?h=hmm-nouveau
>> >
>>
>> Nice to see that.
>> Btw, do you have any plan for a CDM-HMM driver? CPU can write to
>> Device memory directly without extra copy.
>
> Yes nouveau CDM support on PPC (which is the only CDM platform commercialy
> available today) is on the TODO list. Note that the driver changes for CDM
> are minimal (probably less than 100 lines of code). From the driver point
> of view this is memory and it doesn't matter if it is CDM or not.
>

It seems have to migrate/copy memory between system-memory and
device-memory even in HMM-CDM solution.
Because device-memory is not added into buddy system, the page fault
for normal malloc() always allocate memory from system-memory!!
If the device then access the same virtual address, the data is copied
to device-memory.

Correct me if I misunderstand something.
@Balbir, how do you plan to make zero-copy work if using HMM-CDM?

--
Thanks,
Bob

[toc] | [prev] | [next] | [standalone]

#1740005

On Tue, Sep 26, 2017 at 05:56:26PM +0800, Bob Liu wrote:
> On Tue, Sep 12, 2017 at 7:36 AM, Jerome Glisse <jglisse@redhat.com> wrote:
> > On Sun, Sep 10, 2017 at 07:22:58AM +0800, Bob Liu wrote:
> >> On Wed, Sep 6, 2017 at 3:36 AM, Jerome Glisse <jglisse@redhat.com> wrote:
> >> > On Thu, Jul 20, 2017 at 08:48:20PM -0700, Dan Williams wrote:
> >> >> On Thu, Jul 20, 2017 at 6:41 PM, Jerome Glisse <jglisse@redhat.com> wrote:
> >> >> > On Fri, Jul 21, 2017 at 09:15:29AM +0800, Bob Liu wrote:
> >> >> >> On 2017/7/20 23:03, Jerome Glisse wrote:
> >> >> >> > On Wed, Jul 19, 2017 at 05:09:04PM +0800, Bob Liu wrote:
> >> >> >> >> On 2017/7/19 10:25, Jerome Glisse wrote:
> >> >> >> >>> On Wed, Jul 19, 2017 at 09:46:10AM +0800, Bob Liu wrote:
> >> >> >> >>>> On 2017/7/18 23:38, Jerome Glisse wrote:
> >> >> >> >>>>> On Tue, Jul 18, 2017 at 11:26:51AM +0800, Bob Liu wrote:
> >> >> >> >>>>>> On 2017/7/14 5:15, Jérôme Glisse wrote:
> >> >
> >> > [...]
> >> >
> >> >> >> > Second device driver are not integrated that closely within mm and the
> >> >> >> > scheduler kernel code to allow to efficiently plug in device access
> >> >> >> > notification to page (ie to update struct page so that numa worker
> >> >> >> > thread can migrate memory base on accurate informations).
> >> >> >> >
> >> >> >> > Third it can be hard to decide who win between CPU and device access
> >> >> >> > when it comes to updating thing like last CPU id.
> >> >> >> >
> >> >> >> > Fourth there is no such thing like device id ie equivalent of CPU id.
> >> >> >> > If we were to add something the CPU id field in flags of struct page
> >> >> >> > would not be big enough so this can have repercusion on struct page
> >> >> >> > size. This is not an easy sell.
> >> >> >> >
> >> >> >> > They are other issues i can't think of right now. I think for now it
> >> >> >>
> >> >> >> My opinion is most of the issues are the same no matter use CDM or HMM-CDM.
> >> >> >> I just care about a more complete solution no matter CDM,HMM-CDM or other ways.
> >> >> >> HMM or HMM-CDM depends on device driver, but haven't see a public/full driver to
> >> >> >> demonstrate the whole solution works fine.
> >> >> >
> >> >> > I am working with NVidia close source driver team to make sure that it works
> >> >> > well for them. I am also working on nouveau open source driver for same NVidia
> >> >> > hardware thought it will be of less use as what is missing there is a solid
> >> >> > open source userspace to leverage this. Nonetheless open source driver are in
> >> >> > the work.
> >> >>
> >> >> Can you point to the nouveau patches? I still find these HMM patches
> >> >> un-reviewable without an upstream consumer.
> >> >
> >> > So i pushed a branch with WIP for nouveau to use HMM:
> >> >
> >> > https://cgit.freedesktop.org/~glisse/linux/log/?h=hmm-nouveau
> >> >
> >>
> >> Nice to see that.
> >> Btw, do you have any plan for a CDM-HMM driver? CPU can write to
> >> Device memory directly without extra copy.
> >
> > Yes nouveau CDM support on PPC (which is the only CDM platform commercialy
> > available today) is on the TODO list. Note that the driver changes for CDM
> > are minimal (probably less than 100 lines of code). From the driver point
> > of view this is memory and it doesn't matter if it is CDM or not.
> >
> 
> It seems have to migrate/copy memory between system-memory and
> device-memory even in HMM-CDM solution.
> Because device-memory is not added into buddy system, the page fault
> for normal malloc() always allocate memory from system-memory!!
> If the device then access the same virtual address, the data is copied
> to device-memory.
> 
> Correct me if I misunderstand something.
> @Balbir, how do you plan to make zero-copy work if using HMM-CDM?

Device can access system memory so copy to device is _not_ mandatory. Copying
data to device is for performance only ie the device driver take hint from
userspace and monitor device activity to decide which memory should be migrated
to device memory to maximize performance.

Moreover in some previous version of the HMM patchset we had an helper that
allowed to directly allocate device memory on device page fault. I intend to
post this helper again. With that helper you can have zero copy when device
is the first to access the memory.

Plan is to get what we have today work properly with the open source driver
and make it perform well. Once we get some experience with real workload we
might look into allowing CPU page fault to be directed to device memory but
at this time i don't think we need this.

Cheers,
Jérôme

[toc] | [prev] | [next] | [standalone]

#1742463

On 2017/9/27 0:16, Jerome Glisse wrote:
> On Tue, Sep 26, 2017 at 05:56:26PM +0800, Bob Liu wrote:
>> On Tue, Sep 12, 2017 at 7:36 AM, Jerome Glisse <jglisse@redhat.com> wrote:
>>> On Sun, Sep 10, 2017 at 07:22:58AM +0800, Bob Liu wrote:
>>>> On Wed, Sep 6, 2017 at 3:36 AM, Jerome Glisse <jglisse@redhat.com> wrote:
>>>>> On Thu, Jul 20, 2017 at 08:48:20PM -0700, Dan Williams wrote:
>>>>>> On Thu, Jul 20, 2017 at 6:41 PM, Jerome Glisse <jglisse@redhat.com> wrote:
[...]
>>>>> So i pushed a branch with WIP for nouveau to use HMM:
>>>>>
>>>>> https://cgit.freedesktop.org/~glisse/linux/log/?h=hmm-nouveau
>>>>>
>>>>
>>>> Nice to see that.
>>>> Btw, do you have any plan for a CDM-HMM driver? CPU can write to
>>>> Device memory directly without extra copy.
>>>
>>> Yes nouveau CDM support on PPC (which is the only CDM platform commercialy
>>> available today) is on the TODO list. Note that the driver changes for CDM
>>> are minimal (probably less than 100 lines of code). From the driver point
>>> of view this is memory and it doesn't matter if it is CDM or not.
>>>
>>
>> It seems have to migrate/copy memory between system-memory and
>> device-memory even in HMM-CDM solution.
>> Because device-memory is not added into buddy system, the page fault
>> for normal malloc() always allocate memory from system-memory!!
>> If the device then access the same virtual address, the data is copied
>> to device-memory.
>>
>> Correct me if I misunderstand something.
>> @Balbir, how do you plan to make zero-copy work if using HMM-CDM?
> 
> Device can access system memory so copy to device is _not_ mandatory. Copying
> data to device is for performance only ie the device driver take hint from
> userspace and monitor device activity to decide which memory should be migrated
> to device memory to maximize performance.
> 
> Moreover in some previous version of the HMM patchset we had an helper that

Could you point in which version? I'd like to have a look.

> allowed to directly allocate device memory on device page fault. I intend to
> post this helper again. With that helper you can have zero copy when device
> is the first to access the memory.
> 
> Plan is to get what we have today work properly with the open source driver
> and make it perform well. Once we get some experience with real workload we
> might look into allowing CPU page fault to be directed to device memory but
> at this time i don't think we need this.
> 

For us, we need this feature that CPU page fault can be direct to device memory.
So that don't need to copy data from system memory to device memory.
Do you have any suggestion on the implementation? I'll try to make a prototype patch.

--
Thanks,
Bob

[toc] | [prev] | [next] | [standalone]

#1742701

On Sat, Sep 30, 2017 at 10:57:38AM +0800, Bob Liu wrote:
> On 2017/9/27 0:16, Jerome Glisse wrote:
> > On Tue, Sep 26, 2017 at 05:56:26PM +0800, Bob Liu wrote:
> >> On Tue, Sep 12, 2017 at 7:36 AM, Jerome Glisse <jglisse@redhat.com> wrote:
> >>> On Sun, Sep 10, 2017 at 07:22:58AM +0800, Bob Liu wrote:
> >>>> On Wed, Sep 6, 2017 at 3:36 AM, Jerome Glisse <jglisse@redhat.com> wrote:
> >>>>> On Thu, Jul 20, 2017 at 08:48:20PM -0700, Dan Williams wrote:
> >>>>>> On Thu, Jul 20, 2017 at 6:41 PM, Jerome Glisse <jglisse@redhat.com> wrote:
> [...]
> >>>>> So i pushed a branch with WIP for nouveau to use HMM:
> >>>>>
> >>>>> https://cgit.freedesktop.org/~glisse/linux/log/?h=hmm-nouveau
> >>>>>
> >>>>
> >>>> Nice to see that.
> >>>> Btw, do you have any plan for a CDM-HMM driver? CPU can write to
> >>>> Device memory directly without extra copy.
> >>>
> >>> Yes nouveau CDM support on PPC (which is the only CDM platform commercialy
> >>> available today) is on the TODO list. Note that the driver changes for CDM
> >>> are minimal (probably less than 100 lines of code). From the driver point
> >>> of view this is memory and it doesn't matter if it is CDM or not.
> >>>
> >>
> >> It seems have to migrate/copy memory between system-memory and
> >> device-memory even in HMM-CDM solution.
> >> Because device-memory is not added into buddy system, the page fault
> >> for normal malloc() always allocate memory from system-memory!!
> >> If the device then access the same virtual address, the data is copied
> >> to device-memory.
> >>
> >> Correct me if I misunderstand something.
> >> @Balbir, how do you plan to make zero-copy work if using HMM-CDM?
> > 
> > Device can access system memory so copy to device is _not_ mandatory. Copying
> > data to device is for performance only ie the device driver take hint from
> > userspace and monitor device activity to decide which memory should be migrated
> > to device memory to maximize performance.
> > 
> > Moreover in some previous version of the HMM patchset we had an helper that
> 
> Could you point in which version? I'd like to have a look.

I will need to dig in.

> 
> > allowed to directly allocate device memory on device page fault. I intend to
> > post this helper again. With that helper you can have zero copy when device
> > is the first to access the memory.
> > 
> > Plan is to get what we have today work properly with the open source driver
> > and make it perform well. Once we get some experience with real workload we
> > might look into allowing CPU page fault to be directed to device memory but
> > at this time i don't think we need this.
> > 
> 
> For us, we need this feature that CPU page fault can be direct to device memory.
> So that don't need to copy data from system memory to device memory.
> Do you have any suggestion on the implementation? I'll try to make a prototype patch.

Why do you need that ? What is the device and what are the requirement ?

Jérôme

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web