Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1500094 > unrolled thread

Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support  for Xen "Jan Beulich" <JBeulich@suse.com> - 2016-10-13 10:40 +0200
    Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for  Xen Haozhong Zhang <haozhong.zhang@intel.com> - 2016-10-13 11:00 +0200
      Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for  Xen Haozhong Zhang <haozhong.zhang@intel.com> - 2016-10-13 11:10 +0200
      Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support  for Xen "Jan Beulich" <JBeulich@suse.com> - 2016-10-13 11:10 +0200
        Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for  Xen Haozhong Zhang <haozhong.zhang@intel.com> - 2016-10-13 17:50 +0200
          Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support  for Xen "Jan Beulich" <JBeulich@suse.com> - 2016-10-14 12:20 +0200
            Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for  Xen Haozhong Zhang <haozhong.zhang@intel.com> - 2016-10-20 11:20 +0200
        Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Dan Williams <dan.j.williams@intel.com> - 2016-10-13 17:50 +0200
          Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for  Xen Andrew Cooper <andrew.cooper3@citrix.com> - 2016-10-13 18:10 +0200
            Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Dan Williams <dan.j.williams@intel.com> - 2016-10-13 21:10 +0200
              Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for  Xen Andrew Cooper <andrew.cooper3@citrix.com> - 2016-10-13 21:40 +0200
                Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for  Xen Haozhong Zhang <haozhong.zhang@intel.com> - 2016-10-14 09:10 +0200
                  Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for  Xen Andrew Cooper <andrew.cooper3@citrix.com> - 2016-10-14 14:20 +0200
                    Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for  Xen Haozhong Zhang <haozhong.zhang@intel.com> - 2016-10-20 11:20 +0200
                      Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for  Xen Andrew Cooper <andrew.cooper3@citrix.com> - 2016-10-21 00:10 +0200
          Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support  for Xen "Jan Beulich" <JBeulich@suse.com> - 2016-10-14 12:10 +0200

#1500094 — Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen

>>> On 12.10.16 at 18:19, <dan.j.williams@intel.com> wrote:
> On Wed, Oct 12, 2016 at 9:01 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>> On 12.10.16 at 17:42, <dan.j.williams@intel.com> wrote:
>>> On Wed, Oct 12, 2016 at 8:39 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>>> On 12.10.16 at 16:58, <haozhong.zhang@intel.com> wrote:
>>>>> On 10/12/16 05:32 -0600, Jan Beulich wrote:
>>>>>>>>> On 12.10.16 at 12:33, <haozhong.zhang@intel.com> wrote:
>>>>>>> The layout is shown as the following diagram.
>>>>>>>
>>>>>>> +---------------+-----------+-------+----------+--------------+
>>>>>>> | whatever used | Partition | Super | Reserved | /dev/pmem0p1 |
>>>>>>> |  by kernel    |   Table   | Block | for Xen  |              |
>>>>>>> +---------------+-----------+-------+----------+--------------+
>>>>>>>                 \_____________________ _______________________/
>>>>>>>                                   V
>>>>>>>                              /dev/pmem0
>>>>>>
>>>>>>I have to admit that I dislike this, for not being OS-agnostic.
>>>>>>Neither should there be any Xen-specific region, nor should the
>>>>>>"whatever used by kernel" one be restricted to just Linux. What
>>>>>>I could see is an OS-reserved area ahead of the partition table,
>>>>>>the exact usage of which depends on which OS is currently
>>>>>>running (and in the Xen case this might be both Xen _and_ the
>>>>>>Dom0 kernel, arbitrated by a tbd protocol). After all, when
>>>>>>running under Xen, the Dom0 may not have a need for as much
>>>>>>control data as it has when running on bare hardware, for it
>>>>>>controlling less (if any) of the actual memory ranges when Xen
>>>>>>is present.
>>>>>>
>>>>>
>>>>> Isn't this OS-reserved area still not OS-agnostic, as it requires OS
>>>>> to know where the reserved area is?  Or do you mean it's not if it's
>>>>> defined by a protocol that is accepted by all OSes?
>>>>
>>>> The latter - we clearly won't get away without some agreement on
>>>> where to retrieve position and size of this area. I was simply
>>>> assuming that such a protocol already exists.
>>>>
>>>
>>> No, we should not mix the struct page reservation that the Dom0 kernel
>>> may actively use with the Xen reservation that the Dom0 kernel does
>>> not consume.  Explain again what is wrong with the partition approach?
>>
>> Not sure what was unclear in my previous reply. I don't think there
>> should be apriori knowledge of whether Xen is (going to be) used on
>> a system, and even if it gets used, but just occasionally, it would
>> (apart from the abstract considerations already given) be a waste
>> of resources to set something aside that could be used for other
>> purposes while Xen is not running. Static partitioning should only be
>> needed for persistent data.
> 
> The reservation needs to be persistent / static even if the data is
> volatile, as is the case with struct page, because we can't have the
> size of the device change depending on use.  So, from the aspect of
> wasting space while Xen is not in use, both partitions and the
> intrinsic reservation approach suffer the same problem. Setting that
> aside I don't want to mix 2 different use cases into the same
> reservation.

Then you didn't understand what I've said: I certainly didn't mean
the reservation to vary from a device perspective. However, when
Xen is in use I don't see why part of that static reservation couldn't
be used by Xen, and another part by the Dom0 kernel. The kernel
obviously would need to ask the hypervisor how much of the space
is left, and where that area starts.

> The kernel needs to know about the struct page reservation because it
> needs to manage the lifetime of page references vs the lifetime of the
> device.  It does not have the same relationship with a Xen reservation
> which is why I'm proposing they be managed separately.

I don't think I understand the difference you try to point out here.
Linux'es struct page and Xen's struct page_info serve the same
fundamental purpose.

Jan

[toc] | [next] | [standalone]

#1500103 — Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen

On 10/13/16 02:34 -0600, Jan Beulich wrote:
>>>> On 12.10.16 at 18:19, <dan.j.williams@intel.com> wrote:
>> On Wed, Oct 12, 2016 at 9:01 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>> On 12.10.16 at 17:42, <dan.j.williams@intel.com> wrote:
>>>> On Wed, Oct 12, 2016 at 8:39 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>>>> On 12.10.16 at 16:58, <haozhong.zhang@intel.com> wrote:
>>>>>> On 10/12/16 05:32 -0600, Jan Beulich wrote:
>>>>>>>>>> On 12.10.16 at 12:33, <haozhong.zhang@intel.com> wrote:
>>>>>>>> The layout is shown as the following diagram.
>>>>>>>>
>>>>>>>> +---------------+-----------+-------+----------+--------------+
>>>>>>>> | whatever used | Partition | Super | Reserved | /dev/pmem0p1 |
>>>>>>>> |  by kernel    |   Table   | Block | for Xen  |              |
>>>>>>>> +---------------+-----------+-------+----------+--------------+
>>>>>>>>                 \_____________________ _______________________/
>>>>>>>>                                   V
>>>>>>>>                              /dev/pmem0
>>>>>>>
>>>>>>>I have to admit that I dislike this, for not being OS-agnostic.
>>>>>>>Neither should there be any Xen-specific region, nor should the
>>>>>>>"whatever used by kernel" one be restricted to just Linux. What
>>>>>>>I could see is an OS-reserved area ahead of the partition table,
>>>>>>>the exact usage of which depends on which OS is currently
>>>>>>>running (and in the Xen case this might be both Xen _and_ the
>>>>>>>Dom0 kernel, arbitrated by a tbd protocol). After all, when
>>>>>>>running under Xen, the Dom0 may not have a need for as much
>>>>>>>control data as it has when running on bare hardware, for it
>>>>>>>controlling less (if any) of the actual memory ranges when Xen
>>>>>>>is present.
>>>>>>>
>>>>>>
>>>>>> Isn't this OS-reserved area still not OS-agnostic, as it requires OS
>>>>>> to know where the reserved area is?  Or do you mean it's not if it's
>>>>>> defined by a protocol that is accepted by all OSes?
>>>>>
>>>>> The latter - we clearly won't get away without some agreement on
>>>>> where to retrieve position and size of this area. I was simply
>>>>> assuming that such a protocol already exists.
>>>>>
>>>>
>>>> No, we should not mix the struct page reservation that the Dom0 kernel
>>>> may actively use with the Xen reservation that the Dom0 kernel does
>>>> not consume.  Explain again what is wrong with the partition approach?
>>>
>>> Not sure what was unclear in my previous reply. I don't think there
>>> should be apriori knowledge of whether Xen is (going to be) used on
>>> a system, and even if it gets used, but just occasionally, it would
>>> (apart from the abstract considerations already given) be a waste
>>> of resources to set something aside that could be used for other
>>> purposes while Xen is not running. Static partitioning should only be
>>> needed for persistent data.
>>
>> The reservation needs to be persistent / static even if the data is
>> volatile, as is the case with struct page, because we can't have the
>> size of the device change depending on use.  So, from the aspect of
>> wasting space while Xen is not in use, both partitions and the
>> intrinsic reservation approach suffer the same problem. Setting that
>> aside I don't want to mix 2 different use cases into the same
>> reservation.
>
>Then you didn't understand what I've said: I certainly didn't mean
>the reservation to vary from a device perspective. However, when
>Xen is in use I don't see why part of that static reservation couldn't
>be used by Xen, and another part by the Dom0 kernel. The kernel
>obviously would need to ask the hypervisor how much of the space
>is left, and where that area starts.
>

I think Dan means that there should be a clear separation between
reservations for different usages (kernel/xen/...). The libnvdimm
driver is for the linux kernel and only needs to maintain the
reservation for kernel functionality. For others including xen/dm/...,
if they want reservation for their own purpose, they should maintain
their own reservations out of libnvdimm driver and avoid bothering the
libnvdimm driver (e.g. add specific handling in libnvdimm driver).

IIUC, one existing example is device-mapper device (dm) which needs to
reserve on-device area for its own meta-data. Its choice is to store
the meta-data on the block device (/dev/pmemN) provided by the
libnvdimm driver.

I think we can do the similar for Xen, like to lay another pseudo
device on /dev/pmem and do the reservation, like 2. in my previous
reply.

Thanks,
Haozhong

>> The kernel needs to know about the struct page reservation because it
>> needs to manage the lifetime of page references vs the lifetime of the
>> device.  It does not have the same relationship with a Xen reservation
>> which is why I'm proposing they be managed separately.
>
>I don't think I understand the difference you try to point out here.
>Linux'es struct page and Xen's struct page_info serve the same
>fundamental purpose.
>
>Jan
>

[toc] | [prev] | [next] | [standalone]

#1500113 — Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen

+Dan Williams

I accidentally dropped him in my last reply. Add him back.

On 10/13/16 16:53 +0800, Haozhong Zhang wrote:
>On 10/13/16 02:34 -0600, Jan Beulich wrote:
>>>>>On 12.10.16 at 18:19, <dan.j.williams@intel.com> wrote:
>>>On Wed, Oct 12, 2016 at 9:01 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>>>On 12.10.16 at 17:42, <dan.j.williams@intel.com> wrote:
>>>>>On Wed, Oct 12, 2016 at 8:39 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>>>>>On 12.10.16 at 16:58, <haozhong.zhang@intel.com> wrote:
>>>>>>>On 10/12/16 05:32 -0600, Jan Beulich wrote:
>>>>>>>>>>>On 12.10.16 at 12:33, <haozhong.zhang@intel.com> wrote:
>>>>>>>>>The layout is shown as the following diagram.
>>>>>>>>>
>>>>>>>>>+---------------+-----------+-------+----------+--------------+
>>>>>>>>>| whatever used | Partition | Super | Reserved | /dev/pmem0p1 |
>>>>>>>>>|  by kernel    |   Table   | Block | for Xen  |              |
>>>>>>>>>+---------------+-----------+-------+----------+--------------+
>>>>>>>>>                \_____________________ _______________________/
>>>>>>>>>                                  V
>>>>>>>>>                             /dev/pmem0
>>>>>>>>
>>>>>>>>I have to admit that I dislike this, for not being OS-agnostic.
>>>>>>>>Neither should there be any Xen-specific region, nor should the
>>>>>>>>"whatever used by kernel" one be restricted to just Linux. What
>>>>>>>>I could see is an OS-reserved area ahead of the partition table,
>>>>>>>>the exact usage of which depends on which OS is currently
>>>>>>>>running (and in the Xen case this might be both Xen _and_ the
>>>>>>>>Dom0 kernel, arbitrated by a tbd protocol). After all, when
>>>>>>>>running under Xen, the Dom0 may not have a need for as much
>>>>>>>>control data as it has when running on bare hardware, for it
>>>>>>>>controlling less (if any) of the actual memory ranges when Xen
>>>>>>>>is present.
>>>>>>>>
>>>>>>>
>>>>>>>Isn't this OS-reserved area still not OS-agnostic, as it requires OS
>>>>>>>to know where the reserved area is?  Or do you mean it's not if it's
>>>>>>>defined by a protocol that is accepted by all OSes?
>>>>>>
>>>>>>The latter - we clearly won't get away without some agreement on
>>>>>>where to retrieve position and size of this area. I was simply
>>>>>>assuming that such a protocol already exists.
>>>>>>
>>>>>
>>>>>No, we should not mix the struct page reservation that the Dom0 kernel
>>>>>may actively use with the Xen reservation that the Dom0 kernel does
>>>>>not consume.  Explain again what is wrong with the partition approach?
>>>>
>>>>Not sure what was unclear in my previous reply. I don't think there
>>>>should be apriori knowledge of whether Xen is (going to be) used on
>>>>a system, and even if it gets used, but just occasionally, it would
>>>>(apart from the abstract considerations already given) be a waste
>>>>of resources to set something aside that could be used for other
>>>>purposes while Xen is not running. Static partitioning should only be
>>>>needed for persistent data.
>>>
>>>The reservation needs to be persistent / static even if the data is
>>>volatile, as is the case with struct page, because we can't have the
>>>size of the device change depending on use.  So, from the aspect of
>>>wasting space while Xen is not in use, both partitions and the
>>>intrinsic reservation approach suffer the same problem. Setting that
>>>aside I don't want to mix 2 different use cases into the same
>>>reservation.
>>
>>Then you didn't understand what I've said: I certainly didn't mean
>>the reservation to vary from a device perspective. However, when
>>Xen is in use I don't see why part of that static reservation couldn't
>>be used by Xen, and another part by the Dom0 kernel. The kernel
>>obviously would need to ask the hypervisor how much of the space
>>is left, and where that area starts.
>>
>
>I think Dan means that there should be a clear separation between
>reservations for different usages (kernel/xen/...). The libnvdimm
>driver is for the linux kernel and only needs to maintain the
>reservation for kernel functionality. For others including xen/dm/...,
>if they want reservation for their own purpose, they should maintain
>their own reservations out of libnvdimm driver and avoid bothering the
>libnvdimm driver (e.g. add specific handling in libnvdimm driver).
>
>IIUC, one existing example is device-mapper device (dm) which needs to
>reserve on-device area for its own meta-data. Its choice is to store
>the meta-data on the block device (/dev/pmemN) provided by the
>libnvdimm driver.
>
>I think we can do the similar for Xen, like to lay another pseudo
>device on /dev/pmem and do the reservation, like 2. in my previous
>reply.
>
>Thanks,
>Haozhong
>
>>>The kernel needs to know about the struct page reservation because it
>>>needs to manage the lifetime of page references vs the lifetime of the
>>>device.  It does not have the same relationship with a Xen reservation
>>>which is why I'm proposing they be managed separately.
>>
>>I don't think I understand the difference you try to point out here.
>>Linux'es struct page and Xen's struct page_info serve the same
>>fundamental purpose.
>>
>>Jan
>>
>_______________________________________________
>Linux-nvdimm mailing list
>Linux-nvdimm@lists.01.org
>https://lists.01.org/mailman/listinfo/linux-nvdimm

[toc] | [prev] | [next] | [standalone]

#1500114

>>> On 13.10.16 at 10:53, <haozhong.zhang@intel.com> wrote:
> On 10/13/16 02:34 -0600, Jan Beulich wrote:
>>>>> On 12.10.16 at 18:19, <dan.j.williams@intel.com> wrote:
>>> On Wed, Oct 12, 2016 at 9:01 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>>> On 12.10.16 at 17:42, <dan.j.williams@intel.com> wrote:
>>>>> On Wed, Oct 12, 2016 at 8:39 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>>>>> On 12.10.16 at 16:58, <haozhong.zhang@intel.com> wrote:
>>>>>>> On 10/12/16 05:32 -0600, Jan Beulich wrote:
>>>>>>>>>>> On 12.10.16 at 12:33, <haozhong.zhang@intel.com> wrote:
>>>>>>>>> The layout is shown as the following diagram.
>>>>>>>>>
>>>>>>>>> +---------------+-----------+-------+----------+--------------+
>>>>>>>>> | whatever used | Partition | Super | Reserved | /dev/pmem0p1 |
>>>>>>>>> |  by kernel    |   Table   | Block | for Xen  |              |
>>>>>>>>> +---------------+-----------+-------+----------+--------------+
>>>>>>>>>                 \_____________________ _______________________/
>>>>>>>>>                                   V
>>>>>>>>>                              /dev/pmem0
>>>>>>>>
>>>>>>>>I have to admit that I dislike this, for not being OS-agnostic.
>>>>>>>>Neither should there be any Xen-specific region, nor should the
>>>>>>>>"whatever used by kernel" one be restricted to just Linux. What
>>>>>>>>I could see is an OS-reserved area ahead of the partition table,
>>>>>>>>the exact usage of which depends on which OS is currently
>>>>>>>>running (and in the Xen case this might be both Xen _and_ the
>>>>>>>>Dom0 kernel, arbitrated by a tbd protocol). After all, when
>>>>>>>>running under Xen, the Dom0 may not have a need for as much
>>>>>>>>control data as it has when running on bare hardware, for it
>>>>>>>>controlling less (if any) of the actual memory ranges when Xen
>>>>>>>>is present.
>>>>>>>>
>>>>>>>
>>>>>>> Isn't this OS-reserved area still not OS-agnostic, as it requires OS
>>>>>>> to know where the reserved area is?  Or do you mean it's not if it's
>>>>>>> defined by a protocol that is accepted by all OSes?
>>>>>>
>>>>>> The latter - we clearly won't get away without some agreement on
>>>>>> where to retrieve position and size of this area. I was simply
>>>>>> assuming that such a protocol already exists.
>>>>>>
>>>>>
>>>>> No, we should not mix the struct page reservation that the Dom0 kernel
>>>>> may actively use with the Xen reservation that the Dom0 kernel does
>>>>> not consume.  Explain again what is wrong with the partition approach?
>>>>
>>>> Not sure what was unclear in my previous reply. I don't think there
>>>> should be apriori knowledge of whether Xen is (going to be) used on
>>>> a system, and even if it gets used, but just occasionally, it would
>>>> (apart from the abstract considerations already given) be a waste
>>>> of resources to set something aside that could be used for other
>>>> purposes while Xen is not running. Static partitioning should only be
>>>> needed for persistent data.
>>>
>>> The reservation needs to be persistent / static even if the data is
>>> volatile, as is the case with struct page, because we can't have the
>>> size of the device change depending on use.  So, from the aspect of
>>> wasting space while Xen is not in use, both partitions and the
>>> intrinsic reservation approach suffer the same problem. Setting that
>>> aside I don't want to mix 2 different use cases into the same
>>> reservation.
>>
>>Then you didn't understand what I've said: I certainly didn't mean
>>the reservation to vary from a device perspective. However, when
>>Xen is in use I don't see why part of that static reservation couldn't
>>be used by Xen, and another part by the Dom0 kernel. The kernel
>>obviously would need to ask the hypervisor how much of the space
>>is left, and where that area starts.
>>
> 
> I think Dan means that there should be a clear separation between
> reservations for different usages (kernel/xen/...). The libnvdimm
> driver is for the linux kernel and only needs to maintain the
> reservation for kernel functionality. For others including xen/dm/...,
> if they want reservation for their own purpose, they should maintain
> their own reservations out of libnvdimm driver and avoid bothering the
> libnvdimm driver (e.g. add specific handling in libnvdimm driver).
> 
> IIUC, one existing example is device-mapper device (dm) which needs to
> reserve on-device area for its own meta-data. Its choice is to store
> the meta-data on the block device (/dev/pmemN) provided by the
> libnvdimm driver.
> 
> I think we can do the similar for Xen, like to lay another pseudo
> device on /dev/pmem and do the reservation, like 2. in my previous
> reply.

Well, my opinion certainly doesn't count much here, but I continue to
consider this a bad idea. For entities like drivers it may well be
appropriate, but I think there ought to be an independent concept
of "OS reserved", and in the Xen case this could then be shared
between hypervisor and Dom0 kernel. Or if we were to consider Dom0
"just a guest", things should even be the other way around: Xen gets
all of the OS reserved space, and Dom0 needs something custom.

Jan

[toc] | [prev] | [next] | [standalone]

#1500372 — Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen

On 10/13/16 03:08 -0600, Jan Beulich wrote:
>>>> On 13.10.16 at 10:53, <haozhong.zhang@intel.com> wrote:
>> On 10/13/16 02:34 -0600, Jan Beulich wrote:
>>>>>> On 12.10.16 at 18:19, <dan.j.williams@intel.com> wrote:
>>>> On Wed, Oct 12, 2016 at 9:01 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>>>> On 12.10.16 at 17:42, <dan.j.williams@intel.com> wrote:
>>>>>> On Wed, Oct 12, 2016 at 8:39 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>>>>>> On 12.10.16 at 16:58, <haozhong.zhang@intel.com> wrote:
>>>>>>>> On 10/12/16 05:32 -0600, Jan Beulich wrote:
>>>>>>>>>>>> On 12.10.16 at 12:33, <haozhong.zhang@intel.com> wrote:
>>>>>>>>>> The layout is shown as the following diagram.
>>>>>>>>>>
>>>>>>>>>> +---------------+-----------+-------+----------+--------------+
>>>>>>>>>> | whatever used | Partition | Super | Reserved | /dev/pmem0p1 |
>>>>>>>>>> |  by kernel    |   Table   | Block | for Xen  |              |
>>>>>>>>>> +---------------+-----------+-------+----------+--------------+
>>>>>>>>>>                 \_____________________ _______________________/
>>>>>>>>>>                                   V
>>>>>>>>>>                              /dev/pmem0
>>>>>>>>>
>>>>>>>>>I have to admit that I dislike this, for not being OS-agnostic.
>>>>>>>>>Neither should there be any Xen-specific region, nor should the
>>>>>>>>>"whatever used by kernel" one be restricted to just Linux. What
>>>>>>>>>I could see is an OS-reserved area ahead of the partition table,
>>>>>>>>>the exact usage of which depends on which OS is currently
>>>>>>>>>running (and in the Xen case this might be both Xen _and_ the
>>>>>>>>>Dom0 kernel, arbitrated by a tbd protocol). After all, when
>>>>>>>>>running under Xen, the Dom0 may not have a need for as much
>>>>>>>>>control data as it has when running on bare hardware, for it
>>>>>>>>>controlling less (if any) of the actual memory ranges when Xen
>>>>>>>>>is present.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Isn't this OS-reserved area still not OS-agnostic, as it requires OS
>>>>>>>> to know where the reserved area is?  Or do you mean it's not if it's
>>>>>>>> defined by a protocol that is accepted by all OSes?
>>>>>>>
>>>>>>> The latter - we clearly won't get away without some agreement on
>>>>>>> where to retrieve position and size of this area. I was simply
>>>>>>> assuming that such a protocol already exists.
>>>>>>>
>>>>>>
>>>>>> No, we should not mix the struct page reservation that the Dom0 kernel
>>>>>> may actively use with the Xen reservation that the Dom0 kernel does
>>>>>> not consume.  Explain again what is wrong with the partition approach?
>>>>>
>>>>> Not sure what was unclear in my previous reply. I don't think there
>>>>> should be apriori knowledge of whether Xen is (going to be) used on
>>>>> a system, and even if it gets used, but just occasionally, it would
>>>>> (apart from the abstract considerations already given) be a waste
>>>>> of resources to set something aside that could be used for other
>>>>> purposes while Xen is not running. Static partitioning should only be
>>>>> needed for persistent data.
>>>>
>>>> The reservation needs to be persistent / static even if the data is
>>>> volatile, as is the case with struct page, because we can't have the
>>>> size of the device change depending on use.  So, from the aspect of
>>>> wasting space while Xen is not in use, both partitions and the
>>>> intrinsic reservation approach suffer the same problem. Setting that
>>>> aside I don't want to mix 2 different use cases into the same
>>>> reservation.
>>>
>>>Then you didn't understand what I've said: I certainly didn't mean
>>>the reservation to vary from a device perspective. However, when
>>>Xen is in use I don't see why part of that static reservation couldn't
>>>be used by Xen, and another part by the Dom0 kernel. The kernel
>>>obviously would need to ask the hypervisor how much of the space
>>>is left, and where that area starts.
>>>
>>
>> I think Dan means that there should be a clear separation between
>> reservations for different usages (kernel/xen/...). The libnvdimm
>> driver is for the linux kernel and only needs to maintain the
>> reservation for kernel functionality. For others including xen/dm/...,
>> if they want reservation for their own purpose, they should maintain
>> their own reservations out of libnvdimm driver and avoid bothering the
>> libnvdimm driver (e.g. add specific handling in libnvdimm driver).
>>
>> IIUC, one existing example is device-mapper device (dm) which needs to
>> reserve on-device area for its own meta-data. Its choice is to store
>> the meta-data on the block device (/dev/pmemN) provided by the
>> libnvdimm driver.
>>
>> I think we can do the similar for Xen, like to lay another pseudo
>> device on /dev/pmem and do the reservation, like 2. in my previous
>> reply.
>
>Well, my opinion certainly doesn't count much here, but I continue to
>consider this a bad idea. For entities like drivers it may well be
>appropriate, but I think there ought to be an independent concept
>of "OS reserved", and in the Xen case this could then be shared
>between hypervisor and Dom0 kernel.

No such independent concept seems exist right now. It may be hard to
define such concept, because it's hard to know the common requirements
(e.g. size/alignment/...)  from ALL OSes. Making each component to
maintain its own reservation in its own way seems more flexible.

>Or if we were to consider Dom0
>"just a guest", things should even be the other way around: Xen gets
>all of the OS reserved space, and Dom0 needs something custom.
>

Sure, it's possible to implement the driver in a way that if the
driver finds it runs on Xen, then it just leaves the OS reserved area
for Xen and itself goes to other reservation. Are there some
differences in practice from the way that Xen goes to other
reservation that makes we have to do so? If not and it's possible to
not touch the existing libnvdimm driver, why don't we just use the
existing libnvdimm driver and let xen driver make the reservation on
what the libnvdimm driver provides?

In addition (not sure it's related), my Xen patch series (specially
patch 3) does not have many requirements for the location of the
reserved area, as long as it's in the nvdimm. I mean if we find a
better way for the reservation in future, there should be no changes
to Xen. For now, I think we could just choose the way to not touch the
libnvdimm driver.

Thanks,
Haozhong

[toc] | [prev] | [next] | [standalone]

#1500821

>>> On 13.10.16 at 17:46, <haozhong.zhang@intel.com> wrote:
> On 10/13/16 03:08 -0600, Jan Beulich wrote:
>>>>> On 13.10.16 at 10:53, <haozhong.zhang@intel.com> wrote:
>>> On 10/13/16 02:34 -0600, Jan Beulich wrote:
>>>>>>> On 12.10.16 at 18:19, <dan.j.williams@intel.com> wrote:
>>>>> On Wed, Oct 12, 2016 at 9:01 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>>>>> On 12.10.16 at 17:42, <dan.j.williams@intel.com> wrote:
>>>>>>> On Wed, Oct 12, 2016 at 8:39 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>>>>>>> On 12.10.16 at 16:58, <haozhong.zhang@intel.com> wrote:
>>>>>>>>> On 10/12/16 05:32 -0600, Jan Beulich wrote:
>>>>>>>>>>>>> On 12.10.16 at 12:33, <haozhong.zhang@intel.com> wrote:
>>>>>>>>>>> The layout is shown as the following diagram.
>>>>>>>>>>>
>>>>>>>>>>> +---------------+-----------+-------+----------+--------------+
>>>>>>>>>>> | whatever used | Partition | Super | Reserved | /dev/pmem0p1 |
>>>>>>>>>>> |  by kernel    |   Table   | Block | for Xen  |              |
>>>>>>>>>>> +---------------+-----------+-------+----------+--------------+
>>>>>>>>>>>                 \_____________________ _______________________/
>>>>>>>>>>>                                   V
>>>>>>>>>>>                              /dev/pmem0
>>>>>>>>>>
>>>>>>>>>>I have to admit that I dislike this, for not being OS-agnostic.
>>>>>>>>>>Neither should there be any Xen-specific region, nor should the
>>>>>>>>>>"whatever used by kernel" one be restricted to just Linux. What
>>>>>>>>>>I could see is an OS-reserved area ahead of the partition table,
>>>>>>>>>>the exact usage of which depends on which OS is currently
>>>>>>>>>>running (and in the Xen case this might be both Xen _and_ the
>>>>>>>>>>Dom0 kernel, arbitrated by a tbd protocol). After all, when
>>>>>>>>>>running under Xen, the Dom0 may not have a need for as much
>>>>>>>>>>control data as it has when running on bare hardware, for it
>>>>>>>>>>controlling less (if any) of the actual memory ranges when Xen
>>>>>>>>>>is present.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Isn't this OS-reserved area still not OS-agnostic, as it requires OS
>>>>>>>>> to know where the reserved area is?  Or do you mean it's not if it's
>>>>>>>>> defined by a protocol that is accepted by all OSes?
>>>>>>>>
>>>>>>>> The latter - we clearly won't get away without some agreement on
>>>>>>>> where to retrieve position and size of this area. I was simply
>>>>>>>> assuming that such a protocol already exists.
>>>>>>>>
>>>>>>>
>>>>>>> No, we should not mix the struct page reservation that the Dom0 kernel
>>>>>>> may actively use with the Xen reservation that the Dom0 kernel does
>>>>>>> not consume.  Explain again what is wrong with the partition approach?
>>>>>>
>>>>>> Not sure what was unclear in my previous reply. I don't think there
>>>>>> should be apriori knowledge of whether Xen is (going to be) used on
>>>>>> a system, and even if it gets used, but just occasionally, it would
>>>>>> (apart from the abstract considerations already given) be a waste
>>>>>> of resources to set something aside that could be used for other
>>>>>> purposes while Xen is not running. Static partitioning should only be
>>>>>> needed for persistent data.
>>>>>
>>>>> The reservation needs to be persistent / static even if the data is
>>>>> volatile, as is the case with struct page, because we can't have the
>>>>> size of the device change depending on use.  So, from the aspect of
>>>>> wasting space while Xen is not in use, both partitions and the
>>>>> intrinsic reservation approach suffer the same problem. Setting that
>>>>> aside I don't want to mix 2 different use cases into the same
>>>>> reservation.
>>>>
>>>>Then you didn't understand what I've said: I certainly didn't mean
>>>>the reservation to vary from a device perspective. However, when
>>>>Xen is in use I don't see why part of that static reservation couldn't
>>>>be used by Xen, and another part by the Dom0 kernel. The kernel
>>>>obviously would need to ask the hypervisor how much of the space
>>>>is left, and where that area starts.
>>>>
>>>
>>> I think Dan means that there should be a clear separation between
>>> reservations for different usages (kernel/xen/...). The libnvdimm
>>> driver is for the linux kernel and only needs to maintain the
>>> reservation for kernel functionality. For others including xen/dm/...,
>>> if they want reservation for their own purpose, they should maintain
>>> their own reservations out of libnvdimm driver and avoid bothering the
>>> libnvdimm driver (e.g. add specific handling in libnvdimm driver).
>>>
>>> IIUC, one existing example is device-mapper device (dm) which needs to
>>> reserve on-device area for its own meta-data. Its choice is to store
>>> the meta-data on the block device (/dev/pmemN) provided by the
>>> libnvdimm driver.
>>>
>>> I think we can do the similar for Xen, like to lay another pseudo
>>> device on /dev/pmem and do the reservation, like 2. in my previous
>>> reply.
>>
>>Well, my opinion certainly doesn't count much here, but I continue to
>>consider this a bad idea. For entities like drivers it may well be
>>appropriate, but I think there ought to be an independent concept
>>of "OS reserved", and in the Xen case this could then be shared
>>between hypervisor and Dom0 kernel.
> 
> No such independent concept seems exist right now. It may be hard to
> define such concept, because it's hard to know the common requirements
> (e.g. size/alignment/...)  from ALL OSes. Making each component to
> maintain its own reservation in its own way seems more flexible.
> 
>>Or if we were to consider Dom0
>>"just a guest", things should even be the other way around: Xen gets
>>all of the OS reserved space, and Dom0 needs something custom.
>>
> 
> Sure, it's possible to implement the driver in a way that if the
> driver finds it runs on Xen, then it just leaves the OS reserved area
> for Xen and itself goes to other reservation. Are there some
> differences in practice from the way that Xen goes to other
> reservation that makes we have to do so? If not and it's possible to
> not touch the existing libnvdimm driver, why don't we just use the
> existing libnvdimm driver and let xen driver make the reservation on
> what the libnvdimm driver provides?

It continues to feel like you're trying to make the second step before
the first: You talk about implementation, whereas I talk about the
concept that should underly your implementation. One could view it
the way that how said driver works with Xen in the picture should
have been decided before it got implemented, and then the question
of whether it would be okay to leave the existing implementation
alone would not have appeared in the first place.

With that, I can't reasonably answer your questions.

Jan

[toc] | [prev] | [next] | [standalone]

#1504695 — Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen

On 10/14/16 04:16 -0600, Jan Beulich wrote:
>>>> On 13.10.16 at 17:46, <haozhong.zhang@intel.com> wrote:
>> On 10/13/16 03:08 -0600, Jan Beulich wrote:
>>>>>> On 13.10.16 at 10:53, <haozhong.zhang@intel.com> wrote:
>>>> On 10/13/16 02:34 -0600, Jan Beulich wrote:
>>>>>>>> On 12.10.16 at 18:19, <dan.j.williams@intel.com> wrote:
>>>>>> On Wed, Oct 12, 2016 at 9:01 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>>>>>> On 12.10.16 at 17:42, <dan.j.williams@intel.com> wrote:
>>>>>>>> On Wed, Oct 12, 2016 at 8:39 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>>>>>>>> On 12.10.16 at 16:58, <haozhong.zhang@intel.com> wrote:
>>>>>>>>>> On 10/12/16 05:32 -0600, Jan Beulich wrote:
>>>>>>>>>>>>>> On 12.10.16 at 12:33, <haozhong.zhang@intel.com> wrote:
>>>>>>>>>>>> The layout is shown as the following diagram.
>>>>>>>>>>>>
>>>>>>>>>>>> +---------------+-----------+-------+----------+--------------+
>>>>>>>>>>>> | whatever used | Partition | Super | Reserved | /dev/pmem0p1 |
>>>>>>>>>>>> |  by kernel    |   Table   | Block | for Xen  |              |
>>>>>>>>>>>> +---------------+-----------+-------+----------+--------------+
>>>>>>>>>>>>                 \_____________________ _______________________/
>>>>>>>>>>>>                                   V
>>>>>>>>>>>>                              /dev/pmem0
>>>>>>>>>>>
>>>>>>>>>>>I have to admit that I dislike this, for not being OS-agnostic.
>>>>>>>>>>>Neither should there be any Xen-specific region, nor should the
>>>>>>>>>>>"whatever used by kernel" one be restricted to just Linux. What
>>>>>>>>>>>I could see is an OS-reserved area ahead of the partition table,
>>>>>>>>>>>the exact usage of which depends on which OS is currently
>>>>>>>>>>>running (and in the Xen case this might be both Xen _and_ the
>>>>>>>>>>>Dom0 kernel, arbitrated by a tbd protocol). After all, when
>>>>>>>>>>>running under Xen, the Dom0 may not have a need for as much
>>>>>>>>>>>control data as it has when running on bare hardware, for it
>>>>>>>>>>>controlling less (if any) of the actual memory ranges when Xen
>>>>>>>>>>>is present.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Isn't this OS-reserved area still not OS-agnostic, as it requires OS
>>>>>>>>>> to know where the reserved area is?  Or do you mean it's not if it's
>>>>>>>>>> defined by a protocol that is accepted by all OSes?
>>>>>>>>>
>>>>>>>>> The latter - we clearly won't get away without some agreement on
>>>>>>>>> where to retrieve position and size of this area. I was simply
>>>>>>>>> assuming that such a protocol already exists.
>>>>>>>>>
>>>>>>>>
>>>>>>>> No, we should not mix the struct page reservation that the Dom0 kernel
>>>>>>>> may actively use with the Xen reservation that the Dom0 kernel does
>>>>>>>> not consume.  Explain again what is wrong with the partition approach?
>>>>>>>
>>>>>>> Not sure what was unclear in my previous reply. I don't think there
>>>>>>> should be apriori knowledge of whether Xen is (going to be) used on
>>>>>>> a system, and even if it gets used, but just occasionally, it would
>>>>>>> (apart from the abstract considerations already given) be a waste
>>>>>>> of resources to set something aside that could be used for other
>>>>>>> purposes while Xen is not running. Static partitioning should only be
>>>>>>> needed for persistent data.
>>>>>>
>>>>>> The reservation needs to be persistent / static even if the data is
>>>>>> volatile, as is the case with struct page, because we can't have the
>>>>>> size of the device change depending on use.  So, from the aspect of
>>>>>> wasting space while Xen is not in use, both partitions and the
>>>>>> intrinsic reservation approach suffer the same problem. Setting that
>>>>>> aside I don't want to mix 2 different use cases into the same
>>>>>> reservation.
>>>>>
>>>>>Then you didn't understand what I've said: I certainly didn't mean
>>>>>the reservation to vary from a device perspective. However, when
>>>>>Xen is in use I don't see why part of that static reservation couldn't
>>>>>be used by Xen, and another part by the Dom0 kernel. The kernel
>>>>>obviously would need to ask the hypervisor how much of the space
>>>>>is left, and where that area starts.
>>>>>
>>>>
>>>> I think Dan means that there should be a clear separation between
>>>> reservations for different usages (kernel/xen/...). The libnvdimm
>>>> driver is for the linux kernel and only needs to maintain the
>>>> reservation for kernel functionality. For others including xen/dm/...,
>>>> if they want reservation for their own purpose, they should maintain
>>>> their own reservations out of libnvdimm driver and avoid bothering the
>>>> libnvdimm driver (e.g. add specific handling in libnvdimm driver).
>>>>
>>>> IIUC, one existing example is device-mapper device (dm) which needs to
>>>> reserve on-device area for its own meta-data. Its choice is to store
>>>> the meta-data on the block device (/dev/pmemN) provided by the
>>>> libnvdimm driver.
>>>>
>>>> I think we can do the similar for Xen, like to lay another pseudo
>>>> device on /dev/pmem and do the reservation, like 2. in my previous
>>>> reply.
>>>
>>>Well, my opinion certainly doesn't count much here, but I continue to
>>>consider this a bad idea. For entities like drivers it may well be
>>>appropriate, but I think there ought to be an independent concept
>>>of "OS reserved", and in the Xen case this could then be shared
>>>between hypervisor and Dom0 kernel.
>>
>> No such independent concept seems exist right now. It may be hard to
>> define such concept, because it's hard to know the common requirements
>> (e.g. size/alignment/...)  from ALL OSes. Making each component to
>> maintain its own reservation in its own way seems more flexible.
>>
>>>Or if we were to consider Dom0
>>>"just a guest", things should even be the other way around: Xen gets
>>>all of the OS reserved space, and Dom0 needs something custom.
>>>
>>
>> Sure, it's possible to implement the driver in a way that if the
>> driver finds it runs on Xen, then it just leaves the OS reserved area
>> for Xen and itself goes to other reservation. Are there some
>> differences in practice from the way that Xen goes to other
>> reservation that makes we have to do so? If not and it's possible to
>> not touch the existing libnvdimm driver, why don't we just use the
>> existing libnvdimm driver and let xen driver make the reservation on
>> what the libnvdimm driver provides?
>
>It continues to feel like you're trying to make the second step before
>the first: You talk about implementation, whereas I talk about the
>concept that should underly your implementation. One could view it
>the way that how said driver works with Xen in the picture should
>have been decided before it got implemented, and then the question
>of whether it would be okay to leave the existing implementation
>alone would not have appeared in the first place.
>
>With that, I can't reasonably answer your questions.
>

Ideally, Xen should step in first to do whatever needed for the
resource management (e.g. taking all OS-reserved space), and then let
other kernel components work on the remaining resource (e.g. libnvdimm
driver to use other reserved space). However, the libnvdimm driver was
merged in the kernel long before I started to design and implement Xen
vNVDIMM. The implementation of the ideal design above needs to modify
the existing driver which is objected by its maintainer. Therefore, I
looked to the alternative design and implementation.

Thanks,
Haozhong

[toc] | [prev] | [next] | [standalone]

#1500378 — Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen

On Thu, Oct 13, 2016 at 2:08 AM, Jan Beulich <JBeulich@suse.com> wrote:
[..]
>> I think we can do the similar for Xen, like to lay another pseudo
>> device on /dev/pmem and do the reservation, like 2. in my previous
>> reply.
>
> Well, my opinion certainly doesn't count much here, but I continue to
> consider this a bad idea. For entities like drivers it may well be
> appropriate, but I think there ought to be an independent concept
> of "OS reserved", and in the Xen case this could then be shared
> between hypervisor and Dom0 kernel. Or if we were to consider Dom0
> "just a guest", things should even be the other way around: Xen gets
> all of the OS reserved space, and Dom0 needs something custom.

You haven't made the case why Xen is special and other applications of
persistent memory are not.  The current struct page reservation
supports fundamental address-ability of persistent memory namespaces
for the rest of the kernel.  The Xen reservation is application
specific.  XFS, EXT4, and DM also have application specific usages of
persistent memory and consume metadata space out of a block device. If
we don't need an XFS-mode nvdimm device, why do we need Xen-mode?

[toc] | [prev] | [next] | [standalone]

#1500403 — Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen

On 13/10/16 16:40, Dan Williams wrote:
> On Thu, Oct 13, 2016 at 2:08 AM, Jan Beulich <JBeulich@suse.com> wrote:
> [..]
>>> I think we can do the similar for Xen, like to lay another pseudo
>>> device on /dev/pmem and do the reservation, like 2. in my previous
>>> reply.
>> Well, my opinion certainly doesn't count much here, but I continue to
>> consider this a bad idea. For entities like drivers it may well be
>> appropriate, but I think there ought to be an independent concept
>> of "OS reserved", and in the Xen case this could then be shared
>> between hypervisor and Dom0 kernel. Or if we were to consider Dom0
>> "just a guest", things should even be the other way around: Xen gets
>> all of the OS reserved space, and Dom0 needs something custom.
> You haven't made the case why Xen is special and other applications of
> persistent memory are not.

In a Xen system, Xen runs in the baremetal root-mode ring0, and dom0 is
a VM running in ring1/3 with the nvdimm driver.  This is the opposite
way around to the KVM model.

Dom0, being the hardware domain, has default ownership of all the
hardware, but to gain access in the first place, it must request a
mapping from Xen.  Xen therefore needs to know and cope with being able
to give dom0 a mapping to the nvdimms, without touching the content of
the nvidmm itself (so as to avoid corrupting data).

Once dom0 has a mapping of the nvdimm, the nvdimm driver can go to work
and figure out what is on the DIMM, and which areas are safe to use.

At this point, a Xen subsystem in Linux could choose one or more areas
to hand back to the hypervisor to use as RAM/other.

~Andrew

[toc] | [prev] | [next] | [standalone]

#1500505 — Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen

On Thu, Oct 13, 2016 at 9:01 AM, Andrew Cooper
<andrew.cooper3@citrix.com> wrote:
> On 13/10/16 16:40, Dan Williams wrote:
>> On Thu, Oct 13, 2016 at 2:08 AM, Jan Beulich <JBeulich@suse.com> wrote:
>> [..]
>>>> I think we can do the similar for Xen, like to lay another pseudo
>>>> device on /dev/pmem and do the reservation, like 2. in my previous
>>>> reply.
>>> Well, my opinion certainly doesn't count much here, but I continue to
>>> consider this a bad idea. For entities like drivers it may well be
>>> appropriate, but I think there ought to be an independent concept
>>> of "OS reserved", and in the Xen case this could then be shared
>>> between hypervisor and Dom0 kernel. Or if we were to consider Dom0
>>> "just a guest", things should even be the other way around: Xen gets
>>> all of the OS reserved space, and Dom0 needs something custom.
>> You haven't made the case why Xen is special and other applications of
>> persistent memory are not.
>
> In a Xen system, Xen runs in the baremetal root-mode ring0, and dom0 is
> a VM running in ring1/3 with the nvdimm driver.  This is the opposite
> way around to the KVM model.
>
> Dom0, being the hardware domain, has default ownership of all the
> hardware, but to gain access in the first place, it must request a
> mapping from Xen.

This is where my understanding the Xen model breaks down.  Are you
saying dom0 can't access the persistent memory range unless the ring0
agent has metadata storage space for tracking what it maps into dom0?
That can't be true because then PCI memory ranges would not work
without metadata reserve space.  Dom0 still needs to map and write the
DIMMs to even set up the struct page reservation, it isn't established
by default.

> Xen therefore needs to know and cope with being able
> to give dom0 a mapping to the nvdimms, without touching the content of
> the nvidmm itself (so as to avoid corrupting data).

Is it true that this metadata only comes into use when remapping the
dom0 discovered range(s) into a guest VM?

> Once dom0 has a mapping of the nvdimm, the nvdimm driver can go to work
> and figure out what is on the DIMM, and which areas are safe to use.

I don't understand this ordering of events.  Dom0 needs to have a
mapping to even write the on-media structure to indicate a
reservation.  So, initial dom0 access can't depend on metadata
reservation already being present.

> At this point, a Xen subsystem in Linux could choose one or more areas
> to hand back to the hypervisor to use as RAM/other.

To me all this configuration seems to come after the fact.  After dom0
sees /dev/pmemX devices, then it can go to work carving it up and
writing Xen specific metadata to the range(s).  The struct page
reservation never comes into the picture.  In fact, a raw mode
namespace (one without a reservation) could be used in this model, the
nvdimm core never needs to know what is happening.

[toc] | [prev] | [next] | [standalone]

#1500515 — Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen

On 13/10/16 19:59, Dan Williams wrote:
> On Thu, Oct 13, 2016 at 9:01 AM, Andrew Cooper
> <andrew.cooper3@citrix.com> wrote:
>> On 13/10/16 16:40, Dan Williams wrote:
>>> On Thu, Oct 13, 2016 at 2:08 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>> [..]
>>>>> I think we can do the similar for Xen, like to lay another pseudo
>>>>> device on /dev/pmem and do the reservation, like 2. in my previous
>>>>> reply.
>>>> Well, my opinion certainly doesn't count much here, but I continue to
>>>> consider this a bad idea. For entities like drivers it may well be
>>>> appropriate, but I think there ought to be an independent concept
>>>> of "OS reserved", and in the Xen case this could then be shared
>>>> between hypervisor and Dom0 kernel. Or if we were to consider Dom0
>>>> "just a guest", things should even be the other way around: Xen gets
>>>> all of the OS reserved space, and Dom0 needs something custom.
>>> You haven't made the case why Xen is special and other applications of
>>> persistent memory are not.
>> In a Xen system, Xen runs in the baremetal root-mode ring0, and dom0 is
>> a VM running in ring1/3 with the nvdimm driver.  This is the opposite
>> way around to the KVM model.
>>
>> Dom0, being the hardware domain, has default ownership of all the
>> hardware, but to gain access in the first place, it must request a
>> mapping from Xen.
> This is where my understanding the Xen model breaks down.  Are you
> saying dom0 can't access the persistent memory range unless the ring0
> agent has metadata storage space for tracking what it maps into dom0?

No.  I am trying to point out that the current suggestion wont work, and
needs re-designing.

Xen *must* be able to properly configure mappings of the NVDIMM for
dom0, *without* modifying any content on the NVDIMM.  Otherwise, data
corruption will occur.

Whether this means no Xen metadata, or the metadata living elsewhere in
regular ram, such as the main frametable, is an implementation detail.

>
>> Once dom0 has a mapping of the nvdimm, the nvdimm driver can go to work
>> and figure out what is on the DIMM, and which areas are safe to use.
> I don't understand this ordering of events.  Dom0 needs to have a
> mapping to even write the on-media structure to indicate a
> reservation.  So, initial dom0 access can't depend on metadata
> reservation already being present.

I agree.

Overall, I think the following is needed.

* Xen starts up.
** Xen might find some NVDIMM SPA/MFN ranges in the NFIT table, and
needs to note this information somehow.
** Xen might find some Type 7 E820 regions, and needs to note this
information somehow.
* Xen starts dom0.
* Once OSPM is running, a Xen component in Linux needs to collect and
report all NVDIMM SPA/MFN regions it knowns about.
** This covers the AML-only case, and the hotplug case.
* Dom0 requests a mapping of the NVDIMMs via the usual mechanism.
** This should work, as Xen is aware that there is something there to be
mapped (rather than just empty physical address space).
* Dom0 finds that some NVDIMM ranges are now available for use (probably
modelled as hotplug events).
* /dev/pmem $STUFF starts happening as normal.

At some pointer later after dom0 policy decisions are made (ultimately,
by the host administrator):
* If an area of NVDIMM is chosen for Xen to use, Dom0 needs to inform
Xen of the SPA/MFN regions which are safe to use.
* Xen then incorporates these regions into its idea of RAM, and starts
using them for whatever.

~Andrew

[toc] | [prev] | [next] | [standalone]

#1500718 — Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen

On 10/13/16 20:33 +0100, Andrew Cooper wrote:
>On 13/10/16 19:59, Dan Williams wrote:
>> On Thu, Oct 13, 2016 at 9:01 AM, Andrew Cooper
>> <andrew.cooper3@citrix.com> wrote:
>>> On 13/10/16 16:40, Dan Williams wrote:
>>>> On Thu, Oct 13, 2016 at 2:08 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> [..]
>>>>>> I think we can do the similar for Xen, like to lay another pseudo
>>>>>> device on /dev/pmem and do the reservation, like 2. in my previous
>>>>>> reply.
>>>>> Well, my opinion certainly doesn't count much here, but I continue to
>>>>> consider this a bad idea. For entities like drivers it may well be
>>>>> appropriate, but I think there ought to be an independent concept
>>>>> of "OS reserved", and in the Xen case this could then be shared
>>>>> between hypervisor and Dom0 kernel. Or if we were to consider Dom0
>>>>> "just a guest", things should even be the other way around: Xen gets
>>>>> all of the OS reserved space, and Dom0 needs something custom.
>>>> You haven't made the case why Xen is special and other applications of
>>>> persistent memory are not.
>>> In a Xen system, Xen runs in the baremetal root-mode ring0, and dom0 is
>>> a VM running in ring1/3 with the nvdimm driver.  This is the opposite
>>> way around to the KVM model.
>>>
>>> Dom0, being the hardware domain, has default ownership of all the
>>> hardware, but to gain access in the first place, it must request a
>>> mapping from Xen.
>> This is where my understanding the Xen model breaks down.  Are you
>> saying dom0 can't access the persistent memory range unless the ring0
>> agent has metadata storage space for tracking what it maps into dom0?
>
>No.  I am trying to point out that the current suggestion wont work, and
>needs re-designing.
>
>Xen *must* be able to properly configure mappings of the NVDIMM for
>dom0, *without* modifying any content on the NVDIMM.  Otherwise, data
>corruption will occur.
>
>Whether this means no Xen metadata, or the metadata living elsewhere in
>regular ram, such as the main frametable, is an implementation detail.
>
>>
>>> Once dom0 has a mapping of the nvdimm, the nvdimm driver can go to work
>>> and figure out what is on the DIMM, and which areas are safe to use.
>> I don't understand this ordering of events.  Dom0 needs to have a
>> mapping to even write the on-media structure to indicate a
>> reservation.  So, initial dom0 access can't depend on metadata
>> reservation already being present.
>
>I agree.
>
>Overall, I think the following is needed.
>
>* Xen starts up.
>** Xen might find some NVDIMM SPA/MFN ranges in the NFIT table, and
>needs to note this information somehow.
>** Xen might find some Type 7 E820 regions, and needs to note this
>information somehow.

IIUC, this is to collect MFNs and no need to create frame table and
M2P at this stage. If so, what is different from ...

>* Xen starts dom0.
>* Once OSPM is running, a Xen component in Linux needs to collect and
>report all NVDIMM SPA/MFN regions it knowns about.
>** This covers the AML-only case, and the hotplug case.

... the MFNs reported here, especially that the former is a subset
(hotplug ones not included in the former) of latter.

(There is no E820 hole or SRAT entries to tell which address range is
 reserved for hotplugged NVDIMM)

>* Dom0 requests a mapping of the NVDIMMs via the usual mechanism.

Two questions:
1. Why is this request necessary? Even without such requests like what
   my current implementation, Dom0 can still access NVDIMM.

   Or do you mean Xen hypervisor should by default disallow Dom0 to
   access MFNs reported in previous step until they are requested?

2. Who initiates the requests? If it's the libnvdimm driver, that
   means we still need to introduce Xen specific code to the driver.

   Or the requests are issued by OSPM (or the Xen component you
   mentioned above) when they probe new dimms?

   For the latter, Dan, do you think it's acceptable in NFIT code to
   call the Xen component to request the access permission of the pmem
   regions, e.g. in apic_nfit_insert_resource(). Of course, it's only
   used for Dom0 case.

>** This should work, as Xen is aware that there is something there to be
>mapped (rather than just empty physical address space).
>* Dom0 finds that some NVDIMM ranges are now available for use (probably
>modelled as hotplug events).
>* /dev/pmem $STUFF starts happening as normal.
>
>At some pointer later after dom0 policy decisions are made (ultimately,
>by the host administrator):
>* If an area of NVDIMM is chosen for Xen to use, Dom0 needs to inform
>Xen of the SPA/MFN regions which are safe to use.
>* Xen then incorporates these regions into its idea of RAM, and starts
>using them for whatever.
>

Agree. I think we may not need to fix the way/format/... to make the
reservation, and instead let the users (host administrators), who have
better understanding of their data, make the proper decision.

In a worse case that no reservation is made, Xen hypervisor could turn
to use RAM for management structures for NVDIMM, with the cost of less
RAM for guests.

Thanks,
Haozhong

[toc] | [prev] | [next] | [standalone]

#1500883 — Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen

On 14/10/16 08:08, Haozhong Zhang wrote:
> On 10/13/16 20:33 +0100, Andrew Cooper wrote:
>> On 13/10/16 19:59, Dan Williams wrote:
>>> On Thu, Oct 13, 2016 at 9:01 AM, Andrew Cooper
>>> <andrew.cooper3@citrix.com> wrote:
>>>> On 13/10/16 16:40, Dan Williams wrote:
>>>>> On Thu, Oct 13, 2016 at 2:08 AM, Jan Beulich <JBeulich@suse.com>
>>>>> wrote:
>>>>> [..]
>>>>>>> I think we can do the similar for Xen, like to lay another pseudo
>>>>>>> device on /dev/pmem and do the reservation, like 2. in my previous
>>>>>>> reply.
>>>>>> Well, my opinion certainly doesn't count much here, but I
>>>>>> continue to
>>>>>> consider this a bad idea. For entities like drivers it may well be
>>>>>> appropriate, but I think there ought to be an independent concept
>>>>>> of "OS reserved", and in the Xen case this could then be shared
>>>>>> between hypervisor and Dom0 kernel. Or if we were to consider Dom0
>>>>>> "just a guest", things should even be the other way around: Xen gets
>>>>>> all of the OS reserved space, and Dom0 needs something custom.
>>>>> You haven't made the case why Xen is special and other
>>>>> applications of
>>>>> persistent memory are not.
>>>> In a Xen system, Xen runs in the baremetal root-mode ring0, and
>>>> dom0 is
>>>> a VM running in ring1/3 with the nvdimm driver.  This is the opposite
>>>> way around to the KVM model.
>>>>
>>>> Dom0, being the hardware domain, has default ownership of all the
>>>> hardware, but to gain access in the first place, it must request a
>>>> mapping from Xen.
>>> This is where my understanding the Xen model breaks down.  Are you
>>> saying dom0 can't access the persistent memory range unless the ring0
>>> agent has metadata storage space for tracking what it maps into dom0?
>>
>> No.  I am trying to point out that the current suggestion wont work, and
>> needs re-designing.
>>
>> Xen *must* be able to properly configure mappings of the NVDIMM for
>> dom0, *without* modifying any content on the NVDIMM.  Otherwise, data
>> corruption will occur.
>>
>> Whether this means no Xen metadata, or the metadata living elsewhere in
>> regular ram, such as the main frametable, is an implementation detail.
>>
>>>
>>>> Once dom0 has a mapping of the nvdimm, the nvdimm driver can go to
>>>> work
>>>> and figure out what is on the DIMM, and which areas are safe to use.
>>> I don't understand this ordering of events.  Dom0 needs to have a
>>> mapping to even write the on-media structure to indicate a
>>> reservation.  So, initial dom0 access can't depend on metadata
>>> reservation already being present.
>>
>> I agree.
>>
>> Overall, I think the following is needed.
>>
>> * Xen starts up.
>> ** Xen might find some NVDIMM SPA/MFN ranges in the NFIT table, and
>> needs to note this information somehow.
>> ** Xen might find some Type 7 E820 regions, and needs to note this
>> information somehow.
>
> IIUC, this is to collect MFNs and no need to create frame table and
> M2P at this stage. If so, what is different from ...
>
>> * Xen starts dom0.
>> * Once OSPM is running, a Xen component in Linux needs to collect and
>> report all NVDIMM SPA/MFN regions it knowns about.
>> ** This covers the AML-only case, and the hotplug case.
>
> ... the MFNs reported here, especially that the former is a subset
> (hotplug ones not included in the former) of latter.

Hopefully nothing.  However, Xen shouldn't exclusively rely on the dom0
when it is capable of working things out itself, (which can aid with
debugging one half of this arrangement).  Also, the MFNS found by Xen
alone can be present in the default memory map for dom0.

>
> (There is no E820 hole or SRAT entries to tell which address range is
> reserved for hotplugged NVDIMM)
>
>> * Dom0 requests a mapping of the NVDIMMs via the usual mechanism.
>
> Two questions:
> 1. Why is this request necessary? Even without such requests like what
>   my current implementation, Dom0 can still access NVDIMM.

Can it?  (if so, great, but I don't think this holds in the general
case.)  Is that a side effect of the NVDIMM being covered by a hole in
the E820?  The current logic for what dom0 may access by default is
somewhat ad-hoc, and I have a gut feeling that it won't work with E820
type 7 regions.

>
>   Or do you mean Xen hypervisor should by default disallow Dom0 to
>   access MFNs reported in previous step until they are requested?

No - I am not suggesting this.

>
> 2. Who initiates the requests? If it's the libnvdimm driver, that
>   means we still need to introduce Xen specific code to the driver.
>
>   Or the requests are issued by OSPM (or the Xen component you
>   mentioned above) when they probe new dimms?
>
>   For the latter, Dan, do you think it's acceptable in NFIT code to
>   call the Xen component to request the access permission of the pmem
>   regions, e.g. in apic_nfit_insert_resource(). Of course, it's only
>   used for Dom0 case.

The libnvdimm driver should continue to use ioremap() or whatever it
currently does.  There shouldn't be Xen modifications like that.

The one issue will come if libnvdimm tries to ioremap()/other an area
which Xen is unaware is an NVDIMM, and rejects the mapping request. 
Somehow, a Xen component will need to find the MFN/SPA layout and
register this information with Xen, before the ioremap() call made by
the libnvdimm driver.  Perhaps a notifier mechanism out from the ACPI
subsystem might be the best way to make this work in a clean way.

>
>> ** This should work, as Xen is aware that there is something there to be
>> mapped (rather than just empty physical address space).
>> * Dom0 finds that some NVDIMM ranges are now available for use (probably
>> modelled as hotplug events).
>> * /dev/pmem $STUFF starts happening as normal.
>>
>> At some pointer later after dom0 policy decisions are made (ultimately,
>> by the host administrator):
>> * If an area of NVDIMM is chosen for Xen to use, Dom0 needs to inform
>> Xen of the SPA/MFN regions which are safe to use.
>> * Xen then incorporates these regions into its idea of RAM, and starts
>> using them for whatever.
>>
>
> Agree. I think we may not need to fix the way/format/... to make the
> reservation, and instead let the users (host administrators), who have
> better understanding of their data, make the proper decision.

Yes.  This is the best course of action.

>
> In a worse case that no reservation is made, Xen hypervisor could turn
> to use RAM for management structures for NVDIMM, with the cost of less
> RAM for guests.

Or simply not manage the NVDIMM at all.

OTOH, a different usecase might be to register a small area for Xen to
use to crash log into.

~Andrew

[toc] | [prev] | [next] | [standalone]

#1504684 — Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen

On 10/14/16 13:18 +0100, Andrew Cooper wrote:
>On 14/10/16 08:08, Haozhong Zhang wrote:
>> On 10/13/16 20:33 +0100, Andrew Cooper wrote:
>>> On 13/10/16 19:59, Dan Williams wrote:
>>>> On Thu, Oct 13, 2016 at 9:01 AM, Andrew Cooper
>>>> <andrew.cooper3@citrix.com> wrote:
>>>>> On 13/10/16 16:40, Dan Williams wrote:
>>>>>> On Thu, Oct 13, 2016 at 2:08 AM, Jan Beulich <JBeulich@suse.com>
>>>>>> wrote:
>>>>>> [..]
>>>>>>>> I think we can do the similar for Xen, like to lay another pseudo
>>>>>>>> device on /dev/pmem and do the reservation, like 2. in my previous
>>>>>>>> reply.
>>>>>>> Well, my opinion certainly doesn't count much here, but I
>>>>>>> continue to
>>>>>>> consider this a bad idea. For entities like drivers it may well be
>>>>>>> appropriate, but I think there ought to be an independent concept
>>>>>>> of "OS reserved", and in the Xen case this could then be shared
>>>>>>> between hypervisor and Dom0 kernel. Or if we were to consider Dom0
>>>>>>> "just a guest", things should even be the other way around: Xen gets
>>>>>>> all of the OS reserved space, and Dom0 needs something custom.
>>>>>> You haven't made the case why Xen is special and other
>>>>>> applications of
>>>>>> persistent memory are not.
>>>>> In a Xen system, Xen runs in the baremetal root-mode ring0, and
>>>>> dom0 is
>>>>> a VM running in ring1/3 with the nvdimm driver.  This is the opposite
>>>>> way around to the KVM model.
>>>>>
>>>>> Dom0, being the hardware domain, has default ownership of all the
>>>>> hardware, but to gain access in the first place, it must request a
>>>>> mapping from Xen.
>>>> This is where my understanding the Xen model breaks down.  Are you
>>>> saying dom0 can't access the persistent memory range unless the ring0
>>>> agent has metadata storage space for tracking what it maps into dom0?
>>>
>>> No.  I am trying to point out that the current suggestion wont work, and
>>> needs re-designing.
>>>
>>> Xen *must* be able to properly configure mappings of the NVDIMM for
>>> dom0, *without* modifying any content on the NVDIMM.  Otherwise, data
>>> corruption will occur.
>>>
>>> Whether this means no Xen metadata, or the metadata living elsewhere in
>>> regular ram, such as the main frametable, is an implementation detail.
>>>
>>>>
>>>>> Once dom0 has a mapping of the nvdimm, the nvdimm driver can go to
>>>>> work
>>>>> and figure out what is on the DIMM, and which areas are safe to use.
>>>> I don't understand this ordering of events.  Dom0 needs to have a
>>>> mapping to even write the on-media structure to indicate a
>>>> reservation.  So, initial dom0 access can't depend on metadata
>>>> reservation already being present.
>>>
>>> I agree.
>>>
>>> Overall, I think the following is needed.
>>>
>>> * Xen starts up.
>>> ** Xen might find some NVDIMM SPA/MFN ranges in the NFIT table, and
>>> needs to note this information somehow.
>>> ** Xen might find some Type 7 E820 regions, and needs to note this
>>> information somehow.
>>
>> IIUC, this is to collect MFNs and no need to create frame table and
>> M2P at this stage. If so, what is different from ...
>>
>>> * Xen starts dom0.
>>> * Once OSPM is running, a Xen component in Linux needs to collect and
>>> report all NVDIMM SPA/MFN regions it knowns about.
>>> ** This covers the AML-only case, and the hotplug case.
>>
>> ... the MFNs reported here, especially that the former is a subset
>> (hotplug ones not included in the former) of latter.
>
>Hopefully nothing.  However, Xen shouldn't exclusively rely on the dom0
>when it is capable of working things out itself, (which can aid with
>debugging one half of this arrangement).  Also, the MFNS found by Xen
>alone can be present in the default memory map for dom0.
>

Sure, I'll add code to parsing NFIT in Xen to discover statically
plugged pmem mode NVDIMM and their MFNs.

By the default memory map for dom0, do you mean making
XENMEM_memory_map returns above MFNs in Dom0 E820?

>>
>> (There is no E820 hole or SRAT entries to tell which address range is
>> reserved for hotplugged NVDIMM)
>>
>>> * Dom0 requests a mapping of the NVDIMMs via the usual mechanism.
>>
>> Two questions:
>> 1. Why is this request necessary? Even without such requests like what
>>   my current implementation, Dom0 can still access NVDIMM.
>
>Can it?  (if so, great, but I don't think this holds in the general
>case.)  Is that a side effect of the NVDIMM being covered by a hole in
>the E820?

In my development environment, NVDIMM MFNs are not covered by any E820
entry and appear after RAM MFNs.

Can you explain more about this point? Why can it work if covered by
E820 hole?

>The current logic for what dom0 may access by default is
>somewhat ad-hoc, and I have a gut feeling that it won't work with E820
>type 7 regions.
>
>>
>>   Or do you mean Xen hypervisor should by default disallow Dom0 to
>>   access MFNs reported in previous step until they are requested?
>
>No - I am not suggesting this.
>
>>
>> 2. Who initiates the requests? If it's the libnvdimm driver, that
>>   means we still need to introduce Xen specific code to the driver.
>>
>>   Or the requests are issued by OSPM (or the Xen component you
>>   mentioned above) when they probe new dimms?
>>
>>   For the latter, Dan, do you think it's acceptable in NFIT code to
>>   call the Xen component to request the access permission of the pmem
>>   regions, e.g. in apic_nfit_insert_resource(). Of course, it's only
>>   used for Dom0 case.
>
>The libnvdimm driver should continue to use ioremap() or whatever it
>currently does.  There shouldn't be Xen modifications like that.
>
>The one issue will come if libnvdimm tries to ioremap()/other an area
>which Xen is unaware is an NVDIMM, and rejects the mapping request.
>Somehow, a Xen component will need to find the MFN/SPA layout and
>register this information with Xen, before the ioremap() call made by
>the libnvdimm driver.  Perhaps a notifier mechanism out from the ACPI
>subsystem might be the best way to make this work in a clean way.
>

Yes, this is necessary for hotplugged NVDIMM.

>>
>>> ** This should work, as Xen is aware that there is something there to be
>>> mapped (rather than just empty physical address space).
>>> * Dom0 finds that some NVDIMM ranges are now available for use (probably
>>> modelled as hotplug events).
>>> * /dev/pmem $STUFF starts happening as normal.
>>>
>>> At some pointer later after dom0 policy decisions are made (ultimately,
>>> by the host administrator):
>>> * If an area of NVDIMM is chosen for Xen to use, Dom0 needs to inform
>>> Xen of the SPA/MFN regions which are safe to use.
>>> * Xen then incorporates these regions into its idea of RAM, and starts
>>> using them for whatever.
>>>
>>
>> Agree. I think we may not need to fix the way/format/... to make the
>> reservation, and instead let the users (host administrators), who have
>> better understanding of their data, make the proper decision.
>
>Yes.  This is the best course of action.
>
>>
>> In a worse case that no reservation is made, Xen hypervisor could turn
>> to use RAM for management structures for NVDIMM, with the cost of less
>> RAM for guests.
>
>Or simply not manage the NVDIMM at all.
>
>OTOH, a different usecase might be to register a small area for Xen to
>use to crash log into.
>

an interesting usage, but I'd like to put it in the future work.

Thanks,
Haozhong

[toc] | [prev] | [next] | [standalone]

#1505286 — Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen

On 20/10/2016 10:14, Haozhong Zhang wrote:
>
>>>>>
>>>>>> Once dom0 has a mapping of the nvdimm, the nvdimm driver can go to
>>>>>> work
>>>>>> and figure out what is on the DIMM, and which areas are safe to use.
>>>>> I don't understand this ordering of events.  Dom0 needs to have a
>>>>> mapping to even write the on-media structure to indicate a
>>>>> reservation.  So, initial dom0 access can't depend on metadata
>>>>> reservation already being present.
>>>>
>>>> I agree.
>>>>
>>>> Overall, I think the following is needed.
>>>>
>>>> * Xen starts up.
>>>> ** Xen might find some NVDIMM SPA/MFN ranges in the NFIT table, and
>>>> needs to note this information somehow.
>>>> ** Xen might find some Type 7 E820 regions, and needs to note this
>>>> information somehow.
>>>
>>> IIUC, this is to collect MFNs and no need to create frame table and
>>> M2P at this stage. If so, what is different from ...
>>>
>>>> * Xen starts dom0.
>>>> * Once OSPM is running, a Xen component in Linux needs to collect and
>>>> report all NVDIMM SPA/MFN regions it knowns about.
>>>> ** This covers the AML-only case, and the hotplug case.
>>>
>>> ... the MFNs reported here, especially that the former is a subset
>>> (hotplug ones not included in the former) of latter.
>>
>> Hopefully nothing.  However, Xen shouldn't exclusively rely on the dom0
>> when it is capable of working things out itself, (which can aid with
>> debugging one half of this arrangement).  Also, the MFNS found by Xen
>> alone can be present in the default memory map for dom0.
>>
>
> Sure, I'll add code to parsing NFIT in Xen to discover statically
> plugged pmem mode NVDIMM and their MFNs.
>
> By the default memory map for dom0, do you mean making
> XENMEM_memory_map returns above MFNs in Dom0 E820?

Potentially, yes.  Particularly if type 7 is reserved for NVDIMM, it
would be good to report this information properly.

>
>>>
>>> (There is no E820 hole or SRAT entries to tell which address range is
>>> reserved for hotplugged NVDIMM)
>>>
>>>> * Dom0 requests a mapping of the NVDIMMs via the usual mechanism.
>>>
>>> Two questions:
>>> 1. Why is this request necessary? Even without such requests like what
>>>   my current implementation, Dom0 can still access NVDIMM.
>>
>> Can it?  (if so, great, but I don't think this holds in the general
>> case.)  Is that a side effect of the NVDIMM being covered by a hole in
>> the E820?
>
> In my development environment, NVDIMM MFNs are not covered by any E820
> entry and appear after RAM MFNs.
>
> Can you explain more about this point? Why can it work if covered by
> E820 hole?

It is a question, not a statement.  If things currently work fine then
great.  However,  there does seem to be a lot of flexibility in how the
regions are reported, so please be mindful to this when developing the code.

>
>>
>>>
>>> 2. Who initiates the requests? If it's the libnvdimm driver, that
>>>   means we still need to introduce Xen specific code to the driver.
>>>
>>>   Or the requests are issued by OSPM (or the Xen component you
>>>   mentioned above) when they probe new dimms?
>>>
>>>   For the latter, Dan, do you think it's acceptable in NFIT code to
>>>   call the Xen component to request the access permission of the pmem
>>>   regions, e.g. in apic_nfit_insert_resource(). Of course, it's only
>>>   used for Dom0 case.
>>
>> The libnvdimm driver should continue to use ioremap() or whatever it
>> currently does.  There shouldn't be Xen modifications like that.
>>
>> The one issue will come if libnvdimm tries to ioremap()/other an area
>> which Xen is unaware is an NVDIMM, and rejects the mapping request.
>> Somehow, a Xen component will need to find the MFN/SPA layout and
>> register this information with Xen, before the ioremap() call made by
>> the libnvdimm driver.  Perhaps a notifier mechanism out from the ACPI
>> subsystem might be the best way to make this work in a clean way.
>>
>
> Yes, this is necessary for hotplugged NVDIMM.

Ok.

>
>>>
>>>> ** This should work, as Xen is aware that there is something there
>>>> to be
>>>> mapped (rather than just empty physical address space).
>>>> * Dom0 finds that some NVDIMM ranges are now available for use
>>>> (probably
>>>> modelled as hotplug events).
>>>> * /dev/pmem $STUFF starts happening as normal.
>>>>
>>>> At some pointer later after dom0 policy decisions are made
>>>> (ultimately,
>>>> by the host administrator):
>>>> * If an area of NVDIMM is chosen for Xen to use, Dom0 needs to inform
>>>> Xen of the SPA/MFN regions which are safe to use.
>>>> * Xen then incorporates these regions into its idea of RAM, and starts
>>>> using them for whatever.
>>>>
>>>
>>> Agree. I think we may not need to fix the way/format/... to make the
>>> reservation, and instead let the users (host administrators), who have
>>> better understanding of their data, make the proper decision.
>>
>> Yes.  This is the best course of action.
>>
>>>
>>> In a worse case that no reservation is made, Xen hypervisor could turn
>>> to use RAM for management structures for NVDIMM, with the cost of less
>>> RAM for guests.
>>
>> Or simply not manage the NVDIMM at all.
>>
>> OTOH, a different usecase might be to register a small area for Xen to
>> use to crash log into.
>>
>
> an interesting usage, but I'd like to put it in the future work.

Absolutely.  I didn't wish to suggest implementing this now.  It was
just pointing out an alternative usecase.

Leaving this for future work will be perfectly fine.

~Andrew

[toc] | [prev] | [next] | [standalone]

#1500817

>>> On 13.10.16 at 17:40, <dan.j.williams@intel.com> wrote:
> On Thu, Oct 13, 2016 at 2:08 AM, Jan Beulich <JBeulich@suse.com> wrote:
> [..]
>>> I think we can do the similar for Xen, like to lay another pseudo
>>> device on /dev/pmem and do the reservation, like 2. in my previous
>>> reply.
>>
>> Well, my opinion certainly doesn't count much here, but I continue to
>> consider this a bad idea. For entities like drivers it may well be
>> appropriate, but I think there ought to be an independent concept
>> of "OS reserved", and in the Xen case this could then be shared
>> between hypervisor and Dom0 kernel. Or if we were to consider Dom0
>> "just a guest", things should even be the other way around: Xen gets
>> all of the OS reserved space, and Dom0 needs something custom.
> 
> You haven't made the case why Xen is special and other applications of
> persistent memory are not.

Well, I'm implying this from there being a special Linux reservation.
Xen (as explained by Andrew) sitting underneath the Dom0 kernel
(other than ...

>  The current struct page reservation
> supports fundamental address-ability of persistent memory namespaces
> for the rest of the kernel.  The Xen reservation is application
> specific.  XFS, EXT4, and DM also have application specific usages of
> persistent memory and consume metadata space out of a block device. If
> we don't need an XFS-mode nvdimm device, why do we need Xen-mode?

... all the examples you give) by implication is special then too. If
you made the kernel be no different than the other examples you
give, Xen probably shouldn't be any different anymore either.

Jan

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web