Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1498046 > unrolled thread
| Started by | Haozhong Zhang <haozhong.zhang@intel.com> |
|---|---|
| First post | 2016-10-10 02:40 +0200 |
| Last post | 2016-10-12 09:30 +0200 |
| Articles | 20 on this page of 46 — 6 participants |
Back to article view | Back to linux.kernel
[RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Haozhong Zhang <haozhong.zhang@intel.com> - 2016-10-10 02:40 +0200
Re: [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Dan Williams <dan.j.williams@intel.com> - 2016-10-10 05:50 +0200
Re: [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Haozhong Zhang <haozhong.zhang@intel.com> - 2016-10-10 08:40 +0200
Re: [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Dan Williams <dan.j.williams@intel.com> - 2016-10-10 18:30 +0200
Re: [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Haozhong Zhang <haozhong.zhang@intel.com> - 2016-10-11 09:20 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Andrew Cooper <andrew.cooper3@citrix.com> - 2016-10-10 18:50 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Haozhong Zhang <haozhong.zhang@intel.com> - 2016-10-11 08:00 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> - 2016-10-11 20:50 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> - 2016-10-11 21:00 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Andrew Cooper <andrew.cooper3@citrix.com> - 2016-10-11 21:00 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen "Jan Beulich" <jbeulich@suse.com> - 2016-10-11 15:10 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Dan Williams <dan.j.williams@intel.com> - 2016-10-11 18:00 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> - 2016-10-11 19:00 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Dan Williams <dan.j.williams@intel.com> - 2016-10-11 20:00 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Andrew Cooper <andrew.cooper3@citrix.com> - 2016-10-11 20:30 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> - 2016-10-11 20:50 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> - 2016-10-11 21:50 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> - 2016-10-11 20:40 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Dan Williams <dan.j.williams@intel.com> - 2016-10-11 21:30 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> - 2016-10-11 22:00 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Andrew Cooper <andrew.cooper3@citrix.com> - 2016-10-11 22:20 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Dan Williams <dan.j.williams@intel.com> - 2016-10-11 22:20 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Haozhong Zhang <haozhong.zhang@intel.com> - 2016-10-12 12:40 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen "Jan Beulich" <JBeulich@suse.com> - 2016-10-12 13:40 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Haozhong Zhang <haozhong.zhang@intel.com> - 2016-10-12 17:10 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen "Jan Beulich" <JBeulich@suse.com> - 2016-10-12 17:40 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Dan Williams <dan.j.williams@intel.com> - 2016-10-12 17:50 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen "Jan Beulich" <JBeulich@suse.com> - 2016-10-12 18:30 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Dan Williams <dan.j.williams@intel.com> - 2016-10-12 18:30 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen "Jan Beulich" <JBeulich@suse.com> - 2016-10-13 10:40 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Haozhong Zhang <haozhong.zhang@intel.com> - 2016-10-13 11:00 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Haozhong Zhang <haozhong.zhang@intel.com> - 2016-10-13 11:10 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen "Jan Beulich" <JBeulich@suse.com> - 2016-10-13 11:10 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Haozhong Zhang <haozhong.zhang@intel.com> - 2016-10-13 17:50 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen "Jan Beulich" <JBeulich@suse.com> - 2016-10-14 12:20 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Haozhong Zhang <haozhong.zhang@intel.com> - 2016-10-20 11:20 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Dan Williams <dan.j.williams@intel.com> - 2016-10-13 17:50 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Andrew Cooper <andrew.cooper3@citrix.com> - 2016-10-13 18:10 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Dan Williams <dan.j.williams@intel.com> - 2016-10-13 21:10 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Andrew Cooper <andrew.cooper3@citrix.com> - 2016-10-13 21:40 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Haozhong Zhang <haozhong.zhang@intel.com> - 2016-10-14 09:10 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Andrew Cooper <andrew.cooper3@citrix.com> - 2016-10-14 14:20 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Haozhong Zhang <haozhong.zhang@intel.com> - 2016-10-20 11:20 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen Andrew Cooper <andrew.cooper3@citrix.com> - 2016-10-21 00:10 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen "Jan Beulich" <JBeulich@suse.com> - 2016-10-14 12:10 +0200
Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen "Jan Beulich" <JBeulich@suse.com> - 2016-10-12 09:30 +0200
Page 2 of 3 — ← Prev page 1 [2] 3 Next page →
| From | Andrew Cooper <andrew.cooper3@citrix.com> |
|---|---|
| Date | 2016-10-11 22:20 +0200 |
| Subject | Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen |
| Message-ID | <srbGN-2Nu-11@gated-at.bofh.it> |
| In reply to | #1499177 |
On 11/10/16 20:48, Konrad Rzeszutek Wilk wrote: > On Tue, Oct 11, 2016 at 12:28:56PM -0700, Dan Williams wrote: >> On Tue, Oct 11, 2016 at 11:33 AM, Konrad Rzeszutek Wilk >> <konrad.wilk@oracle.com> wrote: >>> On Tue, Oct 11, 2016 at 10:51:19AM -0700, Dan Williams wrote: >> [..] >>>> Right, but why does the libnvdimm core need to know about this >>>> specific Xen reservation? For example, if Xen wants some in-kernel >>> Let me turn this around - why does the libnvdimm core need to know about >>> Linux specific parts? Shouldn't this be OS agnostic, so that FreeBSD >>> for example can also poke a hole in this and fill it with its >>> OS-management meta-data? >> Specifically the core needs to know so that it can answer the Linux >> specific question of whether the pfn returned by ->direct_access() has >> a corresponding struct page or not. It's tied to the lifetime of the >> device and the usage of the reservation needs to be coordinated >> against the references of those pages. If FreeBSD decides it needs to >> reserve "struct page" capacity at the start of the device, I would >> hope that it reuses the same on-device info block that Linux is using >> and not create a new "FreeBSD-mode" device type. > The issue here (as I understand, I may be missing something new) > is that the size of this special namespace may be different. That is > the 'struct page' on FreeBSD could be 256 bytes while on Linux it is > 64 bytes (numbers pulled out of the sky). > > Hence one would have to expand or such to re-use this. >> To be honest I do not yet understand what metadata Xen wants to store >> in the device, but it seems the producer and consumer of that metadata >> is Xen itself and not the wider Linux kernel as is the case with >> struct page. Can you fill me in on what problem Xen solves with this > Exactly! >> reservation? > The same as Linux - its variant of 'struct page'. Which I think is > smaller than the Linux one, but perhaps it is not? There is still a bootstrapping issue though, which looks (in its current form) to cause data corruption. I hope I am mistaken, and apologies if I am, but clearly we cannot build a solution that has data corruption in anything other than an exceptional circumstance. So far, the sequence of boot operations appears to look like this: Xen boots, and may find some NVDIMM SPA/MFN ranges via the NFIT table. Any ranges available only from AML need dynamically reporting back to Xen at a later point, once OSPM is up and running. The NVDIMMs must be mappable by dom0 so the contents can be inspected and deemed to be safe by the nvdimm driver/host admin, before Xen starts writing to any of it (for whatever reason). If this isn't the case, then simply booting a Xen/dom0 combo will end up corrupting a region before working out that it is safe to do so. ~Andrew
[toc] | [prev] | [next] | [standalone]
| From | Dan Williams <dan.j.williams@intel.com> |
|---|---|
| Date | 2016-10-11 22:20 +0200 |
| Subject | Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen |
| Message-ID | <srbGN-2Nu-1@gated-at.bofh.it> |
| In reply to | #1499177 |
On Tue, Oct 11, 2016 at 12:48 PM, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote: > On Tue, Oct 11, 2016 at 12:28:56PM -0700, Dan Williams wrote: >> On Tue, Oct 11, 2016 at 11:33 AM, Konrad Rzeszutek Wilk >> <konrad.wilk@oracle.com> wrote: >> > On Tue, Oct 11, 2016 at 10:51:19AM -0700, Dan Williams wrote: >> [..] >> >> Right, but why does the libnvdimm core need to know about this >> >> specific Xen reservation? For example, if Xen wants some in-kernel >> > >> > Let me turn this around - why does the libnvdimm core need to know about >> > Linux specific parts? Shouldn't this be OS agnostic, so that FreeBSD >> > for example can also poke a hole in this and fill it with its >> > OS-management meta-data? >> >> Specifically the core needs to know so that it can answer the Linux >> specific question of whether the pfn returned by ->direct_access() has >> a corresponding struct page or not. It's tied to the lifetime of the >> device and the usage of the reservation needs to be coordinated >> against the references of those pages. If FreeBSD decides it needs to >> reserve "struct page" capacity at the start of the device, I would >> hope that it reuses the same on-device info block that Linux is using >> and not create a new "FreeBSD-mode" device type. > > The issue here (as I understand, I may be missing something new) > is that the size of this special namespace may be different. That is > the 'struct page' on FreeBSD could be 256 bytes while on Linux it is > 64 bytes (numbers pulled out of the sky). > > Hence one would have to expand or such to re-use this. Sure, but we could support that today. If FreeBSD lays down the info block it is free to make a bigger reservation and Linux would be happy to use a smaller subset. If we, as an industry, want this "struct page" reservation to be common we can take it to a standards body to make as a cross-OS guarantee... but I think this is separate from the Xen reservation. >> To be honest I do not yet understand what metadata Xen wants to store >> in the device, but it seems the producer and consumer of that metadata >> is Xen itself and not the wider Linux kernel as is the case with >> struct page. Can you fill me in on what problem Xen solves with this > > Exactly! >> reservation? > > The same as Linux - its variant of 'struct page'. Which I think is > smaller than the Linux one, but perhaps it is not? > If the hypervisor needs to know where it can store some metadata, can that be satisfied with userspace tooling in Dom0? Something like, "/dev/pmem0p1 == Xen metadata" and "/dev/pmem0p2 == DAX filesystem with files to hand to guests". So my question is not about the rationale for having metadata, it's why does the Linux kernel need to know about the Xen reservation? As far as I can see it is independent / opaque to the kernel.
[toc] | [prev] | [next] | [standalone]
| From | Haozhong Zhang <haozhong.zhang@intel.com> |
|---|---|
| Date | 2016-10-12 12:40 +0200 |
| Subject | Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen |
| Message-ID | <srp73-3aC-9@gated-at.bofh.it> |
| In reply to | #1499185 |
On 10/11/16 13:17 -0700, Dan Williams wrote:
>On Tue, Oct 11, 2016 at 12:48 PM, Konrad Rzeszutek Wilk
><konrad.wilk@oracle.com> wrote:
>> On Tue, Oct 11, 2016 at 12:28:56PM -0700, Dan Williams wrote:
>>> On Tue, Oct 11, 2016 at 11:33 AM, Konrad Rzeszutek Wilk
>>> <konrad.wilk@oracle.com> wrote:
>>> > On Tue, Oct 11, 2016 at 10:51:19AM -0700, Dan Williams wrote:
>>> [..]
>>> >> Right, but why does the libnvdimm core need to know about this
>>> >> specific Xen reservation? For example, if Xen wants some in-kernel
>>> >
>>> > Let me turn this around - why does the libnvdimm core need to know about
>>> > Linux specific parts? Shouldn't this be OS agnostic, so that FreeBSD
>>> > for example can also poke a hole in this and fill it with its
>>> > OS-management meta-data?
>>>
>>> Specifically the core needs to know so that it can answer the Linux
>>> specific question of whether the pfn returned by ->direct_access() has
>>> a corresponding struct page or not. It's tied to the lifetime of the
>>> device and the usage of the reservation needs to be coordinated
>>> against the references of those pages. If FreeBSD decides it needs to
>>> reserve "struct page" capacity at the start of the device, I would
>>> hope that it reuses the same on-device info block that Linux is using
>>> and not create a new "FreeBSD-mode" device type.
>>
>> The issue here (as I understand, I may be missing something new)
>> is that the size of this special namespace may be different. That is
>> the 'struct page' on FreeBSD could be 256 bytes while on Linux it is
>> 64 bytes (numbers pulled out of the sky).
>>
>> Hence one would have to expand or such to re-use this.
>
>Sure, but we could support that today. If FreeBSD lays down the info
>block it is free to make a bigger reservation and Linux would be happy
>to use a smaller subset. If we, as an industry, want this "struct
>page" reservation to be common we can take it to a standards body to
>make as a cross-OS guarantee... but I think this is separate from the
>Xen reservation.
>
>>> To be honest I do not yet understand what metadata Xen wants to store
>>> in the device, but it seems the producer and consumer of that metadata
>>> is Xen itself and not the wider Linux kernel as is the case with
>>> struct page. Can you fill me in on what problem Xen solves with this
>>
>> Exactly!
>>> reservation?
>>
>> The same as Linux - its variant of 'struct page'. Which I think is
>> smaller than the Linux one, but perhaps it is not?
>>
>
>If the hypervisor needs to know where it can store some metadata, can
>that be satisfied with userspace tooling in Dom0? Something like,
>"/dev/pmem0p1 == Xen metadata" and "/dev/pmem0p2 == DAX filesystem
>with files to hand to guests". So my question is not about the
>rationale for having metadata, it's why does the Linux kernel need to
>know about the Xen reservation? As far as I can see it is independent
>/ opaque to the kernel.
Thank everyone for all these comments!
How about doing the reservation in the following way:
1. Create partition(s) on /dev/pmemX and make sure space besides the
partition table and potential padding before the first partition is
large enough to hold Xen's management structures and a super block
introduced in step 2. The space besides the partition table,
padding and the super block will be used as the reserved area.
2. Write a super block before above reserved area. The super block
records the base address and the size of the reserved area. It also
contains a signature and a checksum to identify itself.
The layout is shown as the following diagram.
+---------------+-----------+-------+----------+--------------+
| whatever used | Partition | Super | Reserved | /dev/pmem0p1 |
| by kernel | Table | Block | for Xen | |
+---------------+-----------+-------+----------+--------------+
\_____________________ _______________________/
V
/dev/pmem0
Above two steps can be done via a userspace program and do not need
Xen hypervisor running. The partitions on the device can be used
regardless of the existence of Xen hypervisor.
3. When Xen is running, implement a function in Dom0 Linux xen driver
(drivers/xen/) to response to udevd events that notify the
detection of the pmem regions.
This function searches on the pmem region for the super block
created in step 2. If one is found, it will know this pmem region
has been prepared for Xen usage.
Then it gets the base address and size of the reserved area (from
super block) and the entire address ranges of the pmem region (from
pmem driver), and reports them to Xen hypervisor.
The implementation of this step can be completely included in the
kernel Xen driver. (It may also be implemented as a udevd service in
userspace, if it's not considered as unsafe)
Thanks,
Haozhong
[toc] | [prev] | [next] | [standalone]
| From | "Jan Beulich" <JBeulich@suse.com> |
|---|---|
| Date | 2016-10-12 13:40 +0200 |
| Subject | Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen |
| Message-ID | <srq37-3LI-3@gated-at.bofh.it> |
| In reply to | #1499474 |
>>> On 12.10.16 at 12:33, <haozhong.zhang@intel.com> wrote: > The layout is shown as the following diagram. > > +---------------+-----------+-------+----------+--------------+ > | whatever used | Partition | Super | Reserved | /dev/pmem0p1 | > | by kernel | Table | Block | for Xen | | > +---------------+-----------+-------+----------+--------------+ > \_____________________ _______________________/ > V > /dev/pmem0 I have to admit that I dislike this, for not being OS-agnostic. Neither should there be any Xen-specific region, nor should the "whatever used by kernel" one be restricted to just Linux. What I could see is an OS-reserved area ahead of the partition table, the exact usage of which depends on which OS is currently running (and in the Xen case this might be both Xen _and_ the Dom0 kernel, arbitrated by a tbd protocol). After all, when running under Xen, the Dom0 may not have a need for as much control data as it has when running on bare hardware, for it controlling less (if any) of the actual memory ranges when Xen is present. The assumption of course is that the reserved area holds no persistent data. If that assumption didn't hold, you'd have to have per-OS reserved areas anyway (as many of them as there might be OSes [planned to get] installed on a particular system). Jan
[toc] | [prev] | [next] | [standalone]
| From | Haozhong Zhang <haozhong.zhang@intel.com> |
|---|---|
| Date | 2016-10-12 17:10 +0200 |
| Subject | Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen |
| Message-ID | <srtkl-688-5@gated-at.bofh.it> |
| In reply to | #1499495 |
On 10/12/16 05:32 -0600, Jan Beulich wrote:
>>>> On 12.10.16 at 12:33, <haozhong.zhang@intel.com> wrote:
>> The layout is shown as the following diagram.
>>
>> +---------------+-----------+-------+----------+--------------+
>> | whatever used | Partition | Super | Reserved | /dev/pmem0p1 |
>> | by kernel | Table | Block | for Xen | |
>> +---------------+-----------+-------+----------+--------------+
>> \_____________________ _______________________/
>> V
>> /dev/pmem0
>
>I have to admit that I dislike this, for not being OS-agnostic.
>Neither should there be any Xen-specific region, nor should the
>"whatever used by kernel" one be restricted to just Linux. What
>I could see is an OS-reserved area ahead of the partition table,
>the exact usage of which depends on which OS is currently
>running (and in the Xen case this might be both Xen _and_ the
>Dom0 kernel, arbitrated by a tbd protocol). After all, when
>running under Xen, the Dom0 may not have a need for as much
>control data as it has when running on bare hardware, for it
>controlling less (if any) of the actual memory ranges when Xen
>is present.
>
Isn't this OS-reserved area still not OS-agnostic, as it requires OS
to know where the reserved area is? Or do you mean it's not if it's
defined by a protocol that is accepted by all OSes?
Let me list another two methods just coming to my mind.
1. The first method extends the usage of the super block used by
current Linux kernel to reserve space on pmem.
Current Linux kernel places a super block of the following
structure near the beginning of a pmem namespace.
struct nd_pfn_sb {
u8 signature[PFN_SIG_LEN];
u8 uuid[16];
u8 parent_uuid[16];
__le32 flags;
__le16 version_major;
__le16 version_minor;
__le64 dataoff; /* relative to namespace_base + start_pad */
__le64 npfns;
__le32 mode;
/* minor-version-1 additions for section alignment */
__le32 start_pad;
__le32 end_trunc;
/* minor-version-2 record the base alignment of the mapping */
__le32 align;
u8 padding[4000];
__le64 checksum;
}
Two interesting fields here are 'dataoff' and 'mode':
- 'dataoff' indicates the offset where the data area starts,
ie. IIUC, the part that can be accessed via /dev/pmemN or
/dev/daxN.
- 'mode' indicates whether Linux puts struct page for this
namespace in the ram (= PFN_MODE_RAM) or on the device (=
PFN_MODE_PMEM).
Currently for Linux, only 'mode' is customizable, while 'dataoff'
is not. If mode == PFN_MODE_RAM, no reservation for struct page is
made on the device, and dataoff starts almost immediately after
the super block except a small reserved area in between for other
structures and alignment. If mode == PFN_MODE_PMEM, the size of
the reservation is decided by kernel, i.e. 64 bytes per struct
page.
I propose to make the size of the reserved area customizable,
e.g. via ioctl and ndctl.
- If mode == PFN_MODE_PMEM and
* if the given reserved size is large enough to hold what an OS
(not limited to Linux) wants to put in, then the OS just
starts use it as desired;
* if the given reserved size is not enough, then the OS reports
error and may take other fallback actions.
- If mode == PFN_MODE_RAM and
* if the reserved size is zero, then it's the current way that
Linux uses the device;
* if the reserved size is non-zero, I would like to reserve this
case for hypervisor (right now, namely Xen hypervisor)
usage. That is, the OS should not use the reserved area. For
Xen, we could add a function in xen driver in kernel to report
the reserved area to hypervisor.
I guess this might be the OS-agnostic way Jan expects, but Dan may
object to.
2. Lay another pseudo device on the block device (e.g. /dev/pmemN)
provided by the NVDIMM driver.
This pseudo device can reserve the size according to user's
requirement. The reservation information can be persistently
recorded in a super block before the reserved area.
This pseudo device also implements another pseudo block device to
allow the non-reserved area be accessed as a block device (we can
even implement it as DAX-capable).
pseudo block device
/---------^-----------\
+------------------+-------+---------------+-----------------------+
| whatever used | Super | reserved by | |
| by NVDIMM driver | Block | pseudo device | |
+------------------+-------+---------------+-----------------------+
\_____________________ _______________________/
V
/dev/pmem0
(provided by NVDIMM driver)
In order to make it work across difference OSes, it requires other
OS recognizes the same types of pmem block devices made by Linux,
and implements the driver for the pseudo device.
This is inspired by Dan's reply at
https://lists.xenproject.org/archives/html/xen-devel/2016-10/msg00651.html.
However, it's essentially the same as my partition solution, so I guess
Jan will still dislike.
Any comments?
>The assumption of course is that the reserved area holds no
>persistent data. If that assumption didn't hold, you'd have to
>have per-OS reserved areas anyway (as many of them as
>there might be OSes [planned to get] installed on a particular
>system).
>
No persistent data should be placed in the reserved area.
Thanks,
Haozhong
[toc] | [prev] | [next] | [standalone]
| From | "Jan Beulich" <JBeulich@suse.com> |
|---|---|
| Date | 2016-10-12 17:40 +0200 |
| Subject | Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen |
| Message-ID | <srtNo-6jC-13@gated-at.bofh.it> |
| In reply to | #1499750 |
>>> On 12.10.16 at 16:58, <haozhong.zhang@intel.com> wrote: > On 10/12/16 05:32 -0600, Jan Beulich wrote: >>>>> On 12.10.16 at 12:33, <haozhong.zhang@intel.com> wrote: >>> The layout is shown as the following diagram. >>> >>> +---------------+-----------+-------+----------+--------------+ >>> | whatever used | Partition | Super | Reserved | /dev/pmem0p1 | >>> | by kernel | Table | Block | for Xen | | >>> +---------------+-----------+-------+----------+--------------+ >>> \_____________________ _______________________/ >>> V >>> /dev/pmem0 >> >>I have to admit that I dislike this, for not being OS-agnostic. >>Neither should there be any Xen-specific region, nor should the >>"whatever used by kernel" one be restricted to just Linux. What >>I could see is an OS-reserved area ahead of the partition table, >>the exact usage of which depends on which OS is currently >>running (and in the Xen case this might be both Xen _and_ the >>Dom0 kernel, arbitrated by a tbd protocol). After all, when >>running under Xen, the Dom0 may not have a need for as much >>control data as it has when running on bare hardware, for it >>controlling less (if any) of the actual memory ranges when Xen >>is present. >> > > Isn't this OS-reserved area still not OS-agnostic, as it requires OS > to know where the reserved area is? Or do you mean it's not if it's > defined by a protocol that is accepted by all OSes? The latter - we clearly won't get away without some agreement on where to retrieve position and size of this area. I was simply assuming that such a protocol already exists. Jan
[toc] | [prev] | [next] | [standalone]
| From | Dan Williams <dan.j.williams@intel.com> |
|---|---|
| Date | 2016-10-12 17:50 +0200 |
| Subject | Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen |
| Message-ID | <srtX3-6nc-7@gated-at.bofh.it> |
| In reply to | #1499779 |
On Wed, Oct 12, 2016 at 8:39 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>> On 12.10.16 at 16:58, <haozhong.zhang@intel.com> wrote: >> On 10/12/16 05:32 -0600, Jan Beulich wrote: >>>>>> On 12.10.16 at 12:33, <haozhong.zhang@intel.com> wrote: >>>> The layout is shown as the following diagram. >>>> >>>> +---------------+-----------+-------+----------+--------------+ >>>> | whatever used | Partition | Super | Reserved | /dev/pmem0p1 | >>>> | by kernel | Table | Block | for Xen | | >>>> +---------------+-----------+-------+----------+--------------+ >>>> \_____________________ _______________________/ >>>> V >>>> /dev/pmem0 >>> >>>I have to admit that I dislike this, for not being OS-agnostic. >>>Neither should there be any Xen-specific region, nor should the >>>"whatever used by kernel" one be restricted to just Linux. What >>>I could see is an OS-reserved area ahead of the partition table, >>>the exact usage of which depends on which OS is currently >>>running (and in the Xen case this might be both Xen _and_ the >>>Dom0 kernel, arbitrated by a tbd protocol). After all, when >>>running under Xen, the Dom0 may not have a need for as much >>>control data as it has when running on bare hardware, for it >>>controlling less (if any) of the actual memory ranges when Xen >>>is present. >>> >> >> Isn't this OS-reserved area still not OS-agnostic, as it requires OS >> to know where the reserved area is? Or do you mean it's not if it's >> defined by a protocol that is accepted by all OSes? > > The latter - we clearly won't get away without some agreement on > where to retrieve position and size of this area. I was simply > assuming that such a protocol already exists. > No, we should not mix the struct page reservation that the Dom0 kernel may actively use with the Xen reservation that the Dom0 kernel does not consume. Explain again what is wrong with the partition approach?
[toc] | [prev] | [next] | [standalone]
| From | "Jan Beulich" <JBeulich@suse.com> |
|---|---|
| Date | 2016-10-12 18:30 +0200 |
| Subject | Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen |
| Message-ID | <sruzM-6Ti-9@gated-at.bofh.it> |
| In reply to | #1499784 |
>>> On 12.10.16 at 17:42, <dan.j.williams@intel.com> wrote: > On Wed, Oct 12, 2016 at 8:39 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>>> On 12.10.16 at 16:58, <haozhong.zhang@intel.com> wrote: >>> On 10/12/16 05:32 -0600, Jan Beulich wrote: >>>>>>> On 12.10.16 at 12:33, <haozhong.zhang@intel.com> wrote: >>>>> The layout is shown as the following diagram. >>>>> >>>>> +---------------+-----------+-------+----------+--------------+ >>>>> | whatever used | Partition | Super | Reserved | /dev/pmem0p1 | >>>>> | by kernel | Table | Block | for Xen | | >>>>> +---------------+-----------+-------+----------+--------------+ >>>>> \_____________________ _______________________/ >>>>> V >>>>> /dev/pmem0 >>>> >>>>I have to admit that I dislike this, for not being OS-agnostic. >>>>Neither should there be any Xen-specific region, nor should the >>>>"whatever used by kernel" one be restricted to just Linux. What >>>>I could see is an OS-reserved area ahead of the partition table, >>>>the exact usage of which depends on which OS is currently >>>>running (and in the Xen case this might be both Xen _and_ the >>>>Dom0 kernel, arbitrated by a tbd protocol). After all, when >>>>running under Xen, the Dom0 may not have a need for as much >>>>control data as it has when running on bare hardware, for it >>>>controlling less (if any) of the actual memory ranges when Xen >>>>is present. >>>> >>> >>> Isn't this OS-reserved area still not OS-agnostic, as it requires OS >>> to know where the reserved area is? Or do you mean it's not if it's >>> defined by a protocol that is accepted by all OSes? >> >> The latter - we clearly won't get away without some agreement on >> where to retrieve position and size of this area. I was simply >> assuming that such a protocol already exists. >> > > No, we should not mix the struct page reservation that the Dom0 kernel > may actively use with the Xen reservation that the Dom0 kernel does > not consume. Explain again what is wrong with the partition approach? Not sure what was unclear in my previous reply. I don't think there should be apriori knowledge of whether Xen is (going to be) used on a system, and even if it gets used, but just occasionally, it would (apart from the abstract considerations already given) be a waste of resources to set something aside that could be used for other purposes while Xen is not running. Static partitioning should only be needed for persistent data. Jan
[toc] | [prev] | [next] | [standalone]
| From | Dan Williams <dan.j.williams@intel.com> |
|---|---|
| Date | 2016-10-12 18:30 +0200 |
| Subject | Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen |
| Message-ID | <sruzM-6Ti-25@gated-at.bofh.it> |
| In reply to | #1499807 |
On Wed, Oct 12, 2016 at 9:01 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>> On 12.10.16 at 17:42, <dan.j.williams@intel.com> wrote: >> On Wed, Oct 12, 2016 at 8:39 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>>>> On 12.10.16 at 16:58, <haozhong.zhang@intel.com> wrote: >>>> On 10/12/16 05:32 -0600, Jan Beulich wrote: >>>>>>>> On 12.10.16 at 12:33, <haozhong.zhang@intel.com> wrote: >>>>>> The layout is shown as the following diagram. >>>>>> >>>>>> +---------------+-----------+-------+----------+--------------+ >>>>>> | whatever used | Partition | Super | Reserved | /dev/pmem0p1 | >>>>>> | by kernel | Table | Block | for Xen | | >>>>>> +---------------+-----------+-------+----------+--------------+ >>>>>> \_____________________ _______________________/ >>>>>> V >>>>>> /dev/pmem0 >>>>> >>>>>I have to admit that I dislike this, for not being OS-agnostic. >>>>>Neither should there be any Xen-specific region, nor should the >>>>>"whatever used by kernel" one be restricted to just Linux. What >>>>>I could see is an OS-reserved area ahead of the partition table, >>>>>the exact usage of which depends on which OS is currently >>>>>running (and in the Xen case this might be both Xen _and_ the >>>>>Dom0 kernel, arbitrated by a tbd protocol). After all, when >>>>>running under Xen, the Dom0 may not have a need for as much >>>>>control data as it has when running on bare hardware, for it >>>>>controlling less (if any) of the actual memory ranges when Xen >>>>>is present. >>>>> >>>> >>>> Isn't this OS-reserved area still not OS-agnostic, as it requires OS >>>> to know where the reserved area is? Or do you mean it's not if it's >>>> defined by a protocol that is accepted by all OSes? >>> >>> The latter - we clearly won't get away without some agreement on >>> where to retrieve position and size of this area. I was simply >>> assuming that such a protocol already exists. >>> >> >> No, we should not mix the struct page reservation that the Dom0 kernel >> may actively use with the Xen reservation that the Dom0 kernel does >> not consume. Explain again what is wrong with the partition approach? > > Not sure what was unclear in my previous reply. I don't think there > should be apriori knowledge of whether Xen is (going to be) used on > a system, and even if it gets used, but just occasionally, it would > (apart from the abstract considerations already given) be a waste > of resources to set something aside that could be used for other > purposes while Xen is not running. Static partitioning should only be > needed for persistent data. > The reservation needs to be persistent / static even if the data is volatile, as is the case with struct page, because we can't have the size of the device change depending on use. So, from the aspect of wasting space while Xen is not in use, both partitions and the intrinsic reservation approach suffer the same problem. Setting that aside I don't want to mix 2 different use cases into the same reservation. The kernel needs to know about the struct page reservation because it needs to manage the lifetime of page references vs the lifetime of the device. It does not have the same relationship with a Xen reservation which is why I'm proposing they be managed separately. Note that Toshi and Mike added DM for DAX. This enabling ends up writing DM metadata on the device without adding new reservation mechanisms to the nvdimm core. I'm struggling to see how the Xen use case is materially different DM. In the end it's an application specific metadata space.
[toc] | [prev] | [next] | [standalone]
| From | "Jan Beulich" <JBeulich@suse.com> |
|---|---|
| Date | 2016-10-13 10:40 +0200 |
| Subject | Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen |
| Message-ID | <srJIt-152-3@gated-at.bofh.it> |
| In reply to | #1499813 |
>>> On 12.10.16 at 18:19, <dan.j.williams@intel.com> wrote: > On Wed, Oct 12, 2016 at 9:01 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>>> On 12.10.16 at 17:42, <dan.j.williams@intel.com> wrote: >>> On Wed, Oct 12, 2016 at 8:39 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>>>>> On 12.10.16 at 16:58, <haozhong.zhang@intel.com> wrote: >>>>> On 10/12/16 05:32 -0600, Jan Beulich wrote: >>>>>>>>> On 12.10.16 at 12:33, <haozhong.zhang@intel.com> wrote: >>>>>>> The layout is shown as the following diagram. >>>>>>> >>>>>>> +---------------+-----------+-------+----------+--------------+ >>>>>>> | whatever used | Partition | Super | Reserved | /dev/pmem0p1 | >>>>>>> | by kernel | Table | Block | for Xen | | >>>>>>> +---------------+-----------+-------+----------+--------------+ >>>>>>> \_____________________ _______________________/ >>>>>>> V >>>>>>> /dev/pmem0 >>>>>> >>>>>>I have to admit that I dislike this, for not being OS-agnostic. >>>>>>Neither should there be any Xen-specific region, nor should the >>>>>>"whatever used by kernel" one be restricted to just Linux. What >>>>>>I could see is an OS-reserved area ahead of the partition table, >>>>>>the exact usage of which depends on which OS is currently >>>>>>running (and in the Xen case this might be both Xen _and_ the >>>>>>Dom0 kernel, arbitrated by a tbd protocol). After all, when >>>>>>running under Xen, the Dom0 may not have a need for as much >>>>>>control data as it has when running on bare hardware, for it >>>>>>controlling less (if any) of the actual memory ranges when Xen >>>>>>is present. >>>>>> >>>>> >>>>> Isn't this OS-reserved area still not OS-agnostic, as it requires OS >>>>> to know where the reserved area is? Or do you mean it's not if it's >>>>> defined by a protocol that is accepted by all OSes? >>>> >>>> The latter - we clearly won't get away without some agreement on >>>> where to retrieve position and size of this area. I was simply >>>> assuming that such a protocol already exists. >>>> >>> >>> No, we should not mix the struct page reservation that the Dom0 kernel >>> may actively use with the Xen reservation that the Dom0 kernel does >>> not consume. Explain again what is wrong with the partition approach? >> >> Not sure what was unclear in my previous reply. I don't think there >> should be apriori knowledge of whether Xen is (going to be) used on >> a system, and even if it gets used, but just occasionally, it would >> (apart from the abstract considerations already given) be a waste >> of resources to set something aside that could be used for other >> purposes while Xen is not running. Static partitioning should only be >> needed for persistent data. > > The reservation needs to be persistent / static even if the data is > volatile, as is the case with struct page, because we can't have the > size of the device change depending on use. So, from the aspect of > wasting space while Xen is not in use, both partitions and the > intrinsic reservation approach suffer the same problem. Setting that > aside I don't want to mix 2 different use cases into the same > reservation. Then you didn't understand what I've said: I certainly didn't mean the reservation to vary from a device perspective. However, when Xen is in use I don't see why part of that static reservation couldn't be used by Xen, and another part by the Dom0 kernel. The kernel obviously would need to ask the hypervisor how much of the space is left, and where that area starts. > The kernel needs to know about the struct page reservation because it > needs to manage the lifetime of page references vs the lifetime of the > device. It does not have the same relationship with a Xen reservation > which is why I'm proposing they be managed separately. I don't think I understand the difference you try to point out here. Linux'es struct page and Xen's struct page_info serve the same fundamental purpose. Jan
[toc] | [prev] | [next] | [standalone]
| From | Haozhong Zhang <haozhong.zhang@intel.com> |
|---|---|
| Date | 2016-10-13 11:00 +0200 |
| Subject | Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen |
| Message-ID | <srK1Q-1cz-11@gated-at.bofh.it> |
| In reply to | #1500094 |
On 10/13/16 02:34 -0600, Jan Beulich wrote: >>>> On 12.10.16 at 18:19, <dan.j.williams@intel.com> wrote: >> On Wed, Oct 12, 2016 at 9:01 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>>>> On 12.10.16 at 17:42, <dan.j.williams@intel.com> wrote: >>>> On Wed, Oct 12, 2016 at 8:39 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>>>>>> On 12.10.16 at 16:58, <haozhong.zhang@intel.com> wrote: >>>>>> On 10/12/16 05:32 -0600, Jan Beulich wrote: >>>>>>>>>> On 12.10.16 at 12:33, <haozhong.zhang@intel.com> wrote: >>>>>>>> The layout is shown as the following diagram. >>>>>>>> >>>>>>>> +---------------+-----------+-------+----------+--------------+ >>>>>>>> | whatever used | Partition | Super | Reserved | /dev/pmem0p1 | >>>>>>>> | by kernel | Table | Block | for Xen | | >>>>>>>> +---------------+-----------+-------+----------+--------------+ >>>>>>>> \_____________________ _______________________/ >>>>>>>> V >>>>>>>> /dev/pmem0 >>>>>>> >>>>>>>I have to admit that I dislike this, for not being OS-agnostic. >>>>>>>Neither should there be any Xen-specific region, nor should the >>>>>>>"whatever used by kernel" one be restricted to just Linux. What >>>>>>>I could see is an OS-reserved area ahead of the partition table, >>>>>>>the exact usage of which depends on which OS is currently >>>>>>>running (and in the Xen case this might be both Xen _and_ the >>>>>>>Dom0 kernel, arbitrated by a tbd protocol). After all, when >>>>>>>running under Xen, the Dom0 may not have a need for as much >>>>>>>control data as it has when running on bare hardware, for it >>>>>>>controlling less (if any) of the actual memory ranges when Xen >>>>>>>is present. >>>>>>> >>>>>> >>>>>> Isn't this OS-reserved area still not OS-agnostic, as it requires OS >>>>>> to know where the reserved area is? Or do you mean it's not if it's >>>>>> defined by a protocol that is accepted by all OSes? >>>>> >>>>> The latter - we clearly won't get away without some agreement on >>>>> where to retrieve position and size of this area. I was simply >>>>> assuming that such a protocol already exists. >>>>> >>>> >>>> No, we should not mix the struct page reservation that the Dom0 kernel >>>> may actively use with the Xen reservation that the Dom0 kernel does >>>> not consume. Explain again what is wrong with the partition approach? >>> >>> Not sure what was unclear in my previous reply. I don't think there >>> should be apriori knowledge of whether Xen is (going to be) used on >>> a system, and even if it gets used, but just occasionally, it would >>> (apart from the abstract considerations already given) be a waste >>> of resources to set something aside that could be used for other >>> purposes while Xen is not running. Static partitioning should only be >>> needed for persistent data. >> >> The reservation needs to be persistent / static even if the data is >> volatile, as is the case with struct page, because we can't have the >> size of the device change depending on use. So, from the aspect of >> wasting space while Xen is not in use, both partitions and the >> intrinsic reservation approach suffer the same problem. Setting that >> aside I don't want to mix 2 different use cases into the same >> reservation. > >Then you didn't understand what I've said: I certainly didn't mean >the reservation to vary from a device perspective. However, when >Xen is in use I don't see why part of that static reservation couldn't >be used by Xen, and another part by the Dom0 kernel. The kernel >obviously would need to ask the hypervisor how much of the space >is left, and where that area starts. > I think Dan means that there should be a clear separation between reservations for different usages (kernel/xen/...). The libnvdimm driver is for the linux kernel and only needs to maintain the reservation for kernel functionality. For others including xen/dm/..., if they want reservation for their own purpose, they should maintain their own reservations out of libnvdimm driver and avoid bothering the libnvdimm driver (e.g. add specific handling in libnvdimm driver). IIUC, one existing example is device-mapper device (dm) which needs to reserve on-device area for its own meta-data. Its choice is to store the meta-data on the block device (/dev/pmemN) provided by the libnvdimm driver. I think we can do the similar for Xen, like to lay another pseudo device on /dev/pmem and do the reservation, like 2. in my previous reply. Thanks, Haozhong >> The kernel needs to know about the struct page reservation because it >> needs to manage the lifetime of page references vs the lifetime of the >> device. It does not have the same relationship with a Xen reservation >> which is why I'm proposing they be managed separately. > >I don't think I understand the difference you try to point out here. >Linux'es struct page and Xen's struct page_info serve the same >fundamental purpose. > >Jan >
[toc] | [prev] | [next] | [standalone]
| From | Haozhong Zhang <haozhong.zhang@intel.com> |
|---|---|
| Date | 2016-10-13 11:10 +0200 |
| Subject | Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen |
| Message-ID | <srKbv-1uX-3@gated-at.bofh.it> |
| In reply to | #1500103 |
+Dan Williams I accidentally dropped him in my last reply. Add him back. On 10/13/16 16:53 +0800, Haozhong Zhang wrote: >On 10/13/16 02:34 -0600, Jan Beulich wrote: >>>>>On 12.10.16 at 18:19, <dan.j.williams@intel.com> wrote: >>>On Wed, Oct 12, 2016 at 9:01 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>>>>>On 12.10.16 at 17:42, <dan.j.williams@intel.com> wrote: >>>>>On Wed, Oct 12, 2016 at 8:39 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>>>>>>>On 12.10.16 at 16:58, <haozhong.zhang@intel.com> wrote: >>>>>>>On 10/12/16 05:32 -0600, Jan Beulich wrote: >>>>>>>>>>>On 12.10.16 at 12:33, <haozhong.zhang@intel.com> wrote: >>>>>>>>>The layout is shown as the following diagram. >>>>>>>>> >>>>>>>>>+---------------+-----------+-------+----------+--------------+ >>>>>>>>>| whatever used | Partition | Super | Reserved | /dev/pmem0p1 | >>>>>>>>>| by kernel | Table | Block | for Xen | | >>>>>>>>>+---------------+-----------+-------+----------+--------------+ >>>>>>>>> \_____________________ _______________________/ >>>>>>>>> V >>>>>>>>> /dev/pmem0 >>>>>>>> >>>>>>>>I have to admit that I dislike this, for not being OS-agnostic. >>>>>>>>Neither should there be any Xen-specific region, nor should the >>>>>>>>"whatever used by kernel" one be restricted to just Linux. What >>>>>>>>I could see is an OS-reserved area ahead of the partition table, >>>>>>>>the exact usage of which depends on which OS is currently >>>>>>>>running (and in the Xen case this might be both Xen _and_ the >>>>>>>>Dom0 kernel, arbitrated by a tbd protocol). After all, when >>>>>>>>running under Xen, the Dom0 may not have a need for as much >>>>>>>>control data as it has when running on bare hardware, for it >>>>>>>>controlling less (if any) of the actual memory ranges when Xen >>>>>>>>is present. >>>>>>>> >>>>>>> >>>>>>>Isn't this OS-reserved area still not OS-agnostic, as it requires OS >>>>>>>to know where the reserved area is? Or do you mean it's not if it's >>>>>>>defined by a protocol that is accepted by all OSes? >>>>>> >>>>>>The latter - we clearly won't get away without some agreement on >>>>>>where to retrieve position and size of this area. I was simply >>>>>>assuming that such a protocol already exists. >>>>>> >>>>> >>>>>No, we should not mix the struct page reservation that the Dom0 kernel >>>>>may actively use with the Xen reservation that the Dom0 kernel does >>>>>not consume. Explain again what is wrong with the partition approach? >>>> >>>>Not sure what was unclear in my previous reply. I don't think there >>>>should be apriori knowledge of whether Xen is (going to be) used on >>>>a system, and even if it gets used, but just occasionally, it would >>>>(apart from the abstract considerations already given) be a waste >>>>of resources to set something aside that could be used for other >>>>purposes while Xen is not running. Static partitioning should only be >>>>needed for persistent data. >>> >>>The reservation needs to be persistent / static even if the data is >>>volatile, as is the case with struct page, because we can't have the >>>size of the device change depending on use. So, from the aspect of >>>wasting space while Xen is not in use, both partitions and the >>>intrinsic reservation approach suffer the same problem. Setting that >>>aside I don't want to mix 2 different use cases into the same >>>reservation. >> >>Then you didn't understand what I've said: I certainly didn't mean >>the reservation to vary from a device perspective. However, when >>Xen is in use I don't see why part of that static reservation couldn't >>be used by Xen, and another part by the Dom0 kernel. The kernel >>obviously would need to ask the hypervisor how much of the space >>is left, and where that area starts. >> > >I think Dan means that there should be a clear separation between >reservations for different usages (kernel/xen/...). The libnvdimm >driver is for the linux kernel and only needs to maintain the >reservation for kernel functionality. For others including xen/dm/..., >if they want reservation for their own purpose, they should maintain >their own reservations out of libnvdimm driver and avoid bothering the >libnvdimm driver (e.g. add specific handling in libnvdimm driver). > >IIUC, one existing example is device-mapper device (dm) which needs to >reserve on-device area for its own meta-data. Its choice is to store >the meta-data on the block device (/dev/pmemN) provided by the >libnvdimm driver. > >I think we can do the similar for Xen, like to lay another pseudo >device on /dev/pmem and do the reservation, like 2. in my previous >reply. > >Thanks, >Haozhong > >>>The kernel needs to know about the struct page reservation because it >>>needs to manage the lifetime of page references vs the lifetime of the >>>device. It does not have the same relationship with a Xen reservation >>>which is why I'm proposing they be managed separately. >> >>I don't think I understand the difference you try to point out here. >>Linux'es struct page and Xen's struct page_info serve the same >>fundamental purpose. >> >>Jan >> >_______________________________________________ >Linux-nvdimm mailing list >Linux-nvdimm@lists.01.org >https://lists.01.org/mailman/listinfo/linux-nvdimm
[toc] | [prev] | [next] | [standalone]
| From | "Jan Beulich" <JBeulich@suse.com> |
|---|---|
| Date | 2016-10-13 11:10 +0200 |
| Subject | Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen |
| Message-ID | <srKbv-1uX-5@gated-at.bofh.it> |
| In reply to | #1500103 |
>>> On 13.10.16 at 10:53, <haozhong.zhang@intel.com> wrote: > On 10/13/16 02:34 -0600, Jan Beulich wrote: >>>>> On 12.10.16 at 18:19, <dan.j.williams@intel.com> wrote: >>> On Wed, Oct 12, 2016 at 9:01 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>>>>> On 12.10.16 at 17:42, <dan.j.williams@intel.com> wrote: >>>>> On Wed, Oct 12, 2016 at 8:39 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>>>>>>> On 12.10.16 at 16:58, <haozhong.zhang@intel.com> wrote: >>>>>>> On 10/12/16 05:32 -0600, Jan Beulich wrote: >>>>>>>>>>> On 12.10.16 at 12:33, <haozhong.zhang@intel.com> wrote: >>>>>>>>> The layout is shown as the following diagram. >>>>>>>>> >>>>>>>>> +---------------+-----------+-------+----------+--------------+ >>>>>>>>> | whatever used | Partition | Super | Reserved | /dev/pmem0p1 | >>>>>>>>> | by kernel | Table | Block | for Xen | | >>>>>>>>> +---------------+-----------+-------+----------+--------------+ >>>>>>>>> \_____________________ _______________________/ >>>>>>>>> V >>>>>>>>> /dev/pmem0 >>>>>>>> >>>>>>>>I have to admit that I dislike this, for not being OS-agnostic. >>>>>>>>Neither should there be any Xen-specific region, nor should the >>>>>>>>"whatever used by kernel" one be restricted to just Linux. What >>>>>>>>I could see is an OS-reserved area ahead of the partition table, >>>>>>>>the exact usage of which depends on which OS is currently >>>>>>>>running (and in the Xen case this might be both Xen _and_ the >>>>>>>>Dom0 kernel, arbitrated by a tbd protocol). After all, when >>>>>>>>running under Xen, the Dom0 may not have a need for as much >>>>>>>>control data as it has when running on bare hardware, for it >>>>>>>>controlling less (if any) of the actual memory ranges when Xen >>>>>>>>is present. >>>>>>>> >>>>>>> >>>>>>> Isn't this OS-reserved area still not OS-agnostic, as it requires OS >>>>>>> to know where the reserved area is? Or do you mean it's not if it's >>>>>>> defined by a protocol that is accepted by all OSes? >>>>>> >>>>>> The latter - we clearly won't get away without some agreement on >>>>>> where to retrieve position and size of this area. I was simply >>>>>> assuming that such a protocol already exists. >>>>>> >>>>> >>>>> No, we should not mix the struct page reservation that the Dom0 kernel >>>>> may actively use with the Xen reservation that the Dom0 kernel does >>>>> not consume. Explain again what is wrong with the partition approach? >>>> >>>> Not sure what was unclear in my previous reply. I don't think there >>>> should be apriori knowledge of whether Xen is (going to be) used on >>>> a system, and even if it gets used, but just occasionally, it would >>>> (apart from the abstract considerations already given) be a waste >>>> of resources to set something aside that could be used for other >>>> purposes while Xen is not running. Static partitioning should only be >>>> needed for persistent data. >>> >>> The reservation needs to be persistent / static even if the data is >>> volatile, as is the case with struct page, because we can't have the >>> size of the device change depending on use. So, from the aspect of >>> wasting space while Xen is not in use, both partitions and the >>> intrinsic reservation approach suffer the same problem. Setting that >>> aside I don't want to mix 2 different use cases into the same >>> reservation. >> >>Then you didn't understand what I've said: I certainly didn't mean >>the reservation to vary from a device perspective. However, when >>Xen is in use I don't see why part of that static reservation couldn't >>be used by Xen, and another part by the Dom0 kernel. The kernel >>obviously would need to ask the hypervisor how much of the space >>is left, and where that area starts. >> > > I think Dan means that there should be a clear separation between > reservations for different usages (kernel/xen/...). The libnvdimm > driver is for the linux kernel and only needs to maintain the > reservation for kernel functionality. For others including xen/dm/..., > if they want reservation for their own purpose, they should maintain > their own reservations out of libnvdimm driver and avoid bothering the > libnvdimm driver (e.g. add specific handling in libnvdimm driver). > > IIUC, one existing example is device-mapper device (dm) which needs to > reserve on-device area for its own meta-data. Its choice is to store > the meta-data on the block device (/dev/pmemN) provided by the > libnvdimm driver. > > I think we can do the similar for Xen, like to lay another pseudo > device on /dev/pmem and do the reservation, like 2. in my previous > reply. Well, my opinion certainly doesn't count much here, but I continue to consider this a bad idea. For entities like drivers it may well be appropriate, but I think there ought to be an independent concept of "OS reserved", and in the Xen case this could then be shared between hypervisor and Dom0 kernel. Or if we were to consider Dom0 "just a guest", things should even be the other way around: Xen gets all of the OS reserved space, and Dom0 needs something custom. Jan
[toc] | [prev] | [next] | [standalone]
| From | Haozhong Zhang <haozhong.zhang@intel.com> |
|---|---|
| Date | 2016-10-13 17:50 +0200 |
| Subject | Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen |
| Message-ID | <srQqB-5mY-3@gated-at.bofh.it> |
| In reply to | #1500114 |
On 10/13/16 03:08 -0600, Jan Beulich wrote: >>>> On 13.10.16 at 10:53, <haozhong.zhang@intel.com> wrote: >> On 10/13/16 02:34 -0600, Jan Beulich wrote: >>>>>> On 12.10.16 at 18:19, <dan.j.williams@intel.com> wrote: >>>> On Wed, Oct 12, 2016 at 9:01 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>>>>>> On 12.10.16 at 17:42, <dan.j.williams@intel.com> wrote: >>>>>> On Wed, Oct 12, 2016 at 8:39 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>>>>>>>> On 12.10.16 at 16:58, <haozhong.zhang@intel.com> wrote: >>>>>>>> On 10/12/16 05:32 -0600, Jan Beulich wrote: >>>>>>>>>>>> On 12.10.16 at 12:33, <haozhong.zhang@intel.com> wrote: >>>>>>>>>> The layout is shown as the following diagram. >>>>>>>>>> >>>>>>>>>> +---------------+-----------+-------+----------+--------------+ >>>>>>>>>> | whatever used | Partition | Super | Reserved | /dev/pmem0p1 | >>>>>>>>>> | by kernel | Table | Block | for Xen | | >>>>>>>>>> +---------------+-----------+-------+----------+--------------+ >>>>>>>>>> \_____________________ _______________________/ >>>>>>>>>> V >>>>>>>>>> /dev/pmem0 >>>>>>>>> >>>>>>>>>I have to admit that I dislike this, for not being OS-agnostic. >>>>>>>>>Neither should there be any Xen-specific region, nor should the >>>>>>>>>"whatever used by kernel" one be restricted to just Linux. What >>>>>>>>>I could see is an OS-reserved area ahead of the partition table, >>>>>>>>>the exact usage of which depends on which OS is currently >>>>>>>>>running (and in the Xen case this might be both Xen _and_ the >>>>>>>>>Dom0 kernel, arbitrated by a tbd protocol). After all, when >>>>>>>>>running under Xen, the Dom0 may not have a need for as much >>>>>>>>>control data as it has when running on bare hardware, for it >>>>>>>>>controlling less (if any) of the actual memory ranges when Xen >>>>>>>>>is present. >>>>>>>>> >>>>>>>> >>>>>>>> Isn't this OS-reserved area still not OS-agnostic, as it requires OS >>>>>>>> to know where the reserved area is? Or do you mean it's not if it's >>>>>>>> defined by a protocol that is accepted by all OSes? >>>>>>> >>>>>>> The latter - we clearly won't get away without some agreement on >>>>>>> where to retrieve position and size of this area. I was simply >>>>>>> assuming that such a protocol already exists. >>>>>>> >>>>>> >>>>>> No, we should not mix the struct page reservation that the Dom0 kernel >>>>>> may actively use with the Xen reservation that the Dom0 kernel does >>>>>> not consume. Explain again what is wrong with the partition approach? >>>>> >>>>> Not sure what was unclear in my previous reply. I don't think there >>>>> should be apriori knowledge of whether Xen is (going to be) used on >>>>> a system, and even if it gets used, but just occasionally, it would >>>>> (apart from the abstract considerations already given) be a waste >>>>> of resources to set something aside that could be used for other >>>>> purposes while Xen is not running. Static partitioning should only be >>>>> needed for persistent data. >>>> >>>> The reservation needs to be persistent / static even if the data is >>>> volatile, as is the case with struct page, because we can't have the >>>> size of the device change depending on use. So, from the aspect of >>>> wasting space while Xen is not in use, both partitions and the >>>> intrinsic reservation approach suffer the same problem. Setting that >>>> aside I don't want to mix 2 different use cases into the same >>>> reservation. >>> >>>Then you didn't understand what I've said: I certainly didn't mean >>>the reservation to vary from a device perspective. However, when >>>Xen is in use I don't see why part of that static reservation couldn't >>>be used by Xen, and another part by the Dom0 kernel. The kernel >>>obviously would need to ask the hypervisor how much of the space >>>is left, and where that area starts. >>> >> >> I think Dan means that there should be a clear separation between >> reservations for different usages (kernel/xen/...). The libnvdimm >> driver is for the linux kernel and only needs to maintain the >> reservation for kernel functionality. For others including xen/dm/..., >> if they want reservation for their own purpose, they should maintain >> their own reservations out of libnvdimm driver and avoid bothering the >> libnvdimm driver (e.g. add specific handling in libnvdimm driver). >> >> IIUC, one existing example is device-mapper device (dm) which needs to >> reserve on-device area for its own meta-data. Its choice is to store >> the meta-data on the block device (/dev/pmemN) provided by the >> libnvdimm driver. >> >> I think we can do the similar for Xen, like to lay another pseudo >> device on /dev/pmem and do the reservation, like 2. in my previous >> reply. > >Well, my opinion certainly doesn't count much here, but I continue to >consider this a bad idea. For entities like drivers it may well be >appropriate, but I think there ought to be an independent concept >of "OS reserved", and in the Xen case this could then be shared >between hypervisor and Dom0 kernel. No such independent concept seems exist right now. It may be hard to define such concept, because it's hard to know the common requirements (e.g. size/alignment/...) from ALL OSes. Making each component to maintain its own reservation in its own way seems more flexible. >Or if we were to consider Dom0 >"just a guest", things should even be the other way around: Xen gets >all of the OS reserved space, and Dom0 needs something custom. > Sure, it's possible to implement the driver in a way that if the driver finds it runs on Xen, then it just leaves the OS reserved area for Xen and itself goes to other reservation. Are there some differences in practice from the way that Xen goes to other reservation that makes we have to do so? If not and it's possible to not touch the existing libnvdimm driver, why don't we just use the existing libnvdimm driver and let xen driver make the reservation on what the libnvdimm driver provides? In addition (not sure it's related), my Xen patch series (specially patch 3) does not have many requirements for the location of the reserved area, as long as it's in the nvdimm. I mean if we find a better way for the reservation in future, there should be no changes to Xen. For now, I think we could just choose the way to not touch the libnvdimm driver. Thanks, Haozhong
[toc] | [prev] | [next] | [standalone]
| From | "Jan Beulich" <JBeulich@suse.com> |
|---|---|
| Date | 2016-10-14 12:20 +0200 |
| Subject | Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen |
| Message-ID | <ss7KN-8r9-1@gated-at.bofh.it> |
| In reply to | #1500372 |
>>> On 13.10.16 at 17:46, <haozhong.zhang@intel.com> wrote: > On 10/13/16 03:08 -0600, Jan Beulich wrote: >>>>> On 13.10.16 at 10:53, <haozhong.zhang@intel.com> wrote: >>> On 10/13/16 02:34 -0600, Jan Beulich wrote: >>>>>>> On 12.10.16 at 18:19, <dan.j.williams@intel.com> wrote: >>>>> On Wed, Oct 12, 2016 at 9:01 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>>>>>>> On 12.10.16 at 17:42, <dan.j.williams@intel.com> wrote: >>>>>>> On Wed, Oct 12, 2016 at 8:39 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>>>>>>>>> On 12.10.16 at 16:58, <haozhong.zhang@intel.com> wrote: >>>>>>>>> On 10/12/16 05:32 -0600, Jan Beulich wrote: >>>>>>>>>>>>> On 12.10.16 at 12:33, <haozhong.zhang@intel.com> wrote: >>>>>>>>>>> The layout is shown as the following diagram. >>>>>>>>>>> >>>>>>>>>>> +---------------+-----------+-------+----------+--------------+ >>>>>>>>>>> | whatever used | Partition | Super | Reserved | /dev/pmem0p1 | >>>>>>>>>>> | by kernel | Table | Block | for Xen | | >>>>>>>>>>> +---------------+-----------+-------+----------+--------------+ >>>>>>>>>>> \_____________________ _______________________/ >>>>>>>>>>> V >>>>>>>>>>> /dev/pmem0 >>>>>>>>>> >>>>>>>>>>I have to admit that I dislike this, for not being OS-agnostic. >>>>>>>>>>Neither should there be any Xen-specific region, nor should the >>>>>>>>>>"whatever used by kernel" one be restricted to just Linux. What >>>>>>>>>>I could see is an OS-reserved area ahead of the partition table, >>>>>>>>>>the exact usage of which depends on which OS is currently >>>>>>>>>>running (and in the Xen case this might be both Xen _and_ the >>>>>>>>>>Dom0 kernel, arbitrated by a tbd protocol). After all, when >>>>>>>>>>running under Xen, the Dom0 may not have a need for as much >>>>>>>>>>control data as it has when running on bare hardware, for it >>>>>>>>>>controlling less (if any) of the actual memory ranges when Xen >>>>>>>>>>is present. >>>>>>>>>> >>>>>>>>> >>>>>>>>> Isn't this OS-reserved area still not OS-agnostic, as it requires OS >>>>>>>>> to know where the reserved area is? Or do you mean it's not if it's >>>>>>>>> defined by a protocol that is accepted by all OSes? >>>>>>>> >>>>>>>> The latter - we clearly won't get away without some agreement on >>>>>>>> where to retrieve position and size of this area. I was simply >>>>>>>> assuming that such a protocol already exists. >>>>>>>> >>>>>>> >>>>>>> No, we should not mix the struct page reservation that the Dom0 kernel >>>>>>> may actively use with the Xen reservation that the Dom0 kernel does >>>>>>> not consume. Explain again what is wrong with the partition approach? >>>>>> >>>>>> Not sure what was unclear in my previous reply. I don't think there >>>>>> should be apriori knowledge of whether Xen is (going to be) used on >>>>>> a system, and even if it gets used, but just occasionally, it would >>>>>> (apart from the abstract considerations already given) be a waste >>>>>> of resources to set something aside that could be used for other >>>>>> purposes while Xen is not running. Static partitioning should only be >>>>>> needed for persistent data. >>>>> >>>>> The reservation needs to be persistent / static even if the data is >>>>> volatile, as is the case with struct page, because we can't have the >>>>> size of the device change depending on use. So, from the aspect of >>>>> wasting space while Xen is not in use, both partitions and the >>>>> intrinsic reservation approach suffer the same problem. Setting that >>>>> aside I don't want to mix 2 different use cases into the same >>>>> reservation. >>>> >>>>Then you didn't understand what I've said: I certainly didn't mean >>>>the reservation to vary from a device perspective. However, when >>>>Xen is in use I don't see why part of that static reservation couldn't >>>>be used by Xen, and another part by the Dom0 kernel. The kernel >>>>obviously would need to ask the hypervisor how much of the space >>>>is left, and where that area starts. >>>> >>> >>> I think Dan means that there should be a clear separation between >>> reservations for different usages (kernel/xen/...). The libnvdimm >>> driver is for the linux kernel and only needs to maintain the >>> reservation for kernel functionality. For others including xen/dm/..., >>> if they want reservation for their own purpose, they should maintain >>> their own reservations out of libnvdimm driver and avoid bothering the >>> libnvdimm driver (e.g. add specific handling in libnvdimm driver). >>> >>> IIUC, one existing example is device-mapper device (dm) which needs to >>> reserve on-device area for its own meta-data. Its choice is to store >>> the meta-data on the block device (/dev/pmemN) provided by the >>> libnvdimm driver. >>> >>> I think we can do the similar for Xen, like to lay another pseudo >>> device on /dev/pmem and do the reservation, like 2. in my previous >>> reply. >> >>Well, my opinion certainly doesn't count much here, but I continue to >>consider this a bad idea. For entities like drivers it may well be >>appropriate, but I think there ought to be an independent concept >>of "OS reserved", and in the Xen case this could then be shared >>between hypervisor and Dom0 kernel. > > No such independent concept seems exist right now. It may be hard to > define such concept, because it's hard to know the common requirements > (e.g. size/alignment/...) from ALL OSes. Making each component to > maintain its own reservation in its own way seems more flexible. > >>Or if we were to consider Dom0 >>"just a guest", things should even be the other way around: Xen gets >>all of the OS reserved space, and Dom0 needs something custom. >> > > Sure, it's possible to implement the driver in a way that if the > driver finds it runs on Xen, then it just leaves the OS reserved area > for Xen and itself goes to other reservation. Are there some > differences in practice from the way that Xen goes to other > reservation that makes we have to do so? If not and it's possible to > not touch the existing libnvdimm driver, why don't we just use the > existing libnvdimm driver and let xen driver make the reservation on > what the libnvdimm driver provides? It continues to feel like you're trying to make the second step before the first: You talk about implementation, whereas I talk about the concept that should underly your implementation. One could view it the way that how said driver works with Xen in the picture should have been decided before it got implemented, and then the question of whether it would be okay to leave the existing implementation alone would not have appeared in the first place. With that, I can't reasonably answer your questions. Jan
[toc] | [prev] | [next] | [standalone]
| From | Haozhong Zhang <haozhong.zhang@intel.com> |
|---|---|
| Date | 2016-10-20 11:20 +0200 |
| Subject | Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen |
| Message-ID | <suhG3-5zw-43@gated-at.bofh.it> |
| In reply to | #1500821 |
On 10/14/16 04:16 -0600, Jan Beulich wrote: >>>> On 13.10.16 at 17:46, <haozhong.zhang@intel.com> wrote: >> On 10/13/16 03:08 -0600, Jan Beulich wrote: >>>>>> On 13.10.16 at 10:53, <haozhong.zhang@intel.com> wrote: >>>> On 10/13/16 02:34 -0600, Jan Beulich wrote: >>>>>>>> On 12.10.16 at 18:19, <dan.j.williams@intel.com> wrote: >>>>>> On Wed, Oct 12, 2016 at 9:01 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>>>>>>>> On 12.10.16 at 17:42, <dan.j.williams@intel.com> wrote: >>>>>>>> On Wed, Oct 12, 2016 at 8:39 AM, Jan Beulich <JBeulich@suse.com> wrote: >>>>>>>>>>>> On 12.10.16 at 16:58, <haozhong.zhang@intel.com> wrote: >>>>>>>>>> On 10/12/16 05:32 -0600, Jan Beulich wrote: >>>>>>>>>>>>>> On 12.10.16 at 12:33, <haozhong.zhang@intel.com> wrote: >>>>>>>>>>>> The layout is shown as the following diagram. >>>>>>>>>>>> >>>>>>>>>>>> +---------------+-----------+-------+----------+--------------+ >>>>>>>>>>>> | whatever used | Partition | Super | Reserved | /dev/pmem0p1 | >>>>>>>>>>>> | by kernel | Table | Block | for Xen | | >>>>>>>>>>>> +---------------+-----------+-------+----------+--------------+ >>>>>>>>>>>> \_____________________ _______________________/ >>>>>>>>>>>> V >>>>>>>>>>>> /dev/pmem0 >>>>>>>>>>> >>>>>>>>>>>I have to admit that I dislike this, for not being OS-agnostic. >>>>>>>>>>>Neither should there be any Xen-specific region, nor should the >>>>>>>>>>>"whatever used by kernel" one be restricted to just Linux. What >>>>>>>>>>>I could see is an OS-reserved area ahead of the partition table, >>>>>>>>>>>the exact usage of which depends on which OS is currently >>>>>>>>>>>running (and in the Xen case this might be both Xen _and_ the >>>>>>>>>>>Dom0 kernel, arbitrated by a tbd protocol). After all, when >>>>>>>>>>>running under Xen, the Dom0 may not have a need for as much >>>>>>>>>>>control data as it has when running on bare hardware, for it >>>>>>>>>>>controlling less (if any) of the actual memory ranges when Xen >>>>>>>>>>>is present. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Isn't this OS-reserved area still not OS-agnostic, as it requires OS >>>>>>>>>> to know where the reserved area is? Or do you mean it's not if it's >>>>>>>>>> defined by a protocol that is accepted by all OSes? >>>>>>>>> >>>>>>>>> The latter - we clearly won't get away without some agreement on >>>>>>>>> where to retrieve position and size of this area. I was simply >>>>>>>>> assuming that such a protocol already exists. >>>>>>>>> >>>>>>>> >>>>>>>> No, we should not mix the struct page reservation that the Dom0 kernel >>>>>>>> may actively use with the Xen reservation that the Dom0 kernel does >>>>>>>> not consume. Explain again what is wrong with the partition approach? >>>>>>> >>>>>>> Not sure what was unclear in my previous reply. I don't think there >>>>>>> should be apriori knowledge of whether Xen is (going to be) used on >>>>>>> a system, and even if it gets used, but just occasionally, it would >>>>>>> (apart from the abstract considerations already given) be a waste >>>>>>> of resources to set something aside that could be used for other >>>>>>> purposes while Xen is not running. Static partitioning should only be >>>>>>> needed for persistent data. >>>>>> >>>>>> The reservation needs to be persistent / static even if the data is >>>>>> volatile, as is the case with struct page, because we can't have the >>>>>> size of the device change depending on use. So, from the aspect of >>>>>> wasting space while Xen is not in use, both partitions and the >>>>>> intrinsic reservation approach suffer the same problem. Setting that >>>>>> aside I don't want to mix 2 different use cases into the same >>>>>> reservation. >>>>> >>>>>Then you didn't understand what I've said: I certainly didn't mean >>>>>the reservation to vary from a device perspective. However, when >>>>>Xen is in use I don't see why part of that static reservation couldn't >>>>>be used by Xen, and another part by the Dom0 kernel. The kernel >>>>>obviously would need to ask the hypervisor how much of the space >>>>>is left, and where that area starts. >>>>> >>>> >>>> I think Dan means that there should be a clear separation between >>>> reservations for different usages (kernel/xen/...). The libnvdimm >>>> driver is for the linux kernel and only needs to maintain the >>>> reservation for kernel functionality. For others including xen/dm/..., >>>> if they want reservation for their own purpose, they should maintain >>>> their own reservations out of libnvdimm driver and avoid bothering the >>>> libnvdimm driver (e.g. add specific handling in libnvdimm driver). >>>> >>>> IIUC, one existing example is device-mapper device (dm) which needs to >>>> reserve on-device area for its own meta-data. Its choice is to store >>>> the meta-data on the block device (/dev/pmemN) provided by the >>>> libnvdimm driver. >>>> >>>> I think we can do the similar for Xen, like to lay another pseudo >>>> device on /dev/pmem and do the reservation, like 2. in my previous >>>> reply. >>> >>>Well, my opinion certainly doesn't count much here, but I continue to >>>consider this a bad idea. For entities like drivers it may well be >>>appropriate, but I think there ought to be an independent concept >>>of "OS reserved", and in the Xen case this could then be shared >>>between hypervisor and Dom0 kernel. >> >> No such independent concept seems exist right now. It may be hard to >> define such concept, because it's hard to know the common requirements >> (e.g. size/alignment/...) from ALL OSes. Making each component to >> maintain its own reservation in its own way seems more flexible. >> >>>Or if we were to consider Dom0 >>>"just a guest", things should even be the other way around: Xen gets >>>all of the OS reserved space, and Dom0 needs something custom. >>> >> >> Sure, it's possible to implement the driver in a way that if the >> driver finds it runs on Xen, then it just leaves the OS reserved area >> for Xen and itself goes to other reservation. Are there some >> differences in practice from the way that Xen goes to other >> reservation that makes we have to do so? If not and it's possible to >> not touch the existing libnvdimm driver, why don't we just use the >> existing libnvdimm driver and let xen driver make the reservation on >> what the libnvdimm driver provides? > >It continues to feel like you're trying to make the second step before >the first: You talk about implementation, whereas I talk about the >concept that should underly your implementation. One could view it >the way that how said driver works with Xen in the picture should >have been decided before it got implemented, and then the question >of whether it would be okay to leave the existing implementation >alone would not have appeared in the first place. > >With that, I can't reasonably answer your questions. > Ideally, Xen should step in first to do whatever needed for the resource management (e.g. taking all OS-reserved space), and then let other kernel components work on the remaining resource (e.g. libnvdimm driver to use other reserved space). However, the libnvdimm driver was merged in the kernel long before I started to design and implement Xen vNVDIMM. The implementation of the ideal design above needs to modify the existing driver which is objected by its maintainer. Therefore, I looked to the alternative design and implementation. Thanks, Haozhong
[toc] | [prev] | [next] | [standalone]
| From | Dan Williams <dan.j.williams@intel.com> |
|---|---|
| Date | 2016-10-13 17:50 +0200 |
| Subject | Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen |
| Message-ID | <srQqC-5mY-27@gated-at.bofh.it> |
| In reply to | #1500114 |
On Thu, Oct 13, 2016 at 2:08 AM, Jan Beulich <JBeulich@suse.com> wrote: [..] >> I think we can do the similar for Xen, like to lay another pseudo >> device on /dev/pmem and do the reservation, like 2. in my previous >> reply. > > Well, my opinion certainly doesn't count much here, but I continue to > consider this a bad idea. For entities like drivers it may well be > appropriate, but I think there ought to be an independent concept > of "OS reserved", and in the Xen case this could then be shared > between hypervisor and Dom0 kernel. Or if we were to consider Dom0 > "just a guest", things should even be the other way around: Xen gets > all of the OS reserved space, and Dom0 needs something custom. You haven't made the case why Xen is special and other applications of persistent memory are not. The current struct page reservation supports fundamental address-ability of persistent memory namespaces for the rest of the kernel. The Xen reservation is application specific. XFS, EXT4, and DM also have application specific usages of persistent memory and consume metadata space out of a block device. If we don't need an XFS-mode nvdimm device, why do we need Xen-mode?
[toc] | [prev] | [next] | [standalone]
| From | Andrew Cooper <andrew.cooper3@citrix.com> |
|---|---|
| Date | 2016-10-13 18:10 +0200 |
| Subject | Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen |
| Message-ID | <srQJX-5Jj-9@gated-at.bofh.it> |
| In reply to | #1500378 |
On 13/10/16 16:40, Dan Williams wrote: > On Thu, Oct 13, 2016 at 2:08 AM, Jan Beulich <JBeulich@suse.com> wrote: > [..] >>> I think we can do the similar for Xen, like to lay another pseudo >>> device on /dev/pmem and do the reservation, like 2. in my previous >>> reply. >> Well, my opinion certainly doesn't count much here, but I continue to >> consider this a bad idea. For entities like drivers it may well be >> appropriate, but I think there ought to be an independent concept >> of "OS reserved", and in the Xen case this could then be shared >> between hypervisor and Dom0 kernel. Or if we were to consider Dom0 >> "just a guest", things should even be the other way around: Xen gets >> all of the OS reserved space, and Dom0 needs something custom. > You haven't made the case why Xen is special and other applications of > persistent memory are not. In a Xen system, Xen runs in the baremetal root-mode ring0, and dom0 is a VM running in ring1/3 with the nvdimm driver. This is the opposite way around to the KVM model. Dom0, being the hardware domain, has default ownership of all the hardware, but to gain access in the first place, it must request a mapping from Xen. Xen therefore needs to know and cope with being able to give dom0 a mapping to the nvdimms, without touching the content of the nvidmm itself (so as to avoid corrupting data). Once dom0 has a mapping of the nvdimm, the nvdimm driver can go to work and figure out what is on the DIMM, and which areas are safe to use. At this point, a Xen subsystem in Linux could choose one or more areas to hand back to the hypervisor to use as RAM/other. ~Andrew
[toc] | [prev] | [next] | [standalone]
| From | Dan Williams <dan.j.williams@intel.com> |
|---|---|
| Date | 2016-10-13 21:10 +0200 |
| Subject | Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen |
| Message-ID | <srTya-7E5-9@gated-at.bofh.it> |
| In reply to | #1500403 |
On Thu, Oct 13, 2016 at 9:01 AM, Andrew Cooper <andrew.cooper3@citrix.com> wrote: > On 13/10/16 16:40, Dan Williams wrote: >> On Thu, Oct 13, 2016 at 2:08 AM, Jan Beulich <JBeulich@suse.com> wrote: >> [..] >>>> I think we can do the similar for Xen, like to lay another pseudo >>>> device on /dev/pmem and do the reservation, like 2. in my previous >>>> reply. >>> Well, my opinion certainly doesn't count much here, but I continue to >>> consider this a bad idea. For entities like drivers it may well be >>> appropriate, but I think there ought to be an independent concept >>> of "OS reserved", and in the Xen case this could then be shared >>> between hypervisor and Dom0 kernel. Or if we were to consider Dom0 >>> "just a guest", things should even be the other way around: Xen gets >>> all of the OS reserved space, and Dom0 needs something custom. >> You haven't made the case why Xen is special and other applications of >> persistent memory are not. > > In a Xen system, Xen runs in the baremetal root-mode ring0, and dom0 is > a VM running in ring1/3 with the nvdimm driver. This is the opposite > way around to the KVM model. > > Dom0, being the hardware domain, has default ownership of all the > hardware, but to gain access in the first place, it must request a > mapping from Xen. This is where my understanding the Xen model breaks down. Are you saying dom0 can't access the persistent memory range unless the ring0 agent has metadata storage space for tracking what it maps into dom0? That can't be true because then PCI memory ranges would not work without metadata reserve space. Dom0 still needs to map and write the DIMMs to even set up the struct page reservation, it isn't established by default. > Xen therefore needs to know and cope with being able > to give dom0 a mapping to the nvdimms, without touching the content of > the nvidmm itself (so as to avoid corrupting data). Is it true that this metadata only comes into use when remapping the dom0 discovered range(s) into a guest VM? > Once dom0 has a mapping of the nvdimm, the nvdimm driver can go to work > and figure out what is on the DIMM, and which areas are safe to use. I don't understand this ordering of events. Dom0 needs to have a mapping to even write the on-media structure to indicate a reservation. So, initial dom0 access can't depend on metadata reservation already being present. > At this point, a Xen subsystem in Linux could choose one or more areas > to hand back to the hypervisor to use as RAM/other. To me all this configuration seems to come after the fact. After dom0 sees /dev/pmemX devices, then it can go to work carving it up and writing Xen specific metadata to the range(s). The struct page reservation never comes into the picture. In fact, a raw mode namespace (one without a reservation) could be used in this model, the nvdimm core never needs to know what is happening.
[toc] | [prev] | [next] | [standalone]
| From | Andrew Cooper <andrew.cooper3@citrix.com> |
|---|---|
| Date | 2016-10-13 21:40 +0200 |
| Subject | Re: [Xen-devel] [RFC KERNEL PATCH 0/2] Add Dom0 NVDIMM support for Xen |
| Message-ID | <srU1b-7Po-9@gated-at.bofh.it> |
| In reply to | #1500505 |
On 13/10/16 19:59, Dan Williams wrote: > On Thu, Oct 13, 2016 at 9:01 AM, Andrew Cooper > <andrew.cooper3@citrix.com> wrote: >> On 13/10/16 16:40, Dan Williams wrote: >>> On Thu, Oct 13, 2016 at 2:08 AM, Jan Beulich <JBeulich@suse.com> wrote: >>> [..] >>>>> I think we can do the similar for Xen, like to lay another pseudo >>>>> device on /dev/pmem and do the reservation, like 2. in my previous >>>>> reply. >>>> Well, my opinion certainly doesn't count much here, but I continue to >>>> consider this a bad idea. For entities like drivers it may well be >>>> appropriate, but I think there ought to be an independent concept >>>> of "OS reserved", and in the Xen case this could then be shared >>>> between hypervisor and Dom0 kernel. Or if we were to consider Dom0 >>>> "just a guest", things should even be the other way around: Xen gets >>>> all of the OS reserved space, and Dom0 needs something custom. >>> You haven't made the case why Xen is special and other applications of >>> persistent memory are not. >> In a Xen system, Xen runs in the baremetal root-mode ring0, and dom0 is >> a VM running in ring1/3 with the nvdimm driver. This is the opposite >> way around to the KVM model. >> >> Dom0, being the hardware domain, has default ownership of all the >> hardware, but to gain access in the first place, it must request a >> mapping from Xen. > This is where my understanding the Xen model breaks down. Are you > saying dom0 can't access the persistent memory range unless the ring0 > agent has metadata storage space for tracking what it maps into dom0? No. I am trying to point out that the current suggestion wont work, and needs re-designing. Xen *must* be able to properly configure mappings of the NVDIMM for dom0, *without* modifying any content on the NVDIMM. Otherwise, data corruption will occur. Whether this means no Xen metadata, or the metadata living elsewhere in regular ram, such as the main frametable, is an implementation detail. > >> Once dom0 has a mapping of the nvdimm, the nvdimm driver can go to work >> and figure out what is on the DIMM, and which areas are safe to use. > I don't understand this ordering of events. Dom0 needs to have a > mapping to even write the on-media structure to indicate a > reservation. So, initial dom0 access can't depend on metadata > reservation already being present. I agree. Overall, I think the following is needed. * Xen starts up. ** Xen might find some NVDIMM SPA/MFN ranges in the NFIT table, and needs to note this information somehow. ** Xen might find some Type 7 E820 regions, and needs to note this information somehow. * Xen starts dom0. * Once OSPM is running, a Xen component in Linux needs to collect and report all NVDIMM SPA/MFN regions it knowns about. ** This covers the AML-only case, and the hotplug case. * Dom0 requests a mapping of the NVDIMMs via the usual mechanism. ** This should work, as Xen is aware that there is something there to be mapped (rather than just empty physical address space). * Dom0 finds that some NVDIMM ranges are now available for use (probably modelled as hotplug events). * /dev/pmem $STUFF starts happening as normal. At some pointer later after dom0 policy decisions are made (ultimately, by the host administrator): * If an area of NVDIMM is chosen for Xen to use, Dom0 needs to inform Xen of the SPA/MFN regions which are safe to use. * Xen then incorporates these regions into its idea of RAM, and starts using them for whatever. ~Andrew
[toc] | [prev] | [next] | [standalone]
Page 2 of 3 — ← Prev page 1 [2] 3 Next page →
Back to top | Article view | linux.kernel
csiph-web