Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1220282 > unrolled thread

[PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux

Back to article view | Back to linux.kernel

Contents

  [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux Julien Grall <julien.grall@citrix.com> - 2015-09-07 17:40 +0200
    [PATCH v4 12/20] xen/balloon: Don't rely on the page granularity is the same for Xen and Linux Julien Grall <julien.grall@citrix.com> - 2015-09-07 17:40 +0200
      Re: [PATCH v4 12/20] xen/balloon: Don't rely on the page granularity  is the same for Xen and Linux Stefano Stabellini <stefano.stabellini@eu.citrix.com> - 2015-09-07 18:50 +0200
    [PATCH v4 15/20] block/xen-blkfront: Make it running on 64KB page granularity Julien Grall <julien.grall@citrix.com> - 2015-09-07 17:50 +0200
    [PATCH v4 20/20] arm/xen: Add support for 64KB page granularity Julien Grall <julien.grall@citrix.com> - 2015-09-07 17:50 +0200
    [PATCH v4 17/20] net/xen-netfront: Make it running on 64KB page granularity Julien Grall <julien.grall@citrix.com> - 2015-09-07 17:50 +0200
    [PATCH v4 10/20] xen/xenbus: Use Xen page definition Julien Grall <julien.grall@citrix.com> - 2015-09-07 17:50 +0200
    [PATCH v4 16/20] block/xen-blkback: Make it running on 64KB page granularity Julien Grall <julien.grall@citrix.com> - 2015-09-07 17:50 +0200
    [PATCH v4 18/20] net/xen-netback: Make it running on 64KB page granularity Julien Grall <julien.grall@citrix.com> - 2015-09-07 17:50 +0200
      Re: [PATCH v4 18/20] net/xen-netback: Make it running on 64KB page  granularity Wei Liu <wei.liu2@citrix.com> - 2015-09-07 19:00 +0200
        Re: [Xen-devel] [PATCH v4 18/20] net/xen-netback: Make it running on  64KB page granularity Wei Liu <wei.liu2@citrix.com> - 2015-09-08 13:10 +0200
        Re: [Xen-devel] [PATCH v4 18/20] net/xen-netback: Make it running  on 64KB page granularity Julien Grall <julien.grall@citrix.com> - 2015-09-08 13:10 +0200
    Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB  page in Linux Julien Grall <julien.grall@citrix.com> - 2015-09-11 21:50 +0200
    Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux Roger Pau Monné <roger.pau@citrix.com> - 2015-09-14 11:00 +0200
      Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux Julien Grall <julien.grall@citrix.com> - 2015-09-14 12:50 +0200
        Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux Roger Pau Monné <roger.pau@citrix.com> - 2015-09-14 13:10 +0200
          Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux Julien Grall <julien.grall@citrix.com> - 2015-09-14 13:30 +0200
            Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux Roger Pau Monné <roger.pau@citrix.com> - 2015-09-14 14:10 +0200
              Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux Julien Grall <julien.grall@citrix.com> - 2015-09-14 14:50 +0200
                Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux Roger Pau Monné <roger.pau@citrix.com> - 2015-09-14 16:40 +0200
                  Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux Julien Grall <julien.grall@citrix.com> - 2015-09-14 16:50 +0200
                  Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux Stefano Stabellini <stefano.stabellini@eu.citrix.com> - 2015-09-14 17:00 +0200
                    Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux Roger Pau Monné <roger.pau@citrix.com> - 2015-09-14 17:30 +0200
                      Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux Ian Campbell <ian.campbell@citrix.com> - 2015-09-22 13:20 +0200
              Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux Julien Grall <julien.grall@citrix.com> - 2015-09-18 16:20 +0200
          Re: [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux Arnd Bergmann <arnd@arndb.de> - 2015-09-14 13:40 +0200
            Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB  page in Linux David Vrabel <david.vrabel@citrix.com> - 2015-09-15 15:20 +0200
              Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux Arnd Bergmann <arnd@arndb.de> - 2015-09-15 15:30 +0200
    Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB  page in Linux David Vrabel <david.vrabel@citrix.com> - 2015-09-29 18:30 +0200
      Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB  page in Linux David Vrabel <david.vrabel@citrix.com> - 2015-09-29 18:40 +0200
      Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB  page in Linux Julien Grall <julien.grall@citrix.com> - 2015-09-29 18:40 +0200

Page 2 of 2 — ← Prev page 1 [2]

#1224208

On 14/09/15 15:29, Roger Pau Monné wrote:
>> To give you an example, Centos 7, which will support Xen and only 64KB
>> page granularity, will be supported for years. Dropping any splitting in
>> a short future (3-5 years) will just break those guests to boot on Xen.
> 
> Can't the patches to support 64KB grants be backported to CentOS? In the
> past you said that distros had no problem in picking the needed Xen
> patches for 64KB support, I don't see how that would be different.

Centos 7 is not yet released so it's more easy to get patch in. I don't
know what will be position when the guest is shipped.

I bet they will still want to support 64KB with older Xen (i.e where
64KB grant is not supported).

Anyway, I don't expect to see the 64KB grants support before next year
as my TODO list is already pretty full. So let's talk about it when I
(or someone else) will send an RFC to extend the grant size.

Regards,

-- 
Julien Grall
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1224218

[Multipart message — attachments visible in raw view] — view raw

On Mon, 14 Sep 2015, Roger Pau Monné wrote:
> IMHO this splitting is just a workaround for the fact that we don't have
> a 64KB PV block protocol, and this is the real problem that should be
> solved.

64K is a pure one guest kernel configuration option, not a platform wide
option. The hypervisor interfaces are still the same, the ABI is the
same and all the other guests are still the same, the Xen binary is
still the same.

A 64K block protocol could be a good performance imprevement, but should
not be required to run kernels which have different config options.


> In the long term this will put a burden on all blkfronts (if 64KB pages
> are also used by other OSes), while introducing a 64KB PV block protocol
> will make the blkfront implementation in all OSes very similar to what
> we have now, without replicating the splitting code amongst all the
> possible blkfront implementations.
> 
> Granted that some changes to blkback will be needed in order to support
> mapping 64KB grants, but there are much fewer blkback implementations
> out there than blkfronts.

I don't think we can rely on blkback having something in order to run
new guests, otherwise we break compatibility: new guests won't run on
old hypervisors.

[toc] | [prev] | [next] | [standalone]

#1224249

El 14/09/15 a les 16.54, Stefano Stabellini ha escrit:
> On Mon, 14 Sep 2015, Roger Pau Monné wrote:
>> IMHO this splitting is just a workaround for the fact that we don't have
>> a 64KB PV block protocol, and this is the real problem that should be
>> solved.
> 
> 64K is a pure one guest kernel configuration option, not a platform wide
> option. The hypervisor interfaces are still the same, the ABI is the
> same and all the other guests are still the same, the Xen binary is
> still the same.

Yes, I understand that, but the PV block protocol is missing 64KB page
support, and that's a fact that cannot be ignored. To put an example, is
there a hardware SATA controller on ARM that doesn't support 64KB pages
and needs a similar workaround?

> A 64K block protocol could be a good performance imprevement, but should
> not be required to run kernels which have different config options.
> 
>> In the long term this will put a burden on all blkfronts (if 64KB pages
>> are also used by other OSes), while introducing a 64KB PV block protocol
>> will make the blkfront implementation in all OSes very similar to what
>> we have now, without replicating the splitting code amongst all the
>> possible blkfront implementations.
>>
>> Granted that some changes to blkback will be needed in order to support
>> mapping 64KB grants, but there are much fewer blkback implementations
>> out there than blkfronts.
> 
> I don't think we can rely on blkback having something in order to run
> new guests, otherwise we break compatibility: new guests won't run on
> old hypervisors.

I agree that this is far from ideal, but I don't think it's so
outrageous. For example Linux PVOPS Dom0 kernels require Xen 4.0.1 at
least in order to run, because previous versions lack the necessary
IOAPIC setup hypercall. Also, it won't prevent guests from booting, it
would just prevent them from using blkfront, but you can still get a
root filesystem using iSCSI, NFS or other means.

I'm not saying that we shouldn't take those patches, I'm just saying
that IMHO this is a workaround, and I would like to see a plan and
somebody committed to have it fixed in a proper way, by introducing a
64KB PV block protocol.

Roger.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1230083

On Mon, 2015-09-14 at 17:23 +0200, Roger Pau Monné wrote:
> I'm not saying that we shouldn't take those patches, I'm just saying
> that IMHO this is a workaround, and I would like to see a plan and
> somebody committed to have it fixed in a proper way, by introducing a
> 64KB PV block protocol.

In my view the basic unit of operation for all Xen interfaces (on x86 and
arm at least[0]) is 4K. The peer at either end of a PV protocol should
therefore be entitled to assume that at a minimum the other end supports
operation in 4K mode (i.e. 4K is the baseline).

Operations in larger sizes (which would necessarily be multiples of 4K) are
then an extension which is subject to a negotiation between the two ends,
it doesn't really matter if those larger sizes arise because of the use of
superpages or because the guest is internally using some larger basic page
size (which ARM calls a "granule", and where 64K comes from here).

I think this line of reasoning applies just as strongly to the hypercall
ABI as well BTW, they all use 4K as their basic unit, but might support
extensions to operation on multiples of that (negotiated either via a
specific error return and fallback or via the use of XENFEAT).

Ian.

[0] It's not implausible that some other architecture may exsit which
doesn't use 4K as the basic page szie)
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1227905

Hi Roger,

On 14/09/15 13:08, Roger Pau Monné wrote:
> Well, absolute numbers together with the standard deviation are IMHO the
> best way to provide those figures (ie: see ministat(1) output for
> example), but percentages should also be fine.
> 
> I'm just interested in knowing the performance difference between having
> this patches applied or not when using 4KB pages on the frontend and the
> backend.

I did some benchmark:
	DOM0: 1 VCPU, 4G of RAM based on 4.3-rc1 without this series
	GUEST: 4 VPUs, 4G of RAM, second disk associate to a nullbk device

I used fio with the following options

42sh> fio --name=test --ioengine=libaio --rw=read --numjobs=8 \
--iodepth=32 --time_based=1 --runtime=30 --bs=4KB \
--filename=/dev/xvdb--direct=1 --group_reporting=1 --iodepth_batch=16

The guest is also based on 4.3-rc1 with and without the series.

The overhead with my series is about 0.56%.

Regards,

-- 
Julien Grall
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1224038

On Monday 14 September 2015 13:04:59 Roger Pau Monné wrote:
> > TBH, I'm expecting a small impact to the performance. It would be hard
> > to get the exactly the same performance as today if we keep the helpers
> > to avoid the backend dealing himself with the splitting and page
> > granularity.
> > 
> > Although, if the performance impact is not acceptable, it may be
> > possible to optimize gnttab_foreach_grant_in_range by moving the
> > function inline. The current way to the loop is the fastest I've found
> > (I've wrote a small program to test different way) and we will need it
> > when different of size will be supported.
> 
> I don't expect the performance to drop massively with this patches
> applied, but it would be good to al least have an idea of the impact.

Note that using 64kb pages in Linux tends to destroy performance
in Linux in any case, as the memory consumption for most workloads
explodes. In a virtualized environment you already tend to be
memory constrained, so any measurement should take that into account
and put the extra overhead into perspective to the massive overhead
of running 64kb pages when RAM is tight.

	Arnd
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1225087 — Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux

On 14/09/15 12:32, Arnd Bergmann wrote:
> On Monday 14 September 2015 13:04:59 Roger Pau Monné wrote:
>>> TBH, I'm expecting a small impact to the performance. It would be hard
>>> to get the exactly the same performance as today if we keep the helpers
>>> to avoid the backend dealing himself with the splitting and page
>>> granularity.
>>>
>>> Although, if the performance impact is not acceptable, it may be
>>> possible to optimize gnttab_foreach_grant_in_range by moving the
>>> function inline. The current way to the loop is the fastest I've found
>>> (I've wrote a small program to test different way) and we will need it
>>> when different of size will be supported.
>>
>> I don't expect the performance to drop massively with this patches
>> applied, but it would be good to al least have an idea of the impact.
> 
> Note that using 64kb pages in Linux tends to destroy performance
> in Linux in any case, as the memory consumption for most workloads
> explodes. In a virtualized environment you already tend to be
> memory constrained, so any measurement should take that into account
> and put the extra overhead into perspective to the massive overhead
> of running 64kb pages when RAM is tight.

If this is the case, why are some distros using 64 KiB pages then?

David
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1225095 — Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux

On Tuesday 15 September 2015 14:14:09 David Vrabel wrote:
> On 14/09/15 12:32, Arnd Bergmann wrote:
> > On Monday 14 September 2015 13:04:59 Roger Pau Monné wrote:
> >>> TBH, I'm expecting a small impact to the performance. It would be hard
> >>> to get the exactly the same performance as today if we keep the helpers
> >>> to avoid the backend dealing himself with the splitting and page
> >>> granularity.
> >>>
> >>> Although, if the performance impact is not acceptable, it may be
> >>> possible to optimize gnttab_foreach_grant_in_range by moving the
> >>> function inline. The current way to the loop is the fastest I've found
> >>> (I've wrote a small program to test different way) and we will need it
> >>> when different of size will be supported.
> >>
> >> I don't expect the performance to drop massively with this patches
> >> applied, but it would be good to al least have an idea of the impact.
> > 
> > Note that using 64kb pages in Linux tends to destroy performance
> > in Linux in any case, as the memory consumption for most workloads
> > explodes. In a virtualized environment you already tend to be
> > memory constrained, so any measurement should take that into account
> > and put the extra overhead into perspective to the massive overhead
> > of running 64kb pages when RAM is tight.
> 
> If this is the case, why are some distros using 64 KiB pages then?

I believe that IBM started this on PowerPC64 when they found that it
performs better in microbenchmarks and for certain database workloads
when you do have enough memory and do not work with small files or
many processes.

I also believe that the AIX memory management code is designed differently
enough that it suffers less from large pages (or suffers more from
small pages), so the hardware designers pushed Linux to do the same as AIX.

Why you'd do it on ARM64 I have no idea at all, other than copying what
IBM did on PowerPC. If your TLB is too small, hardware designers may
of course try to push the OS to using pages as large as possible for the
low-level benchmarks to suck less, but in general, you'd be better off
with transparent hugepages and 4kb or 8kb pages for real workloads.

	Arnd
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1235363 — Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux

On 07/09/15 16:33, Julien Grall wrote:
> 
> A branch based on the latest xentip/for-linus-4.3 can be found here:
> 
> git://xenbits.xen.org/people/julieng/linux-arm.git branch xen-64k-v4

This has too many conflicts with the
git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip.git for-linus-4.4
branch.

Please rebase.

David
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1235378 — Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux

On 29/09/15 17:33, Julien Grall wrote:
> Hi David,
> 
> On 29/09/15 17:27, David Vrabel wrote:
>> On 07/09/15 16:33, Julien Grall wrote:
>>>
>>> A branch based on the latest xentip/for-linus-4.3 can be found here:
>>>
>>> git://xenbits.xen.org/people/julieng/linux-arm.git branch xen-64k-v4
>>
>> This has too many conflicts with the
>> git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip.git for-linus-4.4
>> branch.
>>
>> Please rebase.
> 
> I was about to send a new version rebased (no changes in it so far
> except for rebasing purpose). I will do it on top of for-linus-4.4.
> I will also include the swiotlb changes at the end as it was reviewed by
> Stefano.

Thanks.

> BTW, some of the patches should, strictly speaking, go via the block and
> network subsystem. Although, they heavily depends on other patches in
> this series. Would it be fine to send all through xentip?

Yes, if they've been acked by the appropriate maintainer.

David
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1235380 — Re: [Xen-devel] [PATCH v4 00/20] xen/arm64: Add support for 64KB page in Linux

Hi David,

On 29/09/15 17:27, David Vrabel wrote:
> On 07/09/15 16:33, Julien Grall wrote:
>>
>> A branch based on the latest xentip/for-linus-4.3 can be found here:
>>
>> git://xenbits.xen.org/people/julieng/linux-arm.git branch xen-64k-v4
> 
> This has too many conflicts with the
> git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip.git for-linus-4.4
> branch.
> 
> Please rebase.

I was about to send a new version rebased (no changes in it so far
except for rebasing purpose). I will do it on top of for-linus-4.4.
I will also include the swiotlb changes at the end as it was reviewed by
Stefano.

BTW, some of the patches should, strictly speaking, go via the block and
network subsystem. Although, they heavily depends on other patches in
this series. Would it be fine to send all through xentip?

Regards,

-- 
Julien Grall
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.kernel

csiph-web