Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1591098 > unrolled thread
| Started by | Igor Mammedov <imammedo@redhat.com> |
|---|---|
| First post | 2017-03-02 15:00 +0100 |
| Last post | 2017-03-10 23:10 +0100 |
| Articles | 20 on this page of 29 — 7 participants |
Back to article view | Back to linux.kernel
This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by
below is the oldest one visible, not the original post.
Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks Igor Mammedov <imammedo@redhat.com> - 2017-03-02 15:00 +0100
Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks Michal Hocko <mhocko@kernel.org> - 2017-03-02 16:10 +0100
Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks Igor Mammedov <imammedo@redhat.com> - 2017-03-02 18:20 +0100
Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks Michal Hocko <mhocko@kernel.org> - 2017-03-03 10:10 +0100
Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks Igor Mammedov <imammedo@redhat.com> - 2017-03-03 18:40 +0100
Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks Michal Hocko <mhocko@kernel.org> - 2017-03-06 16:10 +0100
Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks Igor Mammedov <imammedo@redhat.com> - 2017-03-07 13:50 +0100
Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks Michal Hocko <mhocko@kernel.org> - 2017-03-09 14:00 +0100
WTH is going on with memory hotplug sysf interface (was: Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks) Michal Hocko <mhocko@kernel.org> - 2017-03-10 15:00 +0100
Re: WTH is going on with memory hotplug sysf interface (was: Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks) Michal Hocko <mhocko@kernel.org> - 2017-03-10 17:00 +0100
Re: WTH is going on with memory hotplug sysf interface (was: Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks) Reza Arbab <arbab@linux.vnet.ibm.com> - 2017-03-10 20:10 +0100
Re: WTH is going on with memory hotplug sysf interface (was: Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks) Michal Hocko <mhocko@kernel.org> - 2017-03-13 10:30 +0100
Re: WTH is going on with memory hotplug sysf interface (was: Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks) Reza Arbab <arbab@linux.vnet.ibm.com> - 2017-03-13 16:00 +0100
Re: WTH is going on with memory hotplug sysf interface (was: Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks) Michal Hocko <mhocko@kernel.org> - 2017-03-13 16:20 +0100
Re: WTH is going on with memory hotplug sysf interface (was: Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks) Andi Kleen <ak@linux.intel.com> - 2017-03-14 00:20 +0100
Re: WTH is going on with memory hotplug sysf interface Yasuaki Ishimatsu <yasu.isimatu@gmail.com> - 2017-03-10 18:40 +0100
Re: WTH is going on with memory hotplug sysf interface Michal Hocko <mhocko@kernel.org> - 2017-03-13 10:30 +0100
Re: WTH is going on with memory hotplug sysf interface (was: Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks) Igor Mammedov <imammedo@redhat.com> - 2017-03-13 11:40 +0100
Re: WTH is going on with memory hotplug sysf interface (was: Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks) Michal Hocko <mhocko@kernel.org> - 2017-03-13 11:50 +0100
Re: WTH is going on with memory hotplug sysf interface (was: Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks) Igor Mammedov <imammedo@redhat.com> - 2017-03-13 15:00 +0100
Re: WTH is going on with memory hotplug sysf interface (was: Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks) Michal Hocko <mhocko@kernel.org> - 2017-03-13 15:40 +0100
Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks Igor Mammedov <imammedo@redhat.com> - 2017-03-13 12:00 +0100
Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks Michal Hocko <mhocko@kernel.org> - 2017-03-13 13:30 +0100
Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks Vitaly Kuznetsov <vkuznets@redhat.com> - 2017-03-13 14:00 +0100
Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks Michal Hocko <mhocko@kernel.org> - 2017-03-13 14:20 +0100
Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks Vitaly Kuznetsov <vkuznets@redhat.com> - 2017-03-13 14:50 +0100
Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks Michal Hocko <mhocko@kernel.org> - 2017-03-13 15:40 +0100
Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks Vitaly Kuznetsov <vkuznets@redhat.com> - 2017-03-13 16:20 +0100
Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks Daniel Kiper <daniel.kiper@oracle.com> - 2017-03-10 23:10 +0100
Page 1 of 2 [1] 2 Next page →
| From | Igor Mammedov <imammedo@redhat.com> |
|---|---|
| Date | 2017-03-02 15:00 +0100 |
| Subject | Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks |
| Message-ID | <tgzqW-2ZE-19@gated-at.bofh.it> |
On Mon 27-02-17 16:43:04, Michal Hocko wrote:
> On Mon 27-02-17 12:25:10, Heiko Carstens wrote:
> > On Mon, Feb 27, 2017 at 11:02:09AM +0100, Vitaly Kuznetsov wrote:
> > > A couple of other thoughts:
> > > 1) Having all newly added memory online ASAP is probably what people
> > > want for all virtual machines.
> >
> > This is not true for s390. On s390 we have "standby" memory that a guest
> > sees and potentially may use if it sets it online. Every guest that sets
> > memory offline contributes to the hypervisor's standby memory pool, while
> > onlining standby memory takes memory away from the standby pool.
> >
> > The use-case is that a system administrator in advance knows the maximum
> > size a guest will ever have and also defines how much memory should be used
> > at boot time. The difference is standby memory.
> >
> > Auto-onlining of standby memory is the last thing we want.
I don't know much about anything other than x86 so all comments
below are from that point of view,
archetectures that don't need auto online can keep current default
> > > Unfortunately, we have additional complexity with memory zones
> > > (ZONE_NORMAL, ZONE_MOVABLE) and in some cases manual intervention is
> > > required. Especially, when further unplug is expected.
> >
> > This also is a reason why auto-onlining doesn't seem be the best way.
When trying to support memory unplug on guest side in RHEL7,
experience shows otherwise. Simplistic udev rule which onlines
added block doesn't work in case one wants to online it as movable.
Hotplugged blocks in current kernel should be onlined in reverse
order to online blocks as movable depending on adjacent blocks zone.
Which means simple udev rule isn't usable since it gets event from
the first to the last hotplugged block order. So now we would have
to write a daemon that would
- watch for all blocks in hotplugged memory appear (how would it know)
- online them in right order (order might also be different depending
on kernel version)
-- it becomes even more complicated in NUMA case when there are
multiple zones and kernel would have to provide user-space
with information about zone maps
In short current experience shows that userspace approach
- doesn't solve issues that Vitaly has been fixing (i.e. onlining
fast and/or under memory pressure) when udev (or something else
might be killed)
- doesn't reduce overall system complexity, it only gets worse
as user-space handler needs to know a lot about kernel internals
and implementation details/kernel versions to work properly
It's might be not easy but doing onlining in kernel on the other hand is:
- faster
- more reliable (can't be killed under memory pressure)
- kernel has access to all info needed for onlining and how it
internals work to implement auto-online correctly
- since there is no need to mantain ABI for user-space
(zones layout/ordering/maybe something else), kernel is
free change internal implemetation without breaking userspace
(currently hotplug+unplug doesn't work reliably and we might
need something more flexible than zones)
That's direction of research in progress, i.e. making kernel
implementation better instead of putting responsibility on
user-space to deal with kernel shortcomings.
> Can you imagine any situation when somebody actually might want to have
> this knob enabled? From what I understand it doesn't seem to be the
> case.
For x86:
* this config option is enabled by default in recent Fedora,
* RHEL6 ships similar downstream patches to do the same thing for years
* RHEL7 has udev rule (because there wasn't kernel side solution at fork time)
that auto-onlines it unconditionally, Vitaly might backport it later
when he has time.
Not linux kernel but still auto online policy is used by Windows
both on baremetal and guest configurations.
That's somewhat shows that current defaults upstream on x86
might be not what end-users wish for.
When auto_online_blocks were introduced, Vitaly has been
conservative and left current upstream defaults where they were
lest it would break someone else setup but allowing downstreams
set their own auto-online policy, eventually we might switch it
upstream too.
[toc] | [next] | [standalone]
| From | Michal Hocko <mhocko@kernel.org> |
|---|---|
| Date | 2017-03-02 16:10 +0100 |
| Message-ID | <tgAwG-3Yt-15@gated-at.bofh.it> |
| In reply to | #1591098 |
On Thu 02-03-17 14:53:48, Igor Mammedov wrote: [...] > When trying to support memory unplug on guest side in RHEL7, > experience shows otherwise. Simplistic udev rule which onlines > added block doesn't work in case one wants to online it as movable. > > Hotplugged blocks in current kernel should be onlined in reverse > order to online blocks as movable depending on adjacent blocks zone. Could you be more specific please? Setting online_movable from the udev rule should just work regardless of the ordering or the state of other memblocks. If that doesn't work I would call it a bug. > Which means simple udev rule isn't usable since it gets event from > the first to the last hotplugged block order. So now we would have > to write a daemon that would > - watch for all blocks in hotplugged memory appear (how would it know) > - online them in right order (order might also be different depending > on kernel version) > -- it becomes even more complicated in NUMA case when there are > multiple zones and kernel would have to provide user-space > with information about zone maps > > In short current experience shows that userspace approach > - doesn't solve issues that Vitaly has been fixing (i.e. onlining > fast and/or under memory pressure) when udev (or something else > might be killed) yeah and that is why the patch does the onlining from the kernel. > > Can you imagine any situation when somebody actually might want to have > > this knob enabled? From what I understand it doesn't seem to be the > > case. > For x86: > * this config option is enabled by default in recent Fedora, How do you want to support usecases which really want to online memory as movable? Do you expect those users to disable the option because unless I am missing something the in kernel auto onlining only supporst regular onlining. -- Michal Hocko SUSE Labs
[toc] | [prev] | [next] | [standalone]
| From | Igor Mammedov <imammedo@redhat.com> |
|---|---|
| Date | 2017-03-02 18:20 +0100 |
| Message-ID | <tgCyt-5uu-1@gated-at.bofh.it> |
| In reply to | #1591173 |
On Thu, 2 Mar 2017 15:28:16 +0100 Michal Hocko <mhocko@kernel.org> wrote: > On Thu 02-03-17 14:53:48, Igor Mammedov wrote: > [...] > > When trying to support memory unplug on guest side in RHEL7, > > experience shows otherwise. Simplistic udev rule which onlines > > added block doesn't work in case one wants to online it as movable. > > > > Hotplugged blocks in current kernel should be onlined in reverse > > order to online blocks as movable depending on adjacent blocks zone. > > Could you be more specific please? Setting online_movable from the udev > rule should just work regardless of the ordering or the state of other > memblocks. If that doesn't work I would call it a bug. It's rather an implementation constrain than a bug for details and workaround patch see [1] https://bugzilla.redhat.com/show_bug.cgi?id=1314306#c7 patch attached there is limited by another memory hotplug issue, which is NORMAL/MOVABLE zone balance, if kernel runs on configuration where the most of memory is hot-removable kernel might experience lack of memory in zone NORMAL. > > > Which means simple udev rule isn't usable since it gets event from > > the first to the last hotplugged block order. So now we would have > > to write a daemon that would > > - watch for all blocks in hotplugged memory appear (how would it know) > > - online them in right order (order might also be different depending > > on kernel version) > > -- it becomes even more complicated in NUMA case when there are > > multiple zones and kernel would have to provide user-space > > with information about zone maps > > > > In short current experience shows that userspace approach > > - doesn't solve issues that Vitaly has been fixing (i.e. onlining > > fast and/or under memory pressure) when udev (or something else > > might be killed) > > yeah and that is why the patch does the onlining from the kernel. onlining in this patch is limited to hyperv and patch breaks auto-online on x86 kvm/vmware/baremetal as they reuse the same hotplug path. > > > Can you imagine any situation when somebody actually might want to have > > > this knob enabled? From what I understand it doesn't seem to be the > > > case. > > For x86: > > * this config option is enabled by default in recent Fedora, > > How do you want to support usecases which really want to online memory > as movable? Do you expect those users to disable the option because > unless I am missing something the in kernel auto onlining only supporst > regular onlining. current auto onlining config option does what it's been designed for, i.e. it onlines hotplugged memory. It's possible for non average Fedora user to override default (commit 86dd995d6) if she/he needs non default behavior (i.e. user knows how to online manually and/or can write a daemon that would handle all of nuances of kernel in use). For the rest when Fedora is used in cloud and user increases memory via management interface of whatever cloud she/he uses, it just works. So it's choice of distribution to pick its own default that makes majority of user-base happy and this patch removes it without taking that in consideration. How to online memory is different issue not related to this patch, current default onlining as ZONE_NORMAL works well for scaling up VMs. Memory unplug is rather new and it doesn't work reliably so far, moving onlining to user-space won't really help. Further work is need to be done so that it would work reliably. Now about the question of onlining removable memory as movable, x86 kernel is able to get info, if hotadded memory is removable, from ACPI subsystem and online it as movable one without any intervention from user-space where it's hard to do so, as patch[1] shows. Problem is still researched and when we figure out how to fix hot-remove issues we might enable auto-onlining by default for x86.
[toc] | [prev] | [next] | [standalone]
| From | Michal Hocko <mhocko@kernel.org> |
|---|---|
| Date | 2017-03-03 10:10 +0100 |
| Message-ID | <tgRnR-7or-49@gated-at.bofh.it> |
| In reply to | #1591301 |
On Thu 02-03-17 18:03:15, Igor Mammedov wrote: > On Thu, 2 Mar 2017 15:28:16 +0100 > Michal Hocko <mhocko@kernel.org> wrote: > > > On Thu 02-03-17 14:53:48, Igor Mammedov wrote: > > [...] > > > When trying to support memory unplug on guest side in RHEL7, > > > experience shows otherwise. Simplistic udev rule which onlines > > > added block doesn't work in case one wants to online it as movable. > > > > > > Hotplugged blocks in current kernel should be onlined in reverse > > > order to online blocks as movable depending on adjacent blocks zone. > > > > Could you be more specific please? Setting online_movable from the udev > > rule should just work regardless of the ordering or the state of other > > memblocks. If that doesn't work I would call it a bug. > It's rather an implementation constrain than a bug > for details and workaround patch see > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1314306#c7 "You are not authorized to access bug #1314306" could you paste the reasoning here please? > patch attached there is limited by another memory hotplug > issue, which is NORMAL/MOVABLE zone balance, if kernel runs > on configuration where the most of memory is hot-removable > kernel might experience lack of memory in zone NORMAL. yes and that is an inherent problem of movable memory. > > > Which means simple udev rule isn't usable since it gets event from > > > the first to the last hotplugged block order. So now we would have > > > to write a daemon that would > > > - watch for all blocks in hotplugged memory appear (how would it know) > > > - online them in right order (order might also be different depending > > > on kernel version) > > > -- it becomes even more complicated in NUMA case when there are > > > multiple zones and kernel would have to provide user-space > > > with information about zone maps > > > > > > In short current experience shows that userspace approach > > > - doesn't solve issues that Vitaly has been fixing (i.e. onlining > > > fast and/or under memory pressure) when udev (or something else > > > might be killed) > > > > yeah and that is why the patch does the onlining from the kernel. > onlining in this patch is limited to hyperv and patch breaks > auto-online on x86 kvm/vmware/baremetal as they reuse the same > hotplug path. Those can use the udev or do you see any reason why they couldn't? > > > > Can you imagine any situation when somebody actually might want to have > > > > this knob enabled? From what I understand it doesn't seem to be the > > > > case. > > > For x86: > > > * this config option is enabled by default in recent Fedora, > > > > How do you want to support usecases which really want to online memory > > as movable? Do you expect those users to disable the option because > > unless I am missing something the in kernel auto onlining only supporst > > regular onlining. > > current auto onlining config option does what it's been designed for, > i.e. it onlines hotplugged memory. > It's possible for non average Fedora user to override default > (commit 86dd995d6) if she/he needs non default behavior > (i.e. user knows how to online manually and/or can write > a daemon that would handle all of nuances of kernel in use). > > For the rest when Fedora is used in cloud and user increases memory > via management interface of whatever cloud she/he uses, it just works. > > So it's choice of distribution to pick its own default that makes > majority of user-base happy and this patch removes it without taking > that in consideration. You still can have a udev rule to achive the same thing for non-ballooning based hotplug. > How to online memory is different issue not related to this patch, > current default onlining as ZONE_NORMAL works well for scaling > up VMs. > > Memory unplug is rather new and it doesn't work reliably so far, > moving onlining to user-space won't really help. Further work > is need to be done so that it would work reliably. The main problem I have with this is that this is a limited usecase driven configuration knob which doesn't work properly for other usecases (namely movable online once your distribution choses to set the config option to auto online). There is a userspace solution for this so this shouldn't have been merged in the first place! It sneaked a proper review process (linux-api wasn't CC to get a broader attenttion) which is really sad. So unless this causes a major regression which would be hard to fix I will submit the patch for inclusion. -- Michal Hocko SUSE Labs
[toc] | [prev] | [next] | [standalone]
| From | Igor Mammedov <imammedo@redhat.com> |
|---|---|
| Date | 2017-03-03 18:40 +0100 |
| Message-ID | <tgZln-4ue-7@gated-at.bofh.it> |
| In reply to | #1591779 |
On Fri, 3 Mar 2017 09:27:23 +0100
Michal Hocko <mhocko@kernel.org> wrote:
> On Thu 02-03-17 18:03:15, Igor Mammedov wrote:
> > On Thu, 2 Mar 2017 15:28:16 +0100
> > Michal Hocko <mhocko@kernel.org> wrote:
> >
> > > On Thu 02-03-17 14:53:48, Igor Mammedov wrote:
> > > [...]
> > > > When trying to support memory unplug on guest side in RHEL7,
> > > > experience shows otherwise. Simplistic udev rule which onlines
> > > > added block doesn't work in case one wants to online it as movable.
> > > >
> > > > Hotplugged blocks in current kernel should be onlined in reverse
> > > > order to online blocks as movable depending on adjacent blocks zone.
> > >
> > > Could you be more specific please? Setting online_movable from the udev
> > > rule should just work regardless of the ordering or the state of other
> > > memblocks. If that doesn't work I would call it a bug.
> > It's rather an implementation constrain than a bug
> > for details and workaround patch see
> > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1314306#c7
>
> "You are not authorized to access bug #1314306"
Sorry,
I've made it public, related comments and patch should be accessible now
(code snippets in BZ are based on older kernel but logic is still the same upstream)
> could you paste the reasoning here please?
sure here is reproducer:
start VM with CLI:
qemu-system-x86_64 -enable-kvm -m size=1G,slots=2,maxmem=4G -numa node \
-object memory-backend-ram,id=m1,size=1G -device pc-dimm,node=0,memdev=m1 \
/path/to/guest_image
then in guest dimm1 blocks are from 32-39
echo online_movable > /sys/devices/system/memory/memory32/state
-bash: echo: write error: Invalid argument
in current mainline kernel it triggers following code path:
online_pages()
...
if (online_type == MMOP_ONLINE_KERNEL) {
if (!zone_can_shift(pfn, nr_pages, ZONE_NORMAL, &zone_shift))
return -EINVAL;
zone_can_shift()
...
if (idx < target) {
/* pages must be at end of current zone */
if (pfn + nr_pages != zone_end_pfn(zone))
return false;
since we are trying to online as movable not the last section in
ZONE_NORMAL.
Here is what makes hotplugged memory end up in ZONE_NORMAL:
acpi_memory_enable_device() -> add_memory -> add_memory_resource ->
-> arch/x86/mm/init_64.c
/*
* Memory is added always to NORMAL zone. This means you will never get
* additional DMA/DMA32 memory.
*/
int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
{
...
struct zone *zone = pgdat->node_zones +
zone_for_memory(nid, start, size, ZONE_NORMAL, for_device);
i.e. all hot-plugged memory modules always go to ZONE_NORMAL
and only the first/last block in zone is allowed to be moved
to another zone. Patch [1] tries to fix issue by assigning
removable memory resource to movable zone so hotplugged+removable
blocks look like:
movable normal, movable, movable
instead of current:
normal, normal, normal movable
but then with this fixed as suggested, auto online by default
should work just fine in kernel with normal and movable zones
without any need for user-space.
> > patch attached there is limited by another memory hotplug
> > issue, which is NORMAL/MOVABLE zone balance, if kernel runs
> > on configuration where the most of memory is hot-removable
> > kernel might experience lack of memory in zone NORMAL.
>
> yes and that is an inherent problem of movable memory.
>
> > > > Which means simple udev rule isn't usable since it gets event from
> > > > the first to the last hotplugged block order. So now we would have
> > > > to write a daemon that would
> > > > - watch for all blocks in hotplugged memory appear (how would it know)
> > > > - online them in right order (order might also be different depending
> > > > on kernel version)
> > > > -- it becomes even more complicated in NUMA case when there are
> > > > multiple zones and kernel would have to provide user-space
> > > > with information about zone maps
> > > >
> > > > In short current experience shows that userspace approach
> > > > - doesn't solve issues that Vitaly has been fixing (i.e. onlining
> > > > fast and/or under memory pressure) when udev (or something else
> > > > might be killed)
> > >
> > > yeah and that is why the patch does the onlining from the kernel.
> > onlining in this patch is limited to hyperv and patch breaks
> > auto-online on x86 kvm/vmware/baremetal as they reuse the same
> > hotplug path.
>
> Those can use the udev or do you see any reason why they couldn't?
Reasons are above, under >>>> and >> quotations, patch breaks
what Vitaly's fixed (including kvm/vmware usecases) i.e. udev/some
user-space process could be killed if hotplugged memory isn't onlined
fast enough leading to service termination and/or memory not
being onlined at all (if udev is killed)
Currently udev rule is not usable and one needs a daemon
which would correctly do onlining and keep zone balance
even for simple case usecase of 1 normal and 1 movable zone.
And it gets more complicated in case of multiple numa nodes
with multiple zones.
> > > > > Can you imagine any situation when somebody actually might want to have
> > > > > this knob enabled? From what I understand it doesn't seem to be the
> > > > > case.
> > > > For x86:
> > > > * this config option is enabled by default in recent Fedora,
> > >
> > > How do you want to support usecases which really want to online memory
> > > as movable? Do you expect those users to disable the option because
> > > unless I am missing something the in kernel auto onlining only supporst
> > > regular onlining.
> >
> > current auto onlining config option does what it's been designed for,
> > i.e. it onlines hotplugged memory.
> > It's possible for non average Fedora user to override default
> > (commit 86dd995d6) if she/he needs non default behavior
> > (i.e. user knows how to online manually and/or can write
> > a daemon that would handle all of nuances of kernel in use).
> >
> > For the rest when Fedora is used in cloud and user increases memory
> > via management interface of whatever cloud she/he uses, it just works.
> >
> > So it's choice of distribution to pick its own default that makes
> > majority of user-base happy and this patch removes it without taking
> > that in consideration.
>
> You still can have a udev rule to achive the same thing for
> non-ballooning based hotplug.
not in case when system is under load, udev path might be slow
and udev might be killed by OOM leading to permanent disablement
of memory onlining.
> > How to online memory is different issue not related to this patch,
> > current default onlining as ZONE_NORMAL works well for scaling
> > up VMs.
> >
> > Memory unplug is rather new and it doesn't work reliably so far,
> > moving onlining to user-space won't really help. Further work
> > is need to be done so that it would work reliably.
>
> The main problem I have with this is that this is a limited usecase
> driven configuration knob which doesn't work properly for other usecases
> (namely movable online once your distribution choses to set the config
> option to auto online).
it works for default usecase in Fedora and non-default
movable can be used with
1) removable memory auto-online as movable in kernel, like
patch [1] would make movable hotplugged memory
(when I have time I'll try to work on it)
2) (or in worst case due to lack of alternative) explicitly
disabled auto-online on kernel CLI + onlining daemon
(since udev isn't working in current kernel due to ordering issue)
> There is a userspace solution for this so this
> shouldn't have been merged in the first place!
Sorry, currently user-space udev solution doesn't work nor
will it work reliably in extreme conditions.
> It sneaked a proper review
> process (linux-api wasn't CC to get a broader attenttion) which is
> really sad.
get_maintainer.pl doesn't lists linux-api for 31bc3858ea3e,
MAINTAINERS should be fixed if linux-api were to be CCed.
> So unless this causes a major regression which would be hard to fix I
> will submit the patch for inclusion.
it will be a major regression due to lack of daemon that
could online fast and can't be killed on OOM. So this
clean up patch does break used feature without providing
a viable alternative.
I wouldn't object to removing config option as in this patch
if memory were onlined for x86 by default but that's not the case yet.
[1] https://bugzilla.redhat.com/attachment.cgi?id=1146332
[toc] | [prev] | [next] | [standalone]
| From | Michal Hocko <mhocko@kernel.org> |
|---|---|
| Date | 2017-03-06 16:10 +0100 |
| Message-ID | <ti2qS-1v4-17@gated-at.bofh.it> |
| In reply to | #1592163 |
On Fri 03-03-17 18:34:22, Igor Mammedov wrote:
> On Fri, 3 Mar 2017 09:27:23 +0100
> Michal Hocko <mhocko@kernel.org> wrote:
>
> > On Thu 02-03-17 18:03:15, Igor Mammedov wrote:
> > > On Thu, 2 Mar 2017 15:28:16 +0100
> > > Michal Hocko <mhocko@kernel.org> wrote:
> > >
> > > > On Thu 02-03-17 14:53:48, Igor Mammedov wrote:
> > > > [...]
> > > > > When trying to support memory unplug on guest side in RHEL7,
> > > > > experience shows otherwise. Simplistic udev rule which onlines
> > > > > added block doesn't work in case one wants to online it as movable.
> > > > >
> > > > > Hotplugged blocks in current kernel should be onlined in reverse
> > > > > order to online blocks as movable depending on adjacent blocks zone.
> > > >
> > > > Could you be more specific please? Setting online_movable from the udev
> > > > rule should just work regardless of the ordering or the state of other
> > > > memblocks. If that doesn't work I would call it a bug.
> > > It's rather an implementation constrain than a bug
> > > for details and workaround patch see
> > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1314306#c7
> >
> > "You are not authorized to access bug #1314306"
> Sorry,
> I've made it public, related comments and patch should be accessible now
> (code snippets in BZ are based on older kernel but logic is still the same upstream)
>
> > could you paste the reasoning here please?
> sure here is reproducer:
> start VM with CLI:
> qemu-system-x86_64 -enable-kvm -m size=1G,slots=2,maxmem=4G -numa node \
> -object memory-backend-ram,id=m1,size=1G -device pc-dimm,node=0,memdev=m1 \
> /path/to/guest_image
>
> then in guest dimm1 blocks are from 32-39
>
> echo online_movable > /sys/devices/system/memory/memory32/state
> -bash: echo: write error: Invalid argument
>
> in current mainline kernel it triggers following code path:
>
> online_pages()
> ...
> if (online_type == MMOP_ONLINE_KERNEL) {
> if (!zone_can_shift(pfn, nr_pages, ZONE_NORMAL, &zone_shift))
> return -EINVAL;
Are you sure? I would expect MMOP_ONLINE_MOVABLE here
> zone_can_shift()
> ...
> if (idx < target) {
> /* pages must be at end of current zone */
> if (pfn + nr_pages != zone_end_pfn(zone))
> return false;
>
> since we are trying to online as movable not the last section in
> ZONE_NORMAL.
>
> Here is what makes hotplugged memory end up in ZONE_NORMAL:
> acpi_memory_enable_device() -> add_memory -> add_memory_resource ->
> -> arch/x86/mm/init_64.c
>
> /*
> * Memory is added always to NORMAL zone. This means you will never get
> * additional DMA/DMA32 memory.
> */
> int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
> {
> ...
> struct zone *zone = pgdat->node_zones +
> zone_for_memory(nid, start, size, ZONE_NORMAL, for_device);
>
> i.e. all hot-plugged memory modules always go to ZONE_NORMAL
> and only the first/last block in zone is allowed to be moved
> to another zone. Patch [1] tries to fix issue by assigning
> removable memory resource to movable zone so hotplugged+removable
> blocks look like:
> movable normal, movable, movable
> instead of current:
> normal, normal, normal movable
Hmm, this code is confusing and clean as mud. I have to stare there some
more but AFAIK zones shouldn't have problems with holes so the only
thing we have to guarantee is that different zones do not overlap. So
this smells like a bug rather than the ineherent implementation
limitation.
[...]
> > > > > Which means simple udev rule isn't usable since it gets event from
> > > > > the first to the last hotplugged block order. So now we would have
> > > > > to write a daemon that would
> > > > > - watch for all blocks in hotplugged memory appear (how would it know)
> > > > > - online them in right order (order might also be different depending
> > > > > on kernel version)
> > > > > -- it becomes even more complicated in NUMA case when there are
> > > > > multiple zones and kernel would have to provide user-space
> > > > > with information about zone maps
> > > > >
> > > > > In short current experience shows that userspace approach
> > > > > - doesn't solve issues that Vitaly has been fixing (i.e. onlining
> > > > > fast and/or under memory pressure) when udev (or something else
> > > > > might be killed)
> > > >
> > > > yeah and that is why the patch does the onlining from the kernel.
> > > onlining in this patch is limited to hyperv and patch breaks
> > > auto-online on x86 kvm/vmware/baremetal as they reuse the same
> > > hotplug path.
> >
> > Those can use the udev or do you see any reason why they couldn't?
>
> Reasons are above, under >>>> and >> quotations, patch breaks
> what Vitaly's fixed (including kvm/vmware usecases) i.e. udev/some
> user-space process could be killed if hotplugged memory isn't onlined
> fast enough leading to service termination and/or memory not
> being onlined at all (if udev is killed)
OK, so from the discussion so far I have learned that this would be
problem _only_ if we are trying to hotplug a _lot_ of memory at once
(~1.5% of the online memory is needed). I am kind of skeptical this is
a reasonable usecase. Who is going to hotadd 8G to 256M machine (which
would eat half of the available memory which is still quite far from
OOM)? Even if the memory balloning uses hotplug then such a grow sounds
a bit excessive.
> Currently udev rule is not usable and one needs a daemon
> which would correctly do onlining and keep zone balance
> even for simple case usecase of 1 normal and 1 movable zone.
> And it gets more complicated in case of multiple numa nodes
> with multiple zones.
That sounds to be more related to the current implementation than
anything else and as such is not a reason to invent specific user
visible api. Btw. you are talking about movable zones byt the auto
onlining doesn't allow to auto online movable memory. So either I miss
your point or I am utterly confused.
[...]
> > > Memory unplug is rather new and it doesn't work reliably so far,
> > > moving onlining to user-space won't really help. Further work
> > > is need to be done so that it would work reliably.
> >
> > The main problem I have with this is that this is a limited usecase
> > driven configuration knob which doesn't work properly for other usecases
> > (namely movable online once your distribution choses to set the config
> > option to auto online).
>
> it works for default usecase in Fedora and non-default
> movable can be used with
> 1) removable memory auto-online as movable in kernel, like
> patch [1] would make movable hotplugged memory
> (when I have time I'll try to work on it)
> 2) (or in worst case due to lack of alternative) explicitly
> disabled auto-online on kernel CLI + onlining daemon
> (since udev isn't working in current kernel due to ordering issue)
So I fail to see how this can work. Say the default will auto online
all the added memory. This will be all in Zone Normal because the auto
onlining doesn't do online_movable. Now I would like to online the
memory as movable but that might fail because some kernel objects might
be already sitting there so the offline would fail.
> > There is a userspace solution for this so this
> > shouldn't have been merged in the first place!
> Sorry, currently user-space udev solution doesn't work nor
> will it work reliably in extreme conditions.
>
> > It sneaked a proper review
> > process (linux-api wasn't CC to get a broader attenttion) which is
> > really sad.
>
> get_maintainer.pl doesn't lists linux-api for 31bc3858ea3e,
> MAINTAINERS should be fixed if linux-api were to be CCed.
user visible APIs _should_ be discussed at this mailing list regardless
what get_maintainer.pl says. This is not about who is the maintainer but
about getting as wide audience for things that would have to be
maintained basically for ever.
> > So unless this causes a major regression which would be hard to fix I
> > will submit the patch for inclusion.
> it will be a major regression due to lack of daemon that
> could online fast and can't be killed on OOM. So this
> clean up patch does break used feature without providing
> a viable alternative.
So let's discuss the current memory hotplug shortcomings and get rid of
the crud which developed on top. I will start by splitting up the patch
into 3 parts. Do the auto online thing from the HyperV and xen balloning
drivers and dropping the config option and finally drop the sysfs knob.
The last patch might be NAKed and I can live with that as long as the
reasoning is proper and there is a general consensus on that.
--
Michal Hocko
SUSE Labs
[toc] | [prev] | [next] | [standalone]
| From | Igor Mammedov <imammedo@redhat.com> |
|---|---|
| Date | 2017-03-07 13:50 +0100 |
| Message-ID | <timIX-7Bl-37@gated-at.bofh.it> |
| In reply to | #1593465 |
On Mon, 6 Mar 2017 15:54:17 +0100
Michal Hocko <mhocko@kernel.org> wrote:
> On Fri 03-03-17 18:34:22, Igor Mammedov wrote:
> > On Fri, 3 Mar 2017 09:27:23 +0100
> > Michal Hocko <mhocko@kernel.org> wrote:
> >
> > > On Thu 02-03-17 18:03:15, Igor Mammedov wrote:
> > > > On Thu, 2 Mar 2017 15:28:16 +0100
> > > > Michal Hocko <mhocko@kernel.org> wrote:
> > > >
> > > > > On Thu 02-03-17 14:53:48, Igor Mammedov wrote:
[...]
> > > > > memblocks. If that doesn't work I would call it a bug.
> > > > It's rather an implementation constrain than a bug
> > > > for details and workaround patch see
> > > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1314306#c7
> > >
> > > "You are not authorized to access bug #1314306"
> > Sorry,
> > I've made it public, related comments and patch should be accessible now
> > (code snippets in BZ are based on older kernel but logic is still the same upstream)
> >
> > > could you paste the reasoning here please?
> > sure here is reproducer:
> > start VM with CLI:
> > qemu-system-x86_64 -enable-kvm -m size=1G,slots=2,maxmem=4G -numa node \
> > -object memory-backend-ram,id=m1,size=1G -device pc-dimm,node=0,memdev=m1 \
> > /path/to/guest_image
> >
> > then in guest dimm1 blocks are from 32-39
> >
> > echo online_movable > /sys/devices/system/memory/memory32/state
> > -bash: echo: write error: Invalid argument
> >
> > in current mainline kernel it triggers following code path:
> >
> > online_pages()
> > ...
> > if (online_type == MMOP_ONLINE_KERNEL) {
> > if (!zone_can_shift(pfn, nr_pages, ZONE_NORMAL, &zone_shift))
> > return -EINVAL;
>
> Are you sure? I would expect MMOP_ONLINE_MOVABLE here
pretty much, reproducer is above so try and see for yourself
[...]
> [...]
> > > > > > Which means simple udev rule isn't usable since it gets event from
> > > > > > the first to the last hotplugged block order. So now we would have
> > > > > > to write a daemon that would
> > > > > > - watch for all blocks in hotplugged memory appear (how would it know)
> > > > > > - online them in right order (order might also be different depending
> > > > > > on kernel version)
> > > > > > -- it becomes even more complicated in NUMA case when there are
> > > > > > multiple zones and kernel would have to provide user-space
> > > > > > with information about zone maps
> > > > > >
> > > > > > In short current experience shows that userspace approach
> > > > > > - doesn't solve issues that Vitaly has been fixing (i.e. onlining
> > > > > > fast and/or under memory pressure) when udev (or something else
> > > > > > might be killed)
> > > > >
> > > > > yeah and that is why the patch does the onlining from the kernel.
> > > > onlining in this patch is limited to hyperv and patch breaks
> > > > auto-online on x86 kvm/vmware/baremetal as they reuse the same
> > > > hotplug path.
> > >
> > > Those can use the udev or do you see any reason why they couldn't?
> >
> > Reasons are above, under >>>> and >> quotations, patch breaks
> > what Vitaly's fixed (including kvm/vmware usecases) i.e. udev/some
> > user-space process could be killed if hotplugged memory isn't onlined
> > fast enough leading to service termination and/or memory not
> > being onlined at all (if udev is killed)
>
> OK, so from the discussion so far I have learned that this would be
> problem _only_ if we are trying to hotplug a _lot_ of memory at once
> (~1.5% of the online memory is needed). I am kind of skeptical this is
> a reasonable usecase. Who is going to hotadd 8G to 256M machine (which
> would eat half of the available memory which is still quite far from
> OOM)? Even if the memory balloning uses hotplug then such a grow sounds
> a bit excessive.
Slow and killable udev issue doesn't really depends on
amount of hotplugged memory since it's onlined in blocks
(128M for x64). Considering that it's currently onlined
as zone normal, kernel doesn't have any issues adding more
follow up blocks of memory.
> > Currently udev rule is not usable and one needs a daemon
> > which would correctly do onlining and keep zone balance
> > even for simple case usecase of 1 normal and 1 movable zone.
> > And it gets more complicated in case of multiple numa nodes
> > with multiple zones.
>
> That sounds to be more related to the current implementation than
> anything else and as such is not a reason to invent specific user
> visible api. Btw. you are talking about movable zones byt the auto
> onlining doesn't allow to auto online movable memory. So either I miss
> your point or I am utterly confused.
in current state neither does udev rule as memory is onlined
as NORMAL (x64 variant at least), which is the same as auto online
does now.
We are discussing 2 different issues here and thread got pretty
hard to follow. I'll try to sum up at results the end.
> [...]
> > > > Memory unplug is rather new and it doesn't work reliably so far,
> > > > moving onlining to user-space won't really help. Further work
> > > > is need to be done so that it would work reliably.
> > >
> > > The main problem I have with this is that this is a limited usecase
> > > driven configuration knob which doesn't work properly for other usecases
> > > (namely movable online once your distribution choses to set the config
> > > option to auto online).
> >
> > it works for default usecase in Fedora and non-default
> > movable can be used with
> > 1) removable memory auto-online as movable in kernel, like
> > patch [1] would make movable hotplugged memory
> > (when I have time I'll try to work on it)
> > 2) (or in worst case due to lack of alternative) explicitly
> > disabled auto-online on kernel CLI + onlining daemon
> > (since udev isn't working in current kernel due to ordering issue)
>
> So I fail to see how this can work. Say the default will auto online
> all the added memory. This will be all in Zone Normal because the auto
> onlining doesn't do online_movable. Now I would like to online the
> memory as movable but that might fail because some kernel objects might
> be already sitting there so the offline would fail.
same happens with udev rule as currently it can online from
the first to last hotplugged block only as Zone Normal.
see
arch/x86/mm/init_64.c:
acpi_memory_enable_device() -> add_memory -> add_memory_resource
both auto online and udev could be fixed by patch
https://bugzilla.redhat.com/attachment.cgi?id=1146332
to online reomovable memory as movable.
> > > There is a userspace solution for this so this
> > > shouldn't have been merged in the first place!
> > Sorry, currently user-space udev solution doesn't work nor
> > will it work reliably in extreme conditions.
> >
> > > It sneaked a proper review
> > > process (linux-api wasn't CC to get a broader attenttion) which is
> > > really sad.
> >
> > get_maintainer.pl doesn't lists linux-api for 31bc3858ea3e,
> > MAINTAINERS should be fixed if linux-api were to be CCed.
>
> user visible APIs _should_ be discussed at this mailing list regardless
> what get_maintainer.pl says. This is not about who is the maintainer but
> about getting as wide audience for things that would have to be
> maintained basically for ever.
How would random contributor know which list to CC?
> > > So unless this causes a major regression which would be hard to fix I
> > > will submit the patch for inclusion.
> > it will be a major regression due to lack of daemon that
> > could online fast and can't be killed on OOM. So this
> > clean up patch does break used feature without providing
> > a viable alternative.
>
> So let's discuss the current memory hotplug shortcomings and get rid of
> the crud which developed on top. I will start by splitting up the patch
> into 3 parts. Do the auto online thing from the HyperV and xen balloning
> drivers and dropping the config option and finally drop the sysfs knob.
> The last patch might be NAKed and I can live with that as long as the
> reasoning is proper and there is a general consensus on that.
PS: CC me on that patches too
It's major regression if you remove auto online in kernels that
run on top of x86 kvm/vmware hypervisors, making API cleanups
while breaking useful functionality doesn't make sense.
I would ACK config option removal if auto online keeps working
for all x86 hypervisors (hyperv/xen isn't the only who needs it)
and keep kernel CLI option to override default.
That doesn't mean that others will agree with flipping default,
that's why config option has been added.
Now to sum up what's been discussed on this thread, there were 2
different issues discussed:
1) memory hotplug: remove in kernel auto online for all
except of hyperv/xen
- suggested RFC is not acceptable from virt point of view
as it regresses guests on top of x86 kvm/vmware which
both use ACPI based memory hotplug.
- udev/userspace solution doesn't work in practice as it's
too slow and unreliable when system is under load which
is quite common in virt usecase. That's why auto online
has been introduced in the first place.
2) memory unplug: online memory as movable
- doesn't work currently with udev rule due to kernel
issues https://bugzilla.redhat.com/show_bug.cgi?id=1314306#c7
- could be fixed both for in kernel auto online and udev
with following patch:
https://bugzilla.redhat.com/attachment.cgi?id=1146332
but fixing it this way exposes zone disbalance issues,
which are not present in current kernel as blocks are
onlined in Zone Normal. So this is area to work and
improve on.
- currently if one wants to use online_movable,
one has to either
* disable auto online in kernel OR
* remove udev rule that distro ships
AND write custom daemon that will be able to online
block in right zone/order. So currently whole
online_movable thing isn't working by default
regardless of who onlines memory.
I'm in favor of implementing that in kernel as it keeps
kernel internals inside kernel and doesn't need
kernel API to be involved (memory blocks in sysfs,
online_kernel, online_movable)
There would be no need in userspace which would have to
deal with kernel zoo and maintain that as well.
[toc] | [prev] | [next] | [standalone]
| From | Michal Hocko <mhocko@kernel.org> |
|---|---|
| Date | 2017-03-09 14:00 +0100 |
| Message-ID | <tj5PJ-5jo-23@gated-at.bofh.it> |
| In reply to | #1594197 |
On Tue 07-03-17 13:40:04, Igor Mammedov wrote:
> On Mon, 6 Mar 2017 15:54:17 +0100
> Michal Hocko <mhocko@kernel.org> wrote:
>
> > On Fri 03-03-17 18:34:22, Igor Mammedov wrote:
[...]
> > > in current mainline kernel it triggers following code path:
> > >
> > > online_pages()
> > > ...
> > > if (online_type == MMOP_ONLINE_KERNEL) {
> > > if (!zone_can_shift(pfn, nr_pages, ZONE_NORMAL, &zone_shift))
> > > return -EINVAL;
> >
> > Are you sure? I would expect MMOP_ONLINE_MOVABLE here
> pretty much, reproducer is above so try and see for yourself
I will play with this...
[...]
> > > get_maintainer.pl doesn't lists linux-api for 31bc3858ea3e,
> > > MAINTAINERS should be fixed if linux-api were to be CCed.
> >
> > user visible APIs _should_ be discussed at this mailing list regardless
> > what get_maintainer.pl says. This is not about who is the maintainer but
> > about getting as wide audience for things that would have to be
> > maintained basically for ever.
>
> How would random contributor know which list to CC?
This should have been brought up during the review process which was
less than sufficient in this case.
> > > > So unless this causes a major regression which would be hard to fix I
> > > > will submit the patch for inclusion.
> > > it will be a major regression due to lack of daemon that
> > > could online fast and can't be killed on OOM. So this
> > > clean up patch does break used feature without providing
> > > a viable alternative.
> >
> > So let's discuss the current memory hotplug shortcomings and get rid of
> > the crud which developed on top. I will start by splitting up the patch
> > into 3 parts. Do the auto online thing from the HyperV and xen balloning
> > drivers and dropping the config option and finally drop the sysfs knob.
> > The last patch might be NAKed and I can live with that as long as the
> > reasoning is proper and there is a general consensus on that.
> PS: CC me on that patches too
>
> It's major regression if you remove auto online in kernels that
> run on top of x86 kvm/vmware hypervisors, making API cleanups
> while breaking useful functionality doesn't make sense.
>
> I would ACK config option removal if auto online keeps working
> for all x86 hypervisors (hyperv/xen isn't the only who needs it)
> and keep kernel CLI option to override default.
>
> That doesn't mean that others will agree with flipping default,
> that's why config option has been added.
>
> Now to sum up what's been discussed on this thread, there were 2
> different issues discussed:
> 1) memory hotplug: remove in kernel auto online for all
> except of hyperv/xen
>
> - suggested RFC is not acceptable from virt point of view
> as it regresses guests on top of x86 kvm/vmware which
> both use ACPI based memory hotplug.
>
> - udev/userspace solution doesn't work in practice as it's
> too slow and unreliable when system is under load which
> is quite common in virt usecase. That's why auto online
> has been introduced in the first place.
Please try to be more specific why "too slow" is a problem. Also how
much slower are we talking about?
> 2) memory unplug: online memory as movable
>
> - doesn't work currently with udev rule due to kernel
> issues https://bugzilla.redhat.com/show_bug.cgi?id=1314306#c7
These should be fixed
> - could be fixed both for in kernel auto online and udev
> with following patch:
> https://bugzilla.redhat.com/attachment.cgi?id=1146332
> but fixing it this way exposes zone disbalance issues,
> which are not present in current kernel as blocks are
> onlined in Zone Normal. So this is area to work and
> improve on.
>
> - currently if one wants to use online_movable,
> one has to either
> * disable auto online in kernel OR
which might not just work because an unmovable allocation could have
made the memblock pinned.
> * remove udev rule that distro ships
> AND write custom daemon that will be able to online
> block in right zone/order. So currently whole
> online_movable thing isn't working by default
> regardless of who onlines memory.
my epxperience with onlining full nodes as movable shows this works just
fine (with all the limitations of the movable zones but that is a
separate thing). I haven't played with configurations where movable
zones are sharing the node with other zones.
> I'm in favor of implementing that in kernel as it keeps
> kernel internals inside kernel and doesn't need
> kernel API to be involved (memory blocks in sysfs,
> online_kernel, online_movable)
> There would be no need in userspace which would have to
> deal with kernel zoo and maintain that as well.
The kernel is supposed to provide a proper API and that is sysfs
currently. I am not entirely happy about it either but pulling a lot of
code into the kernel is not the rigth thing to do. Especially when
different usecases require different treatment.
--
Michal Hocko
SUSE Labs
[toc] | [prev] | [next] | [standalone]
| From | Michal Hocko <mhocko@kernel.org> |
|---|---|
| Date | 2017-03-10 15:00 +0100 |
| Subject | WTH is going on with memory hotplug sysf interface (was: Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks) |
| Message-ID | <tjtfj-4NB-5@gated-at.bofh.it> |
| In reply to | #1596040 |
Let's CC people touching this logic. A short summary is that onlining
memory via udev is currently unusable for online_movable because blocks
are added from lower addresses while movable blocks are allowed from
last blocks. More below.
On Thu 09-03-17 13:54:00, Michal Hocko wrote:
> On Tue 07-03-17 13:40:04, Igor Mammedov wrote:
> > On Mon, 6 Mar 2017 15:54:17 +0100
> > Michal Hocko <mhocko@kernel.org> wrote:
> >
> > > On Fri 03-03-17 18:34:22, Igor Mammedov wrote:
> [...]
> > > > in current mainline kernel it triggers following code path:
> > > >
> > > > online_pages()
> > > > ...
> > > > if (online_type == MMOP_ONLINE_KERNEL) {
> > > > if (!zone_can_shift(pfn, nr_pages, ZONE_NORMAL, &zone_shift))
> > > > return -EINVAL;
> > >
> > > Are you sure? I would expect MMOP_ONLINE_MOVABLE here
> > pretty much, reproducer is above so try and see for yourself
>
> I will play with this...
OK so I did with -m 2G,slots=4,maxmem=4G -numa node,mem=1G -numa node,mem=1G which generated
[...]
[ 0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0x0009ffff]
[ 0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x00100000-0x3fffffff]
[ 0.000000] ACPI: SRAT: Node 1 PXM 1 [mem 0x40000000-0x7fffffff]
[ 0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x27fffffff] hotplug
[ 0.000000] NUMA: Node 0 [mem 0x00000000-0x0009ffff] + [mem 0x00100000-0x3fffffff] -> [mem 0x00000000-0x3fffffff]
[ 0.000000] NODE_DATA(0) allocated [mem 0x3fffc000-0x3fffffff]
[ 0.000000] NODE_DATA(1) allocated [mem 0x7ffdc000-0x7ffdffff]
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff]
[ 0.000000] DMA32 [mem 0x0000000001000000-0x000000007ffdffff]
[ 0.000000] Normal empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff]
[ 0.000000] node 0: [mem 0x0000000000100000-0x000000003fffffff]
[ 0.000000] node 1: [mem 0x0000000040000000-0x000000007ffdffff]
so there is neither any normal zone nor movable one at the boot time.
Then I hotplugged 1G slot
(qemu) object_add memory-backend-ram,id=mem1,size=1G
(qemu) device_add pc-dimm,id=dimm1,memdev=mem1
unfortunatelly the memory didn't show up automatically and I got
[ 116.375781] acpi PNP0C80:00: Enumeration failure
so I had to probe it manually (prbably the BIOS my qemu uses doesn't
support auto probing - I haven't really dug further). Anyway the SRAT
table printed during the boot told that we should start at 0x100000000
# echo 0x100000000 > /sys/devices/system/memory/probe
# grep . /sys/devices/system/memory/memory32/valid_zones
Normal Movable
which looks reasonably right? Both Normal and Movable zones are allowed
# echo $((0x100000000+(128<<20))) > /sys/devices/system/memory/probe
# grep . /sys/devices/system/memory/memory3?/valid_zones
/sys/devices/system/memory/memory32/valid_zones:Normal
/sys/devices/system/memory/memory33/valid_zones:Normal Movable
Huh, so our valid_zones have changed under our feet...
# echo $((0x100000000+2*(128<<20))) > /sys/devices/system/memory/probe
# grep . /sys/devices/system/memory/memory3?/valid_zones
/sys/devices/system/memory/memory32/valid_zones:Normal
/sys/devices/system/memory/memory33/valid_zones:Normal
/sys/devices/system/memory/memory34/valid_zones:Normal Movable
and again. So only the last memblock is considered movable. Let's try to
online them now.
# echo online_movable > /sys/devices/system/memory/memory34/state
# grep . /sys/devices/system/memory/memory3?/valid_zones
/sys/devices/system/memory/memory32/valid_zones:Normal
/sys/devices/system/memory/memory33/valid_zones:Normal Movable
/sys/devices/system/memory/memory34/valid_zones:Movable Normal
This would explain why onlining from the last block actually works but
to me this sounds like a completely crappy behavior. All we need to
guarantee AFAICS is that Normal and Movable zones do not overlap. I
believe there is even no real requirement about ordering of the physical
memory in Normal vs. Movable zones as long as they do not overlap. But
let's keep it simple for the start and always enforce the current status
quo that Normal zone is physically preceeding Movable zone.
Can somebody explain why we cannot have a simple rule for Normal vs.
Movable which would be:
- block [pfn, pfn+block_size] can be Normal if
!zone_populated(MOVABLE) || pfn+block_size < ZONE_MOVABLE->zone_start_pfn
- block [pfn, pfn+block_size] can be Movable if
!zone_populated(NORMAL) || ZONE_NORMAL->zone_end_pfn < pfn
I haven't fully grokked all the restrictions on the movable zone size
based on the kernel parameters (find_zone_movable_pfns_for_nodes) but
this shouldn't really make the situation really much more complicated I
believe because those parameters should be mostly about static
initialization rather than hotplug but I might be easily missing
something.
--
Michal Hocko
SUSE Labs
[toc] | [prev] | [next] | [standalone]
| From | Michal Hocko <mhocko@kernel.org> |
|---|---|
| Date | 2017-03-10 17:00 +0100 |
| Subject | Re: WTH is going on with memory hotplug sysf interface (was: Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks) |
| Message-ID | <tjv7r-66F-9@gated-at.bofh.it> |
| In reply to | #1597687 |
On Fri 10-03-17 14:58:07, Michal Hocko wrote: [...] > This would explain why onlining from the last block actually works but > to me this sounds like a completely crappy behavior. All we need to > guarantee AFAICS is that Normal and Movable zones do not overlap. I > believe there is even no real requirement about ordering of the physical > memory in Normal vs. Movable zones as long as they do not overlap. But > let's keep it simple for the start and always enforce the current status > quo that Normal zone is physically preceeding Movable zone. > Can somebody explain why we cannot have a simple rule for Normal vs. > Movable which would be: > - block [pfn, pfn+block_size] can be Normal if > !zone_populated(MOVABLE) || pfn+block_size < ZONE_MOVABLE->zone_start_pfn > - block [pfn, pfn+block_size] can be Movable if > !zone_populated(NORMAL) || ZONE_NORMAL->zone_end_pfn < pfn OK, so while I was playing with this setup some more I probably got why this is done this way. All new memblocks are added to the zone Normal where they are accounted as spanned but not present. When we do online_movable we just cut from the end of the Normal zone and move it to Movable zone. This sounds really awkward. What was the reason to go this way? Why cannot we simply add those pages to the zone at the online time? -- Michal Hocko SUSE Labs
[toc] | [prev] | [next] | [standalone]
| From | Reza Arbab <arbab@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-03-10 20:10 +0100 |
| Subject | Re: WTH is going on with memory hotplug sysf interface (was: Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks) |
| Message-ID | <tjy5k-8jd-15@gated-at.bofh.it> |
| In reply to | #1597937 |
On Fri, Mar 10, 2017 at 04:53:33PM +0100, Michal Hocko wrote: >OK, so while I was playing with this setup some more I probably got why >this is done this way. All new memblocks are added to the zone Normal >where they are accounted as spanned but not present. It's not always zone Normal. See zone_for_memory(). This leads to a workaround for having to do online_movable in descending block order. Instead of this: 1. probe block 34, probe block 33, probe block 32, ... 2. online_movable 34, online_movable 33, online_movable 32, ... you can online_movable the first block before adding the rest: 1. probe block 32, online_movable 32 2. probe block 33, probe block 34, ... - zone_for_memory() will cause these to start Movable 3. online 33, online 34, ... - they're already in Movable, so online_movable is equivalentr I agree with your general sentiment that this stuff is very nonintuitive. -- Reza Arbab
[toc] | [prev] | [next] | [standalone]
| From | Michal Hocko <mhocko@kernel.org> |
|---|---|
| Date | 2017-03-13 10:30 +0100 |
| Subject | Re: WTH is going on with memory hotplug sysf interface (was: Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks) |
| Message-ID | <tkusH-6Sd-31@gated-at.bofh.it> |
| In reply to | #1598036 |
On Fri 10-03-17 13:00:37, Reza Arbab wrote: > On Fri, Mar 10, 2017 at 04:53:33PM +0100, Michal Hocko wrote: > >OK, so while I was playing with this setup some more I probably got why > >this is done this way. All new memblocks are added to the zone Normal > >where they are accounted as spanned but not present. > > It's not always zone Normal. See zone_for_memory(). This leads to a > workaround for having to do online_movable in descending block order. > Instead of this: > > 1. probe block 34, probe block 33, probe block 32, ... > 2. online_movable 34, online_movable 33, online_movable 32, ... > > you can online_movable the first block before adding the rest: I do I enforce that behavior when the probe happens automagically? > 1. probe block 32, online_movable 32 > 2. probe block 33, probe block 34, ... > - zone_for_memory() will cause these to start Movable > 3. online 33, online 34, ... > - they're already in Movable, so online_movable is equivalentr > > I agree with your general sentiment that this stuff is very nonintuitive. My criterion for nonintuitive is probably different because I would call this _completely_unusable_. Sorry for being so loud about this but the more I look into this area the more WTF code I see. This has seen close to zero review and seems to be building up more single usecase code on top of previous. We need to change this, seriously! -- Michal Hocko SUSE Labs
[toc] | [prev] | [next] | [standalone]
| From | Reza Arbab <arbab@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-03-13 16:00 +0100 |
| Subject | Re: WTH is going on with memory hotplug sysf interface (was: Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks) |
| Message-ID | <tkzC2-1Yl-17@gated-at.bofh.it> |
| In reply to | #1599133 |
On Mon, Mar 13, 2017 at 10:21:45AM +0100, Michal Hocko wrote: >> I agree with your general sentiment that this stuff is very >> nonintuitive. > >My criterion for nonintuitive is probably different because I would >call this _completely_unusable_. Sorry for being so loud about this but >the more I look into this area the more WTF code I see. This has seen >close to zero review and seems to be building up more single usecase >code on top of previous. We need to change this, seriously! No argument here. I'm happy to help however I can. -- Reza Arbab
[toc] | [prev] | [next] | [standalone]
| From | Michal Hocko <mhocko@kernel.org> |
|---|---|
| Date | 2017-03-13 16:20 +0100 |
| Subject | Re: WTH is going on with memory hotplug sysf interface (was: Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks) |
| Message-ID | <tkzVo-2n4-31@gated-at.bofh.it> |
| In reply to | #1597937 |
Let's add Andi
On Fri 10-03-17 16:53:33, Michal Hocko wrote:
> On Fri 10-03-17 14:58:07, Michal Hocko wrote:
> [...]
> > This would explain why onlining from the last block actually works but
> > to me this sounds like a completely crappy behavior. All we need to
> > guarantee AFAICS is that Normal and Movable zones do not overlap. I
> > believe there is even no real requirement about ordering of the physical
> > memory in Normal vs. Movable zones as long as they do not overlap. But
> > let's keep it simple for the start and always enforce the current status
> > quo that Normal zone is physically preceeding Movable zone.
> > Can somebody explain why we cannot have a simple rule for Normal vs.
> > Movable which would be:
> > - block [pfn, pfn+block_size] can be Normal if
> > !zone_populated(MOVABLE) || pfn+block_size < ZONE_MOVABLE->zone_start_pfn
> > - block [pfn, pfn+block_size] can be Movable if
> > !zone_populated(NORMAL) || ZONE_NORMAL->zone_end_pfn < pfn
>
> OK, so while I was playing with this setup some more I probably got why
> this is done this way. All new memblocks are added to the zone Normal
> where they are accounted as spanned but not present. When we do
> online_movable we just cut from the end of the Normal zone and move it
> to Movable zone. This sounds really awkward. What was the reason to go
> this way? Why cannot we simply add those pages to the zone at the online
> time?
Answering to myself. So the reason seems to be 9d99aaa31f59 ("[PATCH]
x86_64: Support memory hotadd without sparsemem") which is no longer
true because
config MEMORY_HOTPLUG
bool "Allow for memory hot-add"
depends on SPARSEMEM || X86_64_ACPI_NUMA
depends on ARCH_ENABLE_MEMORY_HOTPLUG
depends on COMPILE_TEST || !KASAN
so it is either SPARSEMEM or X86_64_ACPI_NUMA that would have to be enabled.
config X86_64_ACPI_NUMA
def_bool y
prompt "ACPI NUMA detection"
depends on X86_64 && NUMA && ACPI && PCI
select ACPI_NUMA
But I do not see any way how to enable anything but SPARSEMEM for x86_64
choice
prompt "Memory model"
depends on SELECT_MEMORY_MODEL
default DISCONTIGMEM_MANUAL if ARCH_DISCONTIGMEM_DEFAULT
default SPARSEMEM_MANUAL if ARCH_SPARSEMEM_DEFAULT
default FLATMEM_MANUAL
ARCH_SPARSEMEM_DEFAULT is 32b only
config ARCH_DISCONTIGMEM_DEFAULT
def_bool y
depends on NUMA && X86_32
and ARCH_SPARSEMEM_DEFAULT is enabeld on 64b. So I guess whatever was
the reason to add this code back in 2006 is not true anymore. So I am
really wondering. Do we absolutely need to assign pages which are not
onlined yet to the ZONE_NORMAL unconditionally? Why cannot we put them
out of any zone and wait for memory online operation to put them where
requested?
--
Michal Hocko
SUSE Labs
[toc] | [prev] | [next] | [standalone]
| From | Andi Kleen <ak@linux.intel.com> |
|---|---|
| Date | 2017-03-14 00:20 +0100 |
| Subject | Re: WTH is going on with memory hotplug sysf interface (was: Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks) |
| Message-ID | <tkHpT-85V-3@gated-at.bofh.it> |
| In reply to | #1599507 |
> and ARCH_SPARSEMEM_DEFAULT is enabeld on 64b. So I guess whatever was > the reason to add this code back in 2006 is not true anymore. So I am > really wondering. Do we absolutely need to assign pages which are not > onlined yet to the ZONE_NORMAL unconditionally? Why cannot we put them > out of any zone and wait for memory online operation to put them where > requested? I don't remember any specific reason. May have just been simplification. Should be fine to use any zone. -Andi
[toc] | [prev] | [next] | [standalone]
| From | Yasuaki Ishimatsu <yasu.isimatu@gmail.com> |
|---|---|
| Date | 2017-03-10 18:40 +0100 |
| Subject | Re: WTH is going on with memory hotplug sysf interface |
| Message-ID | <tjwGd-7eN-11@gated-at.bofh.it> |
| In reply to | #1597687 |
On 03/10/2017 08:58 AM, Michal Hocko wrote:
> Let's CC people touching this logic. A short summary is that onlining
> memory via udev is currently unusable for online_movable because blocks
> are added from lower addresses while movable blocks are allowed from
> last blocks. More below.
>
> On Thu 09-03-17 13:54:00, Michal Hocko wrote:
>> On Tue 07-03-17 13:40:04, Igor Mammedov wrote:
>>> On Mon, 6 Mar 2017 15:54:17 +0100
>>> Michal Hocko <mhocko@kernel.org> wrote:
>>>
>>>> On Fri 03-03-17 18:34:22, Igor Mammedov wrote:
>> [...]
>>>>> in current mainline kernel it triggers following code path:
>>>>>
>>>>> online_pages()
>>>>> ...
>>>>> if (online_type == MMOP_ONLINE_KERNEL) {
>>>>> if (!zone_can_shift(pfn, nr_pages, ZONE_NORMAL, &zone_shift))
>>>>> return -EINVAL;
>>>>
>>>> Are you sure? I would expect MMOP_ONLINE_MOVABLE here
>>> pretty much, reproducer is above so try and see for yourself
>>
>> I will play with this...
>
> OK so I did with -m 2G,slots=4,maxmem=4G -numa node,mem=1G -numa node,mem=1G which generated
> [...]
> [ 0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0x0009ffff]
> [ 0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x00100000-0x3fffffff]
> [ 0.000000] ACPI: SRAT: Node 1 PXM 1 [mem 0x40000000-0x7fffffff]
> [ 0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x27fffffff] hotplug
> [ 0.000000] NUMA: Node 0 [mem 0x00000000-0x0009ffff] + [mem 0x00100000-0x3fffffff] -> [mem 0x00000000-0x3fffffff]
> [ 0.000000] NODE_DATA(0) allocated [mem 0x3fffc000-0x3fffffff]
> [ 0.000000] NODE_DATA(1) allocated [mem 0x7ffdc000-0x7ffdffff]
> [ 0.000000] Zone ranges:
> [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff]
> [ 0.000000] DMA32 [mem 0x0000000001000000-0x000000007ffdffff]
> [ 0.000000] Normal empty
> [ 0.000000] Movable zone start for each node
> [ 0.000000] Early memory node ranges
> [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff]
> [ 0.000000] node 0: [mem 0x0000000000100000-0x000000003fffffff]
> [ 0.000000] node 1: [mem 0x0000000040000000-0x000000007ffdffff]
>
> so there is neither any normal zone nor movable one at the boot time.
> Then I hotplugged 1G slot
> (qemu) object_add memory-backend-ram,id=mem1,size=1G
> (qemu) device_add pc-dimm,id=dimm1,memdev=mem1
>
> unfortunatelly the memory didn't show up automatically and I got
> [ 116.375781] acpi PNP0C80:00: Enumeration failure
>
> so I had to probe it manually (prbably the BIOS my qemu uses doesn't
> support auto probing - I haven't really dug further). Anyway the SRAT
> table printed during the boot told that we should start at 0x100000000
>
> # echo 0x100000000 > /sys/devices/system/memory/probe
> # grep . /sys/devices/system/memory/memory32/valid_zones
> Normal Movable
>
> which looks reasonably right? Both Normal and Movable zones are allowed
>
> # echo $((0x100000000+(128<<20))) > /sys/devices/system/memory/probe
> # grep . /sys/devices/system/memory/memory3?/valid_zones
> /sys/devices/system/memory/memory32/valid_zones:Normal
> /sys/devices/system/memory/memory33/valid_zones:Normal Movable
>
> Huh, so our valid_zones have changed under our feet...
>
> # echo $((0x100000000+2*(128<<20))) > /sys/devices/system/memory/probe
> # grep . /sys/devices/system/memory/memory3?/valid_zones
> /sys/devices/system/memory/memory32/valid_zones:Normal
> /sys/devices/system/memory/memory33/valid_zones:Normal
> /sys/devices/system/memory/memory34/valid_zones:Normal Movable
>
> and again. So only the last memblock is considered movable. Let's try to
> online them now.
>
> # echo online_movable > /sys/devices/system/memory/memory34/state
> # grep . /sys/devices/system/memory/memory3?/valid_zones
> /sys/devices/system/memory/memory32/valid_zones:Normal
> /sys/devices/system/memory/memory33/valid_zones:Normal Movable
> /sys/devices/system/memory/memory34/valid_zones:Movable Normal
>
I think there is no strong reason which kernel has the restriction.
By setting the restrictions, it seems to have made management of
these zone structs simple.
Thanks,
Yasuaki Ishimatsu
> This would explain why onlining from the last block actually works but
> to me this sounds like a completely crappy behavior. All we need to
> guarantee AFAICS is that Normal and Movable zones do not overlap. I
> believe there is even no real requirement about ordering of the physical
> memory in Normal vs. Movable zones as long as they do not overlap. But
> let's keep it simple for the start and always enforce the current status
> quo that Normal zone is physically preceeding Movable zone.
> Can somebody explain why we cannot have a simple rule for Normal vs.
> Movable which would be:
> - block [pfn, pfn+block_size] can be Normal if
> !zone_populated(MOVABLE) || pfn+block_size < ZONE_MOVABLE->zone_start_pfn
> - block [pfn, pfn+block_size] can be Movable if
> !zone_populated(NORMAL) || ZONE_NORMAL->zone_end_pfn < pfn
>
> I haven't fully grokked all the restrictions on the movable zone size
> based on the kernel parameters (find_zone_movable_pfns_for_nodes) but
> this shouldn't really make the situation really much more complicated I
> believe because those parameters should be mostly about static
> initialization rather than hotplug but I might be easily missing
> something.
>
[toc] | [prev] | [next] | [standalone]
| From | Michal Hocko <mhocko@kernel.org> |
|---|---|
| Date | 2017-03-13 10:30 +0100 |
| Subject | Re: WTH is going on with memory hotplug sysf interface |
| Message-ID | <tkusG-6Sd-23@gated-at.bofh.it> |
| In reply to | #1597973 |
On Fri 10-03-17 12:39:27, Yasuaki Ishimatsu wrote: > On 03/10/2017 08:58 AM, Michal Hocko wrote: [...] > >OK so I did with -m 2G,slots=4,maxmem=4G -numa node,mem=1G -numa node,mem=1G which generated > >[...] > >[ 0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0x0009ffff] > >[ 0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x00100000-0x3fffffff] > >[ 0.000000] ACPI: SRAT: Node 1 PXM 1 [mem 0x40000000-0x7fffffff] > >[ 0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x27fffffff] hotplug > >[ 0.000000] NUMA: Node 0 [mem 0x00000000-0x0009ffff] + [mem 0x00100000-0x3fffffff] -> [mem 0x00000000-0x3fffffff] > >[ 0.000000] NODE_DATA(0) allocated [mem 0x3fffc000-0x3fffffff] > >[ 0.000000] NODE_DATA(1) allocated [mem 0x7ffdc000-0x7ffdffff] > >[ 0.000000] Zone ranges: > >[ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] > >[ 0.000000] DMA32 [mem 0x0000000001000000-0x000000007ffdffff] > >[ 0.000000] Normal empty > >[ 0.000000] Movable zone start for each node > >[ 0.000000] Early memory node ranges > >[ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] > >[ 0.000000] node 0: [mem 0x0000000000100000-0x000000003fffffff] > >[ 0.000000] node 1: [mem 0x0000000040000000-0x000000007ffdffff] > > > >so there is neither any normal zone nor movable one at the boot time. > >Then I hotplugged 1G slot > >(qemu) object_add memory-backend-ram,id=mem1,size=1G > >(qemu) device_add pc-dimm,id=dimm1,memdev=mem1 > > > >unfortunatelly the memory didn't show up automatically and I got > >[ 116.375781] acpi PNP0C80:00: Enumeration failure > > > >so I had to probe it manually (prbably the BIOS my qemu uses doesn't > >support auto probing - I haven't really dug further). Anyway the SRAT > >table printed during the boot told that we should start at 0x100000000 > > > ># echo 0x100000000 > /sys/devices/system/memory/probe > ># grep . /sys/devices/system/memory/memory32/valid_zones > >Normal Movable > > > >which looks reasonably right? Both Normal and Movable zones are allowed > > > ># echo $((0x100000000+(128<<20))) > /sys/devices/system/memory/probe > ># grep . /sys/devices/system/memory/memory3?/valid_zones > >/sys/devices/system/memory/memory32/valid_zones:Normal > >/sys/devices/system/memory/memory33/valid_zones:Normal Movable > > > >Huh, so our valid_zones have changed under our feet... > > > ># echo $((0x100000000+2*(128<<20))) > /sys/devices/system/memory/probe > ># grep . /sys/devices/system/memory/memory3?/valid_zones > >/sys/devices/system/memory/memory32/valid_zones:Normal > >/sys/devices/system/memory/memory33/valid_zones:Normal > >/sys/devices/system/memory/memory34/valid_zones:Normal Movable > > > >and again. So only the last memblock is considered movable. Let's try to > >online them now. > > > ># echo online_movable > /sys/devices/system/memory/memory34/state > ># grep . /sys/devices/system/memory/memory3?/valid_zones > >/sys/devices/system/memory/memory32/valid_zones:Normal > >/sys/devices/system/memory/memory33/valid_zones:Normal Movable > >/sys/devices/system/memory/memory34/valid_zones:Movable Normal > > > > I think there is no strong reason which kernel has the restriction. > By setting the restrictions, it seems to have made management of > these zone structs simple. Could you be more specific please? How could this make management any easier when udev is basically racing with the physical hotplug and the result is basically undefined? -- Michal Hocko SUSE Labs
[toc] | [prev] | [next] | [standalone]
| From | Igor Mammedov <imammedo@redhat.com> |
|---|---|
| Date | 2017-03-13 11:40 +0100 |
| Subject | Re: WTH is going on with memory hotplug sysf interface (was: Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks) |
| Message-ID | <tkvyp-7D9-5@gated-at.bofh.it> |
| In reply to | #1597687 |
On Fri, 10 Mar 2017 14:58:07 +0100
Michal Hocko <mhocko@kernel.org> wrote:
> Let's CC people touching this logic. A short summary is that onlining
> memory via udev is currently unusable for online_movable because blocks
> are added from lower addresses while movable blocks are allowed from
> last blocks. More below.
>
> On Thu 09-03-17 13:54:00, Michal Hocko wrote:
> > On Tue 07-03-17 13:40:04, Igor Mammedov wrote:
> > > On Mon, 6 Mar 2017 15:54:17 +0100
> > > Michal Hocko <mhocko@kernel.org> wrote:
> > >
> > > > On Fri 03-03-17 18:34:22, Igor Mammedov wrote:
> > [...]
> > > > > in current mainline kernel it triggers following code path:
> > > > >
> > > > > online_pages()
> > > > > ...
> > > > > if (online_type == MMOP_ONLINE_KERNEL) {
> > > > > if (!zone_can_shift(pfn, nr_pages, ZONE_NORMAL, &zone_shift))
> > > > > return -EINVAL;
> > > >
> > > > Are you sure? I would expect MMOP_ONLINE_MOVABLE here
> > > pretty much, reproducer is above so try and see for yourself
> >
> > I will play with this...
>
> OK so I did with -m 2G,slots=4,maxmem=4G -numa node,mem=1G -numa node,mem=1G which generated
'mem' here distributes boot memory specified by "-m 2G" and does not
include memory specified by -device pc-dimm.
> [...]
> [ 0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0x0009ffff]
> [ 0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x00100000-0x3fffffff]
> [ 0.000000] ACPI: SRAT: Node 1 PXM 1 [mem 0x40000000-0x7fffffff]
> [ 0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x27fffffff] hotplug
> [ 0.000000] NUMA: Node 0 [mem 0x00000000-0x0009ffff] + [mem 0x00100000-0x3fffffff] -> [mem 0x00000000-0x3fffffff]
> [ 0.000000] NODE_DATA(0) allocated [mem 0x3fffc000-0x3fffffff]
> [ 0.000000] NODE_DATA(1) allocated [mem 0x7ffdc000-0x7ffdffff]
> [ 0.000000] Zone ranges:
> [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff]
> [ 0.000000] DMA32 [mem 0x0000000001000000-0x000000007ffdffff]
> [ 0.000000] Normal empty
> [ 0.000000] Movable zone start for each node
> [ 0.000000] Early memory node ranges
> [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff]
> [ 0.000000] node 0: [mem 0x0000000000100000-0x000000003fffffff]
> [ 0.000000] node 1: [mem 0x0000000040000000-0x000000007ffdffff]
>
> so there is neither any normal zone nor movable one at the boot time.
it could be if hotpluggable memory were present at boot time in E802 table
(if I remember right when running on hyperv there is movable zone at boot time),
but in qemu hotpluggable memory isn't put into E820,
so zone is allocated later when memory is enumerated
by ACPI subsystem and onlined.
It causes less issues wrt movable zone and works for
different versions of linux/windows as well.
That's where in kernel auto-onlining could be also useful,
since user would be able to start-up with with small
non removable memory plus several removable DIMMs
and have all the memory onlined/available by the time
initrd is loaded. (missing piece here is onling
removable memory as movable by default).
> Then I hotplugged 1G slot
> (qemu) object_add memory-backend-ram,id=mem1,size=1G
> (qemu) device_add pc-dimm,id=dimm1,memdev=mem1
You can also specify node a pc-dimm goes to with 'node' property
if it should go to other then node 0.
device_add pc-dimm,id=dimm1,memdev=mem1,node=1
> unfortunatelly the memory didn't show up automatically and I got
> [ 116.375781] acpi PNP0C80:00: Enumeration failure
it should work,
do you have CONFIG_ACPI_HOTPLUG_MEMORY enabled?
> so I had to probe it manually (prbably the BIOS my qemu uses doesn't
> support auto probing - I haven't really dug further). Anyway the SRAT
> table printed during the boot told that we should start at 0x100000000
[toc] | [prev] | [next] | [standalone]
| From | Michal Hocko <mhocko@kernel.org> |
|---|---|
| Date | 2017-03-13 11:50 +0100 |
| Subject | Re: WTH is going on with memory hotplug sysf interface (was: Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks) |
| Message-ID | <tkvI7-7IV-31@gated-at.bofh.it> |
| In reply to | #1599184 |
On Mon 13-03-17 11:31:10, Igor Mammedov wrote: > On Fri, 10 Mar 2017 14:58:07 +0100 [...] > > [ 0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0x0009ffff] > > [ 0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x00100000-0x3fffffff] > > [ 0.000000] ACPI: SRAT: Node 1 PXM 1 [mem 0x40000000-0x7fffffff] > > [ 0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x27fffffff] hotplug > > [ 0.000000] NUMA: Node 0 [mem 0x00000000-0x0009ffff] + [mem 0x00100000-0x3fffffff] -> [mem 0x00000000-0x3fffffff] > > [ 0.000000] NODE_DATA(0) allocated [mem 0x3fffc000-0x3fffffff] > > [ 0.000000] NODE_DATA(1) allocated [mem 0x7ffdc000-0x7ffdffff] > > [ 0.000000] Zone ranges: > > [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] > > [ 0.000000] DMA32 [mem 0x0000000001000000-0x000000007ffdffff] > > [ 0.000000] Normal empty > > [ 0.000000] Movable zone start for each node > > [ 0.000000] Early memory node ranges > > [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] > > [ 0.000000] node 0: [mem 0x0000000000100000-0x000000003fffffff] > > [ 0.000000] node 1: [mem 0x0000000040000000-0x000000007ffdffff] > > > > so there is neither any normal zone nor movable one at the boot time. > it could be if hotpluggable memory were present at boot time in E802 table > (if I remember right when running on hyperv there is movable zone at boot time), > > but in qemu hotpluggable memory isn't put into E820, > so zone is allocated later when memory is enumerated > by ACPI subsystem and onlined. > It causes less issues wrt movable zone and works for > different versions of linux/windows as well. > > That's where in kernel auto-onlining could be also useful, > since user would be able to start-up with with small > non removable memory plus several removable DIMMs > and have all the memory onlined/available by the time > initrd is loaded. (missing piece here is onling > removable memory as movable by default). Why we should even care to online that memory that early rather than making it available via e820? > > Then I hotplugged 1G slot > > (qemu) object_add memory-backend-ram,id=mem1,size=1G > > (qemu) device_add pc-dimm,id=dimm1,memdev=mem1 > You can also specify node a pc-dimm goes to with 'node' property > if it should go to other then node 0. > > device_add pc-dimm,id=dimm1,memdev=mem1,node=1 thanks for the tip > > unfortunatelly the memory didn't show up automatically and I got > > [ 116.375781] acpi PNP0C80:00: Enumeration failure > it should work, > do you have CONFIG_ACPI_HOTPLUG_MEMORY enabled? No I didn't. Thanks, good to know! -- Michal Hocko SUSE Labs
[toc] | [prev] | [next] | [standalone]
| From | Igor Mammedov <imammedo@redhat.com> |
|---|---|
| Date | 2017-03-13 15:00 +0100 |
| Subject | Re: WTH is going on with memory hotplug sysf interface (was: Re: [RFC PATCH] mm, hotplug: get rid of auto_online_blocks) |
| Message-ID | <tkyFZ-1ik-27@gated-at.bofh.it> |
| In reply to | #1599198 |
On Mon, 13 Mar 2017 11:43:02 +0100 Michal Hocko <mhocko@kernel.org> wrote: > On Mon 13-03-17 11:31:10, Igor Mammedov wrote: > > On Fri, 10 Mar 2017 14:58:07 +0100 > [...] > > > [ 0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0x0009ffff] > > > [ 0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x00100000-0x3fffffff] > > > [ 0.000000] ACPI: SRAT: Node 1 PXM 1 [mem 0x40000000-0x7fffffff] > > > [ 0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x27fffffff] hotplug > > > [ 0.000000] NUMA: Node 0 [mem 0x00000000-0x0009ffff] + [mem 0x00100000-0x3fffffff] -> [mem 0x00000000-0x3fffffff] > > > [ 0.000000] NODE_DATA(0) allocated [mem 0x3fffc000-0x3fffffff] > > > [ 0.000000] NODE_DATA(1) allocated [mem 0x7ffdc000-0x7ffdffff] > > > [ 0.000000] Zone ranges: > > > [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] > > > [ 0.000000] DMA32 [mem 0x0000000001000000-0x000000007ffdffff] > > > [ 0.000000] Normal empty > > > [ 0.000000] Movable zone start for each node > > > [ 0.000000] Early memory node ranges > > > [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] > > > [ 0.000000] node 0: [mem 0x0000000000100000-0x000000003fffffff] > > > [ 0.000000] node 1: [mem 0x0000000040000000-0x000000007ffdffff] > > > > > > so there is neither any normal zone nor movable one at the boot time. > > it could be if hotpluggable memory were present at boot time in E802 table > > (if I remember right when running on hyperv there is movable zone at boot time), > > > > but in qemu hotpluggable memory isn't put into E820, > > so zone is allocated later when memory is enumerated > > by ACPI subsystem and onlined. > > It causes less issues wrt movable zone and works for > > different versions of linux/windows as well. > > > > That's where in kernel auto-onlining could be also useful, > > since user would be able to start-up with with small > > non removable memory plus several removable DIMMs > > and have all the memory onlined/available by the time > > initrd is loaded. (missing piece here is onling > > removable memory as movable by default). > > Why we should even care to online that memory that early rather than > making it available via e820? It's not forbidden by spec and has less complications when it comes to removable memory. Declaring it in E820 would add following limitations/drawbacks: - firmware should be able to exclude removable memory from its usage (currently SeaBIOS nor EFI have to know/care about it) => less qemu-guest ABI to maintain. - OS should be taught to avoid/move (early) nonmovable allocations from removable address ranges. There were patches targeting that in recent kernels, but it won't work with older kernels that don't have it. So limiting a range of OSes that could run on QEMU and do memory removal. E820 less approach works reasonably well with wide range of guest OSes and less complex that if removable memory were present it E820. Hence I don't have a compelling reason to introduce removable memory in E820 as it only adds to hot(un)plug issues. I have an off-tree QEMU hack that puts hotremovable memory added with "-device pc-dimm" on CLI into E820 to experiment with. It could be useful to play with zone layouts at boot time, so if you are interested I can rebase it on top of current master and post it here to play with.
[toc] | [prev] | [next] | [standalone]
Page 1 of 2 [1] 2 Next page →
Back to top | Article view | linux.kernel
csiph-web