Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1240601 > unrolled thread
| Started by | "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> |
|---|---|
| First post | 2015-10-06 17:40 +0200 |
| Last post | 2015-10-06 17:40 +0200 |
| Articles | 5 — 1 participant |
Back to article view | Back to linux.kernel
[PATCHv12 00/37] THP refcounting redesign "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2015-10-06 17:40 +0200
[PATCHv12 36/37] thp: update documentation "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2015-10-06 17:40 +0200
[PATCHv12 17/37] arm64, thp: remove infrastructure for handling splitting PMDs "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2015-10-06 17:40 +0200
[PATCHv12 01/37] mm, proc: adjust PSS calculation "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2015-10-06 17:40 +0200
[PATCHv12 14/37] futex, thp: remove special case for THP in get_futex_key "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2015-10-06 17:40 +0200
| From | "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> |
|---|---|
| Date | 2015-10-06 17:40 +0200 |
| Subject | [PATCHv12 00/37] THP refcounting redesign |
| Message-ID | <qgClI-1De-3@gated-at.bofh.it> |
Hello everybody,
The THP refcounting has been rebased onto newer -mm tree and few bugs reported
by Sasha were fixed.
Since page->lru in all tail pages occupied by page->compound_head, I had
to adjust the code to use storage of ->mapping and ->index in the second
tail page as list_head to link pages into deferred split queue.
The goal of patchset is to make refcounting on THP pages cheaper with
simpler semantics and allow the same THP compound page to be mapped with
PMD and PTEs. This is required to get reasonable THP-pagecache
implementation.
With the new refcounting design it's much easier to protect against
split_huge_page(): simple reference on a page will make you the deal.
It makes gup_fast() implementation simpler and doesn't require
special-case in futex code to handle tail THP pages.
It should improve THP utilization over the system since splitting THP in
one process doesn't necessary lead to splitting the page in all other
processes have the page mapped.
The patchset drastically lower complexity of get_page()/put_page()
codepaths. I encourage people look on this code before-and-after to
justify time budget on reviewing this patchset.
= Changelog =
v12:
- rebased to mmotm-2015-09-30-16-16;
- couple bug fixes for mlock() (Sasha Levin);
- fix locking inconsitency in split error path
queue_pages_pte_range() (Sasha Levin)
v11:
- rebased to mmotm-2015-09-16-16-34;
- patch to re-enable mlock is included;
v10:
- rebased onto newer -mm tree with compound_head patchset applied.
v9:
- rebased to mmotm-2015-07-15-16-46: fix conflicts with DAX and pagemap
patchsets;
- simplify PG_double_map handling in __split_huge_pmd_locked();
- fix arm64 typo (Suzuki K. Poulose);
- fix build on !THP;
- checkpatch fixes;
- Tested-by/Reviewed-by Aneesh Kumar K.V;
v8:
- rebased to since-4.1;
- fix mmap10 from LTP: make check and clear PG_double_map atomic.
v7:
- avoid situation during split_huge_pmd() where we can temporarily drop
page_mapcount() to zero. It can lead to races e.g. with unmap code;
- update documentation;
- fix NR_ANON_PAGES accounting in page_remove_rmap();
- fix page_mapped();
- optimize page_mapped() and page_mapcount();
- fix PSS calculation for non-shared pages;
v6:
- rebase to since-4.0;
- optimize mapcount handling: significantely reduce overhead for most
common cases.
- split pages on migrate_pages();
- remove infrastructure for handling splitting PMDs on all architectures;
- fix page_mapcount() for hugetlb pages;
v5:
- Tested-by: Sasha Levin!™
- re-split patchset in hope to improve readability;
- rebased on top of page flags and ->mapping sanitizing patchset;
- uncharge compound_mapcount rather than mapcount for hugetlb pages
during removing from rmap;
- differentiate page_mapped() from page_mapcount() for compound pages;
- rework deferred_split_huge_page() to use shrinker interface;
- fix race in page_remove_rmap();
- get rid of __get_page_tail();
- few random bug fixes;
v4:
- fix sizes reported in smaps;
- defines instead of enum for RMAP_{EXCLUSIVE,COMPOUND};
- skip THP pages on munlock_vma_pages_range(): they are never mlocked;
- properly handle huge zero page on FOLL_SPLIT;
- fix lock_page() slow path on tail pages;
- account page_get_anon_vma() fail to THP_SPLIT_PAGE_FAILED;
- fix split_huge_page() on huge page with unmapped head page;
- fix transfering 'write' and 'young' from pmd to ptes on split_huge_pmd;
- call page_remove_rmap() in unfreeze_page under ptl.
= Design overview =
The main reason why we can't map THP with 4k is how refcounting on THP
designed. It built around two requirements:
- split of huge page should never fail;
- we can't change interface of get_user_page();
To be able to split huge page at any point we have to track which tail
page was pinned. It leads to tricky and expensive get_page() on tail pages
and also occupy tail_page->_mapcount.
Most split_huge_page*() users want PMD to be split into table of PTEs and
don't care whether compound page is going to be split or not.
The plan is:
- allow split_huge_page() to fail if the page is pinned. It's trivial to
split non-pinned page and it doesn't require tail page refcounting, so
tail_page->_mapcount is free to be reused.
- introduce new routine -- split_huge_pmd() -- to split PMD into table of
PTEs. It splits only one PMD, not touching other PMDs the page is
mapped with or underlying compound page. Unlike new split_huge_page(),
split_huge_pmd() never fails.
Fortunately, we have only few places where split_huge_page() is needed:
swap out, memory failure, migration, KSM. And all of them can handle
split_huge_page() fail.
In new scheme we use page->_mapcount is used to account how many time
the page is mapped with PTEs. We have separate compound_mapcount() to
count mappings with PMD. page_mapcount() returns sum of PTE and PMD
mappings of the page.
Introducing split_huge_pmd() effectively allows THP to be mapped with 4k.
It may be a surprise to some code to see a PTE which points to tail page
or VMA start/end in the middle of compound page.
munmap() part of THP will split PMD, but doesn't split the huge page. In
order to take memory consumption under control we put partially unmapped
huge page on list. The pages will be split by shrinker if memory pressure
comes. This way we also avoid unnecessary split_huge_page() on exit(2) if
a THP belong to more than one VMA.
= Refcounts and transparent huge pages =
- get_page() and put_page() work *only* on head page's ->_count.
We don't touch tail pages at all for these oprations. We stopped
touching ->_mapcount in tail pages to store it's pins.
- ->_count in tail pages is always zero: get_page_unless_zero() never
succeed on tail pages. Nothing changed in this respect.
- map/unmap of the pages with PTE entry increment/decrement ->_mapcount
on relevent sub-page of the compound page.
- map/unmap of the whole compound page accounted in compound_mapcount
(stored in first tail page).
- PageDoubleMap() indicates that ->_mapcount in all subpages is offset
up by one. This additional reference is required to get race-free
detection of unmap of subpages when we have them mapped with both PMDs
and PTEs.
This is optimization required to lower overhead of per-subpage
mapcount tracking. The alternative is alter ->_mapcount in all
subpages on each map/unmap of the whole compound page.
We set PG_double_map when a PMD of the page got split for the first
time, but still have PMD mapping. The addtional references go away
with last compound_mapcount.
= Benchmarks =
Kernel build benchmark:
baseline v6
Amean user-2 447.76 ( 0.00%) 451.24 ( -0.78%)
Amean user-4 314.94 ( 0.00%) 310.10 ( 1.54%)
Amean user-8 388.91 ( 0.00%) 388.95 ( -0.01%)
Amean user-16 518.68 ( 0.00%) 518.60 ( 0.02%)
Amean user-24 533.58 ( 0.00%) 535.35 ( -0.33%)
Amean syst-2 77.52 ( 0.00%) 70.16 ( 9.49%)
Amean syst-4 51.21 ( 0.00%) 44.55 ( 13.00%)
Amean syst-8 42.12 ( 0.00%) 42.59 ( -1.12%)
Amean syst-16 50.29 ( 0.00%) 50.14 ( 0.30%)
Amean syst-24 49.36 ( 0.00%) 48.54 ( 1.65%)
Amean elsp-2 242.64 ( 0.00%) 244.46 ( -0.75%)
Amean elsp-4 93.78 ( 0.00%) 92.89 ( 0.95%)
Amean elsp-8 61.51 ( 0.00%) 61.92 ( -0.66%)
Amean elsp-16 53.95 ( 0.00%) 53.80 ( 0.29%)
Amean elsp-24 52.75 ( 0.00%) 53.14 ( -0.74%)
Stddev user-2 15.49 ( 0.00%) 13.75 ( 11.24%)
Stddev user-4 7.85 ( 0.00%) 4.42 ( 43.68%)
Stddev user-8 1.29 ( 0.00%) 2.77 (-114.28%)
Stddev user-16 2.56 ( 0.00%) 1.54 ( 39.89%)
Stddev user-24 1.75 ( 0.00%) 1.06 ( 39.75%)
Stddev syst-2 3.02 ( 0.00%) 2.00 ( 33.86%)
Stddev syst-4 1.23 ( 0.00%) 0.91 ( 26.65%)
Stddev syst-8 0.41 ( 0.00%) 0.30 ( 28.32%)
Stddev syst-16 0.51 ( 0.00%) 0.71 (-38.07%)
Stddev syst-24 0.92 ( 0.00%) 0.70 ( 23.86%)
Stddev elsp-2 8.70 ( 0.00%) 7.99 ( 8.13%)
Stddev elsp-4 1.74 ( 0.00%) 0.59 ( 66.08%)
Stddev elsp-8 0.40 ( 0.00%) 0.30 ( 25.11%)
Stddev elsp-16 0.45 ( 0.00%) 0.37 ( 17.73%)
Stddev elsp-24 0.57 ( 0.00%) 0.38 ( 33.64%)
Changes are mostly non-significant. The only noticble part is reduction of
system time for -j2 and -j4: 9.49% and 13.00%.
specjvm
base v6
Ops compiler 569.52 ( 0.00%) 618.74 ( 8.64%)
Ops compress 456.32 ( 0.00%) 469.20 ( 2.82%)
Ops crypto 424.34 ( 0.00%) 413.64 ( -2.52%)
Ops derby 535.15 ( 0.00%) 536.96 ( 0.34%)
Ops mpegaudio 291.03 ( 0.00%) 286.35 ( -1.61%)
Ops scimark.large 75.91 ( 0.00%) 77.21 ( 1.71%)
Ops scimark.small 529.19 ( 0.00%) 527.07 ( -0.40%)
Ops serial 316.13 ( 0.00%) 316.40 ( 0.09%)
Ops sunflow 154.90 ( 0.00%) 154.85 ( -0.03%)
Ops xml 612.94 ( 0.00%) 575.20 ( -6.16%)
Ops compiler.compiler 770.11 ( 0.00%) 878.22 ( 14.04%)
Ops compiler.sunflow 421.17 ( 0.00%) 435.92 ( 3.50%)
Ops compress 456.32 ( 0.00%) 469.20 ( 2.82%)
Ops crypto.aes 153.17 ( 0.00%) 151.30 ( -1.22%)
Ops crypto.rsa 607.43 ( 0.00%) 564.65 ( -7.04%)
Ops crypto.signverify 821.23 ( 0.00%) 828.40 ( 0.87%)
Ops derby 535.15 ( 0.00%) 536.96 ( 0.34%)
Ops mpegaudio 291.03 ( 0.00%) 286.35 ( -1.61%)
Ops scimark.fft.large 69.59 ( 0.00%) 69.81 ( 0.32%)
Ops scimark.lu.large 20.31 ( 0.00%) 20.32 ( 0.05%)
Ops scimark.sor.large 114.57 ( 0.00%) 113.99 ( -0.51%)
Ops scimark.sparse.large 55.56 ( 0.00%) 61.71 ( 11.07%)
Ops scimark.monte_carlo 280.13 ( 0.00%) 275.09 ( -1.80%)
Ops scimark.fft.small 815.19 ( 0.00%) 819.55 ( 0.53%)
Ops scimark.lu.small 1072.62 ( 0.00%) 1081.47 ( 0.83%)
Ops scimark.sor.small 674.47 ( 0.00%) 674.24 ( -0.03%)
Ops scimark.sparse.small 251.20 ( 0.00%) 247.43 ( -1.50%)
Ops serial 316.13 ( 0.00%) 316.40 ( 0.09%)
Ops sunflow 154.90 ( 0.00%) 154.85 ( -0.03%)
Ops xml.transform 538.16 ( 0.00%) 519.64 ( -3.44%)
Ops xml.validation 698.10 ( 0.00%) 636.70 ( -8.80%)
Results are mixed.
= Patches overview =
Patch 1:
We need to look on all subpages of compound page to calculate
correct PSS, because they can have different mapcount.
Patch 2:
With PTE-mapeed THP, rmap cannot rely on PageTransHuge() check to
decide if map small page or THP. We need to get the info from
caller.
Patch 3:
Make memcg aware about new refcounting. Validation needed.
Patch 4:
Adjust conditions when we can re-use the page on write-protection
fault.
Patch 5:
FOLL_SPLIT should be handled on PTE level too.
Patch 6:
Make generic fast GUP implementation aware about PTE-mapped huge
pages.
Patch 7:
Split all pages in mlocked VMA. That should be good enough for
now.
Patch 8:
Make khugepaged aware about PTE-mapped huge pages.
Patch 9:
Rename split_huge_page_pmd() to split_huge_pmd() to reflect that
page is not going to be split, only PMD.
Patch 10:
New THP_SPLIT_* vmstats.
Patch 11:
Up to this point we tried to keep patchset bisectable, but next
patches are going to change how core of THP refcounting work.
That's easier to review change if we would disable THP temporally
and bring it back once everything is ready.
Patch 12:
Remove all split_huge_page()-related code. It also remove need in
tail page refcounting.
Patch 13:
Drop tail page refcounting. Diffstat is nice! :)
Patch 14:
Remove ugly special case if futex happened to be in tail THP page.
With new refcounting it much easier to protect against split.
Patch 15:
Simplify KSM code which handle THP.
Patch 16:
No need in compound_lock anymore.
Patches 17-25:
Drop infrastructure for handling PMD splitting. We don't use it
anymore in split_huge_page().
Patch 26:
Store mapcount for compound pages separately: in the first tail
page ->mapping.
Patch 27:
Let's define page_mapped() to be true for compound pages if any
sub-pages of the compound page is mapped (with PMD or PTE).
Patch 28:
Make numabalancing aware about PTE-mapped THP.
Patch 29:
Implement new split_huge_pmd().
Patch 30-32:
Implement new split_huge_page().
Patch 33:
Split pages instaed of PMDs on migrate_pages.
Patch 34:
Handle partial unmap of THP. We put partially unmapped huge page
list. Pages from list will split via shrinker if memory pressure
comes. This way we also avoid unnecessary split_huge_page() on
exit(2) if a THP belong to more than one VMA.
Patch 35:
Everything is in place. Re-enable THP.
Patch 36:
Documentation update.
Patch 37:
Re-enabled mlock for most THPs.
The patchset also available on git:
git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git thp/refcounting/v12
Please review and consider applying.
Kirill A. Shutemov (37):
mm, proc: adjust PSS calculation
rmap: add argument to charge compound page
memcg: adjust to support new THP refcounting
mm, thp: adjust conditions when we can reuse the page on WP fault
mm: adjust FOLL_SPLIT for new refcounting
mm: handle PTE-mapped tail pages in gerneric fast gup implementaiton
thp, mlock: do not allow huge pages in mlocked area
khugepaged: ignore pmd tables with THP mapped with ptes
thp: rename split_huge_page_pmd() to split_huge_pmd()
mm, vmstats: new THP splitting event
mm: temporally mark THP broken
thp: drop all split_huge_page()-related code
mm: drop tail page refcounting
futex, thp: remove special case for THP in get_futex_key
ksm: prepare to new THP semantics
mm, thp: remove compound_lock
arm64, thp: remove infrastructure for handling splitting PMDs
arm, thp: remove infrastructure for handling splitting PMDs
mips, thp: remove infrastructure for handling splitting PMDs
powerpc, thp: remove infrastructure for handling splitting PMDs
s390, thp: remove infrastructure for handling splitting PMDs
sparc, thp: remove infrastructure for handling splitting PMDs
tile, thp: remove infrastructure for handling splitting PMDs
x86, thp: remove infrastructure for handling splitting PMDs
mm, thp: remove infrastructure for handling splitting PMDs
mm: rework mapcount accounting to enable 4k mapping of THPs
mm: differentiate page_mapped() from page_mapcount() for compound
pages
mm, numa: skip PTE-mapped THP on numa fault
thp: implement split_huge_pmd()
thp: add option to setup migration entries during PMD split
thp, mm: split_huge_page(): caller need to lock page
thp: reintroduce split_huge_page()
migrate_pages: try to split pages on qeueuing
thp: introduce deferred_split_huge_page()
mm: re-enable THP
thp: update documentation
thp: allow mlocked THP again
Documentation/vm/transhuge.txt | 151 ++--
arch/arc/mm/cache.c | 4 +-
arch/arm/include/asm/pgtable-3level.h | 10 -
arch/arm/lib/uaccess_with_memcpy.c | 5 +-
arch/arm/mm/flush.c | 17 +-
arch/arm64/include/asm/pgtable.h | 8 -
arch/arm64/mm/flush.c | 16 -
arch/mips/include/asm/pgtable-bits.h | 6 +-
arch/mips/include/asm/pgtable.h | 18 -
arch/mips/mm/c-r4k.c | 3 +-
arch/mips/mm/cache.c | 2 +-
arch/mips/mm/gup.c | 17 +-
arch/mips/mm/init.c | 6 +-
arch/mips/mm/pgtable-64.c | 14 -
arch/mips/mm/tlbex.c | 1 -
arch/powerpc/include/asm/pgtable-ppc64.h | 25 +-
arch/powerpc/mm/hugepage-hash64.c | 3 -
arch/powerpc/mm/hugetlbpage.c | 17 +-
arch/powerpc/mm/pgtable_64.c | 49 --
arch/powerpc/mm/subpage-prot.c | 2 +-
arch/s390/include/asm/pgtable.h | 15 +-
arch/s390/mm/gup.c | 24 +-
arch/s390/mm/pgtable.c | 16 -
arch/sh/mm/cache-sh4.c | 2 +-
arch/sh/mm/cache.c | 8 +-
arch/sparc/include/asm/pgtable_64.h | 16 -
arch/sparc/mm/fault_64.c | 3 -
arch/sparc/mm/gup.c | 16 +-
arch/tile/include/asm/pgtable.h | 10 -
arch/x86/include/asm/pgtable.h | 9 -
arch/x86/include/asm/pgtable_types.h | 2 -
arch/x86/kernel/vm86_32.c | 6 +-
arch/x86/mm/gup.c | 17 +-
arch/x86/mm/pgtable.c | 14 -
arch/xtensa/mm/tlb.c | 2 +-
fs/proc/page.c | 4 +-
fs/proc/task_mmu.c | 55 +-
include/asm-generic/pgtable.h | 9 -
include/linux/huge_mm.h | 56 +-
include/linux/memcontrol.h | 16 +-
include/linux/mm.h | 116 ++-
include/linux/mm_types.h | 20 +-
include/linux/page-flags.h | 49 +-
include/linux/pagemap.h | 13 +-
include/linux/rmap.h | 16 +-
include/linux/swap.h | 3 +-
include/linux/vm_event_item.h | 4 +-
include/trace/events/huge_memory.h | 1 +
kernel/events/uprobes.c | 11 +-
kernel/futex.c | 61 +-
mm/debug.c | 8 +-
mm/filemap.c | 10 +-
mm/gup.c | 114 +--
mm/huge_memory.c | 1127 +++++++++++++++++-------------
mm/hugetlb.c | 10 +-
mm/internal.h | 70 +-
mm/ksm.c | 61 +-
mm/madvise.c | 2 +-
mm/memcontrol.c | 84 +--
mm/memory-failure.c | 8 +-
mm/memory.c | 73 +-
mm/mempolicy.c | 40 +-
mm/migrate.c | 19 +-
mm/mincore.c | 2 +-
mm/mlock.c | 12 +-
mm/mprotect.c | 2 +-
mm/mremap.c | 15 +-
mm/page_alloc.c | 34 +-
mm/page_idle.c | 3 +-
mm/pagewalk.c | 2 +-
mm/pgtable-generic.c | 14 -
mm/rmap.c | 163 +++--
mm/shmem.c | 21 +-
mm/swap.c | 274 +-------
mm/swapfile.c | 16 +-
mm/userfaultfd.c | 8 +-
mm/vmstat.c | 4 +-
77 files changed, 1391 insertions(+), 1773 deletions(-)
--
2.5.3
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [next] | [standalone]
| From | "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> |
|---|---|
| Date | 2015-10-06 17:40 +0200 |
| Subject | [PATCHv12 36/37] thp: update documentation |
| Message-ID | <qgCvq-1OO-85@gated-at.bofh.it> |
| In reply to | #1240601 |
The patch updates Documentation/vm/transhuge.txt to reflect changes in THP design. Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Jerome Marchand <jmarchan@redhat.com> --- Documentation/vm/transhuge.txt | 151 ++++++++++++++++++++++++++--------------- 1 file changed, 96 insertions(+), 55 deletions(-) diff --git a/Documentation/vm/transhuge.txt b/Documentation/vm/transhuge.txt index 8a282687ee06..21cf34f3ddb2 100644 --- a/Documentation/vm/transhuge.txt +++ b/Documentation/vm/transhuge.txt @@ -35,10 +35,10 @@ miss is going to run faster. == Design == -- "graceful fallback": mm components which don't have transparent - hugepage knowledge fall back to breaking a transparent hugepage and - working on the regular pages and their respective regular pmd/pte - mappings +- "graceful fallback": mm components which don't have transparent hugepage + knowledge fall back to breaking huge pmd mapping into table of ptes and, + if necessary, split a transparent hugepage. Therefore these components + can continue working on the regular pages or regular pte mappings. - if a hugepage allocation fails because of memory fragmentation, regular pages should be gracefully allocated instead and mixed in @@ -221,9 +221,18 @@ thp_collapse_alloc_failed is incremented if khugepaged found a range of pages that should be collapsed into one huge page but failed the allocation. -thp_split is incremented every time a huge page is split into base +thp_split_page is incremented every time a huge page is split into base pages. This can happen for a variety of reasons but a common reason is that a huge page is old and is being reclaimed. + This action implies splitting all PMD the page mapped with. + +thp_split_page_failed is is incremented if kernel fails to split huge + page. This can happen if the page was pinned by somebody. + +thp_split_pmd is incremented every time a PMD split into table of PTEs. + This can happen, for instance, when application calls mprotect() or + munmap() on part of huge page. It doesn't split huge page, only + page table entry. thp_zero_page_alloc is incremented every time a huge zero page is successfully allocated. It includes allocations which where @@ -274,10 +283,8 @@ is complete, so they won't ever notice the fact the page is huge. But if any driver is going to mangle over the page structure of the tail page (like for checking page->mapping or other bits that are relevant for the head page and not the tail page), it should be updated to jump -to check head page instead (while serializing properly against -split_huge_page() to avoid the head and tail pages to disappear from -under it, see the futex code to see an example of that, hugetlbfs also -needed special handling in futex code for similar reasons). +to check head page instead. Taking reference on any head/tail page would +prevent page from being split by anyone. NOTE: these aren't new constraints to the GUP API, and they match the same constrains that applies to hugetlbfs too, so any driver capable @@ -312,9 +319,9 @@ unaffected. libhugetlbfs will also work fine as usual. == Graceful fallback == Code walking pagetables but unware about huge pmds can simply call -split_huge_page_pmd(vma, addr, pmd) where the pmd is the one returned by +split_huge_pmd(vma, pmd, addr) where the pmd is the one returned by pmd_offset. It's trivial to make the code transparent hugepage aware -by just grepping for "pmd_offset" and adding split_huge_page_pmd where +by just grepping for "pmd_offset" and adding split_huge_pmd where missing after pmd_offset returns the pmd. Thanks to the graceful fallback design, with a one liner change, you can avoid to write hundred if not thousand of lines of complex code to make your code @@ -323,7 +330,8 @@ hugepage aware. If you're not walking pagetables but you run into a physical hugepage but you can't handle it natively in your code, you can split it by calling split_huge_page(page). This is what the Linux VM does before -it tries to swapout the hugepage for example. +it tries to swapout the hugepage for example. split_huge_page() can fail +if the page is pinned and you must handle this correctly. Example to make mremap.c transparent hugepage aware with a one liner change: @@ -335,14 +343,14 @@ diff --git a/mm/mremap.c b/mm/mremap.c return NULL; pmd = pmd_offset(pud, addr); -+ split_huge_page_pmd(vma, addr, pmd); ++ split_huge_pmd(vma, pmd, addr); if (pmd_none_or_clear_bad(pmd)) return NULL; == Locking in hugepage aware code == We want as much code as possible hugepage aware, as calling -split_huge_page() or split_huge_page_pmd() has a cost. +split_huge_page() or split_huge_pmd() has a cost. To make pagetable walks huge pmd aware, all you need to do is to call pmd_trans_huge() on the pmd returned by pmd_offset. You must hold the @@ -351,47 +359,80 @@ created from under you by khugepaged (khugepaged collapse_huge_page takes the mmap_sem in write mode in addition to the anon_vma lock). If pmd_trans_huge returns false, you just fallback in the old code paths. If instead pmd_trans_huge returns true, you have to take the -mm->page_table_lock and re-run pmd_trans_huge. Taking the -page_table_lock will prevent the huge pmd to be converted into a -regular pmd from under you (split_huge_page can run in parallel to the +page table lock (pmd_lock()) and re-run pmd_trans_huge. Taking the +page table lock will prevent the huge pmd to be converted into a +regular pmd from under you (split_huge_pmd can run in parallel to the pagetable walk). If the second pmd_trans_huge returns false, you -should just drop the page_table_lock and fallback to the old code as -before. Otherwise you should run pmd_trans_splitting on the pmd. In -case pmd_trans_splitting returns true, it means split_huge_page is -already in the middle of splitting the page. So if pmd_trans_splitting -returns true it's enough to drop the page_table_lock and call -wait_split_huge_page and then fallback the old code paths. You are -guaranteed by the time wait_split_huge_page returns, the pmd isn't -huge anymore. If pmd_trans_splitting returns false, you can proceed to -process the huge pmd and the hugepage natively. Once finished you can -drop the page_table_lock. - -== compound_lock, get_user_pages and put_page == +should just drop the page table lock and fallback to the old code as +before. Otherwise you can proceed to process the huge pmd and the +hugepage natively. Once finished you can drop the page table lock. + +== Refcounts and transparent huge pages == + +Refcounting on THP is mostly consistent with refcounting on other compound +pages: + + - get_page()/put_page() and GUP operate in head page's ->_count. + + - ->_count in tail pages is always zero: get_page_unless_zero() never + succeed on tail pages. + + - map/unmap of the pages with PTE entry increment/decrement ->_mapcount + on relevant sub-page of the compound page. + + - map/unmap of the whole compound page accounted in compound_mapcount + (stored in first tail page). + +PageDoubleMap() indicates that ->_mapcount in all subpages is offset up by one. +This additional reference is required to get race-free detection of unmap of +subpages when we have them mapped with both PMDs and PTEs. + +This is optimization required to lower overhead of per-subpage mapcount +tracking. The alternative is alter ->_mapcount in all subpages on each +map/unmap of the whole compound page. + +We set PG_double_map when a PMD of the page got split for the first time, +but still have PMD mapping. The addtional references go away with last +compound_mapcount. split_huge_page internally has to distribute the refcounts in the head -page to the tail pages before clearing all PG_head/tail bits from the -page structures. It can do that easily for refcounts taken by huge pmd -mappings. But the GUI API as created by hugetlbfs (that returns head -and tail pages if running get_user_pages on an address backed by any -hugepage), requires the refcount to be accounted on the tail pages and -not only in the head pages, if we want to be able to run -split_huge_page while there are gup pins established on any tail -page. Failure to be able to run split_huge_page if there's any gup pin -on any tail page, would mean having to split all hugepages upfront in -get_user_pages which is unacceptable as too many gup users are -performance critical and they must work natively on hugepages like -they work natively on hugetlbfs already (hugetlbfs is simpler because -hugetlbfs pages cannot be split so there wouldn't be requirement of -accounting the pins on the tail pages for hugetlbfs). If we wouldn't -account the gup refcounts on the tail pages during gup, we won't know -anymore which tail page is pinned by gup and which is not while we run -split_huge_page. But we still have to add the gup pin to the head page -too, to know when we can free the compound page in case it's never -split during its lifetime. That requires changing not just -get_page, but put_page as well so that when put_page runs on a tail -page (and only on a tail page) it will find its respective head page, -and then it will decrease the head page refcount in addition to the -tail page refcount. To obtain a head page reliably and to decrease its -refcount without race conditions, put_page has to serialize against -__split_huge_page_refcount using a special per-page lock called -compound_lock. +page to the tail pages before clearing all PG_head/tail bits from the page +structures. It can be done easily for refcounts taken by page table +entries. But we don't have enough information on how to distribute any +additional pins (i.e. from get_user_pages). split_huge_page() fails any +requests to split pinned huge page: it expects page count to be equal to +sum of mapcount of all sub-pages plus one (split_huge_page caller must +have reference for head page). + +split_huge_page uses migration entries to stabilize page->_count and +page->_mapcount. + +We safe against physical memory scanners too: the only legitimate way +scanner can get reference to a page is get_page_unless_zero(). + +All tail pages has zero ->_count until atomic_add(). It prevent scanner +from geting reference to tail page up to the point. After the atomic_add() +we don't care about ->_count value. We already known how many references +with should uncharge from head page. + +For head page get_page_unless_zero() will succeed and we don't mind. It's +clear where reference should go after split: it will stay on head page. + +Note that split_huge_pmd() doesn't have any limitation on refcounting: +pmd can be split at any point and never fails. + +== Partial unmap and deferred_split_huge_page() == + +Unmapping part of THP (with munmap() or other way) is not going to free +memory immediately. Instead, we detect that a subpage of THP is not in use +in page_remove_rmap() and queue the THP for splitting if memory pressure +comes. Splitting will free up unused subpages. + +Splitting the page right away is not an option due to locking context in +the place where we can detect partial unmap. It's also might be +counterproductive since in many cases partial unmap unmap happens during +exit(2) if an THP crosses VMA boundary. + +Function deferred_split_huge_page() is used to queue page for splitting. +The splitting itself will happen when we get memory pressure via shrinker +interface. -- 2.5.3 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> |
|---|---|
| Date | 2015-10-06 17:40 +0200 |
| Subject | [PATCHv12 17/37] arm64, thp: remove infrastructure for handling splitting PMDs |
| Message-ID | <qgCvq-1OO-87@gated-at.bofh.it> |
| In reply to | #1240601 |
With new refcounting we don't need to mark PMDs splitting. Let's drop
code to handle this.
pmdp_splitting_flush() is not needed too: on splitting PMD we will do
pmdp_clear_flush() + set_pte_at(). pmdp_clear_flush() will do IPI as
needed for fast_gup.
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
arch/arm64/include/asm/pgtable.h | 8 --------
arch/arm64/mm/flush.c | 16 ----------------
2 files changed, 24 deletions(-)
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index bd5db28324ba..26c7dea80062 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -274,20 +274,12 @@ static inline pgprot_t mk_sect_prot(pgprot_t prot)
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
#define pmd_trans_huge(pmd) (pmd_val(pmd) && !(pmd_val(pmd) & PMD_TABLE_BIT))
-#define pmd_trans_splitting(pmd) pte_special(pmd_pte(pmd))
-#ifdef CONFIG_HAVE_RCU_TABLE_FREE
-#define __HAVE_ARCH_PMDP_SPLITTING_FLUSH
-struct vm_area_struct;
-void pmdp_splitting_flush(struct vm_area_struct *vma, unsigned long address,
- pmd_t *pmdp);
-#endif /* CONFIG_HAVE_RCU_TABLE_FREE */
#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
#define pmd_dirty(pmd) pte_dirty(pmd_pte(pmd))
#define pmd_young(pmd) pte_young(pmd_pte(pmd))
#define pmd_dirty(pmd) pte_dirty(pmd_pte(pmd))
#define pmd_wrprotect(pmd) pte_pmd(pte_wrprotect(pmd_pte(pmd)))
-#define pmd_mksplitting(pmd) pte_pmd(pte_mkspecial(pmd_pte(pmd)))
#define pmd_mkold(pmd) pte_pmd(pte_mkold(pmd_pte(pmd)))
#define pmd_mkwrite(pmd) pte_pmd(pte_mkwrite(pmd_pte(pmd)))
#define pmd_mkclean(pmd) pte_pmd(pte_mkclean(pmd_pte(pmd)))
diff --git a/arch/arm64/mm/flush.c b/arch/arm64/mm/flush.c
index 4dfa3975ce5b..fc9c657e6f41 100644
--- a/arch/arm64/mm/flush.c
+++ b/arch/arm64/mm/flush.c
@@ -103,19 +103,3 @@ EXPORT_SYMBOL(flush_dcache_page);
* Additional functions defined in assembly.
*/
EXPORT_SYMBOL(flush_icache_range);
-
-#ifdef CONFIG_TRANSPARENT_HUGEPAGE
-#ifdef CONFIG_HAVE_RCU_TABLE_FREE
-void pmdp_splitting_flush(struct vm_area_struct *vma, unsigned long address,
- pmd_t *pmdp)
-{
- pmd_t pmd = pmd_mksplitting(*pmdp);
-
- VM_BUG_ON(address & ~PMD_MASK);
- set_pmd_at(vma->vm_mm, address, pmdp, pmd);
-
- /* dummy IPI to serialise against fast_gup */
- kick_all_cpus_sync();
-}
-#endif /* CONFIG_HAVE_RCU_TABLE_FREE */
-#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
--
2.5.3
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> |
|---|---|
| Date | 2015-10-06 17:40 +0200 |
| Subject | [PATCHv12 01/37] mm, proc: adjust PSS calculation |
| Message-ID | <qgCvr-1OO-91@gated-at.bofh.it> |
| In reply to | #1240601 |
With new refcounting all subpages of the compound page are not necessary
have the same mapcount. We need to take into account mapcount of every
sub-page.
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Tested-by: Sasha Levin <sasha.levin@oracle.com>
Tested-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
Acked-by: Jerome Marchand <jmarchan@redhat.com>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
---
fs/proc/task_mmu.c | 47 +++++++++++++++++++++++++++++++----------------
1 file changed, 31 insertions(+), 16 deletions(-)
diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index bd167675a06f..ace02a4a07db 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -454,9 +454,10 @@ struct mem_size_stats {
};
static void smaps_account(struct mem_size_stats *mss, struct page *page,
- unsigned long size, bool young, bool dirty)
+ bool compound, bool young, bool dirty)
{
- int mapcount;
+ int i, nr = compound ? HPAGE_PMD_NR : 1;
+ unsigned long size = nr * PAGE_SIZE;
if (PageAnon(page))
mss->anonymous += size;
@@ -465,23 +466,37 @@ static void smaps_account(struct mem_size_stats *mss, struct page *page,
/* Accumulate the size in pages that have been accessed. */
if (young || page_is_young(page) || PageReferenced(page))
mss->referenced += size;
- mapcount = page_mapcount(page);
- if (mapcount >= 2) {
- u64 pss_delta;
- if (dirty || PageDirty(page))
- mss->shared_dirty += size;
- else
- mss->shared_clean += size;
- pss_delta = (u64)size << PSS_SHIFT;
- do_div(pss_delta, mapcount);
- mss->pss += pss_delta;
- } else {
+ /*
+ * page_count(page) == 1 guarantees the page is mapped exactly once.
+ * If any subpage of the compound page mapped with PTE it would elevate
+ * page_count().
+ */
+ if (page_count(page) == 1) {
if (dirty || PageDirty(page))
mss->private_dirty += size;
else
mss->private_clean += size;
mss->pss += (u64)size << PSS_SHIFT;
+ return;
+ }
+
+ for (i = 0; i < nr; i++, page++) {
+ int mapcount = page_mapcount(page);
+
+ if (mapcount >= 2) {
+ if (dirty || PageDirty(page))
+ mss->shared_dirty += PAGE_SIZE;
+ else
+ mss->shared_clean += PAGE_SIZE;
+ mss->pss += (PAGE_SIZE << PSS_SHIFT) / mapcount;
+ } else {
+ if (dirty || PageDirty(page))
+ mss->private_dirty += PAGE_SIZE;
+ else
+ mss->private_clean += PAGE_SIZE;
+ mss->pss += PAGE_SIZE << PSS_SHIFT;
+ }
}
}
@@ -516,7 +531,8 @@ static void smaps_pte_entry(pte_t *pte, unsigned long addr,
if (!page)
return;
- smaps_account(mss, page, PAGE_SIZE, pte_young(*pte), pte_dirty(*pte));
+
+ smaps_account(mss, page, false, pte_young(*pte), pte_dirty(*pte));
}
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
@@ -532,8 +548,7 @@ static void smaps_pmd_entry(pmd_t *pmd, unsigned long addr,
if (IS_ERR_OR_NULL(page))
return;
mss->anonymous_thp += HPAGE_PMD_SIZE;
- smaps_account(mss, page, HPAGE_PMD_SIZE,
- pmd_young(*pmd), pmd_dirty(*pmd));
+ smaps_account(mss, page, true, pmd_young(*pmd), pmd_dirty(*pmd));
}
#else
static void smaps_pmd_entry(pmd_t *pmd, unsigned long addr,
--
2.5.3
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> |
|---|---|
| Date | 2015-10-06 17:40 +0200 |
| Subject | [PATCHv12 14/37] futex, thp: remove special case for THP in get_futex_key |
| Message-ID | <qgCvr-1OO-93@gated-at.bofh.it> |
| In reply to | #1240601 |
With new THP refcounting, we don't need tricks to stabilize huge page.
If we've got reference to tail page, it can't split under us.
This patch effectively reverts a5b338f2b0b1.
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Tested-by: Sasha Levin <sasha.levin@oracle.com>
Tested-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
Acked-by: Jerome Marchand <jmarchan@redhat.com>
---
kernel/futex.c | 61 ++++++++++++----------------------------------------------
1 file changed, 12 insertions(+), 49 deletions(-)
diff --git a/kernel/futex.c b/kernel/futex.c
index c4a182f5357e..f9d46c3d9be9 100644
--- a/kernel/futex.c
+++ b/kernel/futex.c
@@ -399,7 +399,7 @@ get_futex_key(u32 __user *uaddr, int fshared, union futex_key *key, int rw)
{
unsigned long address = (unsigned long)uaddr;
struct mm_struct *mm = current->mm;
- struct page *page, *page_head;
+ struct page *page;
int err, ro = 0;
/*
@@ -442,46 +442,9 @@ again:
else
err = 0;
-#ifdef CONFIG_TRANSPARENT_HUGEPAGE
- page_head = page;
- if (unlikely(PageTail(page))) {
- put_page(page);
- /* serialize against __split_huge_page_splitting() */
- local_irq_disable();
- if (likely(__get_user_pages_fast(address, 1, !ro, &page) == 1)) {
- page_head = compound_head(page);
- /*
- * page_head is valid pointer but we must pin
- * it before taking the PG_lock and/or
- * PG_compound_lock. The moment we re-enable
- * irqs __split_huge_page_splitting() can
- * return and the head page can be freed from
- * under us. We can't take the PG_lock and/or
- * PG_compound_lock on a page that could be
- * freed from under us.
- */
- if (page != page_head) {
- get_page(page_head);
- put_page(page);
- }
- local_irq_enable();
- } else {
- local_irq_enable();
- goto again;
- }
- }
-#else
- page_head = compound_head(page);
- if (page != page_head) {
- get_page(page_head);
- put_page(page);
- }
-#endif
-
- lock_page(page_head);
-
+ lock_page(page);
/*
- * If page_head->mapping is NULL, then it cannot be a PageAnon
+ * If page->mapping is NULL, then it cannot be a PageAnon
* page; but it might be the ZERO_PAGE or in the gate area or
* in a special mapping (all cases which we are happy to fail);
* or it may have been a good file page when get_user_pages_fast
@@ -493,12 +456,12 @@ again:
*
* The case we do have to guard against is when memory pressure made
* shmem_writepage move it from filecache to swapcache beneath us:
- * an unlikely race, but we do need to retry for page_head->mapping.
+ * an unlikely race, but we do need to retry for page->mapping.
*/
- if (!page_head->mapping) {
- int shmem_swizzled = PageSwapCache(page_head);
- unlock_page(page_head);
- put_page(page_head);
+ if (!page->mapping) {
+ int shmem_swizzled = PageSwapCache(page);
+ unlock_page(page);
+ put_page(page);
if (shmem_swizzled)
goto again;
return -EFAULT;
@@ -511,7 +474,7 @@ again:
* it's a read-only handle, it's expected that futexes attach to
* the object not the particular process.
*/
- if (PageAnon(page_head)) {
+ if (PageAnon(page)) {
/*
* A RO anonymous page will never change and thus doesn't make
* sense for futex operations.
@@ -526,15 +489,15 @@ again:
key->private.address = address;
} else {
key->both.offset |= FUT_OFF_INODE; /* inode-based key */
- key->shared.inode = page_head->mapping->host;
+ key->shared.inode = page->mapping->host;
key->shared.pgoff = basepage_index(page);
}
get_futex_key_refs(key); /* implies MB (B) */
out:
- unlock_page(page_head);
- put_page(page_head);
+ unlock_page(page);
+ put_page(page);
return err;
}
--
2.5.3
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web