Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1212114 > unrolled thread
| Started by | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| First post | 2015-08-24 14:20 +0200 |
| Last post | 2015-08-25 17:50 +0200 |
| Articles | 18 — 3 participants |
Back to article view | Back to linux.kernel
[PATCH 00/12] Remove zonelist cache and high-order watermark checking v3 Mel Gorman <mgorman@techsingularity.net> - 2015-08-24 14:20 +0200
[PATCH 10/12] mm, page_alloc: Remove MIGRATE_RESERVE Mel Gorman <mgorman@techsingularity.net> - 2015-08-24 14:30 +0200
[PATCH 11/12] mm, page_alloc: Reserve pageblocks for high-order atomic allocations on demand Mel Gorman <mgorman@techsingularity.net> - 2015-08-24 14:40 +0200
Re: [PATCH 11/12] mm, page_alloc: Reserve pageblocks for high-order atomic allocations on demand Vlastimil Babka <vbabka@suse.cz> - 2015-08-26 14:50 +0200
Re: [PATCH 11/12] mm, page_alloc: Reserve pageblocks for high-order atomic allocations on demand Michal Hocko <mhocko@kernel.org> - 2015-08-26 17:00 +0200
Re: [PATCH 11/12] mm, page_alloc: Reserve pageblocks for high-order atomic allocations on demand Mel Gorman <mgorman@techsingularity.net> - 2015-08-26 17:40 +0200
[PATCH 12/12] mm, page_alloc: Only enforce watermarks for order-0 allocations Mel Gorman <mgorman@techsingularity.net> - 2015-08-24 14:40 +0200
Re: [PATCH 12/12] mm, page_alloc: Only enforce watermarks for order-0 allocations Vlastimil Babka <vbabka@suse.cz> - 2015-08-26 15:50 +0200
Re: [PATCH 12/12] mm, page_alloc: Only enforce watermarks for order-0 allocations Mel Gorman <mgorman@techsingularity.net> - 2015-08-26 17:00 +0200
Re: [PATCH 12/12] mm, page_alloc: Only enforce watermarks for order-0 allocations Michal Hocko <mhocko@kernel.org> - 2015-08-28 14:20 +0200
Re: [PATCH 12/12] mm, page_alloc: Only enforce watermarks for order-0 allocations Mel Gorman <mgorman@techsingularity.net> - 2015-08-28 16:20 +0200
Re: [PATCH 07/12] mm, page_alloc: Distinguish between being unable to sleep, unwilling to sleep and avoiding waking kswapd Mel Gorman <mgorman@techsingularity.net> - 2015-08-24 20:30 +0200
Re: [PATCH 07/12] mm, page_alloc: Distinguish between being unable to sleep, unwilling to sleep and avoiding waking kswapd Vlastimil Babka <vbabka@suse.cz> - 2015-08-25 17:40 +0200
Re: [PATCH 07/12] mm, page_alloc: Distinguish between being unable to sleep, unwilling to sleep and avoiding waking kswapd Mel Gorman <mgorman@techsingularity.net> - 2015-08-26 16:50 +0200
Re: [PATCH 07/12] mm, page_alloc: Distinguish between being unable to sleep, unwilling to sleep and avoiding waking kswapd Vlastimil Babka <vbabka@suse.cz> - 2015-08-26 18:30 +0200
Re: [PATCH 07/12] mm, page_alloc: Distinguish between being unable to sleep, unwilling to sleep and avoiding waking kswapd Mel Gorman <mgorman@techsingularity.net> - 2015-08-26 20:20 +0200
Re: [PATCH 07/12] mm, page_alloc: Distinguish between being unable to sleep, unwilling to sleep and avoiding waking kswapd Vlastimil Babka <vbabka@suse.cz> - 2015-08-27 11:20 +0200
Re: [PATCH 07/12] mm, page_alloc: Distinguish between being unable to sleep, unwilling to sleep and avoiding waking kswapd Vlastimil Babka <vbabka@suse.cz> - 2015-08-25 17:50 +0200
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2015-08-24 14:20 +0200 |
| Subject | [PATCH 00/12] Remove zonelist cache and high-order watermark checking v3 |
| Message-ID | <q0YTf-7q8-1@gated-at.bofh.it> |
Changelog since V3 o Covered cases where __GFP_KSWAPD_RECLAIM is needed (vbabka) o Cleaned up trailing references to zlc (vbabka) o Fixed a subtle problem with GFP_TRANSHUGE checks (vbabka) o Split out an unrelated change to its own patch (vbabka) o Reordered series to put GFP flag modifications at start (mhocko) o Added a number of clarifications on reclaim modifications (mhocko) o Only check cpusets when one exists that can limit memory (rientjes) o Applied acks Changelog since V2 o Improve cpusets checks as suggested (rientjes) o Add various acks and reviewed-bys o Rebase to 4.2-rc6 Changelog since V1 o Rebase to 4.2-rc5 o Distinguish between high priority callers and callers that avoid sleep o Remove jump label related damage patches Overall, the intent of this series is to remove the zonelist cache which was introduced to avoid high overhead in the page allocator. Once this is done, it is necessary to reduce the cost of watermark checks. The series starts with minor micro-optimisations. Next it notes that GFP flags that affect watermark checks are bused. __GFP_WAIT historically identified callers that could not sleep and could access reserves. This was later abused to identify callers that simply prefer to avoid sleeping and have other options. A patch distinguishes between atomic callers, high-priority callers and those that simply wish to avoid sleep. The zonelist cache has been around for a long time but it is of dubious merit with a lot of complexity and some issues that are explained. The most important issue is that a failed THP allocation can cause a zone to be treated as "full". This potentially causes unnecessary stalls, reclaim activity or remote fallbacks. The issues could be fixed but it's not worth it. The series places a small number of other micro-optimisations on top before examining GFP flags watermarks. High-order watermarks enforcement can cause high-order allocations to fail even though pages are free. The watermark checks both protect high-order atomic allocations and make kswapd aware of high-order pages but there is a much better way that can be handled using migrate types. This series uses page grouping by mobility to reserve pageblocks for high-order allocations with the size of the reservation depending on demand. kswapd awareness is maintained by examining the free lists. By patch 12 in this series, there are no high-order watermark checks while preserving the properties that motivated the introduction of the watermark checks. Documentation/vm/balance | 14 +- arch/arm/mm/dma-mapping.c | 4 +- arch/arm/xen/mm.c | 2 +- arch/arm64/mm/dma-mapping.c | 4 +- arch/x86/kernel/pci-dma.c | 2 +- block/bio.c | 26 +- block/blk-core.c | 16 +- block/blk-ioc.c | 2 +- block/blk-mq-tag.c | 2 +- block/blk-mq.c | 8 +- block/cfq-iosched.c | 4 +- block/scsi_ioctl.c | 6 +- drivers/block/drbd/drbd_bitmap.c | 2 +- drivers/block/drbd/drbd_receiver.c | 3 +- drivers/block/mtip32xx/mtip32xx.c | 2 +- drivers/block/nvme-core.c | 4 +- drivers/block/osdblk.c | 2 +- drivers/block/paride/pd.c | 2 +- drivers/block/pktcdvd.c | 4 +- drivers/connector/connector.c | 3 +- drivers/firewire/core-cdev.c | 2 +- drivers/gpu/drm/i915/i915_gem.c | 4 +- drivers/ide/ide-atapi.c | 2 +- drivers/ide/ide-cd.c | 2 +- drivers/ide/ide-cd_ioctl.c | 2 +- drivers/ide/ide-devsets.c | 2 +- drivers/ide/ide-disk.c | 2 +- drivers/ide/ide-ioctls.c | 4 +- drivers/ide/ide-park.c | 2 +- drivers/ide/ide-pm.c | 4 +- drivers/ide/ide-tape.c | 4 +- drivers/ide/ide-taskfile.c | 4 +- drivers/infiniband/core/sa_query.c | 2 +- drivers/infiniband/hw/ipath/ipath_file_ops.c | 2 +- drivers/infiniband/hw/qib/qib_init.c | 2 +- drivers/iommu/amd_iommu.c | 2 +- drivers/iommu/intel-iommu.c | 2 +- drivers/md/dm-crypt.c | 6 +- drivers/md/dm-kcopyd.c | 2 +- drivers/media/pci/solo6x10/solo6x10-v4l2-enc.c | 2 +- drivers/media/pci/solo6x10/solo6x10-v4l2.c | 2 +- drivers/media/pci/tw68/tw68-video.c | 2 +- drivers/misc/vmw_balloon.c | 2 +- drivers/mtd/mtdcore.c | 3 +- drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c | 2 +- drivers/scsi/scsi_error.c | 2 +- drivers/scsi/scsi_lib.c | 4 +- drivers/staging/android/ion/ion_system_heap.c | 2 +- .../lustre/include/linux/libcfs/libcfs_private.h | 2 +- drivers/usb/host/u132-hcd.c | 2 +- drivers/video/fbdev/vermilion/vermilion.c | 2 +- fs/btrfs/disk-io.c | 2 +- fs/btrfs/extent_io.c | 14 +- fs/btrfs/volumes.c | 4 +- fs/cachefiles/internal.h | 2 +- fs/direct-io.c | 2 +- fs/ext3/super.c | 2 +- fs/ext4/super.c | 2 +- fs/fscache/cookie.c | 2 +- fs/fscache/page.c | 6 +- fs/jbd/transaction.c | 4 +- fs/jbd2/transaction.c | 4 +- fs/nfs/file.c | 6 +- fs/nilfs2/mdt.h | 2 +- fs/xfs/xfs_qm.c | 2 +- include/linux/cpuset.h | 18 +- include/linux/gfp.h | 70 ++- include/linux/mmzone.h | 88 +--- include/linux/skbuff.h | 6 +- include/net/sock.h | 2 +- include/trace/events/gfpflags.h | 5 +- kernel/audit.c | 6 +- kernel/locking/lockdep.c | 2 +- kernel/power/snapshot.c | 2 +- kernel/power/swap.c | 14 +- kernel/smp.c | 2 +- lib/idr.c | 4 +- lib/percpu_ida.c | 2 +- lib/radix-tree.c | 10 +- mm/backing-dev.c | 2 +- mm/dmapool.c | 2 +- mm/failslab.c | 8 +- mm/filemap.c | 2 +- mm/huge_memory.c | 4 +- mm/internal.h | 1 + mm/memcontrol.c | 8 +- mm/mempool.c | 10 +- mm/migrate.c | 4 +- mm/page_alloc.c | 585 +++++++-------------- mm/slab.c | 18 +- mm/slub.c | 6 +- mm/vmalloc.c | 2 +- mm/vmscan.c | 8 +- mm/vmstat.c | 2 +- mm/zswap.c | 5 +- net/core/skbuff.c | 8 +- net/core/sock.c | 6 +- net/netlink/af_netlink.c | 2 +- net/rxrpc/ar-connection.c | 2 +- net/sctp/associola.c | 2 +- security/integrity/ima/ima_crypto.c | 2 +- 101 files changed, 455 insertions(+), 710 deletions(-) -- 2.4.6 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2015-08-24 14:30 +0200 |
| Subject | [PATCH 10/12] mm, page_alloc: Remove MIGRATE_RESERVE |
| Message-ID | <q0Z2X-7BA-13@gated-at.bofh.it> |
| In reply to | #1212114 |
MIGRATE_RESERVE preserves an old property of the buddy allocator that existed
prior to fragmentation avoidance -- min_free_kbytes worth of pages tended to
remain contiguous until the only alternative was to fail the allocation. At the
time it was discovered that high-order atomic allocations relied on this
property so MIGRATE_RESERVE was introduced. A later patch will introduce
an alternative MIGRATE_HIGHATOMIC so this patch deletes MIGRATE_RESERVE
and supporting code so it'll be easier to review. Note that this patch
in isolation may look like a false regression if someone was bisecting
high-order atomic allocation failures.
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
---
include/linux/mmzone.h | 10 +---
mm/huge_memory.c | 2 +-
mm/page_alloc.c | 148 +++----------------------------------------------
mm/vmstat.c | 1 -
4 files changed, 11 insertions(+), 150 deletions(-)
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index aef62cc11c80..cf643539d640 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -39,8 +39,6 @@ enum {
MIGRATE_UNMOVABLE,
MIGRATE_MOVABLE,
MIGRATE_RECLAIMABLE,
- MIGRATE_PCPTYPES, /* the number of types on the pcp lists */
- MIGRATE_RESERVE = MIGRATE_PCPTYPES,
#ifdef CONFIG_CMA
/*
* MIGRATE_CMA migration type is designed to mimic the way
@@ -63,6 +61,8 @@ enum {
MIGRATE_TYPES
};
+#define MIGRATE_PCPTYPES (MIGRATE_RECLAIMABLE+1)
+
#ifdef CONFIG_CMA
# define is_migrate_cma(migratetype) unlikely((migratetype) == MIGRATE_CMA)
#else
@@ -425,12 +425,6 @@ struct zone {
const char *name;
- /*
- * Number of MIGRATE_RESERVE page block. To maintain for just
- * optimization. Protected by zone->lock.
- */
- int nr_migrate_reserve_block;
-
#ifdef CONFIG_MEMORY_ISOLATION
/*
* Number of isolated pageblock. It is used to solve incorrect
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 36efda9ff8f1..56cfb17169d2 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -113,7 +113,7 @@ static int set_recommended_min_free_kbytes(void)
for_each_populated_zone(zone)
nr_zones++;
- /* Make sure at least 2 hugepages are free for MIGRATE_RESERVE */
+ /* Ensure 2 pageblocks are free to assist fragmentation avoidance */
recommended_min = pageblock_nr_pages * nr_zones * 2;
/*
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index aa52a91a7d44..d5ce050ebe4f 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -792,7 +792,6 @@ static void free_pcppages_bulk(struct zone *zone, int count,
if (unlikely(has_isolate_pageblock(zone)))
mt = get_pageblock_migratetype(page);
- /* MIGRATE_MOVABLE list may include MIGRATE_RESERVEs */
__free_one_page(page, page_to_pfn(page), zone, 0, mt);
trace_mm_page_pcpu_drain(page, 0, mt);
} while (--to_free && --batch_free && !list_empty(list));
@@ -1390,15 +1389,14 @@ struct page *__rmqueue_smallest(struct zone *zone, unsigned int order,
* the free lists for the desirable migrate type are depleted
*/
static int fallbacks[MIGRATE_TYPES][4] = {
- [MIGRATE_UNMOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_MOVABLE, MIGRATE_RESERVE },
- [MIGRATE_RECLAIMABLE] = { MIGRATE_UNMOVABLE, MIGRATE_MOVABLE, MIGRATE_RESERVE },
- [MIGRATE_MOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_UNMOVABLE, MIGRATE_RESERVE },
+ [MIGRATE_UNMOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_MOVABLE, MIGRATE_TYPES },
+ [MIGRATE_RECLAIMABLE] = { MIGRATE_UNMOVABLE, MIGRATE_MOVABLE, MIGRATE_TYPES },
+ [MIGRATE_MOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_UNMOVABLE, MIGRATE_TYPES },
#ifdef CONFIG_CMA
- [MIGRATE_CMA] = { MIGRATE_RESERVE }, /* Never used */
+ [MIGRATE_CMA] = { MIGRATE_TYPES }, /* Never used */
#endif
- [MIGRATE_RESERVE] = { MIGRATE_RESERVE }, /* Never used */
#ifdef CONFIG_MEMORY_ISOLATION
- [MIGRATE_ISOLATE] = { MIGRATE_RESERVE }, /* Never used */
+ [MIGRATE_ISOLATE] = { MIGRATE_TYPES }, /* Never used */
#endif
};
@@ -1572,7 +1570,7 @@ int find_suitable_fallback(struct free_area *area, unsigned int order,
*can_steal = false;
for (i = 0;; i++) {
fallback_mt = fallbacks[migratetype][i];
- if (fallback_mt == MIGRATE_RESERVE)
+ if (fallback_mt == MIGRATE_TYPES)
break;
if (list_empty(&area->free_list[fallback_mt]))
@@ -1651,25 +1649,13 @@ static struct page *__rmqueue(struct zone *zone, unsigned int order,
{
struct page *page;
-retry_reserve:
page = __rmqueue_smallest(zone, order, migratetype);
-
- if (unlikely(!page) && migratetype != MIGRATE_RESERVE) {
+ if (unlikely(!page)) {
if (migratetype == MIGRATE_MOVABLE)
page = __rmqueue_cma_fallback(zone, order);
if (!page)
page = __rmqueue_fallback(zone, order, migratetype);
-
- /*
- * Use MIGRATE_RESERVE rather than fail an allocation. goto
- * is used because __rmqueue_smallest is an inline function
- * and we want just one call site
- */
- if (!page) {
- migratetype = MIGRATE_RESERVE;
- goto retry_reserve;
- }
}
trace_mm_page_alloc_zone_locked(page, order, migratetype);
@@ -3462,7 +3448,6 @@ static void show_migration_types(unsigned char type)
[MIGRATE_UNMOVABLE] = 'U',
[MIGRATE_RECLAIMABLE] = 'E',
[MIGRATE_MOVABLE] = 'M',
- [MIGRATE_RESERVE] = 'R',
#ifdef CONFIG_CMA
[MIGRATE_CMA] = 'C',
#endif
@@ -4273,120 +4258,6 @@ static inline unsigned long wait_table_bits(unsigned long size)
}
/*
- * Check if a pageblock contains reserved pages
- */
-static int pageblock_is_reserved(unsigned long start_pfn, unsigned long end_pfn)
-{
- unsigned long pfn;
-
- for (pfn = start_pfn; pfn < end_pfn; pfn++) {
- if (!pfn_valid_within(pfn) || PageReserved(pfn_to_page(pfn)))
- return 1;
- }
- return 0;
-}
-
-/*
- * Mark a number of pageblocks as MIGRATE_RESERVE. The number
- * of blocks reserved is based on min_wmark_pages(zone). The memory within
- * the reserve will tend to store contiguous free pages. Setting min_free_kbytes
- * higher will lead to a bigger reserve which will get freed as contiguous
- * blocks as reclaim kicks in
- */
-static void setup_zone_migrate_reserve(struct zone *zone)
-{
- unsigned long start_pfn, pfn, end_pfn, block_end_pfn;
- struct page *page;
- unsigned long block_migratetype;
- int reserve;
- int old_reserve;
-
- /*
- * Get the start pfn, end pfn and the number of blocks to reserve
- * We have to be careful to be aligned to pageblock_nr_pages to
- * make sure that we always check pfn_valid for the first page in
- * the block.
- */
- start_pfn = zone->zone_start_pfn;
- end_pfn = zone_end_pfn(zone);
- start_pfn = roundup(start_pfn, pageblock_nr_pages);
- reserve = roundup(min_wmark_pages(zone), pageblock_nr_pages) >>
- pageblock_order;
-
- /*
- * Reserve blocks are generally in place to help high-order atomic
- * allocations that are short-lived. A min_free_kbytes value that
- * would result in more than 2 reserve blocks for atomic allocations
- * is assumed to be in place to help anti-fragmentation for the
- * future allocation of hugepages at runtime.
- */
- reserve = min(2, reserve);
- old_reserve = zone->nr_migrate_reserve_block;
-
- /* When memory hot-add, we almost always need to do nothing */
- if (reserve == old_reserve)
- return;
- zone->nr_migrate_reserve_block = reserve;
-
- for (pfn = start_pfn; pfn < end_pfn; pfn += pageblock_nr_pages) {
- if (!early_page_nid_uninitialised(pfn, zone_to_nid(zone)))
- return;
-
- if (!pfn_valid(pfn))
- continue;
- page = pfn_to_page(pfn);
-
- /* Watch out for overlapping nodes */
- if (page_to_nid(page) != zone_to_nid(zone))
- continue;
-
- block_migratetype = get_pageblock_migratetype(page);
-
- /* Only test what is necessary when the reserves are not met */
- if (reserve > 0) {
- /*
- * Blocks with reserved pages will never free, skip
- * them.
- */
- block_end_pfn = min(pfn + pageblock_nr_pages, end_pfn);
- if (pageblock_is_reserved(pfn, block_end_pfn))
- continue;
-
- /* If this block is reserved, account for it */
- if (block_migratetype == MIGRATE_RESERVE) {
- reserve--;
- continue;
- }
-
- /* Suitable for reserving if this block is movable */
- if (block_migratetype == MIGRATE_MOVABLE) {
- set_pageblock_migratetype(page,
- MIGRATE_RESERVE);
- move_freepages_block(zone, page,
- MIGRATE_RESERVE);
- reserve--;
- continue;
- }
- } else if (!old_reserve) {
- /*
- * At boot time we don't need to scan the whole zone
- * for turning off MIGRATE_RESERVE.
- */
- break;
- }
-
- /*
- * If the reserve is met and this is a previous reserved block,
- * take it back
- */
- if (block_migratetype == MIGRATE_RESERVE) {
- set_pageblock_migratetype(page, MIGRATE_MOVABLE);
- move_freepages_block(zone, page, MIGRATE_MOVABLE);
- }
- }
-}
-
-/*
* Initially all pages are reserved - free ones are freed
* up by free_all_bootmem() once the early boot process is
* done. Non-atomic initialization, single-pass.
@@ -4425,9 +4296,7 @@ void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone,
* movable at startup. This will force kernel allocations
* to reserve their blocks rather than leaking throughout
* the address space during boot when many long-lived
- * kernel allocations are made. Later some blocks near
- * the start are marked MIGRATE_RESERVE by
- * setup_zone_migrate_reserve()
+ * kernel allocations are made.
*
* bitmap is created for zone's valid pfn range. but memmap
* can be created for invalid pages (for alignment)
@@ -5985,7 +5854,6 @@ static void __setup_per_zone_wmarks(void)
high_wmark_pages(zone) - low_wmark_pages(zone) -
atomic_long_read(&zone->vm_stat[NR_ALLOC_BATCH]));
- setup_zone_migrate_reserve(zone);
spin_unlock_irqrestore(&zone->lock, flags);
}
diff --git a/mm/vmstat.c b/mm/vmstat.c
index 4f5cd974e11a..49963aa2dff3 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -901,7 +901,6 @@ static char * const migratetype_names[MIGRATE_TYPES] = {
"Unmovable",
"Reclaimable",
"Movable",
- "Reserve",
#ifdef CONFIG_CMA
"CMA",
#endif
--
2.4.6
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2015-08-24 14:40 +0200 |
| Subject | [PATCH 11/12] mm, page_alloc: Reserve pageblocks for high-order atomic allocations on demand |
| Message-ID | <q0ZcD-7Np-17@gated-at.bofh.it> |
| In reply to | #1212114 |
High-order watermark checking exists for two reasons -- kswapd high-order
awareness and protection for high-order atomic requests. Historically the
kernel depended on MIGRATE_RESERVE to preserve min_free_kbytes as high-order
free pages for as long as possible. This patch introduces MIGRATE_HIGHATOMIC
that reserves pageblocks for high-order atomic allocations on demand and
avoids using those blocks for order-0 allocations. This is more flexible
and reliable than MIGRATE_RESERVE was.
A MIGRATE_HIGHORDER pageblock is created when a high-order allocation
request steals a pageblock but limits the total number to 1% of the zone.
Callers that speculatively abuse atomic allocations for long-lived
high-order allocations to access the reserve will quickly fail. Note that
SLUB is currently not such an abuser as it reclaims at least once. It is
possible that the pageblock stolen has few suitable high-order pages and
will need to steal again in the near future but there would need to be
strong justification to search all pageblocks for an ideal candidate.
The pageblocks are unreserved if an allocation fails after a direct
reclaim attempt.
The watermark checks account for the reserved pageblocks when the allocation
request is not a high-order atomic allocation.
The reserved pageblocks can not be used for order-0 allocations. This may
allow temporary wastage until a failed reclaim reassigns the pageblock. This
is deliberate as the intent of the reservation is to satisfy a limited
number of atomic high-order short-lived requests if the system requires them.
The stutter benchmark was used to evaluate this but while it was running
there was a systemtap script that randomly allocated between 1 high-order
page and 12.5% of memory's worth of order-3 pages using GFP_ATOMIC. This
is much larger than the potential reserve and it does not attempt to be
realistic. It is intended to stress random high-order allocations from
an unknown source, show that there is a reduction in failures without
introducing an anomaly where atomic allocations are more reliable than
regular allocations. The amount of memory reserved varied throughout the
workload as reserves were created and reclaimed under memory pressure. The
allocation failures once the workload warmed up were as follows;
4.2-rc5-vanilla 70%
4.2-rc5-atomic-reserve 56%
The failure rate was also measured while building multiple kernels. The
failure rate was 14% but is 6% with this patch applied.
Overall, this is a small reduction but the reserves are small relative to the
number of allocation requests. In early versions of the patch, the failure
rate reduced by a much larger amount but that required much larger reserves
and perversely made atomic allocations seem more reliable than regular allocations.
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
include/linux/mmzone.h | 6 ++-
mm/page_alloc.c | 117 ++++++++++++++++++++++++++++++++++++++++++++++---
mm/vmstat.c | 1 +
3 files changed, 116 insertions(+), 8 deletions(-)
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index cf643539d640..a9805a85940a 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -39,6 +39,8 @@ enum {
MIGRATE_UNMOVABLE,
MIGRATE_MOVABLE,
MIGRATE_RECLAIMABLE,
+ MIGRATE_PCPTYPES, /* the number of types on the pcp lists */
+ MIGRATE_HIGHATOMIC = MIGRATE_PCPTYPES,
#ifdef CONFIG_CMA
/*
* MIGRATE_CMA migration type is designed to mimic the way
@@ -61,8 +63,6 @@ enum {
MIGRATE_TYPES
};
-#define MIGRATE_PCPTYPES (MIGRATE_RECLAIMABLE+1)
-
#ifdef CONFIG_CMA
# define is_migrate_cma(migratetype) unlikely((migratetype) == MIGRATE_CMA)
#else
@@ -330,6 +330,8 @@ struct zone {
/* zone watermarks, access with *_wmark_pages(zone) macros */
unsigned long watermark[NR_WMARK];
+ unsigned long nr_reserved_highatomic;
+
/*
* We don't know if the memory that we're going to allocate will be freeable
* or/and it will be released eventually, so to avoid totally wasting several
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index d5ce050ebe4f..2415f882b89c 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1589,6 +1589,86 @@ int find_suitable_fallback(struct free_area *area, unsigned int order,
return -1;
}
+/*
+ * Reserve a pageblock for exclusive use of high-order atomic allocations if
+ * there are no empty page blocks that contain a page with a suitable order
+ */
+static void reserve_highatomic_pageblock(struct page *page, struct zone *zone,
+ unsigned int alloc_order)
+{
+ int mt = get_pageblock_migratetype(page);
+ unsigned long max_managed, flags;
+
+ if (mt == MIGRATE_HIGHATOMIC)
+ return;
+
+ /*
+ * Limit the number reserved to 1 pageblock or roughly 1% of a zone.
+ * Check is race-prone but harmless.
+ */
+ max_managed = (zone->managed_pages / 100) + pageblock_nr_pages;
+ if (zone->nr_reserved_highatomic >= max_managed)
+ return;
+
+ /* Yoink! */
+ spin_lock_irqsave(&zone->lock, flags);
+ zone->nr_reserved_highatomic += pageblock_nr_pages;
+ set_pageblock_migratetype(page, MIGRATE_HIGHATOMIC);
+ move_freepages_block(zone, page, MIGRATE_HIGHATOMIC);
+ spin_unlock_irqrestore(&zone->lock, flags);
+}
+
+/*
+ * Used when an allocation is about to fail under memory pressure. This
+ * potentially hurts the reliability of high-order allocations when under
+ * intense memory pressure but failed atomic allocations should be easier
+ * to recover from than an OOM.
+ */
+static void unreserve_highatomic_pageblock(const struct alloc_context *ac)
+{
+ struct zonelist *zonelist = ac->zonelist;
+ unsigned long flags;
+ struct zoneref *z;
+ struct zone *zone;
+ struct page *page;
+ int order;
+
+ for_each_zone_zonelist_nodemask(zone, z, zonelist, ac->high_zoneidx,
+ ac->nodemask) {
+ /* Preserve at least one pageblock */
+ if (zone->nr_reserved_highatomic <= pageblock_nr_pages)
+ continue;
+
+ spin_lock_irqsave(&zone->lock, flags);
+ for (order = 0; order < MAX_ORDER; order++) {
+ struct free_area *area = &(zone->free_area[order]);
+
+ if (list_empty(&area->free_list[MIGRATE_HIGHATOMIC]))
+ continue;
+
+ page = list_entry(area->free_list[MIGRATE_HIGHATOMIC].next,
+ struct page, lru);
+
+ zone->nr_reserved_highatomic -= pageblock_nr_pages;
+
+ /*
+ * Convert to ac->migratetype and avoid the normal
+ * pageblock stealing heuristics. Minimally, the caller
+ * is doing the work and needs the pages. More
+ * importantly, if the block was always converted to
+ * MIGRATE_UNMOVABLE or another type then the number
+ * of pageblocks that cannot be completely freed
+ * may increase.
+ */
+ set_pageblock_migratetype(page, ac->migratetype);
+ move_freepages_block(zone, page, ac->migratetype);
+ spin_unlock_irqrestore(&zone->lock, flags);
+ return;
+ }
+ spin_unlock_irqrestore(&zone->lock, flags);
+ }
+}
+
/* Remove an element from the buddy allocator from the fallback list */
static inline struct page *
__rmqueue_fallback(struct zone *zone, unsigned int order, int start_migratetype)
@@ -1645,10 +1725,16 @@ __rmqueue_fallback(struct zone *zone, unsigned int order, int start_migratetype)
* Call me with the zone->lock already held.
*/
static struct page *__rmqueue(struct zone *zone, unsigned int order,
- int migratetype)
+ int migratetype, gfp_t gfp_flags)
{
struct page *page;
+ if (unlikely(order && (gfp_flags & __GFP_ATOMIC))) {
+ page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);
+ if (page)
+ goto out;
+ }
+
page = __rmqueue_smallest(zone, order, migratetype);
if (unlikely(!page)) {
if (migratetype == MIGRATE_MOVABLE)
@@ -1658,6 +1744,7 @@ static struct page *__rmqueue(struct zone *zone, unsigned int order,
page = __rmqueue_fallback(zone, order, migratetype);
}
+out:
trace_mm_page_alloc_zone_locked(page, order, migratetype);
return page;
}
@@ -1675,7 +1762,7 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order,
spin_lock(&zone->lock);
for (i = 0; i < count; ++i) {
- struct page *page = __rmqueue(zone, order, migratetype);
+ struct page *page = __rmqueue(zone, order, migratetype, 0);
if (unlikely(page == NULL))
break;
@@ -2090,7 +2177,7 @@ struct page *buffered_rmqueue(struct zone *preferred_zone,
WARN_ON_ONCE(order > 1);
}
spin_lock_irqsave(&zone->lock, flags);
- page = __rmqueue(zone, order, migratetype);
+ page = __rmqueue(zone, order, migratetype, gfp_flags);
spin_unlock(&zone->lock);
if (!page)
goto failed;
@@ -2200,15 +2287,23 @@ static bool __zone_watermark_ok(struct zone *z, unsigned int order,
unsigned long mark, int classzone_idx, int alloc_flags,
long free_pages)
{
- /* free_pages may go negative - that's OK */
long min = mark;
int o;
long free_cma = 0;
+ /* free_pages may go negative - that's OK */
free_pages -= (1 << order) - 1;
+
if (alloc_flags & ALLOC_HIGH)
min -= min / 2;
- if (alloc_flags & ALLOC_HARDER)
+
+ /*
+ * If the caller is not atomic then discount the reserves. This will
+ * over-estimate how the atomic reserve but it avoids a search
+ */
+ if (likely(!(alloc_flags & ALLOC_HARDER)))
+ free_pages -= z->nr_reserved_highatomic;
+ else
min -= min / 4;
#ifdef CONFIG_CMA
@@ -2397,6 +2492,14 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
if (page) {
if (prep_new_page(page, order, gfp_mask, alloc_flags))
goto try_this_zone;
+
+ /*
+ * If this is a high-order atomic allocation then check
+ * if the pageblock should be reserved for the future
+ */
+ if (unlikely(order && (alloc_flags & ALLOC_HARDER)))
+ reserve_highatomic_pageblock(page, zone, order);
+
return page;
}
}
@@ -2664,9 +2767,11 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order,
/*
* If an allocation failed after direct reclaim, it could be because
- * pages are pinned on the per-cpu lists. Drain them and try again
+ * pages are pinned on the per-cpu lists or in high alloc reserves.
+ * Shrink them them and try again
*/
if (!page && !drained) {
+ unreserve_highatomic_pageblock(ac);
drain_all_pages(NULL);
drained = true;
goto retry;
diff --git a/mm/vmstat.c b/mm/vmstat.c
index 49963aa2dff3..3427a155f85e 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -901,6 +901,7 @@ static char * const migratetype_names[MIGRATE_TYPES] = {
"Unmovable",
"Reclaimable",
"Movable",
+ "HighAtomic",
#ifdef CONFIG_CMA
"CMA",
#endif
--
2.4.6
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Vlastimil Babka <vbabka@suse.cz> |
|---|---|
| Date | 2015-08-26 14:50 +0200 |
| Subject | Re: [PATCH 11/12] mm, page_alloc: Reserve pageblocks for high-order atomic allocations on demand |
| Message-ID | <q1Ijo-602-21@gated-at.bofh.it> |
| In reply to | #1212125 |
On 08/24/2015 02:29 PM, Mel Gorman wrote:
> High-order watermark checking exists for two reasons -- kswapd high-order
> awareness and protection for high-order atomic requests. Historically the
> kernel depended on MIGRATE_RESERVE to preserve min_free_kbytes as high-order
> free pages for as long as possible. This patch introduces MIGRATE_HIGHATOMIC
> that reserves pageblocks for high-order atomic allocations on demand and
> avoids using those blocks for order-0 allocations. This is more flexible
> and reliable than MIGRATE_RESERVE was.
>
> A MIGRATE_HIGHORDER pageblock is created when a high-order allocation
^ atomic ...
> request steals a pageblock but limits the total number to 1% of the zone.
> Callers that speculatively abuse atomic allocations for long-lived
> high-order allocations to access the reserve will quickly fail. Note that
> SLUB is currently not such an abuser as it reclaims at least once. It is
> possible that the pageblock stolen has few suitable high-order pages and
> will need to steal again in the near future but there would need to be
> strong justification to search all pageblocks for an ideal candidate.
>
> The pageblocks are unreserved if an allocation fails after a direct
> reclaim attempt.
>
> The watermark checks account for the reserved pageblocks when the allocation
> request is not a high-order atomic allocation.
>
> The reserved pageblocks can not be used for order-0 allocations. This may
> allow temporary wastage until a failed reclaim reassigns the pageblock. This
> is deliberate as the intent of the reservation is to satisfy a limited
> number of atomic high-order short-lived requests if the system requires them.
>
> The stutter benchmark was used to evaluate this but while it was running
> there was a systemtap script that randomly allocated between 1 high-order
> page and 12.5% of memory's worth of order-3 pages using GFP_ATOMIC. This
> is much larger than the potential reserve and it does not attempt to be
> realistic. It is intended to stress random high-order allocations from
> an unknown source, show that there is a reduction in failures without
> introducing an anomaly where atomic allocations are more reliable than
> regular allocations. The amount of memory reserved varied throughout the
> workload as reserves were created and reclaimed under memory pressure. The
> allocation failures once the workload warmed up were as follows;
>
> 4.2-rc5-vanilla 70%
> 4.2-rc5-atomic-reserve 56%
>
> The failure rate was also measured while building multiple kernels. The
> failure rate was 14% but is 6% with this patch applied.
>
> Overall, this is a small reduction but the reserves are small relative to the
> number of allocation requests. In early versions of the patch, the failure
> rate reduced by a much larger amount but that required much larger reserves
> and perversely made atomic allocations seem more reliable than regular allocations.
>
> Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Michal Hocko <mhocko@kernel.org> |
|---|---|
| Date | 2015-08-26 17:00 +0200 |
| Subject | Re: [PATCH 11/12] mm, page_alloc: Reserve pageblocks for high-order atomic allocations on demand |
| Message-ID | <q1Klc-qE-25@gated-at.bofh.it> |
| In reply to | #1212125 |
On Mon 24-08-15 13:29:57, Mel Gorman wrote:
> High-order watermark checking exists for two reasons -- kswapd high-order
> awareness and protection for high-order atomic requests. Historically the
> kernel depended on MIGRATE_RESERVE to preserve min_free_kbytes as high-order
> free pages for as long as possible. This patch introduces MIGRATE_HIGHATOMIC
> that reserves pageblocks for high-order atomic allocations on demand and
> avoids using those blocks for order-0 allocations. This is more flexible
> and reliable than MIGRATE_RESERVE was.
>
> A MIGRATE_HIGHORDER pageblock is created when a high-order allocation
> request steals a pageblock but limits the total number to 1% of the zone.
> Callers that speculatively abuse atomic allocations for long-lived
> high-order allocations to access the reserve will quickly fail. Note that
> SLUB is currently not such an abuser as it reclaims at least once. It is
> possible that the pageblock stolen has few suitable high-order pages and
> will need to steal again in the near future but there would need to be
> strong justification to search all pageblocks for an ideal candidate.
>
> The pageblocks are unreserved if an allocation fails after a direct
> reclaim attempt.
>
> The watermark checks account for the reserved pageblocks when the allocation
> request is not a high-order atomic allocation.
>
> The reserved pageblocks can not be used for order-0 allocations. This may
> allow temporary wastage until a failed reclaim reassigns the pageblock. This
> is deliberate as the intent of the reservation is to satisfy a limited
> number of atomic high-order short-lived requests if the system requires them.
>
> The stutter benchmark was used to evaluate this but while it was running
> there was a systemtap script that randomly allocated between 1 high-order
> page and 12.5% of memory's worth of order-3 pages using GFP_ATOMIC. This
> is much larger than the potential reserve and it does not attempt to be
> realistic. It is intended to stress random high-order allocations from
> an unknown source, show that there is a reduction in failures without
> introducing an anomaly where atomic allocations are more reliable than
> regular allocations. The amount of memory reserved varied throughout the
> workload as reserves were created and reclaimed under memory pressure. The
> allocation failures once the workload warmed up were as follows;
>
> 4.2-rc5-vanilla 70%
> 4.2-rc5-atomic-reserve 56%
>
> The failure rate was also measured while building multiple kernels. The
> failure rate was 14% but is 6% with this patch applied.
>
> Overall, this is a small reduction but the reserves are small relative to the
> number of allocation requests. In early versions of the patch, the failure
> rate reduced by a much larger amount but that required much larger reserves
> and perversely made atomic allocations seem more reliable than regular allocations.
Have you considered a counter for vmstat/zoneinfo so that we have an overview
about the memory consumed for this reserve?
> Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
Acked-by: Michal Hocko <mhocko@suse.com>
[...]
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index d5ce050ebe4f..2415f882b89c 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
[...]
> @@ -1645,10 +1725,16 @@ __rmqueue_fallback(struct zone *zone, unsigned int order, int start_migratetype)
> * Call me with the zone->lock already held.
> */
> static struct page *__rmqueue(struct zone *zone, unsigned int order,
> - int migratetype)
> + int migratetype, gfp_t gfp_flags)
> {
> struct page *page;
>
> + if (unlikely(order && (gfp_flags & __GFP_ATOMIC))) {
> + page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);
> + if (page)
> + goto out;
I guess you want to change migratetype to MIGRATE_HIGHATOMIC in the
successful case so the tracepoint reports this properly.
> + }
> +
> page = __rmqueue_smallest(zone, order, migratetype);
> if (unlikely(!page)) {
> if (migratetype == MIGRATE_MOVABLE)
> @@ -1658,6 +1744,7 @@ static struct page *__rmqueue(struct zone *zone, unsigned int order,
> page = __rmqueue_fallback(zone, order, migratetype);
> }
>
> +out:
> trace_mm_page_alloc_zone_locked(page, order, migratetype);
> return page;
--
Michal Hocko
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2015-08-26 17:40 +0200 |
| Subject | Re: [PATCH 11/12] mm, page_alloc: Reserve pageblocks for high-order atomic allocations on demand |
| Message-ID | <q1KXU-1pg-29@gated-at.bofh.it> |
| In reply to | #1213946 |
On Wed, Aug 26, 2015 at 04:53:52PM +0200, Michal Hocko wrote:
> >
> > Overall, this is a small reduction but the reserves are small relative to the
> > number of allocation requests. In early versions of the patch, the failure
> > rate reduced by a much larger amount but that required much larger reserves
> > and perversely made atomic allocations seem more reliable than regular allocations.
>
> Have you considered a counter for vmstat/zoneinfo so that we have an overview
> about the memory consumed for this reserve?
>
It should already be available in /proc/pagetypeinfo
> > Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
>
> Acked-by: Michal Hocko <mhocko@suse.com>
>
> [...]
> > diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> > index d5ce050ebe4f..2415f882b89c 100644
> > --- a/mm/page_alloc.c
> > +++ b/mm/page_alloc.c
> [...]
> > @@ -1645,10 +1725,16 @@ __rmqueue_fallback(struct zone *zone, unsigned int order, int start_migratetype)
> > * Call me with the zone->lock already held.
> > */
> > static struct page *__rmqueue(struct zone *zone, unsigned int order,
> > - int migratetype)
> > + int migratetype, gfp_t gfp_flags)
> > {
> > struct page *page;
> >
> > + if (unlikely(order && (gfp_flags & __GFP_ATOMIC))) {
> > + page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);
> > + if (page)
> > + goto out;
>
> I guess you want to change migratetype to MIGRATE_HIGHATOMIC in the
> successful case so the tracepoint reports this properly.
>
Yes, thanks.
--
Mel Gorman
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2015-08-24 14:40 +0200 |
| Subject | [PATCH 12/12] mm, page_alloc: Only enforce watermarks for order-0 allocations |
| Message-ID | <q0ZcD-7Np-27@gated-at.bofh.it> |
| In reply to | #1212114 |
The primary purpose of watermarks is to ensure that reclaim can always
make forward progress in PF_MEMALLOC context (kswapd and direct reclaim).
These assume that order-0 allocations are all that is necessary for
forward progress.
High-order watermarks serve a different purpose. Kswapd had no high-order
awareness before they were introduced (https://lkml.org/lkml/2004/9/5/9).
This was particularly important when there were high-order atomic requests.
The watermarks both gave kswapd awareness and made a reserve for those
atomic requests.
There are two important side-effects of this. The most important is that
a non-atomic high-order request can fail even though free pages are available
and the order-0 watermarks are ok. The second is that high-order watermark
checks are expensive as the free list counts up to the requested order must
be examined.
With the introduction of MIGRATE_HIGHATOMIC it is no longer necessary to
have high-order watermarks. Kswapd and compaction still need high-order
awareness which is handled by checking that at least one suitable high-order
page is free.
With the patch applied, there was little difference in the allocation
failure rates as the atomic reserves are small relative to the number of
allocation attempts. The expected impact is that there will never be an
allocation failure report that shows suitable pages on the free lists.
The one potential side-effect of this is that in a vanilla kernel, the
watermark checks may have kept a free page for an atomic allocation. Now,
we are 100% relying on the HighAtomic reserves and an early allocation to
have allocated them. If the first high-order atomic allocation is after
the system is already heavily fragmented then it'll fail.
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
mm/page_alloc.c | 38 ++++++++++++++++++++++++--------------
1 file changed, 24 insertions(+), 14 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 2415f882b89c..35dc578730d1 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2280,8 +2280,10 @@ static inline bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order)
#endif /* CONFIG_FAIL_PAGE_ALLOC */
/*
- * Return true if free pages are above 'mark'. This takes into account the order
- * of the allocation.
+ * Return true if free base pages are above 'mark'. For high-order checks it
+ * will return true of the order-0 watermark is reached and there is at least
+ * one free page of a suitable size. Checking now avoids taking the zone lock
+ * to check in the allocation paths if no pages are free.
*/
static bool __zone_watermark_ok(struct zone *z, unsigned int order,
unsigned long mark, int classzone_idx, int alloc_flags,
@@ -2289,7 +2291,7 @@ static bool __zone_watermark_ok(struct zone *z, unsigned int order,
{
long min = mark;
int o;
- long free_cma = 0;
+ const bool atomic = (alloc_flags & ALLOC_HARDER);
/* free_pages may go negative - that's OK */
free_pages -= (1 << order) - 1;
@@ -2301,7 +2303,7 @@ static bool __zone_watermark_ok(struct zone *z, unsigned int order,
* If the caller is not atomic then discount the reserves. This will
* over-estimate how the atomic reserve but it avoids a search
*/
- if (likely(!(alloc_flags & ALLOC_HARDER)))
+ if (likely(!atomic))
free_pages -= z->nr_reserved_highatomic;
else
min -= min / 4;
@@ -2309,22 +2311,30 @@ static bool __zone_watermark_ok(struct zone *z, unsigned int order,
#ifdef CONFIG_CMA
/* If allocation can't use CMA areas don't use free CMA pages */
if (!(alloc_flags & ALLOC_CMA))
- free_cma = zone_page_state(z, NR_FREE_CMA_PAGES);
+ free_pages -= zone_page_state(z, NR_FREE_CMA_PAGES);
#endif
- if (free_pages - free_cma <= min + z->lowmem_reserve[classzone_idx])
+ if (free_pages <= min + z->lowmem_reserve[classzone_idx])
return false;
- for (o = 0; o < order; o++) {
- /* At the next order, this order's pages become unavailable */
- free_pages -= z->free_area[o].nr_free << o;
- /* Require fewer higher order pages to be free */
- min >>= 1;
+ /* order-0 watermarks are ok */
+ if (!order)
+ return true;
+
+ /* Check at least one high-order page is free */
+ for (o = order; o < MAX_ORDER; o++) {
+ struct free_area *area = &z->free_area[o];
+ int mt;
+
+ if (atomic && area->nr_free)
+ return true;
- if (free_pages <= min)
- return false;
+ for (mt = 0; mt < MIGRATE_PCPTYPES; mt++) {
+ if (!list_empty(&area->free_list[mt]))
+ return true;
+ }
}
- return true;
+ return false;
}
bool zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
--
2.4.6
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Vlastimil Babka <vbabka@suse.cz> |
|---|---|
| Date | 2015-08-26 15:50 +0200 |
| Subject | Re: [PATCH 12/12] mm, page_alloc: Only enforce watermarks for order-0 allocations |
| Message-ID | <q1Jft-7l6-29@gated-at.bofh.it> |
| In reply to | #1212128 |
On 08/24/2015 02:30 PM, Mel Gorman wrote:
> The primary purpose of watermarks is to ensure that reclaim can always
> make forward progress in PF_MEMALLOC context (kswapd and direct reclaim).
> These assume that order-0 allocations are all that is necessary for
> forward progress.
>
> High-order watermarks serve a different purpose. Kswapd had no high-order
> awareness before they were introduced (https://lkml.org/lkml/2004/9/5/9).
> This was particularly important when there were high-order atomic requests.
> The watermarks both gave kswapd awareness and made a reserve for those
> atomic requests.
>
> There are two important side-effects of this. The most important is that
> a non-atomic high-order request can fail even though free pages are available
> and the order-0 watermarks are ok. The second is that high-order watermark
> checks are expensive as the free list counts up to the requested order must
> be examined.
>
> With the introduction of MIGRATE_HIGHATOMIC it is no longer necessary to
> have high-order watermarks. Kswapd and compaction still need high-order
> awareness which is handled by checking that at least one suitable high-order
> page is free.
>
> With the patch applied, there was little difference in the allocation
> failure rates as the atomic reserves are small relative to the number of
> allocation attempts. The expected impact is that there will never be an
> allocation failure report that shows suitable pages on the free lists.
>
> The one potential side-effect of this is that in a vanilla kernel, the
> watermark checks may have kept a free page for an atomic allocation. Now,
> we are 100% relying on the HighAtomic reserves and an early allocation to
> have allocated them. If the first high-order atomic allocation is after
> the system is already heavily fragmented then it'll fail.
>
> Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
> ---
> mm/page_alloc.c | 38 ++++++++++++++++++++++++--------------
> 1 file changed, 24 insertions(+), 14 deletions(-)
>
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 2415f882b89c..35dc578730d1 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -2280,8 +2280,10 @@ static inline bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order)
> #endif /* CONFIG_FAIL_PAGE_ALLOC */
>
> /*
> - * Return true if free pages are above 'mark'. This takes into account the order
> - * of the allocation.
> + * Return true if free base pages are above 'mark'. For high-order checks it
> + * will return true of the order-0 watermark is reached and there is at least
> + * one free page of a suitable size. Checking now avoids taking the zone lock
> + * to check in the allocation paths if no pages are free.
> */
> static bool __zone_watermark_ok(struct zone *z, unsigned int order,
> unsigned long mark, int classzone_idx, int alloc_flags,
> @@ -2289,7 +2291,7 @@ static bool __zone_watermark_ok(struct zone *z, unsigned int order,
> {
> long min = mark;
> int o;
> - long free_cma = 0;
> + const bool atomic = (alloc_flags & ALLOC_HARDER);
>
> /* free_pages may go negative - that's OK */
> free_pages -= (1 << order) - 1;
> @@ -2301,7 +2303,7 @@ static bool __zone_watermark_ok(struct zone *z, unsigned int order,
> * If the caller is not atomic then discount the reserves. This will
> * over-estimate how the atomic reserve but it avoids a search
> */
> - if (likely(!(alloc_flags & ALLOC_HARDER)))
> + if (likely(!atomic))
> free_pages -= z->nr_reserved_highatomic;
> else
> min -= min / 4;
> @@ -2309,22 +2311,30 @@ static bool __zone_watermark_ok(struct zone *z, unsigned int order,
> #ifdef CONFIG_CMA
> /* If allocation can't use CMA areas don't use free CMA pages */
> if (!(alloc_flags & ALLOC_CMA))
> - free_cma = zone_page_state(z, NR_FREE_CMA_PAGES);
> + free_pages -= zone_page_state(z, NR_FREE_CMA_PAGES);
> #endif
>
> - if (free_pages - free_cma <= min + z->lowmem_reserve[classzone_idx])
> + if (free_pages <= min + z->lowmem_reserve[classzone_idx])
> return false;
> - for (o = 0; o < order; o++) {
> - /* At the next order, this order's pages become unavailable */
> - free_pages -= z->free_area[o].nr_free << o;
>
> - /* Require fewer higher order pages to be free */
> - min >>= 1;
> + /* order-0 watermarks are ok */
> + if (!order)
> + return true;
> +
> + /* Check at least one high-order page is free */
> + for (o = order; o < MAX_ORDER; o++) {
> + struct free_area *area = &z->free_area[o];
> + int mt;
> +
> + if (atomic && area->nr_free)
> + return true;
>
> - if (free_pages <= min)
> - return false;
> + for (mt = 0; mt < MIGRATE_PCPTYPES; mt++) {
> + if (!list_empty(&area->free_list[mt]))
> + return true;
> + }
I think we really need something like this here:
#ifdef CONFIG_CMA
if (alloc_flags & ALLOC_CMA)) &&
!list_empty(&area->free_list[MIGRATE_CMA])
return true;
#endif
This is not about CMA and high-order atomic allocations being used at
the same time. This is about high-order MIGRATE_MOVABLE allocations
(that set ALLOC_CMA) failing to use MIGRATE_CMA pageblocks, which they
should be allowed to use. It's complementary to the existing free_pages
adjustment above.
Maybe there's not many high-order MIGRATE_MOVABLE allocations today, but
they might increase with the driver migration framework. So why set up
us a bomb.
> }
> - return true;
> + return false;
> }
>
> bool zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2015-08-26 17:00 +0200 |
| Subject | Re: [PATCH 12/12] mm, page_alloc: Only enforce watermarks for order-0 allocations |
| Message-ID | <q1Klc-qE-21@gated-at.bofh.it> |
| In reply to | #1213890 |
On Wed, Aug 26, 2015 at 03:42:23PM +0200, Vlastimil Babka wrote:
> >@@ -2309,22 +2311,30 @@ static bool __zone_watermark_ok(struct zone *z, unsigned int order,
> > #ifdef CONFIG_CMA
> > /* If allocation can't use CMA areas don't use free CMA pages */
> > if (!(alloc_flags & ALLOC_CMA))
> >- free_cma = zone_page_state(z, NR_FREE_CMA_PAGES);
> >+ free_pages -= zone_page_state(z, NR_FREE_CMA_PAGES);
> > #endif
> >
> >- if (free_pages - free_cma <= min + z->lowmem_reserve[classzone_idx])
> >+ if (free_pages <= min + z->lowmem_reserve[classzone_idx])
> > return false;
> >- for (o = 0; o < order; o++) {
> >- /* At the next order, this order's pages become unavailable */
> >- free_pages -= z->free_area[o].nr_free << o;
> >
> >- /* Require fewer higher order pages to be free */
> >- min >>= 1;
> >+ /* order-0 watermarks are ok */
> >+ if (!order)
> >+ return true;
> >+
> >+ /* Check at least one high-order page is free */
> >+ for (o = order; o < MAX_ORDER; o++) {
> >+ struct free_area *area = &z->free_area[o];
> >+ int mt;
> >+
> >+ if (atomic && area->nr_free)
> >+ return true;
> >
> >- if (free_pages <= min)
> >- return false;
> >+ for (mt = 0; mt < MIGRATE_PCPTYPES; mt++) {
> >+ if (!list_empty(&area->free_list[mt]))
> >+ return true;
> >+ }
>
> I think we really need something like this here:
>
> #ifdef CONFIG_CMA
> if (alloc_flags & ALLOC_CMA)) &&
> !list_empty(&area->free_list[MIGRATE_CMA])
> return true;
> #endif
>
> This is not about CMA and high-order atomic allocations being used at the
> same time. This is about high-order MIGRATE_MOVABLE allocations (that set
> ALLOC_CMA) failing to use MIGRATE_CMA pageblocks, which they should be
> allowed to use. It's complementary to the existing free_pages adjustment
> above.
>
> Maybe there's not many high-order MIGRATE_MOVABLE allocations today, but
> they might increase with the driver migration framework. So why set up us a
> bomb.
>
Ok, that seems sensible. Will apply this hunk on top
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 1a4169be1498..10f25bf18665 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2337,6 +2337,13 @@ static bool __zone_watermark_ok(struct zone *z, unsigned int order,
if (!list_empty(&area->free_list[mt]))
return true;
}
+
+#ifdef CONFIG_CMA
+ if ((alloc_flags & ALLOC_CMA) &&
+ !list_empty(&area->free_list[MIGRATE_CMA])) {
+ return true;
+ }
+#endif
}
return false;
}
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Michal Hocko <mhocko@kernel.org> |
|---|---|
| Date | 2015-08-28 14:20 +0200 |
| Subject | Re: [PATCH 12/12] mm, page_alloc: Only enforce watermarks for order-0 allocations |
| Message-ID | <q2qNt-2Z9-13@gated-at.bofh.it> |
| In reply to | #1212128 |
On Mon 24-08-15 13:30:15, Mel Gorman wrote:
> The primary purpose of watermarks is to ensure that reclaim can always
> make forward progress in PF_MEMALLOC context (kswapd and direct reclaim).
> These assume that order-0 allocations are all that is necessary for
> forward progress.
>
> High-order watermarks serve a different purpose. Kswapd had no high-order
> awareness before they were introduced (https://lkml.org/lkml/2004/9/5/9).
lkml.org sucks. Could you plase replace it by something else e.g.
https://lkml.kernel.org/r/413AA7B2.4000907@yahoo.com.au?
> This was particularly important when there were high-order atomic requests.
> The watermarks both gave kswapd awareness and made a reserve for those
> atomic requests.
>
> There are two important side-effects of this. The most important is that
> a non-atomic high-order request can fail even though free pages are available
> and the order-0 watermarks are ok. The second is that high-order watermark
> checks are expensive as the free list counts up to the requested order must
> be examined.
>
> With the introduction of MIGRATE_HIGHATOMIC it is no longer necessary to
> have high-order watermarks. Kswapd and compaction still need high-order
> awareness which is handled by checking that at least one suitable high-order
> page is free.
>
> With the patch applied, there was little difference in the allocation
> failure rates as the atomic reserves are small relative to the number of
> allocation attempts. The expected impact is that there will never be an
> allocation failure report that shows suitable pages on the free lists.
>
> The one potential side-effect of this is that in a vanilla kernel, the
> watermark checks may have kept a free page for an atomic allocation. Now,
> we are 100% relying on the HighAtomic reserves and an early allocation to
> have allocated them. If the first high-order atomic allocation is after
> the system is already heavily fragmented then it'll fail.
>
> Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
Acked-by: Michal Hocko <mhocko@suse.com>
[...]
> @@ -2289,7 +2291,7 @@ static bool __zone_watermark_ok(struct zone *z, unsigned int order,
> {
> long min = mark;
> int o;
> - long free_cma = 0;
> + const bool atomic = (alloc_flags & ALLOC_HARDER);
I just find the naming a bit confusing. ALLOC_HARDER != __GFP_ATOMIC. RT tasks
might get access to this reserve as well.
[...]
> + /* Check at least one high-order page is free */
> + for (o = order; o < MAX_ORDER; o++) {
> + struct free_area *area = &z->free_area[o];
> + int mt;
> +
> + if (atomic && area->nr_free)
> + return true;
Didn't you want
if (atomic) {
if (area->nr_free)
return true;
continue;
}
>
> - if (free_pages <= min)
> - return false;
> + for (mt = 0; mt < MIGRATE_PCPTYPES; mt++) {
> + if (!list_empty(&area->free_list[mt]))
> + return true;
> + }
> }
> - return true;
> + return false;
> }
>
> bool zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
> --
> 2.4.6
--
Michal Hocko
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2015-08-28 16:20 +0200 |
| Subject | Re: [PATCH 12/12] mm, page_alloc: Only enforce watermarks for order-0 allocations |
| Message-ID | <q2sFA-5FU-7@gated-at.bofh.it> |
| In reply to | #1215302 |
On Fri, Aug 28, 2015 at 02:10:51PM +0200, Michal Hocko wrote:
> On Mon 24-08-15 13:30:15, Mel Gorman wrote:
> > The primary purpose of watermarks is to ensure that reclaim can always
> > make forward progress in PF_MEMALLOC context (kswapd and direct reclaim).
> > These assume that order-0 allocations are all that is necessary for
> > forward progress.
> >
> > High-order watermarks serve a different purpose. Kswapd had no high-order
> > awareness before they were introduced (https://lkml.org/lkml/2004/9/5/9).
>
> lkml.org sucks. Could you plase replace it by something else e.g.
> https://lkml.kernel.org/r/413AA7B2.4000907@yahoo.com.au?
>
Done.
> > This was particularly important when there were high-order atomic requests.
> > The watermarks both gave kswapd awareness and made a reserve for those
> > atomic requests.
> >
> > There are two important side-effects of this. The most important is that
> > a non-atomic high-order request can fail even though free pages are available
> > and the order-0 watermarks are ok. The second is that high-order watermark
> > checks are expensive as the free list counts up to the requested order must
> > be examined.
> >
> > With the introduction of MIGRATE_HIGHATOMIC it is no longer necessary to
> > have high-order watermarks. Kswapd and compaction still need high-order
> > awareness which is handled by checking that at least one suitable high-order
> > page is free.
> >
> > With the patch applied, there was little difference in the allocation
> > failure rates as the atomic reserves are small relative to the number of
> > allocation attempts. The expected impact is that there will never be an
> > allocation failure report that shows suitable pages on the free lists.
> >
> > The one potential side-effect of this is that in a vanilla kernel, the
> > watermark checks may have kept a free page for an atomic allocation. Now,
> > we are 100% relying on the HighAtomic reserves and an early allocation to
> > have allocated them. If the first high-order atomic allocation is after
> > the system is already heavily fragmented then it'll fail.
> >
> > Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
>
> Acked-by: Michal Hocko <mhocko@suse.com>
>
Thanks.
> [...]
> > @@ -2289,7 +2291,7 @@ static bool __zone_watermark_ok(struct zone *z, unsigned int order,
> > {
> > long min = mark;
> > int o;
> > - long free_cma = 0;
> > + const bool atomic = (alloc_flags & ALLOC_HARDER);
>
> I just find the naming a bit confusing. ALLOC_HARDER != __GFP_ATOMIC. RT tasks
> might get access to this reserve as well.
>
I'll just call it alloc_harder then.
> [...]
> > + /* Check at least one high-order page is free */
> > + for (o = order; o < MAX_ORDER; o++) {
> > + struct free_area *area = &z->free_area[o];
> > + int mt;
> > +
> > + if (atomic && area->nr_free)
> > + return true;
>
> Didn't you want
> if (atomic) {
> if (area->nr_free)
> return true;
> continue;
> }
>
That is slightly more efficient so yes, I'll use it. Thanks.
--
Mel Gorman
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2015-08-24 20:30 +0200 |
| Subject | Re: [PATCH 07/12] mm, page_alloc: Distinguish between being unable to sleep, unwilling to sleep and avoiding waking kswapd |
| Message-ID | <q14Fj-7f0-13@gated-at.bofh.it> |
| In reply to | #1212114 |
On Mon, Aug 24, 2015 at 01:09:46PM +0100, Mel Gorman wrote:
> diff --git a/lib/radix-tree.c b/lib/radix-tree.c
> index f9ebe1c82060..c3775ee46cd6 100644
> --- a/lib/radix-tree.c
> +++ b/lib/radix-tree.c
> @@ -188,7 +188,7 @@ radix_tree_node_alloc(struct radix_tree_root *root)
> * preloading in the interrupt anyway as all the allocations have to
> * be atomic. So just do normal allocation when in interrupt.
> */
> - if (!(gfp_mask & __GFP_WAIT) && !in_interrupt()) {
> + if (!gfpflags_allow_blocking(gfp_mask) && !in_interrupt()) {
> struct radix_tree_preload *rtp;
>
> /*
> @@ -249,7 +249,7 @@ radix_tree_node_free(struct radix_tree_node *node)
> * with preemption not disabled.
> *
> * To make use of this facility, the radix tree must be initialised without
> - * __GFP_WAIT being passed to INIT_RADIX_TREE().
> + * __GFP_DIRECT_RECLAIM being passed to INIT_RADIX_TREE().
> */
> static int __radix_tree_preload(gfp_t gfp_mask)
> {
> @@ -286,12 +286,12 @@ static int __radix_tree_preload(gfp_t gfp_mask)
> * with preemption not disabled.
> *
> * To make use of this facility, the radix tree must be initialised without
> - * __GFP_WAIT being passed to INIT_RADIX_TREE().
> + * __GFP_DIRECT_RECLAIM being passed to INIT_RADIX_TREE().
> */
> int radix_tree_preload(gfp_t gfp_mask)
> {
> /* Warn on non-sensical use... */
> - WARN_ON_ONCE(!(gfp_mask & __GFP_WAIT));
> + WARN_ON_ONCE(gfpflags_allow_blocking(gfp_mask));
> return __radix_tree_preload(gfp_mask);
> }
> EXPORT_SYMBOL(radix_tree_preload);
This was a last minute conversion related to fixing up direct usages of
__GFP_DIRECT_RECLAIM that is obviously wrong. It needs a
diff --git a/lib/radix-tree.c b/lib/radix-tree.c
index c3775ee46cd6..fcf5d98574ce 100644
--- a/lib/radix-tree.c
+++ b/lib/radix-tree.c
@@ -291,7 +291,7 @@ static int __radix_tree_preload(gfp_t gfp_mask)
int radix_tree_preload(gfp_t gfp_mask)
{
/* Warn on non-sensical use... */
- WARN_ON_ONCE(gfpflags_allow_blocking(gfp_mask));
+ WARN_ON_ONCE(!gfpflags_allow_blocking(gfp_mask));
return __radix_tree_preload(gfp_mask);
}
EXPORT_SYMBOL(radix_tree_preload);
--
Mel Gorman
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Vlastimil Babka <vbabka@suse.cz> |
|---|---|
| Date | 2015-08-25 17:40 +0200 |
| Subject | Re: [PATCH 07/12] mm, page_alloc: Distinguish between being unable to sleep, unwilling to sleep and avoiding waking kswapd |
| Message-ID | <q1oum-2cl-41@gated-at.bofh.it> |
| In reply to | #1212114 |
On 08/24/2015 02:09 PM, Mel Gorman wrote:
> __GFP_WAIT has been used to identify atomic context in callers that hold
> spinlocks or are in interrupts. They are expected to be high priority and
> have access one of two watermarks lower than "min" which can be referred
> to as the "atomic reserve". __GFP_HIGH users get access to the first lower
> watermark and can be called the "high priority reserve".
>
> Over time, callers had a requirement to not block when fallback options
> were available. Some have abused __GFP_WAIT leading to a situation where
> an optimisitic allocation with a fallback option can access atomic reserves.
>
> This patch uses __GFP_ATOMIC to identify callers that are truely atomic,
> cannot sleep and have no alternative. High priority users continue to use
> __GFP_HIGH. __GFP_DIRECT_RECLAIM identifies callers that can sleep and are
> willing to enter direct reclaim. __GFP_KSWAPD_RECLAIM to identify callers
> that want to wake kswapd for background reclaim. __GFP_WAIT is redefined
> as a caller that is willing to enter direct reclaim and wake kswapd for
> background reclaim.
>
> This patch then converts a number of sites
>
> o __GFP_ATOMIC is used by callers that are high priority and have memory
> pools for those requests. GFP_ATOMIC uses this flag.
>
> o Callers that have a limited mempool to guarantee forward progress use
> __GFP_DIRECT_RECLAIM. bio allocations fall into this category where
^ __GFP_KSWAPD_RECLAIM ? (missed it previously)
> kswapd will still be woken but atomic reserves are not used as there
> is a one-entry mempool to guarantee progress.
>
> o Callers that are checking if they are non-blocking should use the
> helper gfpflags_allow_blocking() where possible. This is because
> checking for __GFP_WAIT as was done historically now can trigger false
> positives. Some exceptions like dm-crypt.c exist where the code intent
> is clearer if __GFP_DIRECT_RECLAIM is used instead of the helper due to
> flag manipulations.
>
> o Callers that built their own GFP flags instead of starting with GFP_KERNEL
> and friends now also need to specify __GFP_KSWAPD_RECLAIM.
>
> The first key hazard to watch out for is callers that removed __GFP_WAIT
> and was depending on access to atomic reserves for inconspicuous reasons.
> In some cases it may be appropriate for them to use __GFP_HIGH.
>
> The second key hazard is callers that assembled their own combination of
> GFP flags instead of starting with something like GFP_KERNEL. They may
> now wish to specify __GFP_KSWAPD_RECLAIM. It's almost certainly harmless
> if it's missed in most cases as other activity will wake kswapd.
>
> Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
Thanks for the effort!
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Just last few bits:
> @@ -2158,7 +2158,7 @@ static bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order)
> return false;
> if (fail_page_alloc.ignore_gfp_highmem && (gfp_mask & __GFP_HIGHMEM))
> return false;
> - if (fail_page_alloc.ignore_gfp_wait && (gfp_mask & __GFP_WAIT))
> + if (fail_page_alloc.ignore_gfp_wait && (gfp_mask & (__GFP_ATOMIC|__GFP_DIRECT_RECLAIM)))
> return false;
>
> return should_fail(&fail_page_alloc.attr, 1 << order);
IIUC ignore_gfp_wait tells it to assume that reclaimers will eventually
succeed (for some reason?), so they shouldn't fail. Probably to focus
the testing on atomic allocations. But your change makes atomic
allocation never fail, so that goes against the knob IMHO?
> @@ -2660,7 +2660,7 @@ void warn_alloc_failed(gfp_t gfp_mask, int order, const char *fmt, ...)
> if (test_thread_flag(TIF_MEMDIE) ||
> (current->flags & (PF_MEMALLOC | PF_EXITING)))
> filter &= ~SHOW_MEM_FILTER_NODES;
> - if (in_interrupt() || !(gfp_mask & __GFP_WAIT))
> + if (in_interrupt() || !(gfp_mask & __GFP_WAIT) || (gfp_mask & __GFP_ATOMIC))
> filter &= ~SHOW_MEM_FILTER_NODES;
>
> if (fmt) {
This caught me previously and I convinced myself that it's OK, but now
I'm not anymore. IIUC this is to not filter nodes by mems_allowed during
printing, if the allocation itself wasn't limited? In that case it
should probably only look at __GFP_ATOMIC after this patch? As that's
the only thing that determines ALLOC_CPUSET.
I don't know where in_interrupt() comes from, but it was probably
considered in the past, as can be seen in zlc_setup()?
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2015-08-26 16:50 +0200 |
| Subject | Re: [PATCH 07/12] mm, page_alloc: Distinguish between being unable to sleep, unwilling to sleep and avoiding waking kswapd |
| Message-ID | <q1Kbw-fe-17@gated-at.bofh.it> |
| In reply to | #1213119 |
On Tue, Aug 25, 2015 at 05:37:59PM +0200, Vlastimil Babka wrote:
> On 08/24/2015 02:09 PM, Mel Gorman wrote:
> >__GFP_WAIT has been used to identify atomic context in callers that hold
> >spinlocks or are in interrupts. They are expected to be high priority and
> >have access one of two watermarks lower than "min" which can be referred
> >to as the "atomic reserve". __GFP_HIGH users get access to the first lower
> >watermark and can be called the "high priority reserve".
> >
> >Over time, callers had a requirement to not block when fallback options
> >were available. Some have abused __GFP_WAIT leading to a situation where
> >an optimisitic allocation with a fallback option can access atomic reserves.
> >
> >This patch uses __GFP_ATOMIC to identify callers that are truely atomic,
> >cannot sleep and have no alternative. High priority users continue to use
> >__GFP_HIGH. __GFP_DIRECT_RECLAIM identifies callers that can sleep and are
> >willing to enter direct reclaim. __GFP_KSWAPD_RECLAIM to identify callers
> >that want to wake kswapd for background reclaim. __GFP_WAIT is redefined
> >as a caller that is willing to enter direct reclaim and wake kswapd for
> >background reclaim.
> >
> >This patch then converts a number of sites
> >
> >o __GFP_ATOMIC is used by callers that are high priority and have memory
> > pools for those requests. GFP_ATOMIC uses this flag.
> >
> >o Callers that have a limited mempool to guarantee forward progress use
> > __GFP_DIRECT_RECLAIM. bio allocations fall into this category where
>
> ^ __GFP_KSWAPD_RECLAIM ? (missed it previously)
>
I updated the changelog to make this clearer.
> > kswapd will still be woken but atomic reserves are not used as there
> > is a one-entry mempool to guarantee progress.
> >
> >o Callers that are checking if they are non-blocking should use the
> > helper gfpflags_allow_blocking() where possible. This is because
> > checking for __GFP_WAIT as was done historically now can trigger false
> > positives. Some exceptions like dm-crypt.c exist where the code intent
> > is clearer if __GFP_DIRECT_RECLAIM is used instead of the helper due to
> > flag manipulations.
> >
> >o Callers that built their own GFP flags instead of starting with GFP_KERNEL
> > and friends now also need to specify __GFP_KSWAPD_RECLAIM.
> >
> >The first key hazard to watch out for is callers that removed __GFP_WAIT
> >and was depending on access to atomic reserves for inconspicuous reasons.
> >In some cases it may be appropriate for them to use __GFP_HIGH.
> >
> >The second key hazard is callers that assembled their own combination of
> >GFP flags instead of starting with something like GFP_KERNEL. They may
> >now wish to specify __GFP_KSWAPD_RECLAIM. It's almost certainly harmless
> >if it's missed in most cases as other activity will wake kswapd.
> >
> >Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
>
> Thanks for the effort!
>
> Acked-by: Vlastimil Babka <vbabka@suse.cz>
>
> Just last few bits:
>
> >@@ -2158,7 +2158,7 @@ static bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order)
> > return false;
> > if (fail_page_alloc.ignore_gfp_highmem && (gfp_mask & __GFP_HIGHMEM))
> > return false;
> >- if (fail_page_alloc.ignore_gfp_wait && (gfp_mask & __GFP_WAIT))
> >+ if (fail_page_alloc.ignore_gfp_wait && (gfp_mask & (__GFP_ATOMIC|__GFP_DIRECT_RECLAIM)))
> > return false;
> >
> > return should_fail(&fail_page_alloc.attr, 1 << order);
>
> IIUC ignore_gfp_wait tells it to assume that reclaimers will eventually
> succeed (for some reason?), so they shouldn't fail. Probably to focus the
> testing on atomic allocations. But your change makes atomic allocation never
> fail, so that goes against the knob IMHO?
>
Fair point, I'll remove the __GFP_ATOMIC check. I felt this was a sensible
but then again deliberately failing allocations makes my brain twitch a
bit. In retrospect, someone who cared should add a ignore_gfp_atomic knob.
> >@@ -2660,7 +2660,7 @@ void warn_alloc_failed(gfp_t gfp_mask, int order, const char *fmt, ...)
> > if (test_thread_flag(TIF_MEMDIE) ||
> > (current->flags & (PF_MEMALLOC | PF_EXITING)))
> > filter &= ~SHOW_MEM_FILTER_NODES;
> >- if (in_interrupt() || !(gfp_mask & __GFP_WAIT))
> >+ if (in_interrupt() || !(gfp_mask & __GFP_WAIT) || (gfp_mask & __GFP_ATOMIC))
> > filter &= ~SHOW_MEM_FILTER_NODES;
> >
> > if (fmt) {
>
> This caught me previously and I convinced myself that it's OK, but now I'm
> not anymore. IIUC this is to not filter nodes by mems_allowed during
> printing, if the allocation itself wasn't limited? In that case it should
> probably only look at __GFP_ATOMIC after this patch? As that's the only
> thing that determines ALLOC_CPUSET.
> I don't know where in_interrupt() comes from, but it was probably considered
> in the past, as can be seen in zlc_setup()?
>
I assumed the in_interrupt() thing was simply because cpusets were the
primary means of limiting allocations of interest to the author at the
time.
I guess now that I think about it more that a more sensible check would
be against __GFP_DIRECT_RECLAIM because that covers the interesting
cases.
--
Mel Gorman
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Vlastimil Babka <vbabka@suse.cz> |
|---|---|
| Date | 2015-08-26 18:30 +0200 |
| Subject | Re: [PATCH 07/12] mm, page_alloc: Distinguish between being unable to sleep, unwilling to sleep and avoiding waking kswapd |
| Message-ID | <q1LKh-2zS-7@gated-at.bofh.it> |
| In reply to | #1213935 |
On 08/26/2015 04:45 PM, Mel Gorman wrote:
> On Tue, Aug 25, 2015 at 05:37:59PM +0200, Vlastimil Babka wrote:
>>> @@ -2158,7 +2158,7 @@ static bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order)
>>> return false;
>>> if (fail_page_alloc.ignore_gfp_highmem && (gfp_mask & __GFP_HIGHMEM))
>>> return false;
>>> - if (fail_page_alloc.ignore_gfp_wait && (gfp_mask & __GFP_WAIT))
>>> + if (fail_page_alloc.ignore_gfp_wait && (gfp_mask & (__GFP_ATOMIC|__GFP_DIRECT_RECLAIM)))
>>> return false;
>>>
>>> return should_fail(&fail_page_alloc.attr, 1 << order);
>>
>> IIUC ignore_gfp_wait tells it to assume that reclaimers will eventually
>> succeed (for some reason?), so they shouldn't fail. Probably to focus the
>> testing on atomic allocations. But your change makes atomic allocation never
>> fail, so that goes against the knob IMHO?
>>
>
> Fair point, I'll remove the __GFP_ATOMIC check. I felt this was a sensible
> but then again deliberately failing allocations makes my brain twitch a
> bit. In retrospect, someone who cared should add a ignore_gfp_atomic knob.
Thanks.
>>> @@ -2660,7 +2660,7 @@ void warn_alloc_failed(gfp_t gfp_mask, int order, const char *fmt, ...)
>>> if (test_thread_flag(TIF_MEMDIE) ||
>>> (current->flags & (PF_MEMALLOC | PF_EXITING)))
>>> filter &= ~SHOW_MEM_FILTER_NODES;
>>> - if (in_interrupt() || !(gfp_mask & __GFP_WAIT))
>>> + if (in_interrupt() || !(gfp_mask & __GFP_WAIT) || (gfp_mask & __GFP_ATOMIC))
>>> filter &= ~SHOW_MEM_FILTER_NODES;
>>>
>>> if (fmt) {
>>
>> This caught me previously and I convinced myself that it's OK, but now I'm
>> not anymore. IIUC this is to not filter nodes by mems_allowed during
>> printing, if the allocation itself wasn't limited? In that case it should
>> probably only look at __GFP_ATOMIC after this patch? As that's the only
>> thing that determines ALLOC_CPUSET.
>> I don't know where in_interrupt() comes from, but it was probably considered
>> in the past, as can be seen in zlc_setup()?
>>
>
> I assumed the in_interrupt() thing was simply because cpusets were the
> primary means of limiting allocations of interest to the author at the
> time.
IIUC this hunk is unrelated to the previous one - not about limiting
allocations, but printing allocation warnings. Which includes the state
of nodes where the allocation was allowed to try. And
~SHOW_MEM_FILTER_NODES means it was allowed everywhere, so the printing
won't filter by mems_allowed.
> I guess now that I think about it more that a more sensible check would
> be against __GFP_DIRECT_RECLAIM because that covers the interesting
> cases.
I think the most robust check would be to rely on what was already
prepared by gfp_to_alloc_flags(), instead of repeating it here. So add
alloc_flags parameter to warn_alloc_failed(), and drop the filter when
- ALLOC_CPUSET is not set, as that disables the cpuset checks
- ALLOC_NO_WATERMARKS is set, as that allows calling
__alloc_pages_high_priority() attempt which ignores cpusets
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2015-08-26 20:20 +0200 |
| Subject | Re: [PATCH 07/12] mm, page_alloc: Distinguish between being unable to sleep, unwilling to sleep and avoiding waking kswapd |
| Message-ID | <q1NsK-55Y-31@gated-at.bofh.it> |
| In reply to | #1214023 |
On Wed, Aug 26, 2015 at 06:24:34PM +0200, Vlastimil Babka wrote:
> On 08/26/2015 04:45 PM, Mel Gorman wrote:
> >On Tue, Aug 25, 2015 at 05:37:59PM +0200, Vlastimil Babka wrote:
> >>>@@ -2158,7 +2158,7 @@ static bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order)
> >>> return false;
> >>> if (fail_page_alloc.ignore_gfp_highmem && (gfp_mask & __GFP_HIGHMEM))
> >>> return false;
> >>>- if (fail_page_alloc.ignore_gfp_wait && (gfp_mask & __GFP_WAIT))
> >>>+ if (fail_page_alloc.ignore_gfp_wait && (gfp_mask & (__GFP_ATOMIC|__GFP_DIRECT_RECLAIM)))
> >>> return false;
> >>>
> >>> return should_fail(&fail_page_alloc.attr, 1 << order);
> >>
> >>IIUC ignore_gfp_wait tells it to assume that reclaimers will eventually
> >>succeed (for some reason?), so they shouldn't fail. Probably to focus the
> >>testing on atomic allocations. But your change makes atomic allocation never
> >>fail, so that goes against the knob IMHO?
> >>
> >
> >Fair point, I'll remove the __GFP_ATOMIC check. I felt this was a sensible
> >but then again deliberately failing allocations makes my brain twitch a
> >bit. In retrospect, someone who cared should add a ignore_gfp_atomic knob.
>
> Thanks.
>
> >>>@@ -2660,7 +2660,7 @@ void warn_alloc_failed(gfp_t gfp_mask, int order, const char *fmt, ...)
> >>> if (test_thread_flag(TIF_MEMDIE) ||
> >>> (current->flags & (PF_MEMALLOC | PF_EXITING)))
> >>> filter &= ~SHOW_MEM_FILTER_NODES;
> >>>- if (in_interrupt() || !(gfp_mask & __GFP_WAIT))
> >>>+ if (in_interrupt() || !(gfp_mask & __GFP_WAIT) || (gfp_mask & __GFP_ATOMIC))
> >>> filter &= ~SHOW_MEM_FILTER_NODES;
> >>>
> >>> if (fmt) {
> >>
> >>This caught me previously and I convinced myself that it's OK, but now I'm
> >>not anymore. IIUC this is to not filter nodes by mems_allowed during
> >>printing, if the allocation itself wasn't limited? In that case it should
> >>probably only look at __GFP_ATOMIC after this patch? As that's the only
> >>thing that determines ALLOC_CPUSET.
> >>I don't know where in_interrupt() comes from, but it was probably considered
> >>in the past, as can be seen in zlc_setup()?
> >>
> >
> >I assumed the in_interrupt() thing was simply because cpusets were the
> >primary means of limiting allocations of interest to the author at the
> >time.
>
> IIUC this hunk is unrelated to the previous one - not about limiting
> allocations, but printing allocation warnings. Which includes the state of
> nodes where the allocation was allowed to try. And ~SHOW_MEM_FILTER_NODES
> means it was allowed everywhere, so the printing won't filter by
> mems_allowed.
>
> >I guess now that I think about it more that a more sensible check would
> >be against __GFP_DIRECT_RECLAIM because that covers the interesting
> >cases.
>
> I think the most robust check would be to rely on what was already prepared
> by gfp_to_alloc_flags(), instead of repeating it here. So add alloc_flags
> parameter to warn_alloc_failed(), and drop the filter when
> - ALLOC_CPUSET is not set, as that disables the cpuset checks
> - ALLOC_NO_WATERMARKS is set, as that allows calling
> __alloc_pages_high_priority() attempt which ignores cpusets
>
warn_alloc_failed is used outside of page_alloc.c in a context that does
not have alloc_flags. It could be extended to take an extra parameter
that is ALLOC_CPUSET for the other callers or else split it into
__warn_alloc_failed (takes alloc_flags parameter) and warn_alloc_failed
(calls __warn_alloc_failed with ALLOC_CPUSET) but is it really worth it?
--
Mel Gorman
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Vlastimil Babka <vbabka@suse.cz> |
|---|---|
| Date | 2015-08-27 11:20 +0200 |
| Subject | Re: [PATCH 07/12] mm, page_alloc: Distinguish between being unable to sleep, unwilling to sleep and avoiding waking kswapd |
| Message-ID | <q21vJ-8z-45@gated-at.bofh.it> |
| In reply to | #1214097 |
On 08/26/2015 08:10 PM, Mel Gorman wrote: >> >> I think the most robust check would be to rely on what was already prepared >> by gfp_to_alloc_flags(), instead of repeating it here. So add alloc_flags >> parameter to warn_alloc_failed(), and drop the filter when >> - ALLOC_CPUSET is not set, as that disables the cpuset checks >> - ALLOC_NO_WATERMARKS is set, as that allows calling >> __alloc_pages_high_priority() attempt which ignores cpusets >> > > warn_alloc_failed is used outside of page_alloc.c in a context that does > not have alloc_flags. It could be extended to take an extra parameter > that is ALLOC_CPUSET for the other callers or else split it into > __warn_alloc_failed (takes alloc_flags parameter) and warn_alloc_failed > (calls __warn_alloc_failed with ALLOC_CPUSET) but is it really worth it? Probably not. Testing lack of __GFP_DIRECT_RECLAIM is good enough until somebody cares more. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Vlastimil Babka <vbabka@suse.cz> |
|---|---|
| Date | 2015-08-25 17:50 +0200 |
| Subject | Re: [PATCH 07/12] mm, page_alloc: Distinguish between being unable to sleep, unwilling to sleep and avoiding waking kswapd |
| Message-ID | <q1oE3-2nZ-25@gated-at.bofh.it> |
| In reply to | #1212114 |
On 08/24/2015 02:09 PM, Mel Gorman wrote: > The first key hazard to watch out for is callers that removed __GFP_WAIT > and was depending on access to atomic reserves for inconspicuous reasons. > In some cases it may be appropriate for them to use __GFP_HIGH. Hm so I think this hazard should be expanded. If such caller comes from interrupt and doesn't use __GFP_ATOMIC, the ALLOC_CPUSET with restrictions taken from the interrupted process will also apply to him? -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web