Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1375583

[PATCH 20/22] mm, page_alloc: Check multiple page fields with a single branch

Show all headers | View raw

Every page allocated or freed is checked for sanity to avoid corruptions
that are difficult to detect later.  A bad page could be due to a number of
fields. Instead of using multiple branches, this patch combines multiple
fields into a single branch. A detailed check is only necessary if that
check fails.

Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
 mm/page_alloc.c | 55 +++++++++++++++++++++++++++++++++++++++++++------------
 1 file changed, 43 insertions(+), 12 deletions(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 8f53cb3853fc..59d7c627b29d 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -784,10 +784,42 @@ static inline void __free_one_page(struct page *page,
 	zone->free_area[order].nr_free++;
 }
 
+/*
+ * A bad page could be due to a number of fields. Instead of multiple branches,
+ * try and check multiple fields with one check. The caller must do a detailed
+ * check if necessary.
+ */
+static inline bool page_expected_state(struct page *page,
+					unsigned long check_flags)
+{
+	if (unlikely(atomic_read(&page->_mapcount) != -1))
+		return false;
+
+	if (unlikely((unsigned long)page->mapping |
+			page_ref_count(page) |
+#ifdef CONFIG_MEMCG
+			(unsigned long)page->mem_cgroup |
+#endif
+			(page->flags & check_flags)))
+		return false;
+
+	return true;
+}
+
 static inline int free_pages_check(struct page *page)
 {
-	const char *bad_reason = NULL;
-	unsigned long bad_flags = 0;
+	const char *bad_reason;
+	unsigned long bad_flags;
+
+	if (page_expected_state(page, PAGE_FLAGS_CHECK_AT_FREE)) {
+		page_cpupid_reset_last(page);
+		page->flags &= ~PAGE_FLAGS_CHECK_AT_PREP;
+		return 0;
+	}
+
+	/* Something has gone sideways, find it */
+	bad_reason = NULL;
+	bad_flags = 0;
 
 	if (unlikely(atomic_read(&page->_mapcount) != -1))
 		bad_reason = "nonzero mapcount";
@@ -803,14 +835,8 @@ static inline int free_pages_check(struct page *page)
 	if (unlikely(page->mem_cgroup))
 		bad_reason = "page still charged to cgroup";
 #endif
-	if (unlikely(bad_reason)) {
-		bad_page(page, bad_reason, bad_flags);
-		return 1;
-	}
-	page_cpupid_reset_last(page);
-	if (page->flags & PAGE_FLAGS_CHECK_AT_PREP)
-		page->flags &= ~PAGE_FLAGS_CHECK_AT_PREP;
-	return 0;
+	bad_page(page, bad_reason, bad_flags);
+	return 1;
 }
 
 /*
@@ -1491,9 +1517,14 @@ static inline void expand(struct zone *zone, struct page *page,
  */
 static inline int check_new_page(struct page *page)
 {
-	const char *bad_reason = NULL;
-	unsigned long bad_flags = 0;
+	const char *bad_reason;
+	unsigned long bad_flags;
+
+	if (page_expected_state(page, PAGE_FLAGS_CHECK_AT_PREP|__PG_HWPOISON))
+		return 0;
 
+	bad_reason = NULL;
+	bad_flags = 0;
 	if (unlikely(atomic_read(&page->_mapcount) != -1))
 		bad_reason = "nonzero mapcount";
 	if (unlikely(page->mapping != NULL))
-- 
2.6.4

Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

[PATCH 00/21] Optimise page alloc/free fast paths Mel Gorman <mgorman@techsingularity.net> - 2016-04-11 10:20 +0200
  [PATCH 08/22] mm, page_alloc: Convert alloc_flags to unsigned Mel Gorman <mgorman@techsingularity.net> - 2016-04-11 10:20 +0200
  [PATCH 04/22] mm, page_alloc: Inline zone_statistics Mel Gorman <mgorman@techsingularity.net> - 2016-04-11 10:20 +0200
  [PATCH 21/22] cpuset: use static key better and convert to new API Mel Gorman <mgorman@techsingularity.net> - 2016-04-11 10:20 +0200
  [PATCH 11/22] mm, page_alloc: Remove unnecessary initialisation in get_page_from_freelist Mel Gorman <mgorman@techsingularity.net> - 2016-04-11 10:20 +0200
  [PATCH 14/22] mm, page_alloc: Simplify last cpupid reset Mel Gorman <mgorman@techsingularity.net> - 2016-04-11 10:20 +0200
  [PATCH 13/22] mm, page_alloc: Remove redundant check for empty zonelist Mel Gorman <mgorman@techsingularity.net> - 2016-04-11 10:20 +0200
  [PATCH 09/22] mm, page_alloc: Convert nr_fair_skipped to bool Mel Gorman <mgorman@techsingularity.net> - 2016-04-11 10:20 +0200
  [PATCH 15/22] mm, page_alloc: Move might_sleep_if check to the allocator slowpath Mel Gorman <mgorman@techsingularity.net> - 2016-04-11 10:20 +0200
  [PATCH 20/22] mm, page_alloc: Check multiple page fields with a single branch Mel Gorman <mgorman@techsingularity.net> - 2016-04-11 10:20 +0200
  [PATCH 12/22] mm, page_alloc: Remove unnecessary initialisation from __alloc_pages_nodemask() Mel Gorman <mgorman@techsingularity.net> - 2016-04-11 10:20 +0200
  [PATCH 19/22] mm, page_alloc: Avoid looking up the first zone in a zonelist twice Mel Gorman <mgorman@techsingularity.net> - 2016-04-11 10:20 +0200
  [PATCH 05/22] mm, page_alloc: Inline the fast path of the zonelist iterator Mel Gorman <mgorman@techsingularity.net> - 2016-04-11 10:20 +0200
  [PATCH 02/22] mm, page_alloc: Use new PageAnonHead helper in the free page fast path Mel Gorman <mgorman@techsingularity.net> - 2016-04-11 10:20 +0200
  [PATCH 16/22] mm, page_alloc: Move __GFP_HARDWALL modifications out of the fastpath Mel Gorman <mgorman@techsingularity.net> - 2016-04-11 10:20 +0200
  [PATCH 17/22] mm, page_alloc: Reduce cost of fair zone allocation policy retry Mel Gorman <mgorman@techsingularity.net> - 2016-04-11 10:20 +0200
  [PATCH 18/22] mm, page_alloc: Shortcut watermark checks for order-0 pages Mel Gorman <mgorman@techsingularity.net> - 2016-04-11 10:30 +0200
  Re: [PATCH 00/21] Optimise page alloc/free fast paths Mel Gorman <mgorman@techsingularity.net> - 2016-04-11 10:30 +0200
  [PATCH 06/22] mm, page_alloc: Use __dec_zone_state for order-0 page allocation Mel Gorman <mgorman@techsingularity.net> - 2016-04-11 10:30 +0200
  [PATCH 10/22] mm, page_alloc: Remove unnecessary local variable in get_page_from_freelist Mel Gorman <mgorman@techsingularity.net> - 2016-04-11 10:30 +0200
  [PATCH 07/22] mm, page_alloc: Avoid unnecessary zone lookups during pageblock operations Mel Gorman <mgorman@techsingularity.net> - 2016-04-11 10:30 +0200
  [PATCH 01/22] mm, page_alloc: Only check PageCompound for high-order pages Mel Gorman <mgorman@techsingularity.net> - 2016-04-11 10:40 +0200
  [PATCH 03/22] mm, page_alloc: Reduce branches in zone_statistics Mel Gorman <mgorman@techsingularity.net> - 2016-04-11 10:40 +0200

csiph-web