Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1695246 > unrolled thread

[PATCH v2 00/23] percpu: replace percpu area map allocator with bitmap allocator

Back to article view | Back to linux.kernel

Contents

  [PATCH v2 00/23] percpu: replace percpu area map allocator with bitmap allocator Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:10 +0200
    [PATCH v2 22/23] percpu: update pcpu_find_block_fit to use an iterator Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:10 +0200
      Re: [PATCH v2 22/23] percpu: update pcpu_find_block_fit to use an  iterator Josef Bacik <josef@toxicpanda.com> - 2017-07-25 21:50 +0200
    [PATCH v2 09/23] percpu: combine percpu address checks Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:10 +0200
      Re: [PATCH v2 09/23] percpu: combine percpu address checks Josef Bacik <josef@toxicpanda.com> - 2017-07-25 20:30 +0200
    [PATCH v2 02/23] percpu: introduce start_offset to pcpu_chunk Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:10 +0200
      Re: [PATCH v2 02/23] percpu: introduce start_offset to pcpu_chunk Josef Bacik <josef@toxicpanda.com> - 2017-07-25 20:10 +0200
    [PATCH v2 13/23] percpu: generalize bitmap (un)populated iterators Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:10 +0200
      Re: [PATCH v2 13/23] percpu: generalize bitmap (un)populated  iterators Josef Bacik <josef@toxicpanda.com> - 2017-07-25 20:40 +0200
      Re: [PATCH v2 13/23] percpu: generalize bitmap (un)populated  iterators Tejun Heo <tj@kernel.org> - 2017-07-26 16:10 +0200
    [PATCH v2 08/23] percpu: modify base_addr to be region specific Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:10 +0200
      Re: [PATCH v2 08/23] percpu: modify base_addr to be region specific Josef Bacik <josef@toxicpanda.com> - 2017-07-25 20:30 +0200
    [PATCH v2 17/23] percpu: skip chunks if the alloc does not fit in the contig hint Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:10 +0200
      Re: [PATCH v2 17/23] percpu: skip chunks if the alloc does not fit  in the contig hint Josef Bacik <josef@toxicpanda.com> - 2017-07-25 21:50 +0200
    [PATCH v2 19/23] percpu: update alloc path to only scan if contig hints are broken Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:10 +0200
      Re: [PATCH v2 19/23] percpu: update alloc path to only scan if  contig hints are broken Josef Bacik <josef@toxicpanda.com> - 2017-07-25 21:40 +0200
    [PATCH v2 12/23] percpu: increase minimum percpu allocation size and align first regions Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:10 +0200
      Re: [PATCH v2 12/23] percpu: increase minimum percpu allocation size  and align first regions Josef Bacik <josef@toxicpanda.com> - 2017-07-25 20:40 +0200
    [PATCH v2 01/23] percpu: setup_first_chunk enforce dynamic region must exist Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:20 +0200
      Re: [PATCH v2 01/23] percpu: setup_first_chunk enforce dynamic  region must exist Josef Bacik <josef@toxicpanda.com> - 2017-07-25 20:10 +0200
    [PATCH v2 06/23] percpu: end chunk area maps page aligned for the populated bitmap Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:20 +0200
      Re: [PATCH v2 06/23] percpu: end chunk area maps page aligned for  the populated bitmap Josef Bacik <josef@toxicpanda.com> - 2017-07-25 20:20 +0200
    Re: [PATCH v2 14/23] percpu: replace area map allocator with bitmap  allocator Josef Bacik <josef@toxicpanda.com> - 2017-07-25 21:20 +0200

Page 1 of 2  [1] 2  Next page →

#1695246 — [PATCH v2 00/23] percpu: replace percpu area map allocator with bitmap allocator

Hi everyone,

V2:
There was a good bit of refactoring. Several variable names were
modified to more descriptive. First bit is now kept track of instead of
first block.

Chunk slots are back to being managed in bytes.

The reserved region is no longer expanded to be page aligned. The other
regions in the first chunk are now expanded accordingly to align with the
minimum allocation size. This lets the first chunk allocation code be
combined. The chunks now keep track of start and end offsets to support
the reserved region. Block size should now be changeable within some
constraints.

The unit_size is expected to be a multiple of the bitmap block size. The
LCM(page size, block size) > 1 to map the bitmap correctly to the
backing pages.

Iterators were added for walking the metadata blocks to find free space
and for walking the metadata to update the chunk hints.

The metadata now keeps track of the best starting offsets rather than
just the first one.

Empty page count had an error where the count was being incremented when
it was populating pages for an allocation. Chunk scanning on the free
path now occurs if a page becomes free in the case of large metadata
blocks.

The data presented below has been updated and additional data is
provided to demonstrate the variable performance of the allocation path
for the area map allocator.

---------

The Linux kernel percpu memory allocator is responsible for managing
percpu memory. It allocates memory from chunks of percpu areas and uses a
simple first-fit area allocator to manage allocations inside each chunk.
There now exist use cases where allocating and deallocating a million or
more objects occurs making the current implementation inadequate.

The two primary problems with the current area map allocator are:
  1. The backing data structure is an array of the areas. To manage this
     array, it is possible to need to memmove a large portion of it.
  2. On allocation, chunks are considered based on the contig_hint. It is
     possible that the contig_hint may be large enough while the alignment 
     could not meet the request. This causes scanning over every free
     fragment that could spill over into scanning chunks.

The primary considerations for the new allocator were the following:
 - Remove the memmove operation from the critical path
 - Be conservative with additional use of memory
 - Provide consistency in performance and memory footprint
 - Focus on small allocations < 64 bytes

This patchset introduces a simple bitmap allocator backed by metadata
blocks as a replacement for the area map allocator for percpu memory. Each
chunk has an allocation bitmap, a boundary bitmap, and a set of metadata
blocks. The allocation map serves as the ground truth for allocations
while the boundary map serves as a way to distinguish between consecutive
allocations. The minimum allocation size has been increased to 4-bytes.

The key property behind the bitmap allocator is its static metadata. The
main problem it solves is that a memmove is no longer part of the critical
path for freeing, which was the primary source of latency. This also helps
bound the metadata overhead. The area map allocator prior required an
integer per allocation. This may be beneficial with larger allocations,
but as mentioned, allocating a significant number of small objects is
becoming more common. This causes worst-case scenarios for metadata
overhead.

In an effort to make freeing fast, the only time metadata is updated on
the free path is if a whole block becomes free or the freed area spans
across metadata blocks. This causes the chunk’s contig_hint to be
potentially smaller than what it could allocate by up to a block. If the
chunk’s contig hint is smaller than a block, a check occurs and the hint
is kept accurate. Metadata is always kept accurate on allocation and
therefore the situation where a chunk has a larger contig hint than
available will never occur.

I have primarily done testing against a simple workload of allocation of
1 million objects of varying size. Deallocation was done by in order,
alternating, and in reverse. These numbers were collected after rebasing
ontop of a80099a152. I present the worst-case numbers here:

  Area Map Allocator:

        Object Size | Alloc Time (ms) | Free Time (ms)
        ----------------------------------------------
              4B    |        310      |     4770
             16B    |        557      |     1325
             64B    |        436      |      273
            256B    |        776      |      131
           1024B    |       3280      |      122

  Bitmap Allocator:

        Object Size | Alloc Time (ms) | Free Time (ms)
        ----------------------------------------------
              4B    |        490      |       70
             16B    |        515      |       75
             64B    |        610      |       80
            256B    |        950      |      100
           1024B    |       3520      |      200

This data demonstrates the inability for the area map allocator to
handle less than ideal situations. In the best case of reverse
deallocation, the area map allocator was able to perform within range
of the bitmap allocator. In the worst case situation, freeing took
nearly 5 seconds for 1 million 4-byte objects. The bitmap allocator
dramatically improves the consistency of the free path. The small
allocations performed nearly identical regardless of the freeing
pattern.

While it does add to the allocation latency, the allocation scenario
here is optimal for the area map allocator. The area map allocator runs
into trouble when it is allocating in chunks where the latter half is
full. It is difficult to replicate this, so I present a variant where
the pages are second half filled. Freeing was done sequentially. Below
are the numbers for this scenario:

  Area Map Allocator:

        Object Size | Alloc Time (ms) | Free Time (ms)
        ----------------------------------------------
              4B    |       4118      |     4892
             16B    |       1651      |     1163
             64B    |        598      |      285
            256B    |        771      |      158
           1024B    |       3034      |      160

  Bitmap Allocator:

        Object Size | Alloc Time (ms) | Free Time (ms)
        ----------------------------------------------
              4B    |        481      |       67
             16B    |        506      |       69
             64B    |        636      |       75
            256B    |        892      |       90
           1024B    |       3262      |      147

The data shows a parabolic curve of performance for the area map
allocator. This is due to the memmove operation being the dominant cost
with the lower object sizes as more objects are packed in a chunk and at
higher object sizes, the traversal of the chunk slots is the dominating
cost. The bitmap allocator suffers this problem as well. The above data
shows the inability to scale for the allocation path with the area map
allocator and that the bitmap allocator demonstrates consistent
performance in general.

The second problem of additional scanning can result in the area map
allocator completing in 52 minutes when trying to allocate 1 million
4-byte objects with 8-byte alignment. The same workload takes
approximately 16 seconds to complete for the bitmap allocator.

Alternative implementations were evaluated including: linked lists, trees,
and buddy systems. These all suffer from either high metadata overhead for
small allocations or from the amplified costs of fragmentation with percpu
memory.

This patchset contains the following 23 patches:

  0001-percpu-setup_first_chunk-enforce-dynamic-region-must.patch
  0002-percpu-introduce-start_offset-to-pcpu_chunk.patch
  0003-percpu-remove-has_reserved-from-pcpu_chunk.patch
  0004-percpu-setup_first_chunk-remove-dyn_size-and-consoli.patch
  0005-percpu-unify-allocation-of-schunk-and-dchunk.patch
  0006-percpu-end-chunk-area-maps-page-aligned-for-the-popu.patch
  0007-percpu-setup_first_chunk-rename-schunk-dchunk-to-chu.patch
  0008-percpu-modify-base_addr-to-be-region-specific.patch
  0009-percpu-combine-percpu-address-checks.patch
  0010-percpu-change-the-number-of-pages-marked-in-the-firs.patch
  0011-percpu-introduce-nr_empty_pop_pages-to-help-empty-pa.patch
  0012-percpu-increase-minimum-percpu-allocation-size-and-a.patch
  0013-percpu-generalize-bitmap-un-populated-iterators.patch
  0014-percpu-replace-area-map-allocator-with-bitmap-alloca.patch
  0015-percpu-introduce-bitmap-metadata-blocks.patch
  0016-percpu-add-first_bit-to-keep-track-of-the-first-free.patch
  0017-percpu-skip-chunks-if-the-alloc-does-not-fit-in-the-.patch
  0018-percpu-keep-track-of-the-best-offset-for-contig-hint.patch
  0019-percpu-update-alloc-path-to-only-scan-if-contig-hint.patch
  0020-percpu-update-free-path-to-take-advantage-of-contig-.patch
  0021-percpu-use-metadata-blocks-to-update-the-chunk-conti.patch
  0022-percpu-update-pcpu_find_block_fit-to-use-an-iterator.patch
  0023-percpu-update-header-to-contain-bitmap-allocator-exp.patch

0001-0007 clean up and merge the first chunk allocation paths.
0008-0009 move the base address up and merge the address check logic.
0010 fixes the bitmap size of the first chunk. 0011-0013 are preparatory
patches for the bitmap allocator. 0014 swaps out the area map allocator
for the bitmap allocator. 0015 introduces metadata blocks. 0016-0022
replace temporary scanning functions with faster code. 0023 updates
the header to contain a new copyright and details about the bitmap
allocator.

This patchset is on top of linus#master a80099a152.

diffstats below:

Dennis Zhou (Facebook) (23):
  percpu: setup_first_chunk enforce dynamic region must exist
  percpu: introduce start_offset to pcpu_chunk
  percpu: remove has_reserved from pcpu_chunk
  percpu: setup_first_chunk remove dyn_size and consolidate logic
  percpu: unify allocation of schunk and dchunk
  percpu: end chunk area maps page aligned for the populated bitmap
  percpu: setup_first_chunk rename schunk/dchunk to chunk
  percpu: modify base_addr to be region specific
  percpu: combine percpu address checks
  percpu: change the number of pages marked in the first_chunk pop
    bitmap
  percpu: introduce nr_empty_pop_pages to help empty page accounting
  percpu: increase minimum percpu allocation size and align first
    regions
  percpu: generalize bitmap (un)populated iterators
  percpu: replace area map allocator with bitmap allocator
  percpu: introduce bitmap metadata blocks
  percpu: add first_bit to keep track of the first free in the bitmap
  percpu: skip chunks if the alloc does not fit in the contig hint
  percpu: keep track of the best offset for contig hints
  percpu: update alloc path to only scan if contig hints are broken
  percpu: update free path to take advantage of contig hints
  percpu: use metadata blocks to update the chunk contig hint
  percpu: update pcpu_find_block_fit to use an iterator
  percpu: update header to contain bitmap allocator explanation.

 include/linux/percpu.h |   20 +-
 init/main.c            |    1 -
 mm/percpu-internal.h   |   81 ++-
 mm/percpu-km.c         |    2 +-
 mm/percpu-stats.c      |  100 ++--
 mm/percpu.c            | 1466 ++++++++++++++++++++++++++++++------------------
 6 files changed, 1070 insertions(+), 600 deletions(-)

Thanks,
Dennis

[toc] | [next] | [standalone]

#1695247 — [PATCH v2 22/23] percpu: update pcpu_find_block_fit to use an iterator

From: "Dennis Zhou (Facebook)" <dennisszhou@gmail.com>

The simple, and expensive, way to find a free area is to iterate over
the entire bitmap until an area is found that fits the allocation size
and alignment. This patch makes use of an iterate that find an area to
check by using the block level contig hints. It will only return an area
that can fit the size and alignment request. If the request can fit
inside a block, it returns the first_free bit to start checking from to
see if it can be fulfilled prior to the contig hint. The pcpu_alloc_area
check has a bound of a block size added in case it is wrong.

Signed-off-by: Dennis Zhou <dennisszhou@gmail.com>
---
 mm/percpu.c | 112 +++++++++++++++++++++++++++++++++++++++++++++++++-----------
 1 file changed, 92 insertions(+), 20 deletions(-)

diff --git a/mm/percpu.c b/mm/percpu.c
index 9e4192c..ffa9da7 100644
--- a/mm/percpu.c
+++ b/mm/percpu.c
@@ -355,10 +355,72 @@ static void pcpu_next_md_free_region(struct pcpu_chunk *chunk, int *bit_off,
 	}
 }
 
+/**
+ * pcpu_next_fit_region - finds fit areas for a given allocation request
+ * @chunk: chunk of interest
+ * @alloc_bits: size of allocation
+ * @align: alignment of area (max PAGE_SIZE)
+ * @bit_off: chunk offset
+ * @bits: size of free area
+ *
+ * Finds the next free region that is viable for use with a given size and
+ * alignment.  This only returns if there is a valid area to be used for this
+ * allocation.  block->first_free is returned if the allocation request fits
+ * within the block to see if the request can be fulfilled prior to the contig
+ * hint.
+ */
+static void pcpu_next_fit_region(struct pcpu_chunk *chunk, int alloc_bits,
+				 int align, int *bit_off, int *bits)
+{
+	int i = pcpu_off_to_block_index(*bit_off);
+	int block_off = pcpu_off_to_block_off(*bit_off);
+	struct pcpu_block_md *block;
+
+	*bits = 0;
+	for (block = chunk->md_blocks + i; i < pcpu_chunk_nr_blocks(chunk);
+	     block++, i++) {
+		/* handles contig area across blocks */
+		if (*bits) {
+			*bits += block->left_free;
+			if (*bits >= alloc_bits)
+				return;
+			if (block->left_free == PCPU_BITMAP_BLOCK_BITS)
+				continue;
+		}
+
+		/* check block->contig_hint */
+		*bits = ALIGN(block->contig_hint_start, align) -
+			block->contig_hint_start;
+		/*
+		 * This uses the block offset to determine if this has been
+		 * checked in the prior iteration.
+		 */
+		if (block->contig_hint &&
+		    block->contig_hint_start >= block_off &&
+		    block->contig_hint >= *bits + alloc_bits) {
+			*bits += alloc_bits + block->contig_hint_start -
+				 block->first_free;
+			*bit_off = pcpu_block_off_to_off(i, block->first_free);
+			return;
+		}
+
+		*bit_off = ALIGN(PCPU_BITMAP_BLOCK_BITS - block->right_free,
+				 align);
+		*bits = PCPU_BITMAP_BLOCK_BITS - *bit_off;
+		*bit_off = pcpu_block_off_to_off(i, *bit_off);
+		if (*bits >= alloc_bits)
+			return;
+	}
+
+	/* no valid offsets were found - fail condition */
+	*bit_off = pcpu_chunk_map_bits(chunk);
+}
+
 /*
  * Metadata free area iterators.  These perform aggregation of free areas
  * based on the metadata blocks and return the offset @bit_off and size in
- * bits of the free area @bits.
+ * bits of the free area @bits.  pcpu_for_each_fit_region only returns when
+ * a fit is found for the allocation request.
  */
 #define pcpu_for_each_md_free_region(chunk, bit_off, bits)		\
 	for (pcpu_next_md_free_region((chunk), &(bit_off), &(bits));	\
@@ -366,6 +428,14 @@ static void pcpu_next_md_free_region(struct pcpu_chunk *chunk, int *bit_off,
 	     (bit_off) += (bits) + 1,					\
 	     pcpu_next_md_free_region((chunk), &(bit_off), &(bits)))
 
+#define pcpu_for_each_fit_region(chunk, alloc_bits, align, bit_off, bits)     \
+	for (pcpu_next_fit_region((chunk), (alloc_bits), (align), &(bit_off), \
+				  &(bits));				      \
+	     (bit_off) < pcpu_chunk_map_bits((chunk));			      \
+	     (bit_off) += (bits),					      \
+	     pcpu_next_fit_region((chunk), (alloc_bits), (align), &(bit_off), \
+				  &(bits)))
+
 /**
  * pcpu_mem_zalloc - allocate memory
  * @size: bytes to allocate
@@ -818,6 +888,14 @@ static bool pcpu_is_populated(struct pcpu_chunk *chunk, int bit_off, int bits,
  * @align: alignment of area (max PAGE_SIZE bytes)
  * @pop_only: use populated regions only
  *
+ * Given a chunk and an allocation spec, find the offset to begin searching
+ * for a free region.  This iterates over the bitmap metadata blocks to
+ * find an offset that will be guaranteed to fit the requirements.  It is
+ * not quite first fit as if the allocation does not fit in the contig hint
+ * of a block or chunk, it is skipped.  This errs on the side of caution
+ * to prevent excess iteration.  Poor alignment can cause the allocator to
+ * skip over blocks and chunks that have valid free areas.
+ *
  * RETURNS:
  * The offset in the bitmap to begin searching.
  * -1 if no offset is found.
@@ -825,8 +903,7 @@ static bool pcpu_is_populated(struct pcpu_chunk *chunk, int bit_off, int bits,
 static int pcpu_find_block_fit(struct pcpu_chunk *chunk, int alloc_bits,
 			       size_t align, bool pop_only)
 {
-	int bit_off, bits;
-	int re; /* region end */
+	int bit_off, bits, next_off;
 
 	/*
 	 * Check to see if the allocation can fit in the chunk's contig hint.
@@ -839,22 +916,14 @@ static int pcpu_find_block_fit(struct pcpu_chunk *chunk, int alloc_bits,
 	if (bit_off + alloc_bits > chunk->contig_bits)
 		return -1;
 
-	pcpu_for_each_unpop_region(chunk->alloc_map, bit_off, re,
-				   chunk->first_bit,
-				   pcpu_chunk_map_bits(chunk)) {
-		bits = re - bit_off;
-
-		/* check alignment */
-		bits -= ALIGN(bit_off, align) - bit_off;
-		bit_off = ALIGN(bit_off, align);
-		if (bits < alloc_bits)
-			continue;
-
-		bits = alloc_bits;
+	bit_off = chunk->first_bit;
+	bits = 0;
+	pcpu_for_each_fit_region(chunk, alloc_bits, align, bit_off, bits) {
 		if (!pop_only || pcpu_is_populated(chunk, bit_off, bits,
-						   &bit_off))
+						   &next_off))
 			break;
 
+		bit_off = next_off;
 		bits = 0;
 	}
 
@@ -872,9 +941,12 @@ static int pcpu_find_block_fit(struct pcpu_chunk *chunk, int alloc_bits,
  * @start: bit_off to start searching
  *
  * This function takes in a @start offset to begin searching to fit an
- * allocation of @alloc_bits with alignment @align.  If it confirms a
- * valid free area, it then updates the allocation and boundary maps
- * accordingly.
+ * allocation of @alloc_bits with alignment @align.  It needs to scan
+ * the allocation map because if it fits within the block's contig hint,
+ * @start will be block->first_free. This is an attempt to fill the
+ * allocation prior to breaking the contig hint.  The allocation and
+ * boundary maps are updated accordingly if it confirms a valid
+ * free area.
  *
  * RETURNS:
  * Allocated addr offset in @chunk on success.
@@ -893,7 +965,7 @@ static int pcpu_alloc_area(struct pcpu_chunk *chunk, int alloc_bits,
 	/*
 	 * Search to find a fit.
 	 */
-	end = start + alloc_bits;
+	end = start + alloc_bits + PCPU_BITMAP_BLOCK_BITS;
 	bit_off = bitmap_find_next_zero_area(chunk->alloc_map, end, start,
 					     alloc_bits, align_mask);
 	if (bit_off >= end)
-- 
2.9.3

[toc] | [prev] | [next] | [standalone]

#1696185 — Re: [PATCH v2 22/23] percpu: update pcpu_find_block_fit to use an iterator

On Mon, Jul 24, 2017 at 07:02:19PM -0400, Dennis Zhou wrote:
> From: "Dennis Zhou (Facebook)" <dennisszhou@gmail.com>
> 
> The simple, and expensive, way to find a free area is to iterate over
> the entire bitmap until an area is found that fits the allocation size
> and alignment. This patch makes use of an iterate that find an area to
> check by using the block level contig hints. It will only return an area
> that can fit the size and alignment request. If the request can fit
> inside a block, it returns the first_free bit to start checking from to
> see if it can be fulfilled prior to the contig hint. The pcpu_alloc_area
> check has a bound of a block size added in case it is wrong.
> 
> Signed-off-by: Dennis Zhou <dennisszhou@gmail.com>

Reviewed-by: Josef Bacik <jbacik@fb.com>

Thanks,

Josef

[toc] | [prev] | [next] | [standalone]

#1695248 — [PATCH v2 09/23] percpu: combine percpu address checks

From: "Dennis Zhou (Facebook)" <dennisszhou@gmail.com>

The percpu address checks for the reserved and dynamic region chunks are
now specific to each region. The address checking logic can be combined
taking advantage of the global references to the dynamic and static
region chunks.

Signed-off-by: Dennis Zhou <dennisszhou@gmail.com>
---
 mm/percpu.c | 51 +++++++++++----------------------------------------
 1 file changed, 11 insertions(+), 40 deletions(-)

diff --git a/mm/percpu.c b/mm/percpu.c
index 7c9f0d3..5b1fcef 100644
--- a/mm/percpu.c
+++ b/mm/percpu.c
@@ -182,52 +182,23 @@ static void pcpu_schedule_balance_work(void)
 }
 
 /**
- * pcpu_addr_in_first_chunk - address check for first chunk's dynamic region
- * @addr: percpu address of interest
- *
- * The first chunk is considered to be the dynamic region of the first chunk.
- * While the true first chunk is composed of the static, dynamic, and
- * reserved regions, it is the chunk that serves the dynamic region that is
- * circulated in the chunk slots.
- *
- * The reserved chunk has a separate check and the static region addresses
- * should never be passed into the percpu allocator.
- *
- * RETURNS:
- * True if the address is in the dynamic region of the first chunk.
- */
-static bool pcpu_addr_in_first_chunk(void *addr)
-{
-	void *start_addr = pcpu_first_chunk->base_addr +
-			   pcpu_first_chunk->start_offset;
-	void *end_addr = pcpu_first_chunk->base_addr +
-			 pcpu_first_chunk->nr_pages * PAGE_SIZE -
-			 pcpu_first_chunk->end_offset;
-
-	return addr >= start_addr && addr < end_addr;
-}
-
-/**
- * pcpu_addr_in_reserved_chunk - address check for reserved region
- *
- * The reserved region is a part of the first chunk and primarily serves
- * static percpu variables from kernel modules.
+ * pcpu_addr_in_chunk - check if the address is served from this chunk
+ * @chunk: chunk of interest
+ * @addr: percpu address
  *
  * RETURNS:
- * True if the address is in the reserved region.
+ * True if the address is served from this chunk.
  */
-static bool pcpu_addr_in_reserved_chunk(void *addr)
+static bool pcpu_addr_in_chunk(struct pcpu_chunk *chunk, void *addr)
 {
 	void *start_addr, *end_addr;
 
-	if (!pcpu_reserved_chunk)
+	if (!chunk)
 		return false;
 
-	start_addr = pcpu_reserved_chunk->base_addr +
-		     pcpu_reserved_chunk->start_offset;
-	end_addr = pcpu_reserved_chunk->base_addr +
-		   pcpu_reserved_chunk->nr_pages * PAGE_SIZE -
-		   pcpu_reserved_chunk->end_offset;
+	start_addr = chunk->base_addr + chunk->start_offset;
+	end_addr = chunk->base_addr + chunk->nr_pages * PAGE_SIZE -
+		   chunk->end_offset;
 
 	return addr >= start_addr && addr < end_addr;
 }
@@ -929,11 +900,11 @@ static int __init pcpu_verify_alloc_info(const struct pcpu_alloc_info *ai);
 static struct pcpu_chunk *pcpu_chunk_addr_search(void *addr)
 {
 	/* is it in the dynamic region (first chunk)? */
-	if (pcpu_addr_in_first_chunk(addr))
+	if (pcpu_addr_in_chunk(pcpu_first_chunk, addr))
 		return pcpu_first_chunk;
 
 	/* is it in the reserved region? */
-	if (pcpu_addr_in_reserved_chunk(addr))
+	if (pcpu_addr_in_chunk(pcpu_reserved_chunk, addr))
 		return pcpu_reserved_chunk;
 
 	/*
-- 
2.9.3

[toc] | [prev] | [next] | [standalone]

#1696012 — Re: [PATCH v2 09/23] percpu: combine percpu address checks

On Mon, Jul 24, 2017 at 07:02:06PM -0400, Dennis Zhou wrote:
> From: "Dennis Zhou (Facebook)" <dennisszhou@gmail.com>
> 
> The percpu address checks for the reserved and dynamic region chunks are
> now specific to each region. The address checking logic can be combined
> taking advantage of the global references to the dynamic and static
> region chunks.
> 
> Signed-off-by: Dennis Zhou <dennisszhou@gmail.com>

Reviewed-by: Josef Bacik <jbacik@fb.com>

Tho this probably would have been fine to do in the previous patch.  Thanks,

Josef

[toc] | [prev] | [next] | [standalone]

#1695250 — [PATCH v2 02/23] percpu: introduce start_offset to pcpu_chunk

From: "Dennis Zhou (Facebook)" <dennisszhou@gmail.com>

The reserved chunk arithmetic uses a global variable
pcpu_reserved_chunk_limit that is set in the first chunk init code to
hide a portion of the area map. The bitmap allocator to come will
eventually move the base_addr up and require both the reserved chunk
and static chunk to maintain this offset. pcpu_reserved_chunk_limit is
removed and start_offset is added.

The first chunk that is circulated and is pcpu_first_chunk serves the
dynamic region, the region following the reserved region. The reserved
chunk address check will temporarily use the first chunk to identify its
address range. A following patch will increase the base_addr and remove
this. If there is no reserved chunk, this will check the static region
and return false because those values should never be passed into the
allocator.

Lastly, when linking in the first chunk, make sure to count the right
free region for the number of empty populated pages.

Signed-off-by: Dennis Zhou <dennisszhou@gmail.com>
---
 mm/percpu-internal.h |  3 +++
 mm/percpu.c          | 21 ++++++++++-----------
 2 files changed, 13 insertions(+), 11 deletions(-)

diff --git a/mm/percpu-internal.h b/mm/percpu-internal.h
index c9158a4..92fc012 100644
--- a/mm/percpu-internal.h
+++ b/mm/percpu-internal.h
@@ -28,6 +28,9 @@ struct pcpu_chunk {
 						   contain reservation for static chunk.
 						   Dynamic chunk will contain reservation
 						   for static and reserved chunks. */
+	int			start_offset;	/* the overlap with the previous
+						   region to have a page aligned
+						   base_addr */
 	int			nr_populated;	/* # of populated pages */
 	unsigned long		populated[];	/* populated bitmap */
 };
diff --git a/mm/percpu.c b/mm/percpu.c
index 3602d41..e94f0d1 100644
--- a/mm/percpu.c
+++ b/mm/percpu.c
@@ -145,13 +145,10 @@ struct pcpu_chunk *pcpu_first_chunk __ro_after_init;
 
 /*
  * Optional reserved chunk.  This chunk reserves part of the first
- * chunk and serves it for reserved allocations.  The amount of
- * reserved offset is in pcpu_reserved_chunk_limit.  When reserved
- * area doesn't exist, the following variables contain NULL and 0
- * respectively.
+ * chunk and serves it for reserved allocations.  When the reserved
+ * region doesn't exist, the following variable is NULL.
  */
 struct pcpu_chunk *pcpu_reserved_chunk __ro_after_init;
-static int pcpu_reserved_chunk_limit __ro_after_init;
 
 DEFINE_SPINLOCK(pcpu_lock);	/* all internal data structures */
 static DEFINE_MUTEX(pcpu_alloc_mutex);	/* chunk create/destroy, [de]pop, map ext */
@@ -196,7 +193,7 @@ static bool pcpu_addr_in_reserved_chunk(void *addr)
 	void *first_start = pcpu_first_chunk->base_addr;
 
 	return addr >= first_start &&
-		addr < first_start + pcpu_reserved_chunk_limit;
+		addr < first_start + pcpu_first_chunk->start_offset;
 }
 
 static int __pcpu_size_to_slot(int size)
@@ -1687,6 +1684,7 @@ int __init pcpu_setup_first_chunk(const struct pcpu_alloc_info *ai,
 	INIT_LIST_HEAD(&schunk->list);
 	INIT_LIST_HEAD(&schunk->map_extend_list);
 	schunk->base_addr = base_addr;
+	schunk->start_offset = ai->static_size;
 	schunk->map = smap;
 	schunk->map_alloc = ARRAY_SIZE(smap);
 	schunk->immutable = true;
@@ -1696,7 +1694,6 @@ int __init pcpu_setup_first_chunk(const struct pcpu_alloc_info *ai,
 	if (ai->reserved_size) {
 		schunk->free_size = ai->reserved_size;
 		pcpu_reserved_chunk = schunk;
-		pcpu_reserved_chunk_limit = ai->static_size + ai->reserved_size;
 	} else {
 		schunk->free_size = dyn_size;
 		dyn_size = 0;			/* dynamic area covered */
@@ -1704,7 +1701,7 @@ int __init pcpu_setup_first_chunk(const struct pcpu_alloc_info *ai,
 
 	schunk->contig_hint = schunk->free_size;
 	schunk->map[0] = 1;
-	schunk->map[1] = ai->static_size;
+	schunk->map[1] = schunk->start_offset;
 	schunk->map[2] = (ai->static_size + schunk->free_size) | 1;
 	schunk->map_used = 2;
 	schunk->has_reserved = true;
@@ -1715,6 +1712,7 @@ int __init pcpu_setup_first_chunk(const struct pcpu_alloc_info *ai,
 		INIT_LIST_HEAD(&dchunk->list);
 		INIT_LIST_HEAD(&dchunk->map_extend_list);
 		dchunk->base_addr = base_addr;
+		dchunk->start_offset = ai->static_size + ai->reserved_size;
 		dchunk->map = dmap;
 		dchunk->map_alloc = ARRAY_SIZE(dmap);
 		dchunk->immutable = true;
@@ -1723,16 +1721,17 @@ int __init pcpu_setup_first_chunk(const struct pcpu_alloc_info *ai,
 
 		dchunk->contig_hint = dchunk->free_size = dyn_size;
 		dchunk->map[0] = 1;
-		dchunk->map[1] = pcpu_reserved_chunk_limit;
-		dchunk->map[2] = (pcpu_reserved_chunk_limit + dchunk->free_size) | 1;
+		dchunk->map[1] = dchunk->start_offset;
+		dchunk->map[2] = (dchunk->start_offset + dchunk->free_size) | 1;
 		dchunk->map_used = 2;
 		dchunk->has_reserved = true;
 	}
 
 	/* link the first chunk in */
 	pcpu_first_chunk = dchunk ?: schunk;
+	i = (pcpu_first_chunk->start_offset) ? 1 : 0;
 	pcpu_nr_empty_pop_pages +=
-		pcpu_count_occupied_pages(pcpu_first_chunk, 1);
+		pcpu_count_occupied_pages(pcpu_first_chunk, i);
 	pcpu_chunk_relocate(pcpu_first_chunk, -1);
 
 	pcpu_stats_chunk_alloc();
-- 
2.9.3

[toc] | [prev] | [next] | [standalone]

#1695999 — Re: [PATCH v2 02/23] percpu: introduce start_offset to pcpu_chunk

On Mon, Jul 24, 2017 at 07:01:59PM -0400, Dennis Zhou wrote:
> From: "Dennis Zhou (Facebook)" <dennisszhou@gmail.com>
> 
> The reserved chunk arithmetic uses a global variable
> pcpu_reserved_chunk_limit that is set in the first chunk init code to
> hide a portion of the area map. The bitmap allocator to come will
> eventually move the base_addr up and require both the reserved chunk
> and static chunk to maintain this offset. pcpu_reserved_chunk_limit is
> removed and start_offset is added.
> 
> The first chunk that is circulated and is pcpu_first_chunk serves the
> dynamic region, the region following the reserved region. The reserved
> chunk address check will temporarily use the first chunk to identify its
> address range. A following patch will increase the base_addr and remove
> this. If there is no reserved chunk, this will check the static region
> and return false because those values should never be passed into the
> allocator.
> 
> Lastly, when linking in the first chunk, make sure to count the right
> free region for the number of empty populated pages.
> 
> Signed-off-by: Dennis Zhou <dennisszhou@gmail.com>

Reviewed-by: Josef Bacik <jbacik@fb.com>

Thanks,

Josef

[toc] | [prev] | [next] | [standalone]

#1695251 — [PATCH v2 13/23] percpu: generalize bitmap (un)populated iterators

From: "Dennis Zhou (Facebook)" <dennisszhou@gmail.com>

The area map allocator only used a bitmap for the backing page state.
The new bitmap allocator will use bitmaps to manage the allocation
region in addition to this.

This patch generalizes the bitmap iterators so they can be reused with
the bitmap allocator.

Signed-off-by: Dennis Zhou <dennisszhou@gmail.com>
---
 mm/percpu.c | 49 +++++++++++++++++++++++++------------------------
 1 file changed, 25 insertions(+), 24 deletions(-)

diff --git a/mm/percpu.c b/mm/percpu.c
index dc755721..84cc255 100644
--- a/mm/percpu.c
+++ b/mm/percpu.c
@@ -253,35 +253,32 @@ static unsigned long pcpu_chunk_addr(struct pcpu_chunk *chunk,
 	       pcpu_unit_page_offset(cpu, page_idx);
 }
 
-static void __maybe_unused pcpu_next_unpop(struct pcpu_chunk *chunk,
-					   int *rs, int *re, int end)
+static void pcpu_next_unpop(unsigned long *bitmap, int *rs, int *re, int end)
 {
-	*rs = find_next_zero_bit(chunk->populated, end, *rs);
-	*re = find_next_bit(chunk->populated, end, *rs + 1);
+	*rs = find_next_zero_bit(bitmap, end, *rs);
+	*re = find_next_bit(bitmap, end, *rs + 1);
 }
 
-static void __maybe_unused pcpu_next_pop(struct pcpu_chunk *chunk,
-					 int *rs, int *re, int end)
+static void pcpu_next_pop(unsigned long *bitmap, int *rs, int *re, int end)
 {
-	*rs = find_next_bit(chunk->populated, end, *rs);
-	*re = find_next_zero_bit(chunk->populated, end, *rs + 1);
+	*rs = find_next_bit(bitmap, end, *rs);
+	*re = find_next_zero_bit(bitmap, end, *rs + 1);
 }
 
 /*
- * (Un)populated page region iterators.  Iterate over (un)populated
- * page regions between @start and @end in @chunk.  @rs and @re should
- * be integer variables and will be set to start and end page index of
- * the current region.
+ * Bitmap region iterators.  Iterates over the bitmap between
+ * [@start, @end) in @chunk.  @rs and @re should be integer variables
+ * and will be set to start and end index of the current free region.
  */
-#define pcpu_for_each_unpop_region(chunk, rs, re, start, end)		    \
-	for ((rs) = (start), pcpu_next_unpop((chunk), &(rs), &(re), (end)); \
-	     (rs) < (re);						    \
-	     (rs) = (re) + 1, pcpu_next_unpop((chunk), &(rs), &(re), (end)))
+#define pcpu_for_each_unpop_region(bitmap, rs, re, start, end)		     \
+	for ((rs) = (start), pcpu_next_unpop((bitmap), &(rs), &(re), (end)); \
+	     (rs) < (re);						     \
+	     (rs) = (re) + 1, pcpu_next_unpop((bitmap), &(rs), &(re), (end)))
 
-#define pcpu_for_each_pop_region(chunk, rs, re, start, end)		    \
-	for ((rs) = (start), pcpu_next_pop((chunk), &(rs), &(re), (end));   \
-	     (rs) < (re);						    \
-	     (rs) = (re) + 1, pcpu_next_pop((chunk), &(rs), &(re), (end)))
+#define pcpu_for_each_pop_region(bitmap, rs, re, start, end)		     \
+	for ((rs) = (start), pcpu_next_pop((bitmap), &(rs), &(re), (end));   \
+	     (rs) < (re);						     \
+	     (rs) = (re) + 1, pcpu_next_pop((bitmap), &(rs), &(re), (end)))
 
 /**
  * pcpu_mem_zalloc - allocate memory
@@ -521,7 +518,8 @@ static int pcpu_fit_in_area(struct pcpu_chunk *chunk, int off, int this_size,
 		page_end = PFN_UP(head + off + size);
 
 		rs = page_start;
-		pcpu_next_unpop(chunk, &rs, &re, PFN_UP(off + this_size));
+		pcpu_next_unpop(chunk->populated, &rs, &re,
+				PFN_UP(off + this_size));
 		if (rs >= page_end)
 			return head;
 		cand_off = re * PAGE_SIZE;
@@ -1071,7 +1069,8 @@ static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved,
 		page_start = PFN_DOWN(off);
 		page_end = PFN_UP(off + size);
 
-		pcpu_for_each_unpop_region(chunk, rs, re, page_start, page_end) {
+		pcpu_for_each_unpop_region(chunk->populated, rs, re,
+					   page_start, page_end) {
 			WARN_ON(chunk->immutable);
 
 			ret = pcpu_populate_chunk(chunk, rs, re);
@@ -1221,7 +1220,8 @@ static void pcpu_balance_workfn(struct work_struct *work)
 	list_for_each_entry_safe(chunk, next, &to_free, list) {
 		int rs, re;
 
-		pcpu_for_each_pop_region(chunk, rs, re, 0, chunk->nr_pages) {
+		pcpu_for_each_pop_region(chunk->populated, rs, re, 0,
+					 chunk->nr_pages) {
 			pcpu_depopulate_chunk(chunk, rs, re);
 			spin_lock_irq(&pcpu_lock);
 			pcpu_chunk_depopulated(chunk, rs, re);
@@ -1288,7 +1288,8 @@ static void pcpu_balance_workfn(struct work_struct *work)
 			continue;
 
 		/* @chunk can't go away while pcpu_alloc_mutex is held */
-		pcpu_for_each_unpop_region(chunk, rs, re, 0, chunk->nr_pages) {
+		pcpu_for_each_unpop_region(chunk->populated, rs, re, 0,
+					   chunk->nr_pages) {
 			int nr = min(re - rs, nr_to_pop);
 
 			ret = pcpu_populate_chunk(chunk, rs, rs + nr);
-- 
2.9.3

[toc] | [prev] | [next] | [standalone]

#1696026 — Re: [PATCH v2 13/23] percpu: generalize bitmap (un)populated iterators

On Mon, Jul 24, 2017 at 07:02:10PM -0400, Dennis Zhou wrote:
> From: "Dennis Zhou (Facebook)" <dennisszhou@gmail.com>
> 
> The area map allocator only used a bitmap for the backing page state.
> The new bitmap allocator will use bitmaps to manage the allocation
> region in addition to this.
> 
> This patch generalizes the bitmap iterators so they can be reused with
> the bitmap allocator.
> 
> Signed-off-by: Dennis Zhou <dennisszhou@gmail.com>

Reviewed-by: Josef Bacik <jbacik@fb.com>

Thanks,

Josef

[toc] | [prev] | [next] | [standalone]

#1697130 — Re: [PATCH v2 13/23] percpu: generalize bitmap (un)populated iterators

On Mon, Jul 24, 2017 at 07:02:10PM -0400, Dennis Zhou wrote:
> From: "Dennis Zhou (Facebook)" <dennisszhou@gmail.com>
> 
> The area map allocator only used a bitmap for the backing page state.
> The new bitmap allocator will use bitmaps to manage the allocation
> region in addition to this.
> 
> This patch generalizes the bitmap iterators so they can be reused with
> the bitmap allocator.
> 
> Signed-off-by: Dennis Zhou <dennisszhou@gmail.com>

Applied 1-13 to percpu/for-4.14.

Nice cleanups.  Thanks.

-- 
tejun

[toc] | [prev] | [next] | [standalone]

#1695252 — [PATCH v2 08/23] percpu: modify base_addr to be region specific

From: "Dennis Zhou (Facebook)" <dennisszhou@gmail.com>

Originally, the first chunk was served by one or two chunks, each
given a region they are responsible for. Despite this, the arithmetic
was based off of the true base_addr of the chunk making it be overly
inclusive.

This patch moves the base_addr of chunks that are responsible for the
first chunk. The base_addr must remain page aligned to keep the
address alignment correct, so it is the beginning of the region served
page aligned down. start_offset holds where the region served begins
from this new base_addr.

The corresponding percpu address checks are modified to be more specific
as a result. The first chunk considers only the dynamic region and both
first chunk and reserved chunk checks ignore the static region. The
static region addresses should never be passed into the allocator. There
is no impact here besides distinguishing the first chunk and making the
checks specific.

The percpu pointer to physical address is left intact as addresses are
not given out in the non-allocated portion of percpu memory.

nr_pages is added to pcpu_chunk to keep track of the size of the entire
region served containing both start_offset and end_offset. This variable
will be used to manage the bitmap allocator.

Signed-off-by: Dennis Zhou <dennisszhou@gmail.com>
---
 mm/percpu-internal.h |   2 +
 mm/percpu.c          | 155 +++++++++++++++++++++++++++++++++++++--------------
 2 files changed, 116 insertions(+), 41 deletions(-)

diff --git a/mm/percpu-internal.h b/mm/percpu-internal.h
index f02f31c..34cb979 100644
--- a/mm/percpu-internal.h
+++ b/mm/percpu-internal.h
@@ -29,6 +29,8 @@ struct pcpu_chunk {
 	int			end_offset;	/* additional area required to
 						   have the region end page
 						   aligned */
+
+	int			nr_pages;	/* # of pages served by this chunk */
 	int			nr_populated;	/* # of populated pages */
 	unsigned long		populated[];	/* populated bitmap */
 };
diff --git a/mm/percpu.c b/mm/percpu.c
index e08ed61..7c9f0d3 100644
--- a/mm/percpu.c
+++ b/mm/percpu.c
@@ -181,19 +181,55 @@ static void pcpu_schedule_balance_work(void)
 		schedule_work(&pcpu_balance_work);
 }
 
+/**
+ * pcpu_addr_in_first_chunk - address check for first chunk's dynamic region
+ * @addr: percpu address of interest
+ *
+ * The first chunk is considered to be the dynamic region of the first chunk.
+ * While the true first chunk is composed of the static, dynamic, and
+ * reserved regions, it is the chunk that serves the dynamic region that is
+ * circulated in the chunk slots.
+ *
+ * The reserved chunk has a separate check and the static region addresses
+ * should never be passed into the percpu allocator.
+ *
+ * RETURNS:
+ * True if the address is in the dynamic region of the first chunk.
+ */
 static bool pcpu_addr_in_first_chunk(void *addr)
 {
-	void *first_start = pcpu_first_chunk->base_addr;
+	void *start_addr = pcpu_first_chunk->base_addr +
+			   pcpu_first_chunk->start_offset;
+	void *end_addr = pcpu_first_chunk->base_addr +
+			 pcpu_first_chunk->nr_pages * PAGE_SIZE -
+			 pcpu_first_chunk->end_offset;
 
-	return addr >= first_start && addr < first_start + pcpu_unit_size;
+	return addr >= start_addr && addr < end_addr;
 }
 
+/**
+ * pcpu_addr_in_reserved_chunk - address check for reserved region
+ *
+ * The reserved region is a part of the first chunk and primarily serves
+ * static percpu variables from kernel modules.
+ *
+ * RETURNS:
+ * True if the address is in the reserved region.
+ */
 static bool pcpu_addr_in_reserved_chunk(void *addr)
 {
-	void *first_start = pcpu_first_chunk->base_addr;
+	void *start_addr, *end_addr;
+
+	if (!pcpu_reserved_chunk)
+		return false;
 
-	return addr >= first_start &&
-		addr < first_start + pcpu_first_chunk->start_offset;
+	start_addr = pcpu_reserved_chunk->base_addr +
+		     pcpu_reserved_chunk->start_offset;
+	end_addr = pcpu_reserved_chunk->base_addr +
+		   pcpu_reserved_chunk->nr_pages * PAGE_SIZE -
+		   pcpu_reserved_chunk->end_offset;
+
+	return addr >= start_addr && addr < end_addr;
 }
 
 static int __pcpu_size_to_slot(int size)
@@ -234,11 +270,16 @@ static int __maybe_unused pcpu_page_idx(unsigned int cpu, int page_idx)
 	return pcpu_unit_map[cpu] * pcpu_unit_pages + page_idx;
 }
 
+static unsigned long pcpu_unit_page_offset(unsigned int cpu, int page_idx)
+{
+	return pcpu_unit_offsets[cpu] + (page_idx << PAGE_SHIFT);
+}
+
 static unsigned long pcpu_chunk_addr(struct pcpu_chunk *chunk,
 				     unsigned int cpu, int page_idx)
 {
-	return (unsigned long)chunk->base_addr + pcpu_unit_offsets[cpu] +
-		(page_idx << PAGE_SHIFT);
+	return (unsigned long)chunk->base_addr +
+	       pcpu_unit_page_offset(cpu, page_idx);
 }
 
 static void __maybe_unused pcpu_next_unpop(struct pcpu_chunk *chunk,
@@ -708,23 +749,34 @@ static void pcpu_free_area(struct pcpu_chunk *chunk, int freeme,
 	pcpu_chunk_relocate(chunk, oslot);
 }
 
-static struct pcpu_chunk * __init pcpu_alloc_first_chunk(void *base_addr,
-							 int start_offset,
+static struct pcpu_chunk * __init pcpu_alloc_first_chunk(unsigned long tmp_addr,
 							 int map_size,
 							 int *map,
 							 int init_map_size)
 {
 	struct pcpu_chunk *chunk;
-	int region_size;
+	unsigned long aligned_addr;
+	int start_offset, region_size;
+
+	/* region calculations */
+	aligned_addr = tmp_addr & PAGE_MASK;
+
+	start_offset = tmp_addr - aligned_addr;
 
 	region_size = PFN_ALIGN(start_offset + map_size);
 
+	/* allocate chunk */
 	chunk = memblock_virt_alloc(pcpu_chunk_struct_size, 0);
+
 	INIT_LIST_HEAD(&chunk->list);
 	INIT_LIST_HEAD(&chunk->map_extend_list);
-	chunk->base_addr = base_addr;
+
+	chunk->base_addr = (void *)aligned_addr;
 	chunk->start_offset = start_offset;
 	chunk->end_offset = region_size - chunk->start_offset - map_size;
+
+	chunk->nr_pages = pcpu_unit_pages;
+
 	chunk->map = map;
 	chunk->map_alloc = init_map_size;
 
@@ -734,10 +786,17 @@ static struct pcpu_chunk * __init pcpu_alloc_first_chunk(void *base_addr,
 	chunk->nr_populated = pcpu_unit_pages;
 
 	chunk->contig_hint = chunk->free_size = map_size;
-	chunk->map[0] = 1;
-	chunk->map[1] = chunk->start_offset;
-	chunk->map[2] = (chunk->start_offset + chunk->free_size) | 1;
-	chunk->map_used = 2;
+
+	if (chunk->start_offset) {
+		/* hide the beginning of the bitmap */
+		chunk->map[0] = 1;
+		chunk->map[1] = chunk->start_offset;
+		chunk->map_used = 1;
+	}
+
+	/* set chunk's free region */
+	chunk->map[++chunk->map_used] =
+		(chunk->start_offset + chunk->free_size) | 1;
 
 	if (chunk->end_offset) {
 		/* hide the end of the bitmap */
@@ -772,6 +831,8 @@ static struct pcpu_chunk *pcpu_alloc_chunk(void)
 	chunk->free_size = pcpu_unit_size;
 	chunk->contig_hint = pcpu_unit_size;
 
+	chunk->nr_pages = pcpu_unit_pages;
+
 	return chunk;
 }
 
@@ -859,18 +920,21 @@ static int __init pcpu_verify_alloc_info(const struct pcpu_alloc_info *ai);
  * pcpu_chunk_addr_search - determine chunk containing specified address
  * @addr: address for which the chunk needs to be determined.
  *
+ * This is an internal function that handles all but static allocations.
+ * Static percpu address values should never be passed into the allocator.
+ *
  * RETURNS:
  * The address of the found chunk.
  */
 static struct pcpu_chunk *pcpu_chunk_addr_search(void *addr)
 {
-	/* is it in the first chunk? */
-	if (pcpu_addr_in_first_chunk(addr)) {
-		/* is it in the reserved area? */
-		if (pcpu_addr_in_reserved_chunk(addr))
-			return pcpu_reserved_chunk;
+	/* is it in the dynamic region (first chunk)? */
+	if (pcpu_addr_in_first_chunk(addr))
 		return pcpu_first_chunk;
-	}
+
+	/* is it in the reserved region? */
+	if (pcpu_addr_in_reserved_chunk(addr))
+		return pcpu_reserved_chunk;
 
 	/*
 	 * The address is relative to unit0 which might be unused and
@@ -1401,10 +1465,16 @@ phys_addr_t per_cpu_ptr_to_phys(void *addr)
 	 * The following test on unit_low/high isn't strictly
 	 * necessary but will speed up lookups of addresses which
 	 * aren't in the first chunk.
+	 *
+	 * The address check is against full chunk sizes.  pcpu_base_addr
+	 * points to the beginning of the first chunk including the
+	 * static region.  Assumes good intent as the first chunk may
+	 * not be full (ie. < pcpu_unit_pages in size).
 	 */
-	first_low = pcpu_chunk_addr(pcpu_first_chunk, pcpu_low_unit_cpu, 0);
-	first_high = pcpu_chunk_addr(pcpu_first_chunk, pcpu_high_unit_cpu,
-				     pcpu_unit_pages);
+	first_low = (unsigned long)pcpu_base_addr +
+		    pcpu_unit_page_offset(pcpu_low_unit_cpu, 0);
+	first_high = (unsigned long)pcpu_base_addr +
+		     pcpu_unit_page_offset(pcpu_high_unit_cpu, pcpu_unit_pages);
 	if ((unsigned long)addr >= first_low &&
 	    (unsigned long)addr < first_high) {
 		for_each_possible_cpu(cpu) {
@@ -1586,12 +1656,13 @@ static void pcpu_dump_alloc_info(const char *lvl,
  * The caller should have mapped the first chunk at @base_addr and
  * copied static data to each unit.
  *
- * If the first chunk ends up with both reserved and dynamic areas, it
- * is served by two chunks - one to serve the core static and reserved
- * areas and the other for the dynamic area.  They share the same vm
- * and page map but uses different area allocation map to stay away
- * from each other.  The latter chunk is circulated in the chunk slots
- * and available for dynamic allocation like any other chunks.
+ * The first chunk will always contain a static and a dynamic region.
+ * However, the static region is not managed by any chunk.  If the first
+ * chunk also contains a reserved region, it is served by two chunks -
+ * one for the reserved region and one for the dynamic region.  They
+ * share the same vm, but use offset regions in the area allocation map.
+ * The chunk serving the dynamic region is circulated in the chunk slots
+ * and available for dynamic allocation like any other chunk.
  *
  * RETURNS:
  * 0 on success, -errno on failure.
@@ -1609,7 +1680,8 @@ int __init pcpu_setup_first_chunk(const struct pcpu_alloc_info *ai,
 	unsigned int cpu;
 	int *unit_map;
 	int group, unit, i;
-	int map_size, start_offset;
+	int map_size;
+	unsigned long tmp_addr;
 
 #define PCPU_SETUP_BUG_ON(cond)	do {					\
 	if (unlikely(cond)) {						\
@@ -1712,25 +1784,26 @@ int __init pcpu_setup_first_chunk(const struct pcpu_alloc_info *ai,
 		INIT_LIST_HEAD(&pcpu_slot[i]);
 
 	/*
-	 * Initialize static chunk.  If reserved_size is zero, the
-	 * static chunk covers static area + dynamic allocation area
-	 * in the first chunk.  If reserved_size is not zero, it
-	 * covers static area + reserved area (mostly used for module
-	 * static percpu allocation).
+	 * Initialize first chunk.
+	 * If the reserved_size is non-zero, this initializes the reserved
+	 * chunk.  If the reserved_size is zero, the reserved chunk is NULL
+	 * and the dynamic region is initialized here.  The first chunk,
+	 * pcpu_first_chunk, will always point to the chunk that serves
+	 * the dynamic region.
 	 */
-	start_offset = ai->static_size;
+	tmp_addr = (unsigned long)base_addr + ai->static_size;
 	map_size = ai->reserved_size ?: ai->dyn_size;
-	chunk = pcpu_alloc_first_chunk(base_addr, start_offset, map_size, smap,
+	chunk = pcpu_alloc_first_chunk(tmp_addr, map_size, smap,
 				       ARRAY_SIZE(smap));
 
 	/* init dynamic chunk if necessary */
 	if (ai->reserved_size) {
 		pcpu_reserved_chunk = chunk;
 
-		start_offset = ai->static_size + ai->reserved_size;
+		tmp_addr = (unsigned long)base_addr + ai->static_size +
+			   ai->reserved_size;
 		map_size = ai->dyn_size;
-		chunk = pcpu_alloc_first_chunk(base_addr, start_offset,
-					       map_size, dmap,
+		chunk = pcpu_alloc_first_chunk(tmp_addr, map_size, dmap,
 					       ARRAY_SIZE(dmap));
 	}
 
-- 
2.9.3

[toc] | [prev] | [next] | [standalone]

#1696016 — Re: [PATCH v2 08/23] percpu: modify base_addr to be region specific

On Mon, Jul 24, 2017 at 07:02:05PM -0400, Dennis Zhou wrote:
> From: "Dennis Zhou (Facebook)" <dennisszhou@gmail.com>
> 
> Originally, the first chunk was served by one or two chunks, each
> given a region they are responsible for. Despite this, the arithmetic
> was based off of the true base_addr of the chunk making it be overly
> inclusive.
> 
> This patch moves the base_addr of chunks that are responsible for the
> first chunk. The base_addr must remain page aligned to keep the
> address alignment correct, so it is the beginning of the region served
> page aligned down. start_offset holds where the region served begins
> from this new base_addr.
> 
> The corresponding percpu address checks are modified to be more specific
> as a result. The first chunk considers only the dynamic region and both
> first chunk and reserved chunk checks ignore the static region. The
> static region addresses should never be passed into the allocator. There
> is no impact here besides distinguishing the first chunk and making the
> checks specific.
> 
> The percpu pointer to physical address is left intact as addresses are
> not given out in the non-allocated portion of percpu memory.
> 
> nr_pages is added to pcpu_chunk to keep track of the size of the entire
> region served containing both start_offset and end_offset. This variable
> will be used to manage the bitmap allocator.
> 
> Signed-off-by: Dennis Zhou <dennisszhou@gmail.com>

Reviewed-by: Josef Bacik <jbacik@fb.com>

Thanks,

Josef

[toc] | [prev] | [next] | [standalone]

#1695253 — [PATCH v2 17/23] percpu: skip chunks if the alloc does not fit in the contig hint

From: "Dennis Zhou (Facebook)" <dennisszhou@gmail.com>

This patch adds chunk->contig_bits_start to keep track of the contig
hint's offset and the check to skip the chunk if it does not fit. If
the chunk's contig hint starting offset cannot satisfy an allocation,
the allocator assumes there is enough memory pressure in this chunk to
either use a different chunk or create a new one. This accepts a less
tight packing for a smoother latency curve.

Signed-off-by: Dennis Zhou <dennisszhou@gmail.com>
---
 mm/percpu-internal.h |  2 ++
 mm/percpu.c          | 18 ++++++++++++++++--
 2 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/mm/percpu-internal.h b/mm/percpu-internal.h
index e60e049..7065faf 100644
--- a/mm/percpu-internal.h
+++ b/mm/percpu-internal.h
@@ -29,6 +29,8 @@ struct pcpu_chunk {
 	struct list_head	list;		/* linked to pcpu_slot lists */
 	int			free_bytes;	/* free bytes in the chunk */
 	int			contig_bits;	/* max contiguous size hint */
+	int			contig_bits_start; /* contig_bits starting
+						      offset */
 	void			*base_addr;	/* base address of this chunk */
 
 	unsigned long		*alloc_map;	/* allocation map */
diff --git a/mm/percpu.c b/mm/percpu.c
index ad70c67..3732373 100644
--- a/mm/percpu.c
+++ b/mm/percpu.c
@@ -393,12 +393,14 @@ static inline int pcpu_cnt_pop_pages(struct pcpu_chunk *chunk, int bit_off,
  * @bit_off: chunk offset
  * @bits: size of free area
  *
- * This updates the chunk's contig hint given a free area.
+ * This updates the chunk's contig hint and starting offset given a free area.
  */
 static void pcpu_chunk_update(struct pcpu_chunk *chunk, int bit_off, int bits)
 {
-	if (bits > chunk->contig_bits)
+	if (bits > chunk->contig_bits) {
+		chunk->contig_bits_start = bit_off;
 		chunk->contig_bits = bits;
+	}
 }
 
 /**
@@ -409,6 +411,7 @@ static void pcpu_chunk_update(struct pcpu_chunk *chunk, int bit_off, int bits)
  *
  * Updates:
  *      chunk->contig_bits
+ *      chunk->contig_bits_start
  *      nr_empty_pop_pages
  */
 static void pcpu_chunk_refresh_hint(struct pcpu_chunk *chunk)
@@ -639,6 +642,17 @@ static int pcpu_find_block_fit(struct pcpu_chunk *chunk, int alloc_bits,
 	int bit_off, bits;
 	int re; /* region end */
 
+	/*
+	 * Check to see if the allocation can fit in the chunk's contig hint.
+	 * This is an optimization to prevent scanning by assuming if it
+	 * cannot fit in the global hint, there is memory pressure and creating
+	 * a new chunk would happen soon.
+	 */
+	bit_off = ALIGN(chunk->contig_bits_start, align) -
+		  chunk->contig_bits_start;
+	if (bit_off + alloc_bits > chunk->contig_bits)
+		return -1;
+
 	pcpu_for_each_unpop_region(chunk->alloc_map, bit_off, re,
 				   chunk->first_bit,
 				   pcpu_chunk_map_bits(chunk)) {
-- 
2.9.3

[toc] | [prev] | [next] | [standalone]

#1696186 — Re: [PATCH v2 17/23] percpu: skip chunks if the alloc does not fit in the contig hint

On Mon, Jul 24, 2017 at 07:02:14PM -0400, Dennis Zhou wrote:
> From: "Dennis Zhou (Facebook)" <dennisszhou@gmail.com>
> 
> This patch adds chunk->contig_bits_start to keep track of the contig
> hint's offset and the check to skip the chunk if it does not fit. If
> the chunk's contig hint starting offset cannot satisfy an allocation,
> the allocator assumes there is enough memory pressure in this chunk to
> either use a different chunk or create a new one. This accepts a less
> tight packing for a smoother latency curve.
> 
> Signed-off-by: Dennis Zhou <dennisszhou@gmail.com>

Reviewed-by: Josef Bacik <jbacik@fb.com>

Thanks,

Josef

[toc] | [prev] | [next] | [standalone]

#1695254 — [PATCH v2 19/23] percpu: update alloc path to only scan if contig hints are broken

From: "Dennis Zhou (Facebook)" <dennisszhou@gmail.com>

Metadata is kept per block to keep track of where the contig hints are.
Scanning can be avoided when the contig hints are not broken. In that
case, left and right contigs have to be managed manually.

This patch changes the allocation path hint updating to only scan when
contig hints are broken.

Signed-off-by: Dennis Zhou <dennisszhou@gmail.com>
---
 mm/percpu.c | 59 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 56 insertions(+), 3 deletions(-)

diff --git a/mm/percpu.c b/mm/percpu.c
index aaad747..2bf2cfc 100644
--- a/mm/percpu.c
+++ b/mm/percpu.c
@@ -514,6 +514,10 @@ static void pcpu_block_refresh_hint(struct pcpu_chunk *chunk, int index)
  * @chunk: chunk of interest
  * @bit_off: chunk offset
  * @bits: size of request
+ *
+ * Updates metadata for the allocation path.  The metadata only has to be
+ * refreshed by a full scan iff the chunk's contig hint is broken.  Block level
+ * scans are required if the block's contig hint is broken.
  */
 static void pcpu_block_update_hint_alloc(struct pcpu_chunk *chunk, int bit_off,
 					 int bits)
@@ -538,14 +542,56 @@ static void pcpu_block_update_hint_alloc(struct pcpu_chunk *chunk, int bit_off,
 
 	/*
 	 * Update s_block.
+	 * block->first_free must be updated if the allocation takes its place.
+	 * If the allocation breaks the contig_hint, a scan is required to
+	 * restore this hint.
 	 */
-	pcpu_block_refresh_hint(chunk, s_index);
+	if (s_off == s_block->first_free)
+		s_block->first_free = find_next_zero_bit(
+					pcpu_index_alloc_map(chunk, s_index),
+					PCPU_BITMAP_BLOCK_BITS,
+					s_off + bits);
+
+	if (s_off >= s_block->contig_hint_start &&
+	    s_off < s_block->contig_hint_start + s_block->contig_hint) {
+		/* block contig hint is broken - scan to fix it */
+		pcpu_block_refresh_hint(chunk, s_index);
+	} else {
+		/* update left and right contig manually */
+		s_block->left_free = min(s_block->left_free, s_off);
+		if (s_index == e_index)
+			s_block->right_free = min_t(int, s_block->right_free,
+					PCPU_BITMAP_BLOCK_BITS - e_off);
+		else
+			s_block->right_free = 0;
+	}
 
 	/*
 	 * Update e_block.
 	 */
 	if (s_index != e_index) {
-		pcpu_block_refresh_hint(chunk, e_index);
+		/*
+		 * When the allocation is across blocks, the end is along
+		 * the left part of the e_block.
+		 */
+		e_block->first_free = find_next_zero_bit(
+				pcpu_index_alloc_map(chunk, e_index),
+				PCPU_BITMAP_BLOCK_BITS, e_off);
+
+		if (e_off == PCPU_BITMAP_BLOCK_BITS) {
+			/* reset the block */
+			e_block++;
+		} else {
+			if (e_off > e_block->contig_hint_start) {
+				/* contig hint is broken - scan to fix it */
+				pcpu_block_refresh_hint(chunk, e_index);
+			} else {
+				e_block->left_free = 0;
+				e_block->right_free =
+					min_t(int, e_block->right_free,
+					      PCPU_BITMAP_BLOCK_BITS - e_off);
+			}
+		}
 
 		/* update in-between md_blocks */
 		for (block = s_block + 1; block < e_block; block++) {
@@ -555,7 +601,14 @@ static void pcpu_block_update_hint_alloc(struct pcpu_chunk *chunk, int bit_off,
 		}
 	}
 
-	pcpu_chunk_refresh_hint(chunk);
+	/*
+	 * The only time a full chunk scan is required is if the chunk
+	 * contig hint is broken.  Otherwise, it means a smaller space
+	 * was used and therefore the chunk contig hint is still correct.
+	 */
+	if (bit_off >= chunk->contig_bits_start  &&
+	    bit_off < chunk->contig_bits_start + chunk->contig_bits)
+		pcpu_chunk_refresh_hint(chunk);
 }
 
 /**
-- 
2.9.3

[toc] | [prev] | [next] | [standalone]

#1696145 — Re: [PATCH v2 19/23] percpu: update alloc path to only scan if contig hints are broken

On Mon, Jul 24, 2017 at 07:02:16PM -0400, Dennis Zhou wrote:
> From: "Dennis Zhou (Facebook)" <dennisszhou@gmail.com>
> 
> Metadata is kept per block to keep track of where the contig hints are.
> Scanning can be avoided when the contig hints are not broken. In that
> case, left and right contigs have to be managed manually.
> 
> This patch changes the allocation path hint updating to only scan when
> contig hints are broken.
> 
> Signed-off-by: Dennis Zhou <dennisszhou@gmail.com>

Reviewed-by: Josef Bacik <jbacik@fb.com>

Thanks,

Josef

[toc] | [prev] | [next] | [standalone]

#1695256 — [PATCH v2 12/23] percpu: increase minimum percpu allocation size and align first regions

From: "Dennis Zhou (Facebook)" <dennisszhou@gmail.com>

This patch increases the minimum allocation size of percpu memory to
4-bytes. This change will help minimize the metadata overhead
associated with the bitmap allocator. The assumption is that most
allocations will be of objects or structs greater than 2 bytes with
integers or longs being used rather than shorts.

The first chunk regions are now aligned with the minimum allocation
size. The reserved region is expected to be set as a multiple of the
minimum allocation size. The static region is aligned up and the delta
is removed from the dynamic size. This works because the dynamic size is
increased to be page aligned. If the static size is not minimum
allocation size aligned, then there must be a gap that is added to the
dynamic size. The dynamic size will never be smaller than the set value.

Signed-off-by: Dennis Zhou <dennisszhou@gmail.com>
---
 include/linux/percpu.h |  4 ++++
 mm/percpu.c            | 27 ++++++++++++++++++++-------
 2 files changed, 24 insertions(+), 7 deletions(-)

diff --git a/include/linux/percpu.h b/include/linux/percpu.h
index 491b3f5..90e0cb0 100644
--- a/include/linux/percpu.h
+++ b/include/linux/percpu.h
@@ -21,6 +21,10 @@
 /* minimum unit size, also is the maximum supported allocation size */
 #define PCPU_MIN_UNIT_SIZE		PFN_ALIGN(32 << 10)
 
+/* minimum allocation size and shift in bytes */
+#define PCPU_MIN_ALLOC_SHIFT		2
+#define PCPU_MIN_ALLOC_SIZE		(1 << PCPU_MIN_ALLOC_SHIFT)
+
 /*
  * Percpu allocator can serve percpu allocations before slab is
  * initialized which allows slab to depend on the percpu allocator.
diff --git a/mm/percpu.c b/mm/percpu.c
index 657ab08..dc755721 100644
--- a/mm/percpu.c
+++ b/mm/percpu.c
@@ -956,10 +956,10 @@ static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved,
 	 * We want the lowest bit of offset available for in-use/free
 	 * indicator, so force >= 16bit alignment and make size even.
 	 */
-	if (unlikely(align < 2))
-		align = 2;
+	if (unlikely(align < PCPU_MIN_ALLOC_SIZE))
+		align = PCPU_MIN_ALLOC_SIZE;
 
-	size = ALIGN(size, 2);
+	size = ALIGN(size, PCPU_MIN_ALLOC_SIZE);
 
 	if (unlikely(!size || size > PCPU_MIN_UNIT_SIZE || align > PAGE_SIZE ||
 		     !is_power_of_2(align))) {
@@ -1653,6 +1653,7 @@ int __init pcpu_setup_first_chunk(const struct pcpu_alloc_info *ai,
 	static int smap[PERCPU_DYNAMIC_EARLY_SLOTS] __initdata;
 	static int dmap[PERCPU_DYNAMIC_EARLY_SLOTS] __initdata;
 	size_t size_sum = ai->static_size + ai->reserved_size + ai->dyn_size;
+	size_t static_size, dyn_size;
 	struct pcpu_chunk *chunk;
 	unsigned long *group_offsets;
 	size_t *group_sizes;
@@ -1686,6 +1687,7 @@ int __init pcpu_setup_first_chunk(const struct pcpu_alloc_info *ai,
 	PCPU_SETUP_BUG_ON(ai->unit_size < PCPU_MIN_UNIT_SIZE);
 	PCPU_SETUP_BUG_ON(ai->dyn_size < PERCPU_DYNAMIC_EARLY_SIZE);
 	PCPU_SETUP_BUG_ON(!ai->dyn_size);
+	PCPU_SETUP_BUG_ON(!IS_ALIGNED(ai->reserved_size, PCPU_MIN_ALLOC_SIZE));
 	PCPU_SETUP_BUG_ON(pcpu_verify_alloc_info(ai) < 0);
 
 	/* process group information and build config tables accordingly */
@@ -1764,6 +1766,17 @@ int __init pcpu_setup_first_chunk(const struct pcpu_alloc_info *ai,
 		INIT_LIST_HEAD(&pcpu_slot[i]);
 
 	/*
+	 * The end of the static region needs to be aligned with the
+	 * minimum allocation size as this offsets the reserved and
+	 * dynamic region.  The first chunk ends page aligned by
+	 * expanding the dynamic region, therefore the dynamic region
+	 * can be shrunk to compensate while still staying above the
+	 * configured sizes.
+	 */
+	static_size = ALIGN(ai->static_size, PCPU_MIN_ALLOC_SIZE);
+	dyn_size = ai->dyn_size - (static_size - ai->static_size);
+
+	/*
 	 * Initialize first chunk.
 	 * If the reserved_size is non-zero, this initializes the reserved
 	 * chunk.  If the reserved_size is zero, the reserved chunk is NULL
@@ -1771,8 +1784,8 @@ int __init pcpu_setup_first_chunk(const struct pcpu_alloc_info *ai,
 	 * pcpu_first_chunk, will always point to the chunk that serves
 	 * the dynamic region.
 	 */
-	tmp_addr = (unsigned long)base_addr + ai->static_size;
-	map_size = ai->reserved_size ?: ai->dyn_size;
+	tmp_addr = (unsigned long)base_addr + static_size;
+	map_size = ai->reserved_size ?: dyn_size;
 	chunk = pcpu_alloc_first_chunk(tmp_addr, map_size, smap,
 				       ARRAY_SIZE(smap));
 
@@ -1780,9 +1793,9 @@ int __init pcpu_setup_first_chunk(const struct pcpu_alloc_info *ai,
 	if (ai->reserved_size) {
 		pcpu_reserved_chunk = chunk;
 
-		tmp_addr = (unsigned long)base_addr + ai->static_size +
+		tmp_addr = (unsigned long)base_addr + static_size +
 			   ai->reserved_size;
-		map_size = ai->dyn_size;
+		map_size = dyn_size;
 		chunk = pcpu_alloc_first_chunk(tmp_addr, map_size, dmap,
 					       ARRAY_SIZE(dmap));
 	}
-- 
2.9.3

[toc] | [prev] | [next] | [standalone]

#1696027 — Re: [PATCH v2 12/23] percpu: increase minimum percpu allocation size and align first regions

On Mon, Jul 24, 2017 at 07:02:09PM -0400, Dennis Zhou wrote:
> From: "Dennis Zhou (Facebook)" <dennisszhou@gmail.com>
> 
> This patch increases the minimum allocation size of percpu memory to
> 4-bytes. This change will help minimize the metadata overhead
> associated with the bitmap allocator. The assumption is that most
> allocations will be of objects or structs greater than 2 bytes with
> integers or longs being used rather than shorts.
> 
> The first chunk regions are now aligned with the minimum allocation
> size. The reserved region is expected to be set as a multiple of the
> minimum allocation size. The static region is aligned up and the delta
> is removed from the dynamic size. This works because the dynamic size is
> increased to be page aligned. If the static size is not minimum
> allocation size aligned, then there must be a gap that is added to the
> dynamic size. The dynamic size will never be smaller than the set value.
> 
> Signed-off-by: Dennis Zhou <dennisszhou@gmail.com>

Reviewed-by: Josef Bacik <jbacik@fb.com>

Thanks,

Josef

[toc] | [prev] | [next] | [standalone]

#1695258 — [PATCH v2 01/23] percpu: setup_first_chunk enforce dynamic region must exist

From: "Dennis Zhou (Facebook)" <dennisszhou@gmail.com>

The first chunk is handled as a special case as it is composed of the
static, reserved, and dynamic regions. The code handles each case
individually. The next several patches will merge these code paths and
lay the foundation for the bitmap allocator.

This patch modifies logic to enforce that a dynamic region exists and
changes the area map to account for that. This brings the logic closer
to the dynamic chunk's init logic.

Signed-off-by: Dennis Zhou <dennisszhou@gmail.com>
---
 mm/percpu.c | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/mm/percpu.c b/mm/percpu.c
index 29244fb..3602d41 100644
--- a/mm/percpu.c
+++ b/mm/percpu.c
@@ -1598,6 +1598,7 @@ int __init pcpu_setup_first_chunk(const struct pcpu_alloc_info *ai,
 	PCPU_SETUP_BUG_ON(offset_in_page(ai->unit_size));
 	PCPU_SETUP_BUG_ON(ai->unit_size < PCPU_MIN_UNIT_SIZE);
 	PCPU_SETUP_BUG_ON(ai->dyn_size < PERCPU_DYNAMIC_EARLY_SIZE);
+	PCPU_SETUP_BUG_ON(!ai->dyn_size);
 	PCPU_SETUP_BUG_ON(pcpu_verify_alloc_info(ai) < 0);
 
 	/* process group information and build config tables accordingly */
@@ -1700,14 +1701,12 @@ int __init pcpu_setup_first_chunk(const struct pcpu_alloc_info *ai,
 		schunk->free_size = dyn_size;
 		dyn_size = 0;			/* dynamic area covered */
 	}
-	schunk->contig_hint = schunk->free_size;
 
+	schunk->contig_hint = schunk->free_size;
 	schunk->map[0] = 1;
 	schunk->map[1] = ai->static_size;
-	schunk->map_used = 1;
-	if (schunk->free_size)
-		schunk->map[++schunk->map_used] = ai->static_size + schunk->free_size;
-	schunk->map[schunk->map_used] |= 1;
+	schunk->map[2] = (ai->static_size + schunk->free_size) | 1;
+	schunk->map_used = 2;
 	schunk->has_reserved = true;
 
 	/* init dynamic chunk if necessary */
-- 
2.9.3

[toc] | [prev] | [next] | [standalone]

#1695991 — Re: [PATCH v2 01/23] percpu: setup_first_chunk enforce dynamic region must exist

On Mon, Jul 24, 2017 at 07:01:58PM -0400, Dennis Zhou wrote:
> From: "Dennis Zhou (Facebook)" <dennisszhou@gmail.com>
> 
> The first chunk is handled as a special case as it is composed of the
> static, reserved, and dynamic regions. The code handles each case
> individually. The next several patches will merge these code paths and
> lay the foundation for the bitmap allocator.
> 
> This patch modifies logic to enforce that a dynamic region exists and
> changes the area map to account for that. This brings the logic closer
> to the dynamic chunk's init logic.
> 
> Signed-off-by: Dennis Zhou <dennisszhou@gmail.com>

Reviewed-by: Josef Bacik <jbacik@fb.com>

Thanks,

Josef

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.kernel

csiph-web