Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1695246 > unrolled thread

[PATCH v2 00/23] percpu: replace percpu area map allocator with bitmap allocator

Back to article view | Back to linux.kernel

Contents

  [PATCH v2 00/23] percpu: replace percpu area map allocator with bitmap allocator Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:10 +0200
    [PATCH v2 22/23] percpu: update pcpu_find_block_fit to use an iterator Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:10 +0200
      Re: [PATCH v2 22/23] percpu: update pcpu_find_block_fit to use an  iterator Josef Bacik <josef@toxicpanda.com> - 2017-07-25 21:50 +0200
    [PATCH v2 09/23] percpu: combine percpu address checks Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:10 +0200
      Re: [PATCH v2 09/23] percpu: combine percpu address checks Josef Bacik <josef@toxicpanda.com> - 2017-07-25 20:30 +0200
    [PATCH v2 02/23] percpu: introduce start_offset to pcpu_chunk Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:10 +0200
      Re: [PATCH v2 02/23] percpu: introduce start_offset to pcpu_chunk Josef Bacik <josef@toxicpanda.com> - 2017-07-25 20:10 +0200
    [PATCH v2 13/23] percpu: generalize bitmap (un)populated iterators Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:10 +0200
      Re: [PATCH v2 13/23] percpu: generalize bitmap (un)populated  iterators Josef Bacik <josef@toxicpanda.com> - 2017-07-25 20:40 +0200
      Re: [PATCH v2 13/23] percpu: generalize bitmap (un)populated  iterators Tejun Heo <tj@kernel.org> - 2017-07-26 16:10 +0200
    [PATCH v2 08/23] percpu: modify base_addr to be region specific Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:10 +0200
      Re: [PATCH v2 08/23] percpu: modify base_addr to be region specific Josef Bacik <josef@toxicpanda.com> - 2017-07-25 20:30 +0200
    [PATCH v2 17/23] percpu: skip chunks if the alloc does not fit in the contig hint Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:10 +0200
      Re: [PATCH v2 17/23] percpu: skip chunks if the alloc does not fit  in the contig hint Josef Bacik <josef@toxicpanda.com> - 2017-07-25 21:50 +0200
    [PATCH v2 19/23] percpu: update alloc path to only scan if contig hints are broken Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:10 +0200
      Re: [PATCH v2 19/23] percpu: update alloc path to only scan if  contig hints are broken Josef Bacik <josef@toxicpanda.com> - 2017-07-25 21:40 +0200
    [PATCH v2 12/23] percpu: increase minimum percpu allocation size and align first regions Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:10 +0200
      Re: [PATCH v2 12/23] percpu: increase minimum percpu allocation size  and align first regions Josef Bacik <josef@toxicpanda.com> - 2017-07-25 20:40 +0200
    [PATCH v2 01/23] percpu: setup_first_chunk enforce dynamic region must exist Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:20 +0200
      Re: [PATCH v2 01/23] percpu: setup_first_chunk enforce dynamic  region must exist Josef Bacik <josef@toxicpanda.com> - 2017-07-25 20:10 +0200
    [PATCH v2 06/23] percpu: end chunk area maps page aligned for the populated bitmap Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:20 +0200
      Re: [PATCH v2 06/23] percpu: end chunk area maps page aligned for  the populated bitmap Josef Bacik <josef@toxicpanda.com> - 2017-07-25 20:20 +0200
    Re: [PATCH v2 14/23] percpu: replace area map allocator with bitmap  allocator Josef Bacik <josef@toxicpanda.com> - 2017-07-25 21:20 +0200

Page 2 of 2 — ← Prev page 1 [2]

#1695260 — [PATCH v2 06/23] percpu: end chunk area maps page aligned for the populated bitmap

From: "Dennis Zhou (Facebook)" <dennisszhou@gmail.com>

The area map allocator manages the first chunk area by hiding all but
the region it is responsible for serving in the area map. To align this
with the populated page bitmap, end_offset is introduced to keep track
of the delta to end page aligned. The area map is appended with the
page aligned end when necessary to be in line with how the bitmap
allocator requires the ending to be aligned with the LCM of PAGE_SIZE
and the size of each bitmap block. percpu_stats is updated to ignore
this region when present.

Signed-off-by: Dennis Zhou <dennisszhou@gmail.com>
---
 mm/percpu-internal.h | 3 +++
 mm/percpu-stats.c    | 5 +++--
 mm/percpu.c          | 9 +++++++++
 3 files changed, 15 insertions(+), 2 deletions(-)

diff --git a/mm/percpu-internal.h b/mm/percpu-internal.h
index c876b5b..f02f31c 100644
--- a/mm/percpu-internal.h
+++ b/mm/percpu-internal.h
@@ -26,6 +26,9 @@ struct pcpu_chunk {
 	int			start_offset;	/* the overlap with the previous
 						   region to have a page aligned
 						   base_addr */
+	int			end_offset;	/* additional area required to
+						   have the region end page
+						   aligned */
 	int			nr_populated;	/* # of populated pages */
 	unsigned long		populated[];	/* populated bitmap */
 };
diff --git a/mm/percpu-stats.c b/mm/percpu-stats.c
index 32f3550..ffbdb96 100644
--- a/mm/percpu-stats.c
+++ b/mm/percpu-stats.c
@@ -51,7 +51,7 @@ static int find_max_map_used(void)
 static void chunk_map_stats(struct seq_file *m, struct pcpu_chunk *chunk,
 			    int *buffer)
 {
-	int i, s_index, last_alloc, alloc_sign, as_len;
+	int i, s_index, e_index, last_alloc, alloc_sign, as_len;
 	int *alloc_sizes, *p;
 	/* statistics */
 	int sum_frag = 0, max_frag = 0;
@@ -59,10 +59,11 @@ static void chunk_map_stats(struct seq_file *m, struct pcpu_chunk *chunk,
 
 	alloc_sizes = buffer;
 	s_index = (chunk->start_offset) ? 1 : 0;
+	e_index = chunk->map_used - ((chunk->end_offset) ? 1 : 0);
 
 	/* find last allocation */
 	last_alloc = -1;
-	for (i = chunk->map_used - 1; i >= s_index; i--) {
+	for (i = e_index - 1; i >= s_index; i--) {
 		if (chunk->map[i] & 1) {
 			last_alloc = i;
 			break;
diff --git a/mm/percpu.c b/mm/percpu.c
index 2e785a7..1d2c980 100644
--- a/mm/percpu.c
+++ b/mm/percpu.c
@@ -715,12 +715,16 @@ static struct pcpu_chunk * __init pcpu_alloc_first_chunk(void *base_addr,
 							 int init_map_size)
 {
 	struct pcpu_chunk *chunk;
+	int region_size;
+
+	region_size = PFN_ALIGN(start_offset + map_size);
 
 	chunk = memblock_virt_alloc(pcpu_chunk_struct_size, 0);
 	INIT_LIST_HEAD(&chunk->list);
 	INIT_LIST_HEAD(&chunk->map_extend_list);
 	chunk->base_addr = base_addr;
 	chunk->start_offset = start_offset;
+	chunk->end_offset = region_size - chunk->start_offset - map_size;
 	chunk->map = map;
 	chunk->map_alloc = init_map_size;
 
@@ -735,6 +739,11 @@ static struct pcpu_chunk * __init pcpu_alloc_first_chunk(void *base_addr,
 	chunk->map[2] = (chunk->start_offset + chunk->free_size) | 1;
 	chunk->map_used = 2;
 
+	if (chunk->end_offset) {
+		/* hide the end of the bitmap */
+		chunk->map[++chunk->map_used] = region_size | 1;
+	}
+
 	return chunk;
 }
 
-- 
2.9.3

[toc] | [prev] | [next] | [standalone]

#1696005 — Re: [PATCH v2 06/23] percpu: end chunk area maps page aligned for the populated bitmap

On Mon, Jul 24, 2017 at 07:02:03PM -0400, Dennis Zhou wrote:
> From: "Dennis Zhou (Facebook)" <dennisszhou@gmail.com>
> 
> The area map allocator manages the first chunk area by hiding all but
> the region it is responsible for serving in the area map. To align this
> with the populated page bitmap, end_offset is introduced to keep track
> of the delta to end page aligned. The area map is appended with the
> page aligned end when necessary to be in line with how the bitmap
> allocator requires the ending to be aligned with the LCM of PAGE_SIZE
> and the size of each bitmap block. percpu_stats is updated to ignore
> this region when present.
> 
> Signed-off-by: Dennis Zhou <dennisszhou@gmail.com>

Reviewed-by: Josef Bacik <jbacik@fb.com>

Thanks,

Josef

[toc] | [prev] | [next] | [standalone]

#1696052 — Re: [PATCH v2 14/23] percpu: replace area map allocator with bitmap allocator

On Mon, Jul 24, 2017 at 07:02:11PM -0400, Dennis Zhou wrote:
> From: "Dennis Zhou (Facebook)" <dennisszhou@gmail.com>
> 
> The percpu memory allocator is experiencing scalability issues when
> allocating and freeing large numbers of counters as in BPF.
> Additionally, there is a corner case where iteration is triggered over
> all chunks if the contig_hint is the right size, but wrong alignment.
> 
> This patch replaces the area map allocator with a basic bitmap allocator
> implementation. Each subsequent patch will introduce new features and
> replace full scanning functions with faster non-scanning options when
> possible.
> 
> Implementation:
> This patchset removes the area map allocator in favor of a bitmap
> allocator backed by metadata blocks. The primary goal is to provide
> consistency in performance and memory footprint with a focus on small
> allocations (< 64 bytes). The bitmap removes the heavy memmove from the
> freeing critical path and provides a consistent memory footprint. The
> metadata blocks provide a bound on the amount of scanning required by
> maintaining a set of hints.
> 
> In an effort to make freeing fast, the metadata is updated on the free
> path if the new free area makes a page free, a block free, or spans
> across blocks. This causes the chunk's contig hint to potentially be
> smaller than what it could allocate by up to the smaller of a page or a
> block. If the chunk's contig hint is contained within a block, a check
> occurs and the hint is kept accurate. Metadata is always kept accurate
> on allocation, so there will not be a situation where a chunk has a
> later contig hint than available.
> 
> Evaluation:
> I have primarily done testing against a simple workload of allocation of
> 1 million objects (2^20) of varying size. Deallocation was done by in
> order, alternating, and in reverse. These numbers were collected after
> rebasing ontop of a80099a152. I present the worst-case numbers here:
> 
>   Area Map Allocator:
> 
>         Object Size | Alloc Time (ms) | Free Time (ms)
>         ----------------------------------------------
>               4B    |        310      |     4770
>              16B    |        557      |     1325
>              64B    |        436      |      273
>             256B    |        776      |      131
>            1024B    |       3280      |      122
> 
>   Bitmap Allocator:
> 
>         Object Size | Alloc Time (ms) | Free Time (ms)
>         ----------------------------------------------
>               4B    |        490      |       70
>              16B    |        515      |       75
>              64B    |        610      |       80
>             256B    |        950      |      100
>            1024B    |       3520      |      200
> 
> This data demonstrates the inability for the area map allocator to
> handle less than ideal situations. In the best case of reverse
> deallocation, the area map allocator was able to perform within range
> of the bitmap allocator. In the worst case situation, freeing took
> nearly 5 seconds for 1 million 4-byte objects. The bitmap allocator
> dramatically improves the consistency of the free path. The small
> allocations performed nearly identical regardless of the freeing
> pattern.
> 
> While it does add to the allocation latency, the allocation scenario
> here is optimal for the area map allocator. The area map allocator runs
> into trouble when it is allocating in chunks where the latter half is
> full. It is difficult to replicate this, so I present a variant where
> the pages are second half filled. Freeing was done sequentially. Below
> are the numbers for this scenario:
> 
>   Area Map Allocator:
> 
>         Object Size | Alloc Time (ms) | Free Time (ms)
>         ----------------------------------------------
>               4B    |       4118      |     4892
>              16B    |       1651      |     1163
>              64B    |        598      |      285
>             256B    |        771      |      158
>            1024B    |       3034      |      160
> 
>   Bitmap Allocator:
> 
>         Object Size | Alloc Time (ms) | Free Time (ms)
>         ----------------------------------------------
>               4B    |        481      |       67
>              16B    |        506      |       69
>              64B    |        636      |       75
>             256B    |        892      |       90
>            1024B    |       3262      |      147
> 
> The data shows a parabolic curve of performance for the area map
> allocator. This is due to the memmove operation being the dominant cost
> with the lower object sizes as more objects are packed in a chunk and at
> higher object sizes, the traversal of the chunk slots is the dominating
> cost. The bitmap allocator suffers this problem as well. The above data
> shows the inability to scale for the allocation path with the area map
> allocator and that the bitmap allocator demonstrates consistent
> performance in general.
> 
> The second problem of additional scanning can result in the area map
> allocator completing in 52 minutes when trying to allocate 1 million
> 4-byte objects with 8-byte alignment. The same workload takes
> approximately 16 seconds to complete for the bitmap allocator.
> 
> Signed-off-by: Dennis Zhou <dennisszhou@gmail.com>

Once you fix that init thing and the comment thing you can add

Reviewed-by: Josef Bacik <jbacik@fb.com>

Thanks,

Josef

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.kernel

csiph-web