Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1695256
| From | Dennis Zhou <dennisz@fb.com> |
|---|---|
| Newsgroups | linux.kernel |
| Subject | [PATCH v2 12/23] percpu: increase minimum percpu allocation size and align first regions |
| Date | 2017-07-25 01:10 +0200 |
| Message-ID | <u6UEc-3ig-59@gated-at.bofh.it> (permalink) |
| References | <u6UE9-3ig-3@gated-at.bofh.it> |
| Organization | linux.* mail to news gateway |
From: "Dennis Zhou (Facebook)" <dennisszhou@gmail.com>
This patch increases the minimum allocation size of percpu memory to
4-bytes. This change will help minimize the metadata overhead
associated with the bitmap allocator. The assumption is that most
allocations will be of objects or structs greater than 2 bytes with
integers or longs being used rather than shorts.
The first chunk regions are now aligned with the minimum allocation
size. The reserved region is expected to be set as a multiple of the
minimum allocation size. The static region is aligned up and the delta
is removed from the dynamic size. This works because the dynamic size is
increased to be page aligned. If the static size is not minimum
allocation size aligned, then there must be a gap that is added to the
dynamic size. The dynamic size will never be smaller than the set value.
Signed-off-by: Dennis Zhou <dennisszhou@gmail.com>
---
include/linux/percpu.h | 4 ++++
mm/percpu.c | 27 ++++++++++++++++++++-------
2 files changed, 24 insertions(+), 7 deletions(-)
diff --git a/include/linux/percpu.h b/include/linux/percpu.h
index 491b3f5..90e0cb0 100644
--- a/include/linux/percpu.h
+++ b/include/linux/percpu.h
@@ -21,6 +21,10 @@
/* minimum unit size, also is the maximum supported allocation size */
#define PCPU_MIN_UNIT_SIZE PFN_ALIGN(32 << 10)
+/* minimum allocation size and shift in bytes */
+#define PCPU_MIN_ALLOC_SHIFT 2
+#define PCPU_MIN_ALLOC_SIZE (1 << PCPU_MIN_ALLOC_SHIFT)
+
/*
* Percpu allocator can serve percpu allocations before slab is
* initialized which allows slab to depend on the percpu allocator.
diff --git a/mm/percpu.c b/mm/percpu.c
index 657ab08..dc755721 100644
--- a/mm/percpu.c
+++ b/mm/percpu.c
@@ -956,10 +956,10 @@ static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved,
* We want the lowest bit of offset available for in-use/free
* indicator, so force >= 16bit alignment and make size even.
*/
- if (unlikely(align < 2))
- align = 2;
+ if (unlikely(align < PCPU_MIN_ALLOC_SIZE))
+ align = PCPU_MIN_ALLOC_SIZE;
- size = ALIGN(size, 2);
+ size = ALIGN(size, PCPU_MIN_ALLOC_SIZE);
if (unlikely(!size || size > PCPU_MIN_UNIT_SIZE || align > PAGE_SIZE ||
!is_power_of_2(align))) {
@@ -1653,6 +1653,7 @@ int __init pcpu_setup_first_chunk(const struct pcpu_alloc_info *ai,
static int smap[PERCPU_DYNAMIC_EARLY_SLOTS] __initdata;
static int dmap[PERCPU_DYNAMIC_EARLY_SLOTS] __initdata;
size_t size_sum = ai->static_size + ai->reserved_size + ai->dyn_size;
+ size_t static_size, dyn_size;
struct pcpu_chunk *chunk;
unsigned long *group_offsets;
size_t *group_sizes;
@@ -1686,6 +1687,7 @@ int __init pcpu_setup_first_chunk(const struct pcpu_alloc_info *ai,
PCPU_SETUP_BUG_ON(ai->unit_size < PCPU_MIN_UNIT_SIZE);
PCPU_SETUP_BUG_ON(ai->dyn_size < PERCPU_DYNAMIC_EARLY_SIZE);
PCPU_SETUP_BUG_ON(!ai->dyn_size);
+ PCPU_SETUP_BUG_ON(!IS_ALIGNED(ai->reserved_size, PCPU_MIN_ALLOC_SIZE));
PCPU_SETUP_BUG_ON(pcpu_verify_alloc_info(ai) < 0);
/* process group information and build config tables accordingly */
@@ -1764,6 +1766,17 @@ int __init pcpu_setup_first_chunk(const struct pcpu_alloc_info *ai,
INIT_LIST_HEAD(&pcpu_slot[i]);
/*
+ * The end of the static region needs to be aligned with the
+ * minimum allocation size as this offsets the reserved and
+ * dynamic region. The first chunk ends page aligned by
+ * expanding the dynamic region, therefore the dynamic region
+ * can be shrunk to compensate while still staying above the
+ * configured sizes.
+ */
+ static_size = ALIGN(ai->static_size, PCPU_MIN_ALLOC_SIZE);
+ dyn_size = ai->dyn_size - (static_size - ai->static_size);
+
+ /*
* Initialize first chunk.
* If the reserved_size is non-zero, this initializes the reserved
* chunk. If the reserved_size is zero, the reserved chunk is NULL
@@ -1771,8 +1784,8 @@ int __init pcpu_setup_first_chunk(const struct pcpu_alloc_info *ai,
* pcpu_first_chunk, will always point to the chunk that serves
* the dynamic region.
*/
- tmp_addr = (unsigned long)base_addr + ai->static_size;
- map_size = ai->reserved_size ?: ai->dyn_size;
+ tmp_addr = (unsigned long)base_addr + static_size;
+ map_size = ai->reserved_size ?: dyn_size;
chunk = pcpu_alloc_first_chunk(tmp_addr, map_size, smap,
ARRAY_SIZE(smap));
@@ -1780,9 +1793,9 @@ int __init pcpu_setup_first_chunk(const struct pcpu_alloc_info *ai,
if (ai->reserved_size) {
pcpu_reserved_chunk = chunk;
- tmp_addr = (unsigned long)base_addr + ai->static_size +
+ tmp_addr = (unsigned long)base_addr + static_size +
ai->reserved_size;
- map_size = ai->dyn_size;
+ map_size = dyn_size;
chunk = pcpu_alloc_first_chunk(tmp_addr, map_size, dmap,
ARRAY_SIZE(dmap));
}
--
2.9.3
Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
[PATCH v2 00/23] percpu: replace percpu area map allocator with bitmap allocator Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:10 +0200
[PATCH v2 22/23] percpu: update pcpu_find_block_fit to use an iterator Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:10 +0200
Re: [PATCH v2 22/23] percpu: update pcpu_find_block_fit to use an iterator Josef Bacik <josef@toxicpanda.com> - 2017-07-25 21:50 +0200
[PATCH v2 09/23] percpu: combine percpu address checks Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:10 +0200
Re: [PATCH v2 09/23] percpu: combine percpu address checks Josef Bacik <josef@toxicpanda.com> - 2017-07-25 20:30 +0200
[PATCH v2 02/23] percpu: introduce start_offset to pcpu_chunk Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:10 +0200
Re: [PATCH v2 02/23] percpu: introduce start_offset to pcpu_chunk Josef Bacik <josef@toxicpanda.com> - 2017-07-25 20:10 +0200
[PATCH v2 13/23] percpu: generalize bitmap (un)populated iterators Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:10 +0200
Re: [PATCH v2 13/23] percpu: generalize bitmap (un)populated iterators Josef Bacik <josef@toxicpanda.com> - 2017-07-25 20:40 +0200
Re: [PATCH v2 13/23] percpu: generalize bitmap (un)populated iterators Tejun Heo <tj@kernel.org> - 2017-07-26 16:10 +0200
[PATCH v2 08/23] percpu: modify base_addr to be region specific Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:10 +0200
Re: [PATCH v2 08/23] percpu: modify base_addr to be region specific Josef Bacik <josef@toxicpanda.com> - 2017-07-25 20:30 +0200
[PATCH v2 17/23] percpu: skip chunks if the alloc does not fit in the contig hint Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:10 +0200
Re: [PATCH v2 17/23] percpu: skip chunks if the alloc does not fit in the contig hint Josef Bacik <josef@toxicpanda.com> - 2017-07-25 21:50 +0200
[PATCH v2 19/23] percpu: update alloc path to only scan if contig hints are broken Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:10 +0200
Re: [PATCH v2 19/23] percpu: update alloc path to only scan if contig hints are broken Josef Bacik <josef@toxicpanda.com> - 2017-07-25 21:40 +0200
[PATCH v2 12/23] percpu: increase minimum percpu allocation size and align first regions Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:10 +0200
Re: [PATCH v2 12/23] percpu: increase minimum percpu allocation size and align first regions Josef Bacik <josef@toxicpanda.com> - 2017-07-25 20:40 +0200
[PATCH v2 01/23] percpu: setup_first_chunk enforce dynamic region must exist Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:20 +0200
Re: [PATCH v2 01/23] percpu: setup_first_chunk enforce dynamic region must exist Josef Bacik <josef@toxicpanda.com> - 2017-07-25 20:10 +0200
[PATCH v2 06/23] percpu: end chunk area maps page aligned for the populated bitmap Dennis Zhou <dennisz@fb.com> - 2017-07-25 01:20 +0200
Re: [PATCH v2 06/23] percpu: end chunk area maps page aligned for the populated bitmap Josef Bacik <josef@toxicpanda.com> - 2017-07-25 20:20 +0200
Re: [PATCH v2 14/23] percpu: replace area map allocator with bitmap allocator Josef Bacik <josef@toxicpanda.com> - 2017-07-25 21:20 +0200
csiph-web