Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1531607 > unrolled thread
| Started by | Matthew Wilcox <mawilcox@linuxonhyperv.com> |
|---|---|
| First post | 2016-11-28 21:10 +0100 |
| Last post | 2016-12-06 23:00 +0100 |
| Articles | 20 on this page of 40 — 5 participants |
Back to article view | Back to linux.kernel
[PATCH v3 00/33] Radix tree patches for 4.10 Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:10 +0100
[PATCH v3 26/33] radix-tree: Fix replacement for multiorder entries Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:10 +0100
[PATCH v3 00/33] Radix tree patches for 4.10 Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:10 +0100
[PATCH v3 27/33] radix tree test suite: Check multiorder iteration Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:10 +0100
[PATCH v3 15/33] radix-tree: Move rcu_head into a union with private_list Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:10 +0100
[PATCH v3 23/33] radix-tree: Add radix_tree_join Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:10 +0100
[PATCH v3 09/33] radix tree test suite: Use rcu_barrier Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:10 +0100
[PATCH v3 16/33] radix-tree: Create node_tag_set() Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:10 +0100
[PATCH v3 25/33] radix-tree: Add radix_tree_split_preload() Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:10 +0100
[PATCH v3 09/33] radix tree test suite: Use rcu_barrier Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:10 +0100
[PATCH v3 12/33] tools: Add more bitmap functions Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:10 +0100
[PATCH v3 16/33] radix-tree: Create node_tag_set() Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:10 +0100
[PATCH v3 27/33] radix tree test suite: Check multiorder iteration Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:10 +0100
[PATCH v3 21/33] radix-tree: Delete radix_tree_locate_item() Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:10 +0100
[PATCH v3 08/33] radix tree test suite: benchmark for iterator Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:20 +0100
[PATCH v3 22/33] radix-tree: Delete radix_tree_range_tag_if_tagged() Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:20 +0100
[PATCH v3 17/33] radix-tree: Make radix_tree_find_next_bit more useful Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:20 +0100
[PATCH v3 23/33] radix-tree: Add radix_tree_join Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:20 +0100
[PATCH v3 18/33] radix-tree: Improve dump output Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:20 +0100
[PATCH v3 05/33] radix tree test suite: Free preallocated nodes Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:20 +0100
[PATCH v3 06/33] radix tree test suite: Make runs more reproducible Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:20 +0100
[PATCH v3 20/33] radix-tree: Improve multiorder iterators Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:20 +0100
[PATCH v3 02/33] tools: Add WARN_ON_ONCE Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:20 +0100
[PATCH v3 14/33] radix-tree: Fix typo Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:20 +0100
[PATCH v3 03/33] radix tree test suite: Allow GFP_ATOMIC allocations to fail Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:20 +0100
[PATCH v3 12/33] tools: Add more bitmap functions Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:20 +0100
[PATCH v3 10/33] radix tree test suite: Handle exceptional entries Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:20 +0100
[PATCH v3 24/33] radix-tree: Add radix_tree_split Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:20 +0100
Re: [PATCH v3 24/33] radix-tree: Add radix_tree_split Randy Dunlap <rdunlap@infradead.org> - 2016-11-29 01:10 +0100
RE: [PATCH v3 24/33] radix-tree: Add radix_tree_split Matthew Wilcox <mawilcox@microsoft.com> - 2016-11-29 16:00 +0100
[PATCH v3 15/33] radix-tree: Move rcu_head into a union with private_list Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:20 +0100
[PATCH v3 11/33] radix tree test suite: record order in each item Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:20 +0100
[PATCH v3 01/33] radix tree test suite: Fix compilation Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:20 +0100
[PATCH v3 08/33] radix tree test suite: benchmark for iterator Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:20 +0100
[PATCH v3 17/33] radix-tree: Make radix_tree_find_next_bit more useful Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:20 +0100
[PATCH v3 10/33] radix tree test suite: Handle exceptional entries Matthew Wilcox <mawilcox@linuxonhyperv.com> - 2016-11-28 21:20 +0100
Re: [PATCH v3 00/33] Radix tree patches for 4.10 "Kirill A. Shutemov" <kirill@shutemov.name> - 2016-11-29 12:30 +0100
Re: [PATCH v3 33/33] Reimplement IDR and IDA using the radix tree Andrew Morton <akpm@linux-foundation.org> - 2016-12-06 21:50 +0100
Re: [PATCH v3 33/33] Reimplement IDR and IDA using the radix tree Andrew Morton <akpm@linux-foundation.org> - 2016-12-06 22:30 +0100
RE: [PATCH v3 33/33] Reimplement IDR and IDA using the radix tree Matthew Wilcox <mawilcox@microsoft.com> - 2016-12-06 23:00 +0100
Page 1 of 2 [1] 2 Next page →
| From | Matthew Wilcox <mawilcox@linuxonhyperv.com> |
|---|---|
| Date | 2016-11-28 21:10 +0100 |
| Subject | [PATCH v3 00/33] Radix tree patches for 4.10 |
| Message-ID | <sIAfL-6sT-3@gated-at.bofh.it> |
From: Matthew Wilcox <mawilcox@microsoft.com> Hi Andrew, Please include these patches in the -mm tree for 4.10. Mostly these are improvements; the only bug fixes in here relate to multiorder entries (which are unused in the 4.9 tree). The IDR rewrite has the highest potential for causing mayhem as the test suite is quite spartan. We have an Outreachy intern scheduled to work on the test suite for the 2016 winter season, so hopefully it will improve soon. I did not include Konstantin's suggested change to the API for radix_tree_iter_resume(). Many of the callers do not currently care about the size of the entry they are consuming, and determining that information is not always trivial. Since this is not a performance critical API (it's called when we've paused iterating through a tree in order to schedule for a higher priority task), I think it's more important to have a simpler interface. I'd like to thank Kiryl for all the testing he's been doing. He's found at least two bugs which weren't related to the API extensions that he really wanted from this patch set. Noteworthy changes from v2: - Rebased on latest mmots tree - Various fixes to accommodate Johannes' exceptional entry accounting changes - Renamed radix_tree_iter_next() to radix_tree_iter_resume() - Add kerneldoc to radix_tree_iter_join(), radix_tree_iter_split(), and partially to struct radix_tree_node. - Factor set_iter_tags() out of radix_tree_next_chunk() and use it in __radix_tree_next_slot(). - Rename __radix_tree_iter_next() to skip_siblings() and only compile it in when MULTIORDER is supported. Test suite: - Record item order - Added an IDR replace test - Improve iteration test to handle multiorder entries - More tests to check exceptional entry accounting - Handle exceptional entries in item_kill_tree() Konstantin Khlebnikov (1): radix tree test suite: benchmark for iterator Matthew Wilcox (32): radix tree test suite: Fix compilation tools: Add WARN_ON_ONCE radix tree test suite: Allow GFP_ATOMIC allocations to fail radix tree test suite: Track preempt_count radix tree test suite: Free preallocated nodes radix tree test suite: Make runs more reproducible radix tree test suite: iteration test misuses RCU radix tree test suite: Use rcu_barrier radix tree test suite: Handle exceptional entries radix tree test suite: record order in each item tools: Add more bitmap functions radix tree test suite: Use common find-bit code radix-tree: Fix typo radix-tree: Move rcu_head into a union with private_list radix-tree: Create node_tag_set() radix-tree: Make radix_tree_find_next_bit more useful radix-tree: Improve dump output btrfs: Fix race in btrfs_free_dummy_fs_info() radix-tree: Improve multiorder iterators radix-tree: Delete radix_tree_locate_item() radix-tree: Delete radix_tree_range_tag_if_tagged() radix-tree: Add radix_tree_join radix-tree: Add radix_tree_split radix-tree: Add radix_tree_split_preload() radix-tree: Fix replacement for multiorder entries radix tree test suite: Check multiorder iteration idr: Add ida_is_empty tpm: Use idr_find(), not idr_find_slowpath() rxrpc: Abstract away knowledge of IDR internals idr: Reduce the number of bits per level from 8 to 6 radix tree test suite: Add some more functionality Reimplement IDR and IDA using the radix tree drivers/char/tpm/tpm-chip.c | 4 +- drivers/usb/gadget/function/f_hid.c | 6 +- drivers/usb/gadget/function/f_printer.c | 6 +- fs/btrfs/tests/btrfs-tests.c | 1 + include/linux/idr.h | 156 +- include/linux/radix-tree.h | 179 +-- init/main.c | 3 +- lib/idr.c | 1078 -------------- lib/radix-tree.c | 1526 +++++++++++++++----- mm/khugepaged.c | 7 +- mm/page-writeback.c | 28 +- mm/shmem.c | 32 +- net/rxrpc/af_rxrpc.c | 11 +- net/rxrpc/conn_client.c | 4 +- tools/include/asm/bug.h | 11 + tools/include/linux/bitmap.h | 26 + tools/testing/radix-tree/Makefile | 18 +- tools/testing/radix-tree/benchmark.c | 98 ++ tools/testing/radix-tree/find_next_bit.c | 57 - tools/testing/radix-tree/idr.c | 160 ++ tools/testing/radix-tree/iteration_check.c | 123 +- tools/testing/radix-tree/linux.c | 23 +- tools/testing/radix-tree/linux/bitops.h | 40 +- tools/testing/radix-tree/linux/bitops/non-atomic.h | 13 +- tools/testing/radix-tree/linux/bug.h | 2 +- tools/testing/radix-tree/linux/cpu.h | 22 +- tools/testing/radix-tree/linux/gfp.h | 22 +- tools/testing/radix-tree/linux/idr.h | 1 + tools/testing/radix-tree/linux/kernel.h | 20 + tools/testing/radix-tree/linux/notifier.h | 8 - tools/testing/radix-tree/linux/preempt.h | 6 +- tools/testing/radix-tree/linux/slab.h | 6 +- tools/testing/radix-tree/linux/types.h | 2 - tools/testing/radix-tree/main.c | 83 +- tools/testing/radix-tree/multiorder.c | 289 +++- tools/testing/radix-tree/regression2.c | 3 +- tools/testing/radix-tree/regression3.c | 8 +- tools/testing/radix-tree/tag_check.c | 9 +- tools/testing/radix-tree/test.c | 92 +- tools/testing/radix-tree/test.h | 23 +- 40 files changed, 2311 insertions(+), 1895 deletions(-) create mode 100644 tools/testing/radix-tree/benchmark.c delete mode 100644 tools/testing/radix-tree/find_next_bit.c create mode 100644 tools/testing/radix-tree/idr.c create mode 100644 tools/testing/radix-tree/linux/idr.h delete mode 100644 tools/testing/radix-tree/linux/notifier.h -- 2.10.2
[toc] | [next] | [standalone]
| From | Matthew Wilcox <mawilcox@linuxonhyperv.com> |
|---|---|
| Date | 2016-11-28 21:10 +0100 |
| Subject | [PATCH v3 26/33] radix-tree: Fix replacement for multiorder entries |
| Message-ID | <sIApt-6LC-59@gated-at.bofh.it> |
| In reply to | #1531607 |
From: Matthew Wilcox <mawilcox@microsoft.com>
When replacing an entry with NULL, we need to delete any sibling entries.
Also account deleting exceptional entries properly.
Also fix a bug with radix_tree_iter_replace() where we would fail to
remove entirely freed nodes.
Also fix accounting bug when switching between normal and exceptional
entries with replace_slot.
Also add testcases for all these bugs.
Signed-off-by: Matthew Wilcox <mawilcox@microsoft.com>
---
lib/radix-tree.c | 60 +++++++++++++++++-------
tools/testing/radix-tree/multiorder.c | 87 ++++++++++++++++++++++++++++++-----
2 files changed, 119 insertions(+), 28 deletions(-)
diff --git a/lib/radix-tree.c b/lib/radix-tree.c
index 9d24bec..a227727 100644
--- a/lib/radix-tree.c
+++ b/lib/radix-tree.c
@@ -976,6 +976,24 @@ void *radix_tree_lookup(struct radix_tree_root *root, unsigned long index)
}
EXPORT_SYMBOL(radix_tree_lookup);
+static inline int slot_count(struct radix_tree_node *node,
+ void **slot)
+{
+ int n = 1;
+#ifdef CONFIG_RADIX_TREE_MULTIORDER
+ void *ptr = node_to_entry(slot);
+ unsigned offset = get_slot_offset(node, slot);
+ int i;
+
+ for (i = 1; offset + i < RADIX_TREE_MAP_SIZE; i++) {
+ if (node->slots[offset + i] != ptr)
+ break;
+ n++;
+ }
+#endif
+ return n;
+}
+
static void replace_slot(struct radix_tree_root *root,
struct radix_tree_node *node,
void **slot, void *item,
@@ -994,12 +1012,35 @@ static void replace_slot(struct radix_tree_root *root,
if (node) {
node->count += count;
- node->exceptional += exceptional;
+ if (exceptional) {
+ exceptional *= slot_count(node, slot);
+ node->exceptional += exceptional;
+ }
}
rcu_assign_pointer(*slot, item);
}
+static inline void delete_sibling_entries(struct radix_tree_node *node,
+ void **slot)
+{
+#ifdef CONFIG_RADIX_TREE_MULTIORDER
+ bool exceptional = radix_tree_exceptional_entry(*slot);
+ void *ptr = node_to_entry(slot);
+ unsigned offset = get_slot_offset(node, slot);
+ int i;
+
+ for (i = 1; offset + i < RADIX_TREE_MAP_SIZE; i++) {
+ if (node->slots[offset + i] != ptr)
+ break;
+ node->slots[offset + i] = NULL;
+ node->count--;
+ if (exceptional)
+ node->exceptional--;
+ }
+#endif
+}
+
/**
* __radix_tree_replace - replace item in a slot
* @root: radix tree root
@@ -1017,6 +1058,8 @@ void __radix_tree_replace(struct radix_tree_root *root,
void **slot, void *item,
radix_tree_update_node_t update_node, void *private)
{
+ if (!item)
+ delete_sibling_entries(node, slot);
/*
* This function supports replacing exceptional entries and
* deleting entries, but that needs accounting against the
@@ -1793,20 +1836,6 @@ void __radix_tree_delete_node(struct radix_tree_root *root,
delete_node(root, node, NULL, NULL);
}
-static inline void delete_sibling_entries(struct radix_tree_node *node,
- void *ptr, unsigned offset)
-{
-#ifdef CONFIG_RADIX_TREE_MULTIORDER
- int i;
- for (i = 1; offset + i < RADIX_TREE_MAP_SIZE; i++) {
- if (node->slots[offset + i] != ptr)
- break;
- node->slots[offset + i] = NULL;
- node->count--;
- }
-#endif
-}
-
/**
* radix_tree_delete_item - delete an item from a radix tree
* @root: radix tree root
@@ -1846,7 +1875,6 @@ void *radix_tree_delete_item(struct radix_tree_root *root,
for (tag = 0; tag < RADIX_TREE_MAX_TAGS; tag++)
node_tag_clear(root, node, tag, offset);
- delete_sibling_entries(node, node_to_entry(slot), offset);
__radix_tree_replace(root, node, slot, NULL, NULL, NULL);
return entry;
diff --git a/tools/testing/radix-tree/multiorder.c b/tools/testing/radix-tree/multiorder.c
index 5421f01..9757b89 100644
--- a/tools/testing/radix-tree/multiorder.c
+++ b/tools/testing/radix-tree/multiorder.c
@@ -410,8 +410,6 @@ static void __multiorder_split(int old_order, int new_order)
RADIX_TREE(tree, GFP_ATOMIC);
void **slot;
struct radix_tree_iter iter;
- struct radix_tree_node *node;
- void *item;
unsigned alloc;
radix_tree_preload(GFP_KERNEL);
@@ -434,58 +432,122 @@ static void __multiorder_split(int old_order, int new_order)
radix_tree_preload_end();
item_kill_tree(&tree);
+}
+
+static void __multiorder_split2(int old_order, int new_order)
+{
+ RADIX_TREE(tree, GFP_KERNEL);
+ void **slot;
+ struct radix_tree_iter iter;
+ struct radix_tree_node *node;
+ void *item;
- radix_tree_preload(GFP_KERNEL);
__radix_tree_insert(&tree, 0, old_order, (void *)0x12);
- radix_tree_preload_end();
item = __radix_tree_lookup(&tree, 0, &node, NULL);
assert(item == (void *)0x12);
assert(node->exceptional > 0);
- radix_tree_split_preload(old_order, new_order, GFP_KERNEL);
radix_tree_split(&tree, 0, new_order);
radix_tree_for_each_slot(slot, &tree, &iter, 0) {
radix_tree_iter_replace(&tree, &iter, slot,
item_create(iter.index, new_order));
}
- radix_tree_preload_end();
item = __radix_tree_lookup(&tree, 0, &node, NULL);
assert(item != (void *)0x12);
assert(node->exceptional == 0);
item_kill_tree(&tree);
+}
+
+static void __multiorder_split3(int old_order, int new_order)
+{
+ RADIX_TREE(tree, GFP_KERNEL);
+ void **slot;
+ struct radix_tree_iter iter;
+ struct radix_tree_node *node;
+ void *item;
- radix_tree_preload(GFP_KERNEL);
__radix_tree_insert(&tree, 0, old_order, (void *)0x12);
- radix_tree_preload_end();
item = __radix_tree_lookup(&tree, 0, &node, NULL);
assert(item == (void *)0x12);
assert(node->exceptional > 0);
- radix_tree_split_preload(old_order, new_order, GFP_KERNEL);
radix_tree_split(&tree, 0, new_order);
radix_tree_for_each_slot(slot, &tree, &iter, 0) {
radix_tree_iter_replace(&tree, &iter, slot, (void *)0x16);
}
- radix_tree_preload_end();
item = __radix_tree_lookup(&tree, 0, &node, NULL);
assert(item == (void *)0x16);
assert(node->exceptional > 0);
item_kill_tree(&tree);
+
+ __radix_tree_insert(&tree, 0, old_order, (void *)0x12);
+
+ item = __radix_tree_lookup(&tree, 0, &node, NULL);
+ assert(item == (void *)0x12);
+ assert(node->exceptional > 0);
+
+ radix_tree_split(&tree, 0, new_order);
+ radix_tree_for_each_slot(slot, &tree, &iter, 0) {
+ if (iter.index == (1 << new_order))
+ radix_tree_iter_replace(&tree, &iter, slot,
+ (void *)0x16);
+ else
+ radix_tree_iter_replace(&tree, &iter, slot, NULL);
+ }
+
+ item = __radix_tree_lookup(&tree, 1 << new_order, &node, NULL);
+ assert(item == (void *)0x16);
+ assert(node->count == node->exceptional);
+ do {
+ node = node->parent;
+ if (!node)
+ break;
+ assert(node->count == 1);
+ assert(node->exceptional == 0);
+ } while (1);
+
+ item_kill_tree(&tree);
}
static void multiorder_split(void)
{
int i, j;
- for (i = 9; i < 19; i++)
- for (j = 0; j < i; j++)
+ for (i = 3; i < 11; i++)
+ for (j = 0; j < i; j++) {
__multiorder_split(i, j);
+ __multiorder_split2(i, j);
+ __multiorder_split3(i, j);
+ }
+}
+
+static void multiorder_account(void)
+{
+ RADIX_TREE(tree, GFP_KERNEL);
+ struct radix_tree_node *node;
+ void **slot;
+
+ item_insert_order(&tree, 0, 5);
+
+ __radix_tree_insert(&tree, 1 << 5, 5, (void *)0x12);
+ __radix_tree_lookup(&tree, 0, &node, NULL);
+ assert(node->count == node->exceptional * 2);
+ radix_tree_delete(&tree, 1 << 5);
+ assert(node->exceptional == 0);
+
+ __radix_tree_insert(&tree, 1 << 5, 5, (void *)0x12);
+ __radix_tree_lookup(&tree, 1 << 5, &node, &slot);
+ assert(node->count == node->exceptional * 2);
+ __radix_tree_replace(&tree, node, slot, NULL, NULL, NULL);
+ assert(node->exceptional == 0);
+
+ item_kill_tree(&tree);
}
void multiorder_checks(void)
@@ -507,6 +569,7 @@ void multiorder_checks(void)
multiorder_tagged_iteration();
multiorder_join();
multiorder_split();
+ multiorder_account();
radix_tree_cpu_dead(0);
}
--
2.10.2
[toc] | [prev] | [next] | [standalone]
| From | Matthew Wilcox <mawilcox@linuxonhyperv.com> |
|---|---|
| Date | 2016-11-28 21:10 +0100 |
| Message-ID | <sIApt-6LC-61@gated-at.bofh.it> |
| In reply to | #1531607 |
From: Matthew Wilcox <mawilcox@microsoft.com> Hi Andrew, Please include these patches in the -mm tree for 4.10. Mostly these are improvements; the only bug fixes in here relate to multiorder entries (which are unused in the 4.9 tree). The IDR rewrite has the highest potential for causing mayhem as the test suite is quite spartan. We have an Outreachy intern scheduled to work on the test suite for the 2016 winter season, so hopefully it will improve soon. I did not include Konstantin's suggested change to the API for radix_tree_iter_resume(). Many of the callers do not currently care about the size of the entry they are consuming, and determining that information is not always trivial. Since this is not a performance critical API (it's called when we've paused iterating through a tree in order to schedule for a higher priority task), I think it's more important to have a simpler interface. I'd like to thank Kiryl for all the testing he's been doing. He's found at least two bugs which weren't related to the API extensions that he really wanted from this patch set. Noteworthy changes from v2: - Rebased on latest mmots tree - Various fixes to accommodate Johannes' exceptional entry accounting changes - Renamed radix_tree_iter_next() to radix_tree_iter_resume() - Add kerneldoc to radix_tree_iter_join(), radix_tree_iter_split(), and partially to struct radix_tree_node. - Factor set_iter_tags() out of radix_tree_next_chunk() and use it in __radix_tree_next_slot(). - Rename __radix_tree_iter_next() to skip_siblings() and only compile it in when MULTIORDER is supported. Test suite: - Record item order - Added an IDR replace test - Improve iteration test to handle multiorder entries - More tests to check exceptional entry accounting - Handle exceptional entries in item_kill_tree() Konstantin Khlebnikov (1): radix tree test suite: benchmark for iterator Matthew Wilcox (32): radix tree test suite: Fix compilation tools: Add WARN_ON_ONCE radix tree test suite: Allow GFP_ATOMIC allocations to fail radix tree test suite: Track preempt_count radix tree test suite: Free preallocated nodes radix tree test suite: Make runs more reproducible radix tree test suite: iteration test misuses RCU radix tree test suite: Use rcu_barrier radix tree test suite: Handle exceptional entries radix tree test suite: record order in each item tools: Add more bitmap functions radix tree test suite: Use common find-bit code radix-tree: Fix typo radix-tree: Move rcu_head into a union with private_list radix-tree: Create node_tag_set() radix-tree: Make radix_tree_find_next_bit more useful radix-tree: Improve dump output btrfs: Fix race in btrfs_free_dummy_fs_info() radix-tree: Improve multiorder iterators radix-tree: Delete radix_tree_locate_item() radix-tree: Delete radix_tree_range_tag_if_tagged() radix-tree: Add radix_tree_join radix-tree: Add radix_tree_split radix-tree: Add radix_tree_split_preload() radix-tree: Fix replacement for multiorder entries radix tree test suite: Check multiorder iteration idr: Add ida_is_empty tpm: Use idr_find(), not idr_find_slowpath() rxrpc: Abstract away knowledge of IDR internals idr: Reduce the number of bits per level from 8 to 6 radix tree test suite: Add some more functionality Reimplement IDR and IDA using the radix tree drivers/char/tpm/tpm-chip.c | 4 +- drivers/usb/gadget/function/f_hid.c | 6 +- drivers/usb/gadget/function/f_printer.c | 6 +- fs/btrfs/tests/btrfs-tests.c | 1 + include/linux/idr.h | 156 +- include/linux/radix-tree.h | 179 +-- init/main.c | 3 +- lib/idr.c | 1078 -------------- lib/radix-tree.c | 1526 +++++++++++++++----- mm/khugepaged.c | 7 +- mm/page-writeback.c | 28 +- mm/shmem.c | 32 +- net/rxrpc/af_rxrpc.c | 11 +- net/rxrpc/conn_client.c | 4 +- tools/include/asm/bug.h | 11 + tools/include/linux/bitmap.h | 26 + tools/testing/radix-tree/Makefile | 18 +- tools/testing/radix-tree/benchmark.c | 98 ++ tools/testing/radix-tree/find_next_bit.c | 57 - tools/testing/radix-tree/idr.c | 160 ++ tools/testing/radix-tree/iteration_check.c | 123 +- tools/testing/radix-tree/linux.c | 23 +- tools/testing/radix-tree/linux/bitops.h | 40 +- tools/testing/radix-tree/linux/bitops/non-atomic.h | 13 +- tools/testing/radix-tree/linux/bug.h | 2 +- tools/testing/radix-tree/linux/cpu.h | 22 +- tools/testing/radix-tree/linux/gfp.h | 22 +- tools/testing/radix-tree/linux/idr.h | 1 + tools/testing/radix-tree/linux/kernel.h | 20 + tools/testing/radix-tree/linux/notifier.h | 8 - tools/testing/radix-tree/linux/preempt.h | 6 +- tools/testing/radix-tree/linux/slab.h | 6 +- tools/testing/radix-tree/linux/types.h | 2 - tools/testing/radix-tree/main.c | 83 +- tools/testing/radix-tree/multiorder.c | 289 +++- tools/testing/radix-tree/regression2.c | 3 +- tools/testing/radix-tree/regression3.c | 8 +- tools/testing/radix-tree/tag_check.c | 9 +- tools/testing/radix-tree/test.c | 92 +- tools/testing/radix-tree/test.h | 23 +- 40 files changed, 2311 insertions(+), 1895 deletions(-) create mode 100644 tools/testing/radix-tree/benchmark.c delete mode 100644 tools/testing/radix-tree/find_next_bit.c create mode 100644 tools/testing/radix-tree/idr.c create mode 100644 tools/testing/radix-tree/linux/idr.h delete mode 100644 tools/testing/radix-tree/linux/notifier.h -- 2.10.2
[toc] | [prev] | [next] | [standalone]
| From | Matthew Wilcox <mawilcox@linuxonhyperv.com> |
|---|---|
| Date | 2016-11-28 21:10 +0100 |
| Subject | [PATCH v3 27/33] radix tree test suite: Check multiorder iteration |
| Message-ID | <sIApt-6LC-65@gated-at.bofh.it> |
| In reply to | #1531607 |
From: Matthew Wilcox <mawilcox@microsoft.com>
The random iteration test only inserts order-0 entries currently.
Update it to insert entries of order between 7 and 0. Also make the
maximum index configurable, make some variables static, make the test
duration variable, remove some useless spinning, and add a fifth thread
which calls tag_tagged_items().
Signed-off-by: Matthew Wilcox <mawilcox@microsoft.com>
---
tools/testing/radix-tree/iteration_check.c | 80 ++++++++++++++++++------------
tools/testing/radix-tree/main.c | 3 +-
tools/testing/radix-tree/multiorder.c | 23 +++++++++
tools/testing/radix-tree/test.h | 2 +-
4 files changed, 73 insertions(+), 35 deletions(-)
diff --git a/tools/testing/radix-tree/iteration_check.c b/tools/testing/radix-tree/iteration_check.c
index f328a66..7572b7e 100644
--- a/tools/testing/radix-tree/iteration_check.c
+++ b/tools/testing/radix-tree/iteration_check.c
@@ -16,26 +16,36 @@
#include <pthread.h>
#include "test.h"
-#define NUM_THREADS 4
-#define TAG 0
+#define NUM_THREADS 5
+#define MAX_IDX 100
+#define TAG 0
+#define NEW_TAG 1
+
static pthread_mutex_t tree_lock = PTHREAD_MUTEX_INITIALIZER;
static pthread_t threads[NUM_THREADS];
static unsigned int seeds[3];
-RADIX_TREE(tree, GFP_KERNEL);
-bool test_complete;
+static RADIX_TREE(tree, GFP_KERNEL);
+static bool test_complete;
+static int max_order;
/* relentlessly fill the tree with tagged entries */
static void *add_entries_fn(void *arg)
{
- int pgoff;
-
rcu_register_thread();
while (!test_complete) {
- for (pgoff = 0; pgoff < 100; pgoff++) {
+ unsigned long pgoff;
+ int order;
+
+ for (pgoff = 0; pgoff < MAX_IDX; pgoff++) {
pthread_mutex_lock(&tree_lock);
- if (item_insert(&tree, pgoff) == 0)
- item_tag_set(&tree, pgoff, TAG);
+ for (order = max_order; order >= 0; order--) {
+ if (item_insert_order(&tree, pgoff, order)
+ == 0) {
+ item_tag_set(&tree, pgoff, TAG);
+ break;
+ }
+ }
pthread_mutex_unlock(&tree_lock);
}
}
@@ -62,14 +72,7 @@ static void *tagged_iteration_fn(void *arg)
while (!test_complete) {
rcu_read_lock();
radix_tree_for_each_tagged(slot, &tree, &iter, 0, TAG) {
- void *entry;
- int i;
-
- /* busy wait to let removals happen */
- for (i = 0; i < 1000000; i++)
- ;
-
- entry = radix_tree_deref_slot(slot);
+ void *entry = radix_tree_deref_slot(slot);
if (unlikely(!entry))
continue;
@@ -110,14 +113,7 @@ static void *untagged_iteration_fn(void *arg)
while (!test_complete) {
rcu_read_lock();
radix_tree_for_each_slot(slot, &tree, &iter, 0) {
- void *entry;
- int i;
-
- /* busy wait to let removals happen */
- for (i = 0; i < 1000000; i++)
- ;
-
- entry = radix_tree_deref_slot(slot);
+ void *entry = radix_tree_deref_slot(slot);
if (unlikely(!entry))
continue;
@@ -152,7 +148,7 @@ static void *remove_entries_fn(void *arg)
while (!test_complete) {
int pgoff;
- pgoff = rand_r(&seeds[2]) % 100;
+ pgoff = rand_r(&seeds[2]) % MAX_IDX;
pthread_mutex_lock(&tree_lock);
item_delete(&tree, pgoff);
@@ -164,36 +160,54 @@ static void *remove_entries_fn(void *arg)
return NULL;
}
+static void *tag_entries_fn(void *arg)
+{
+ rcu_register_thread();
+
+ while (!test_complete) {
+ tag_tagged_items(&tree, &tree_lock, 0, MAX_IDX, 10, TAG,
+ NEW_TAG);
+ }
+ rcu_unregister_thread();
+ return NULL;
+}
+
/* This is a unit test for a bug found by the syzkaller tester */
-void iteration_test(void)
+void iteration_test(unsigned order, unsigned test_duration)
{
int i;
- printf("Running iteration tests for 10 seconds\n");
+ printf("Running %siteration tests for %d seconds\n",
+ order > 0 ? "multiorder " : "", test_duration);
+ max_order = order;
test_complete = false;
for (i = 0; i < 3; i++)
seeds[i] = rand();
if (pthread_create(&threads[0], NULL, tagged_iteration_fn, NULL)) {
- perror("pthread_create");
+ perror("create tagged iteration thread");
exit(1);
}
if (pthread_create(&threads[1], NULL, untagged_iteration_fn, NULL)) {
- perror("pthread_create");
+ perror("create untagged iteration thread");
exit(1);
}
if (pthread_create(&threads[2], NULL, add_entries_fn, NULL)) {
- perror("pthread_create");
+ perror("create add entry thread");
exit(1);
}
if (pthread_create(&threads[3], NULL, remove_entries_fn, NULL)) {
- perror("pthread_create");
+ perror("create remove entry thread");
+ exit(1);
+ }
+ if (pthread_create(&threads[4], NULL, tag_entries_fn, NULL)) {
+ perror("create tag entry thread");
exit(1);
}
- sleep(10);
+ sleep(test_duration);
test_complete = true;
for (i = 0; i < NUM_THREADS; i++) {
diff --git a/tools/testing/radix-tree/main.c b/tools/testing/radix-tree/main.c
index 170175c..f7e9801 100644
--- a/tools/testing/radix-tree/main.c
+++ b/tools/testing/radix-tree/main.c
@@ -350,7 +350,8 @@ int main(int argc, char **argv)
regression1_test();
regression2_test();
regression3_test();
- iteration_test();
+ iteration_test(0, 10);
+ iteration_test(7, 20);
single_thread_tests(long_run);
/* Free any remaining preallocated nodes */
diff --git a/tools/testing/radix-tree/multiorder.c b/tools/testing/radix-tree/multiorder.c
index 9757b89..08b4e16 100644
--- a/tools/testing/radix-tree/multiorder.c
+++ b/tools/testing/radix-tree/multiorder.c
@@ -75,8 +75,27 @@ static void __multiorder_tag_test(int index, int order)
item_kill_tree(&tree);
}
+static void __multiorder_tag_test2(unsigned order, unsigned long index2)
+{
+ RADIX_TREE(tree, GFP_KERNEL);
+ unsigned long index = (1 << order);
+ index2 += index;
+
+ assert(item_insert_order(&tree, 0, order) == 0);
+ assert(item_insert(&tree, index2) == 0);
+
+ assert(radix_tree_tag_set(&tree, 0, 0));
+ assert(radix_tree_tag_set(&tree, index2, 0));
+
+ assert(tag_tagged_items(&tree, NULL, 0, ~0UL, 10, 0, 1) == 2);
+
+ item_kill_tree(&tree);
+}
+
static void multiorder_tag_tests(void)
{
+ int i, j;
+
/* test multi-order entry for indices 0-7 with no sibling pointers */
__multiorder_tag_test(0, 3);
__multiorder_tag_test(5, 3);
@@ -116,6 +135,10 @@ static void multiorder_tag_tests(void)
__multiorder_tag_test(300, 8);
__multiorder_tag_test(0x12345678UL, 8);
+
+ for (i = 1; i < 10; i++)
+ for (j = 0; j < (10 << i); j++)
+ __multiorder_tag_test2(i, j);
}
static void multiorder_check(unsigned long index, int order)
diff --git a/tools/testing/radix-tree/test.h b/tools/testing/radix-tree/test.h
index 7c2611c..056a23b 100644
--- a/tools/testing/radix-tree/test.h
+++ b/tools/testing/radix-tree/test.h
@@ -32,7 +32,7 @@ unsigned long find_item(struct radix_tree_root *, void *item);
void tag_check(void);
void multiorder_checks(void);
-void iteration_test(void);
+void iteration_test(unsigned order, unsigned duration);
void benchmark(void);
struct item *
--
2.10.2
[toc] | [prev] | [next] | [standalone]
| From | Matthew Wilcox <mawilcox@linuxonhyperv.com> |
|---|---|
| Date | 2016-11-28 21:10 +0100 |
| Subject | [PATCH v3 15/33] radix-tree: Move rcu_head into a union with private_list |
| Message-ID | <sIApt-6LC-71@gated-at.bofh.it> |
| In reply to | #1531607 |
From: Matthew Wilcox <willy@infradead.org>
I want to be able to reference node->parent after freeing node.
Currently node->parent is in a union with rcu_head, so it is overwritten
when the node is put on the RCU list. We know that private_list is not
referenced after the node is freed, so it is safe for these two members
to share space.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
---
include/linux/radix-tree.h | 14 ++++----------
lib/radix-tree.c | 1 +
2 files changed, 5 insertions(+), 10 deletions(-)
diff --git a/include/linux/radix-tree.h b/include/linux/radix-tree.h
index 7444860..d04073a 100644
--- a/include/linux/radix-tree.h
+++ b/include/linux/radix-tree.h
@@ -85,18 +85,12 @@ struct radix_tree_node {
unsigned char offset; /* Slot offset in parent */
unsigned char count; /* Total entry count */
unsigned char exceptional; /* Exceptional entry count */
+ struct radix_tree_node *parent; /* Used when ascending tree */
+ void *private_data; /* For tree user */
union {
- struct {
- /* Used when ascending tree */
- struct radix_tree_node *parent;
- /* For tree user */
- void *private_data;
- };
- /* Used when freeing node */
- struct rcu_head rcu_head;
+ struct list_head private_list; /* For tree user */
+ struct rcu_head rcu_head; /* Used when freeing node */
};
- /* For tree user */
- struct list_head private_list;
void __rcu *slots[RADIX_TREE_MAP_SIZE];
unsigned long tags[RADIX_TREE_MAX_TAGS][RADIX_TREE_TAG_LONGS];
};
diff --git a/lib/radix-tree.c b/lib/radix-tree.c
index b329056..3f49417 100644
--- a/lib/radix-tree.c
+++ b/lib/radix-tree.c
@@ -324,6 +324,7 @@ static void radix_tree_node_rcu_free(struct rcu_head *head)
tag_clear(node, i, 0);
node->slots[0] = NULL;
+ INIT_LIST_HEAD(&node->private_list);
kmem_cache_free(radix_tree_node_cachep, node);
}
--
2.10.2
[toc] | [prev] | [next] | [standalone]
| From | Matthew Wilcox <mawilcox@linuxonhyperv.com> |
|---|---|
| Date | 2016-11-28 21:10 +0100 |
| Subject | [PATCH v3 23/33] radix-tree: Add radix_tree_join |
| Message-ID | <sIApu-6LC-77@gated-at.bofh.it> |
| In reply to | #1531607 |
From: Matthew Wilcox <willy@linux.intel.com>
This new function allows for the replacement of many smaller entries in
the radix tree with one larger multiorder entry. From the point of view
of an RCU walker, they may see a mixture of the smaller entries and the
large entry during the same walk, but they will never see NULL for an
index which was populated before the join.
Signed-off-by: Matthew Wilcox <willy@linux.intel.com>
---
include/linux/radix-tree.h | 3 +
lib/radix-tree.c | 183 ++++++++++++++++++++++++++++------
tools/testing/radix-tree/multiorder.c | 58 +++++++++++
3 files changed, 213 insertions(+), 31 deletions(-)
diff --git a/include/linux/radix-tree.h b/include/linux/radix-tree.h
index 7a8d251..935293a 100644
--- a/include/linux/radix-tree.h
+++ b/include/linux/radix-tree.h
@@ -335,6 +335,9 @@ static inline void radix_tree_preload_end(void)
preempt_enable();
}
+int radix_tree_join(struct radix_tree_root *, unsigned long index,
+ unsigned new_order, void *);
+
#define RADIX_TREE_ITER_TAG_MASK 0x00FF /* tag index in lower byte */
#define RADIX_TREE_ITER_TAGGED 0x0100 /* lookup tagged slots */
#define RADIX_TREE_ITER_CONTIG 0x0200 /* stop at first hole */
diff --git a/lib/radix-tree.c b/lib/radix-tree.c
index e1ed58b..257ad9d 100644
--- a/lib/radix-tree.c
+++ b/lib/radix-tree.c
@@ -338,17 +338,14 @@ static void radix_tree_node_rcu_free(struct rcu_head *head)
{
struct radix_tree_node *node =
container_of(head, struct radix_tree_node, rcu_head);
- int i;
/*
- * must only free zeroed nodes into the slab. radix_tree_shrink
- * can leave us with a non-NULL entry in the first slot, so clear
- * that here to make sure.
+ * Must only free zeroed nodes into the slab. We can be left with
+ * non-NULL entries by radix_tree_free_nodes, so clear the entries
+ * and tags here.
*/
- for (i = 0; i < RADIX_TREE_MAX_TAGS; i++)
- tag_clear(node, i, 0);
-
- node->slots[0] = NULL;
+ memset(node->slots, 0, sizeof(node->slots));
+ memset(node->tags, 0, sizeof(node->tags));
INIT_LIST_HEAD(&node->private_list);
kmem_cache_free(radix_tree_node_cachep, node);
@@ -677,14 +674,14 @@ int __radix_tree_create(struct radix_tree_root *root, unsigned long index,
shift = radix_tree_load_root(root, &child, &maxindex);
/* Make sure the tree is high enough. */
+ if (order > 0 && max == ((1UL << order) - 1))
+ max++;
if (max > maxindex) {
int error = radix_tree_extend(root, max, shift);
if (error < 0)
return error;
shift = error;
child = root->rnode;
- if (order == shift)
- shift += RADIX_TREE_MAP_SHIFT;
}
while (shift > order) {
@@ -696,6 +693,8 @@ int __radix_tree_create(struct radix_tree_root *root, unsigned long index,
return -ENOMEM;
child->shift = shift;
child->offset = offset;
+ child->count = 0;
+ child->exceptional = 0;
child->parent = node;
rcu_assign_pointer(*slot, node_to_entry(child));
if (node)
@@ -709,31 +708,121 @@ int __radix_tree_create(struct radix_tree_root *root, unsigned long index,
slot = &node->slots[offset];
}
+ if (nodep)
+ *nodep = node;
+ if (slotp)
+ *slotp = slot;
+ return 0;
+}
+
#ifdef CONFIG_RADIX_TREE_MULTIORDER
- /* Insert pointers to the canonical entry */
- if (order > shift) {
- unsigned i, n = 1 << (order - shift);
+/*
+ * Free any nodes below this node. The tree is presumed to not need
+ * shrinking, and any user data in the tree is presumed to not need a
+ * destructor called on it. If we need to add a destructor, we can
+ * add that functionality later. Note that we may not clear tags or
+ * slots from the tree as an RCU walker may still have a pointer into
+ * this subtree. We could replace the entries with RADIX_TREE_RETRY,
+ * but we'll still have to clear those in rcu_free.
+ */
+static void radix_tree_free_nodes(struct radix_tree_node *node)
+{
+ unsigned offset = 0;
+ struct radix_tree_node *child = entry_to_node(node);
+
+ for (;;) {
+ void *entry = child->slots[offset];
+ if (radix_tree_is_internal_node(entry) &&
+ !is_sibling_entry(child, entry)) {
+ child = entry_to_node(entry);
+ offset = 0;
+ continue;
+ }
+ offset++;
+ while (offset == RADIX_TREE_MAP_SIZE) {
+ struct radix_tree_node *old = child;
+ offset = child->offset + 1;
+ child = child->parent;
+ radix_tree_node_free(old);
+ if (old == entry_to_node(node))
+ return;
+ }
+ }
+}
+
+static inline int insert_entries(struct radix_tree_node *node, void **slot,
+ void *item, unsigned order, bool replace)
+{
+ struct radix_tree_node *child;
+ unsigned i, n, tag, offset, tags = 0;
+
+ if (node) {
+ n = 1 << (order - node->shift);
+ offset = get_slot_offset(node, slot);
+ } else {
+ n = 1;
+ offset = 0;
+ }
+
+ if (n > 1) {
offset = offset & ~(n - 1);
slot = &node->slots[offset];
- child = node_to_entry(slot);
- for (i = 0; i < n; i++) {
- if (slot[i])
+ }
+ child = node_to_entry(slot);
+
+ for (i = 0; i < n; i++) {
+ if (slot[i]) {
+ if (replace) {
+ node->count--;
+ for (tag = 0; tag < RADIX_TREE_MAX_TAGS; tag++)
+ if (tag_get(node, tag, offset + i))
+ tags |= 1 << tag;
+ } else
return -EEXIST;
}
+ }
- for (i = 1; i < n; i++) {
+ for (i = 0; i < n; i++) {
+ struct radix_tree_node *old = slot[i];
+ if (i) {
rcu_assign_pointer(slot[i], child);
- node->count++;
+ for (tag = 0; tag < RADIX_TREE_MAX_TAGS; tag++)
+ if (tags & (1 << tag))
+ tag_clear(node, tag, offset + i);
+ } else {
+ rcu_assign_pointer(slot[i], item);
+ for (tag = 0; tag < RADIX_TREE_MAX_TAGS; tag++)
+ if (tags & (1 << tag))
+ tag_set(node, tag, offset);
}
+ if (radix_tree_is_internal_node(old) &&
+ !is_sibling_entry(node, old))
+ radix_tree_free_nodes(old);
+ if (radix_tree_exceptional_entry(old))
+ node->exceptional--;
}
-#endif
-
- if (nodep)
- *nodep = node;
- if (slotp)
- *slotp = slot;
- return 0;
+ if (node) {
+ node->count += n;
+ if (radix_tree_exceptional_entry(item))
+ node->exceptional += n;
+ }
+ return n;
}
+#else
+static inline int insert_entries(struct radix_tree_node *node, void **slot,
+ void *item, unsigned order, bool replace)
+{
+ if (*slot)
+ return -EEXIST;
+ rcu_assign_pointer(*slot, item);
+ if (node) {
+ node->count++;
+ if (radix_tree_exceptional_entry(item))
+ node->exceptional++;
+ }
+ return 1;
+}
+#endif
/**
* __radix_tree_insert - insert into a radix tree
@@ -756,15 +845,13 @@ int __radix_tree_insert(struct radix_tree_root *root, unsigned long index,
error = __radix_tree_create(root, index, order, &node, &slot);
if (error)
return error;
- if (*slot != NULL)
- return -EEXIST;
- rcu_assign_pointer(*slot, item);
+
+ error = insert_entries(node, slot, item, order, false);
+ if (error < 0)
+ return error;
if (node) {
unsigned offset = get_slot_offset(node, slot);
- node->count++;
- if (radix_tree_exceptional_entry(item))
- node->exceptional++;
BUG_ON(tag_get(node, 0, offset));
BUG_ON(tag_get(node, 1, offset));
BUG_ON(tag_get(node, 2, offset));
@@ -941,6 +1028,40 @@ void radix_tree_replace_slot(struct radix_tree_root *root,
replace_slot(root, NULL, slot, item, true);
}
+#ifdef CONFIG_RADIX_TREE_MULTIORDER
+/**
+ * radix_tree_join - replace multiple entries with one multiorder entry
+ * @root: radix tree root
+ * @index: an index inside the new entry
+ * @order: order of the new entry
+ * @item: new entry
+ *
+ * Call this function to replace several entries with one larger entry.
+ * The existing entries are presumed to not need freeing as a result of
+ * this call.
+ *
+ * The replacement entry will have all the tags set on it that were set
+ * on any of the entries it is replacing.
+ */
+int radix_tree_join(struct radix_tree_root *root, unsigned long index,
+ unsigned order, void *item)
+{
+ struct radix_tree_node *node;
+ void **slot;
+ int error;
+
+ BUG_ON(radix_tree_is_internal_node(item));
+
+ error = __radix_tree_create(root, index, order, &node, &slot);
+ if (!error)
+ error = insert_entries(node, slot, item, order, true);
+ if (error > 0)
+ error = 0;
+
+ return error;
+}
+#endif
+
/**
* radix_tree_tag_set - set a tag on a radix tree node
* @root: radix tree root
diff --git a/tools/testing/radix-tree/multiorder.c b/tools/testing/radix-tree/multiorder.c
index 86daf23..c9f656c 100644
--- a/tools/testing/radix-tree/multiorder.c
+++ b/tools/testing/radix-tree/multiorder.c
@@ -332,6 +332,63 @@ void multiorder_tagged_iteration(void)
item_kill_tree(&tree);
}
+static void __multiorder_join(unsigned long index,
+ unsigned order1, unsigned order2)
+{
+ unsigned long loc;
+ void *item, *item2 = item_create(index + 1, order1);
+ RADIX_TREE(tree, GFP_KERNEL);
+
+ item_insert_order(&tree, index, order2);
+ item = radix_tree_lookup(&tree, index);
+ radix_tree_join(&tree, index + 1, order1, item2);
+ loc = find_item(&tree, item);
+ if (loc == -1)
+ free(item);
+ item = radix_tree_lookup(&tree, index + 1);
+ assert(item == item2);
+ item_kill_tree(&tree);
+}
+
+static void __multiorder_join2(unsigned order1, unsigned order2)
+{
+ RADIX_TREE(tree, GFP_KERNEL);
+ struct radix_tree_node *node;
+ void *item1 = item_create(0, order1);
+ void *item2;
+
+ item_insert_order(&tree, 0, order2);
+ radix_tree_insert(&tree, 1 << order2, (void *)0x12UL);
+ item2 = __radix_tree_lookup(&tree, 1 << order2, &node, NULL);
+ assert(item2 == (void *)0x12UL);
+ assert(node->exceptional == 1);
+
+ radix_tree_join(&tree, 0, order1, item1);
+ item2 = __radix_tree_lookup(&tree, 1 << order2, &node, NULL);
+ assert(item2 == item1);
+ assert(node->exceptional == 0);
+ item_kill_tree(&tree);
+}
+
+static void multiorder_join(void)
+{
+ int i, j, idx;
+
+ for (idx = 0; idx < 1024; idx = idx * 2 + 3) {
+ for (i = 1; i < 15; i++) {
+ for (j = 0; j < i; j++) {
+ __multiorder_join(idx, i, j);
+ }
+ }
+ }
+
+ for (i = 1; i < 15; i++) {
+ for (j = 0; j < i; j++) {
+ __multiorder_join2(i, j);
+ }
+ }
+}
+
void multiorder_checks(void)
{
int i;
@@ -349,4 +406,5 @@ void multiorder_checks(void)
multiorder_tag_tests();
multiorder_iteration();
multiorder_tagged_iteration();
+ multiorder_join();
}
--
2.10.2
[toc] | [prev] | [next] | [standalone]
| From | Matthew Wilcox <mawilcox@linuxonhyperv.com> |
|---|---|
| Date | 2016-11-28 21:10 +0100 |
| Subject | [PATCH v3 09/33] radix tree test suite: Use rcu_barrier |
| Message-ID | <sIApu-6LC-79@gated-at.bofh.it> |
| In reply to | #1531607 |
From: Matthew Wilcox <mawilcox@microsoft.com>
Calling rcu_barrier() allows all of the rcu-freed memory to be actually
returned to the pool, and allows nr_allocated to return to 0. As well
as allowing diffs between runs to be more useful, it also lets us
pinpoint leaks more effectively.
Signed-off-by: Matthew Wilcox <mawilcox@microsoft.com>
---
tools/testing/radix-tree/main.c | 12 ++++++++++--
tools/testing/radix-tree/tag_check.c | 5 +++++
2 files changed, 15 insertions(+), 2 deletions(-)
diff --git a/tools/testing/radix-tree/main.c b/tools/testing/radix-tree/main.c
index f1d1e3b..76d9c95 100644
--- a/tools/testing/radix-tree/main.c
+++ b/tools/testing/radix-tree/main.c
@@ -295,24 +295,31 @@ static void single_thread_tests(bool long_run)
printf("starting single_thread_tests: %d allocated, preempt %d\n",
nr_allocated, preempt_count);
multiorder_checks();
+ rcu_barrier();
printf("after multiorder_check: %d allocated, preempt %d\n",
nr_allocated, preempt_count);
locate_check();
+ rcu_barrier();
printf("after locate_check: %d allocated, preempt %d\n",
nr_allocated, preempt_count);
tag_check();
+ rcu_barrier();
printf("after tag_check: %d allocated, preempt %d\n",
nr_allocated, preempt_count);
gang_check();
+ rcu_barrier();
printf("after gang_check: %d allocated, preempt %d\n",
nr_allocated, preempt_count);
add_and_check();
+ rcu_barrier();
printf("after add_and_check: %d allocated, preempt %d\n",
nr_allocated, preempt_count);
dynamic_height_check();
+ rcu_barrier();
printf("after dynamic_height_check: %d allocated, preempt %d\n",
nr_allocated, preempt_count);
big_gang_check(long_run);
+ rcu_barrier();
printf("after big_gang_check: %d allocated, preempt %d\n",
nr_allocated, preempt_count);
for (i = 0; i < (long_run ? 2000 : 3); i++) {
@@ -320,6 +327,7 @@ static void single_thread_tests(bool long_run)
printf("%d ", i);
fflush(stdout);
}
+ rcu_barrier();
printf("after copy_tag_check: %d allocated, preempt %d\n",
nr_allocated, preempt_count);
}
@@ -354,8 +362,8 @@ int main(int argc, char **argv)
benchmark();
- sleep(1);
- printf("after sleep(1): %d allocated, preempt %d\n",
+ rcu_barrier();
+ printf("after rcu_barrier: %d allocated, preempt %d\n",
nr_allocated, preempt_count);
rcu_unregister_thread();
diff --git a/tools/testing/radix-tree/tag_check.c b/tools/testing/radix-tree/tag_check.c
index b0ac057..186f6e4 100644
--- a/tools/testing/radix-tree/tag_check.c
+++ b/tools/testing/radix-tree/tag_check.c
@@ -51,6 +51,7 @@ void simple_checks(void)
verify_tag_consistency(&tree, 1);
printf("before item_kill_tree: %d allocated\n", nr_allocated);
item_kill_tree(&tree);
+ rcu_barrier();
printf("after item_kill_tree: %d allocated\n", nr_allocated);
}
@@ -331,12 +332,16 @@ void tag_check(void)
single_check();
extend_checks();
contract_checks();
+ rcu_barrier();
printf("after extend_checks: %d allocated\n", nr_allocated);
__leak_check();
leak_check();
+ rcu_barrier();
printf("after leak_check: %d allocated\n", nr_allocated);
simple_checks();
+ rcu_barrier();
printf("after simple_checks: %d allocated\n", nr_allocated);
thrash_tags();
+ rcu_barrier();
printf("after thrash_tags: %d allocated\n", nr_allocated);
}
--
2.10.2
[toc] | [prev] | [next] | [standalone]
| From | Matthew Wilcox <mawilcox@linuxonhyperv.com> |
|---|---|
| Date | 2016-11-28 21:10 +0100 |
| Subject | [PATCH v3 16/33] radix-tree: Create node_tag_set() |
| Message-ID | <sIApu-6LC-81@gated-at.bofh.it> |
| In reply to | #1531607 |
From: Matthew Wilcox <willy@linux.intel.com>
Similar to node_tag_clear(), factor node_tag_set() out of
radix_tree_range_tag_if_tagged().
Signed-off-by: Matthew Wilcox <willy@linux.intel.com>
---
lib/radix-tree.c | 41 +++++++++++++++++++----------------------
1 file changed, 19 insertions(+), 22 deletions(-)
diff --git a/lib/radix-tree.c b/lib/radix-tree.c
index 3f49417..c72da89 100644
--- a/lib/radix-tree.c
+++ b/lib/radix-tree.c
@@ -990,6 +990,22 @@ static void node_tag_clear(struct radix_tree_root *root,
root_tag_clear(root, tag);
}
+static void node_tag_set(struct radix_tree_root *root,
+ struct radix_tree_node *node,
+ unsigned int tag, unsigned int offset)
+{
+ while (node) {
+ if (tag_get(node, tag, offset))
+ return;
+ tag_set(node, tag, offset);
+ offset = node->offset;
+ node = node->parent;
+ }
+
+ if (!root_tag_get(root, tag))
+ root_tag_set(root, tag);
+}
+
/**
* radix_tree_tag_clear - clear a tag on a radix tree node
* @root: radix tree root
@@ -1228,7 +1244,7 @@ unsigned long radix_tree_range_tag_if_tagged(struct radix_tree_root *root,
unsigned long nr_to_tag,
unsigned int iftag, unsigned int settag)
{
- struct radix_tree_node *parent, *node, *child;
+ struct radix_tree_node *node, *child;
unsigned long maxindex;
unsigned long tagged = 0;
unsigned long index = *first_indexp;
@@ -1263,22 +1279,8 @@ unsigned long radix_tree_range_tag_if_tagged(struct radix_tree_root *root,
continue;
}
- /* tag the leaf */
tagged++;
- tag_set(node, settag, offset);
-
- /* walk back up the path tagging interior nodes */
- parent = node;
- for (;;) {
- offset = parent->offset;
- parent = parent->parent;
- if (!parent)
- break;
- /* stop if we find a node with the tag already set */
- if (tag_get(parent, settag, offset))
- break;
- tag_set(parent, settag, offset);
- }
+ node_tag_set(root, node, settag, offset);
next:
/* Go to next entry in node */
index = ((index >> node->shift) + 1) << node->shift;
@@ -1300,12 +1302,7 @@ unsigned long radix_tree_range_tag_if_tagged(struct radix_tree_root *root,
if (tagged >= nr_to_tag)
break;
}
- /*
- * We need not to tag the root tag if there is no tag which is set with
- * settag within the range from *first_indexp to last_index.
- */
- if (tagged > 0)
- root_tag_set(root, settag);
+
*first_indexp = index;
return tagged;
--
2.10.2
[toc] | [prev] | [next] | [standalone]
| From | Matthew Wilcox <mawilcox@linuxonhyperv.com> |
|---|---|
| Date | 2016-11-28 21:10 +0100 |
| Subject | [PATCH v3 25/33] radix-tree: Add radix_tree_split_preload() |
| Message-ID | <sIApu-6LC-83@gated-at.bofh.it> |
| In reply to | #1531607 |
From: Matthew Wilcox <willy@linux.intel.com>
Calculate how many nodes we need to allocate to split an old_order entry
into multiple entries, each of size new_order. The test suite checks that
we allocated exactly the right number of nodes; neither too many (checked
by rtp->nr == 0), nor too few (checked by comparing nr_allocated before
and after the call to radix_tree_split()).
Signed-off-by: Matthew Wilcox <willy@linux.intel.com>
---
include/linux/radix-tree.h | 1 +
lib/radix-tree.c | 24 +++++++++++++++++++-
tools/testing/radix-tree/multiorder.c | 42 +++++++++++++++++++++++++++++++++--
tools/testing/radix-tree/test.h | 5 +++++
4 files changed, 69 insertions(+), 3 deletions(-)
diff --git a/include/linux/radix-tree.h b/include/linux/radix-tree.h
index 1f4b561..5dea8f6 100644
--- a/include/linux/radix-tree.h
+++ b/include/linux/radix-tree.h
@@ -345,6 +345,7 @@ static inline void radix_tree_preload_end(void)
preempt_enable();
}
+int radix_tree_split_preload(unsigned old_order, unsigned new_order, gfp_t);
int radix_tree_split(struct radix_tree_root *, unsigned long index,
unsigned new_order);
int radix_tree_join(struct radix_tree_root *, unsigned long index,
diff --git a/lib/radix-tree.c b/lib/radix-tree.c
index 704201b..9d24bec 100644
--- a/lib/radix-tree.c
+++ b/lib/radix-tree.c
@@ -367,7 +367,7 @@ radix_tree_node_free(struct radix_tree_node *node)
* To make use of this facility, the radix tree must be initialised without
* __GFP_DIRECT_RECLAIM being passed to INIT_RADIX_TREE().
*/
-static int __radix_tree_preload(gfp_t gfp_mask, int nr)
+static int __radix_tree_preload(gfp_t gfp_mask, unsigned nr)
{
struct radix_tree_preload *rtp;
struct radix_tree_node *node;
@@ -433,6 +433,28 @@ int radix_tree_maybe_preload(gfp_t gfp_mask)
}
EXPORT_SYMBOL(radix_tree_maybe_preload);
+#ifdef CONFIG_RADIX_TREE_MULTIORDER
+/*
+ * Preload with enough objects to ensure that we can split a single entry
+ * of order @old_order into many entries of size @new_order
+ */
+int radix_tree_split_preload(unsigned int old_order, unsigned int new_order,
+ gfp_t gfp_mask)
+{
+ unsigned top = 1 << (old_order % RADIX_TREE_MAP_SHIFT);
+ unsigned layers = (old_order / RADIX_TREE_MAP_SHIFT) -
+ (new_order / RADIX_TREE_MAP_SHIFT);
+ unsigned nr = 0;
+
+ WARN_ON_ONCE(!gfpflags_allow_blocking(gfp_mask));
+ BUG_ON(new_order >= old_order);
+
+ while (layers--)
+ nr = nr * RADIX_TREE_MAP_SIZE + 1;
+ return __radix_tree_preload(gfp_mask, top * nr);
+}
+#endif
+
/*
* The same as function above, but preload number of nodes required to insert
* (1 << order) continuous naturally-aligned elements.
diff --git a/tools/testing/radix-tree/multiorder.c b/tools/testing/radix-tree/multiorder.c
index fa6effe..5421f01 100644
--- a/tools/testing/radix-tree/multiorder.c
+++ b/tools/testing/radix-tree/multiorder.c
@@ -389,35 +389,67 @@ static void multiorder_join(void)
}
}
+static void check_mem(unsigned old_order, unsigned new_order, unsigned alloc)
+{
+ struct radix_tree_preload *rtp = &radix_tree_preloads;
+ if (rtp->nr != 0)
+ printf("split(%u %u) remaining %u\n", old_order, new_order,
+ rtp->nr);
+ /*
+ * Can't check for equality here as some nodes may have been
+ * RCU-freed while we ran. But we should never finish with more
+ * nodes allocated since they should have all been preloaded.
+ */
+ if (nr_allocated > alloc)
+ printf("split(%u %u) allocated %u %u\n", old_order, new_order,
+ alloc, nr_allocated);
+}
+
static void __multiorder_split(int old_order, int new_order)
{
- RADIX_TREE(tree, GFP_KERNEL);
+ RADIX_TREE(tree, GFP_ATOMIC);
void **slot;
struct radix_tree_iter iter;
struct radix_tree_node *node;
void *item;
+ unsigned alloc;
+
+ radix_tree_preload(GFP_KERNEL);
+ assert(item_insert_order(&tree, 0, old_order) == 0);
+ radix_tree_preload_end();
+
+ /* Wipe out the preloaded cache or it'll confuse check_mem() */
+ radix_tree_cpu_dead(0);
- item_insert_order(&tree, 0, old_order);
radix_tree_tag_set(&tree, 0, 2);
+
+ radix_tree_split_preload(old_order, new_order, GFP_KERNEL);
+ alloc = nr_allocated;
radix_tree_split(&tree, 0, new_order);
+ check_mem(old_order, new_order, alloc);
radix_tree_for_each_slot(slot, &tree, &iter, 0) {
radix_tree_iter_replace(&tree, &iter, slot,
item_create(iter.index, new_order));
}
+ radix_tree_preload_end();
item_kill_tree(&tree);
+ radix_tree_preload(GFP_KERNEL);
__radix_tree_insert(&tree, 0, old_order, (void *)0x12);
+ radix_tree_preload_end();
item = __radix_tree_lookup(&tree, 0, &node, NULL);
assert(item == (void *)0x12);
assert(node->exceptional > 0);
+ radix_tree_split_preload(old_order, new_order, GFP_KERNEL);
radix_tree_split(&tree, 0, new_order);
radix_tree_for_each_slot(slot, &tree, &iter, 0) {
radix_tree_iter_replace(&tree, &iter, slot,
item_create(iter.index, new_order));
}
+ radix_tree_preload_end();
item = __radix_tree_lookup(&tree, 0, &node, NULL);
assert(item != (void *)0x12);
@@ -425,16 +457,20 @@ static void __multiorder_split(int old_order, int new_order)
item_kill_tree(&tree);
+ radix_tree_preload(GFP_KERNEL);
__radix_tree_insert(&tree, 0, old_order, (void *)0x12);
+ radix_tree_preload_end();
item = __radix_tree_lookup(&tree, 0, &node, NULL);
assert(item == (void *)0x12);
assert(node->exceptional > 0);
+ radix_tree_split_preload(old_order, new_order, GFP_KERNEL);
radix_tree_split(&tree, 0, new_order);
radix_tree_for_each_slot(slot, &tree, &iter, 0) {
radix_tree_iter_replace(&tree, &iter, slot, (void *)0x16);
}
+ radix_tree_preload_end();
item = __radix_tree_lookup(&tree, 0, &node, NULL);
assert(item == (void *)0x16);
@@ -471,4 +507,6 @@ void multiorder_checks(void)
multiorder_tagged_iteration();
multiorder_join();
multiorder_split();
+
+ radix_tree_cpu_dead(0);
}
diff --git a/tools/testing/radix-tree/test.h b/tools/testing/radix-tree/test.h
index e11e4d2..7c2611c 100644
--- a/tools/testing/radix-tree/test.h
+++ b/tools/testing/radix-tree/test.h
@@ -52,3 +52,8 @@ int root_tag_get(struct radix_tree_root *root, unsigned int tag);
unsigned long node_maxindex(struct radix_tree_node *);
unsigned long shift_maxindex(unsigned int shift);
int radix_tree_cpu_dead(unsigned int cpu);
+struct radix_tree_preload {
+ unsigned nr;
+ struct radix_tree_node *nodes;
+};
+extern struct radix_tree_preload radix_tree_preloads;
--
2.10.2
[toc] | [prev] | [next] | [standalone]
| From | Matthew Wilcox <mawilcox@linuxonhyperv.com> |
|---|---|
| Date | 2016-11-28 21:10 +0100 |
| Subject | [PATCH v3 09/33] radix tree test suite: Use rcu_barrier |
| Message-ID | <sIApu-6LC-87@gated-at.bofh.it> |
| In reply to | #1531607 |
From: Matthew Wilcox <mawilcox@microsoft.com>
Calling rcu_barrier() allows all of the rcu-freed memory to be actually
returned to the pool, and allows nr_allocated to return to 0. As well
as allowing diffs between runs to be more useful, it also lets us
pinpoint leaks more effectively.
Signed-off-by: Matthew Wilcox <mawilcox@microsoft.com>
---
tools/testing/radix-tree/main.c | 12 ++++++++++--
tools/testing/radix-tree/tag_check.c | 5 +++++
2 files changed, 15 insertions(+), 2 deletions(-)
diff --git a/tools/testing/radix-tree/main.c b/tools/testing/radix-tree/main.c
index f1d1e3b..76d9c95 100644
--- a/tools/testing/radix-tree/main.c
+++ b/tools/testing/radix-tree/main.c
@@ -295,24 +295,31 @@ static void single_thread_tests(bool long_run)
printf("starting single_thread_tests: %d allocated, preempt %d\n",
nr_allocated, preempt_count);
multiorder_checks();
+ rcu_barrier();
printf("after multiorder_check: %d allocated, preempt %d\n",
nr_allocated, preempt_count);
locate_check();
+ rcu_barrier();
printf("after locate_check: %d allocated, preempt %d\n",
nr_allocated, preempt_count);
tag_check();
+ rcu_barrier();
printf("after tag_check: %d allocated, preempt %d\n",
nr_allocated, preempt_count);
gang_check();
+ rcu_barrier();
printf("after gang_check: %d allocated, preempt %d\n",
nr_allocated, preempt_count);
add_and_check();
+ rcu_barrier();
printf("after add_and_check: %d allocated, preempt %d\n",
nr_allocated, preempt_count);
dynamic_height_check();
+ rcu_barrier();
printf("after dynamic_height_check: %d allocated, preempt %d\n",
nr_allocated, preempt_count);
big_gang_check(long_run);
+ rcu_barrier();
printf("after big_gang_check: %d allocated, preempt %d\n",
nr_allocated, preempt_count);
for (i = 0; i < (long_run ? 2000 : 3); i++) {
@@ -320,6 +327,7 @@ static void single_thread_tests(bool long_run)
printf("%d ", i);
fflush(stdout);
}
+ rcu_barrier();
printf("after copy_tag_check: %d allocated, preempt %d\n",
nr_allocated, preempt_count);
}
@@ -354,8 +362,8 @@ int main(int argc, char **argv)
benchmark();
- sleep(1);
- printf("after sleep(1): %d allocated, preempt %d\n",
+ rcu_barrier();
+ printf("after rcu_barrier: %d allocated, preempt %d\n",
nr_allocated, preempt_count);
rcu_unregister_thread();
diff --git a/tools/testing/radix-tree/tag_check.c b/tools/testing/radix-tree/tag_check.c
index b0ac057..186f6e4 100644
--- a/tools/testing/radix-tree/tag_check.c
+++ b/tools/testing/radix-tree/tag_check.c
@@ -51,6 +51,7 @@ void simple_checks(void)
verify_tag_consistency(&tree, 1);
printf("before item_kill_tree: %d allocated\n", nr_allocated);
item_kill_tree(&tree);
+ rcu_barrier();
printf("after item_kill_tree: %d allocated\n", nr_allocated);
}
@@ -331,12 +332,16 @@ void tag_check(void)
single_check();
extend_checks();
contract_checks();
+ rcu_barrier();
printf("after extend_checks: %d allocated\n", nr_allocated);
__leak_check();
leak_check();
+ rcu_barrier();
printf("after leak_check: %d allocated\n", nr_allocated);
simple_checks();
+ rcu_barrier();
printf("after simple_checks: %d allocated\n", nr_allocated);
thrash_tags();
+ rcu_barrier();
printf("after thrash_tags: %d allocated\n", nr_allocated);
}
--
2.10.2
[toc] | [prev] | [next] | [standalone]
| From | Matthew Wilcox <mawilcox@linuxonhyperv.com> |
|---|---|
| Date | 2016-11-28 21:10 +0100 |
| Subject | [PATCH v3 12/33] tools: Add more bitmap functions |
| Message-ID | <sIApu-6LC-89@gated-at.bofh.it> |
| In reply to | #1531607 |
From: Matthew Wilcox <willy@linux.intel.com>
I need the following functions for the radix tree:
bitmap_fill
bitmap_empty
bitmap_full
Copy the implementations from include/linux/bitmap.h
---
tools/include/linux/bitmap.h | 26 ++++++++++++++++++++++++++
1 file changed, 26 insertions(+)
diff --git a/tools/include/linux/bitmap.h b/tools/include/linux/bitmap.h
index 43c1c50..eef41d5 100644
--- a/tools/include/linux/bitmap.h
+++ b/tools/include/linux/bitmap.h
@@ -35,6 +35,32 @@ static inline void bitmap_zero(unsigned long *dst, int nbits)
}
}
+static inline void bitmap_fill(unsigned long *dst, unsigned int nbits)
+{
+ unsigned int nlongs = BITS_TO_LONGS(nbits);
+ if (!small_const_nbits(nbits)) {
+ unsigned int len = (nlongs - 1) * sizeof(unsigned long);
+ memset(dst, 0xff, len);
+ }
+ dst[nlongs - 1] = BITMAP_LAST_WORD_MASK(nbits);
+}
+
+static inline int bitmap_empty(const unsigned long *src, unsigned nbits)
+{
+ if (small_const_nbits(nbits))
+ return ! (*src & BITMAP_LAST_WORD_MASK(nbits));
+
+ return find_first_bit(src, nbits) == nbits;
+}
+
+static inline int bitmap_full(const unsigned long *src, unsigned int nbits)
+{
+ if (small_const_nbits(nbits))
+ return ! (~(*src) & BITMAP_LAST_WORD_MASK(nbits));
+
+ return find_first_zero_bit(src, nbits) == nbits;
+}
+
static inline int bitmap_weight(const unsigned long *src, int nbits)
{
if (small_const_nbits(nbits))
--
2.10.2
[toc] | [prev] | [next] | [standalone]
| From | Matthew Wilcox <mawilcox@linuxonhyperv.com> |
|---|---|
| Date | 2016-11-28 21:10 +0100 |
| Subject | [PATCH v3 16/33] radix-tree: Create node_tag_set() |
| Message-ID | <sIApu-6LC-91@gated-at.bofh.it> |
| In reply to | #1531607 |
From: Matthew Wilcox <willy@linux.intel.com>
Similar to node_tag_clear(), factor node_tag_set() out of
radix_tree_range_tag_if_tagged().
Signed-off-by: Matthew Wilcox <willy@linux.intel.com>
---
lib/radix-tree.c | 41 +++++++++++++++++++----------------------
1 file changed, 19 insertions(+), 22 deletions(-)
diff --git a/lib/radix-tree.c b/lib/radix-tree.c
index 3f49417..c72da89 100644
--- a/lib/radix-tree.c
+++ b/lib/radix-tree.c
@@ -990,6 +990,22 @@ static void node_tag_clear(struct radix_tree_root *root,
root_tag_clear(root, tag);
}
+static void node_tag_set(struct radix_tree_root *root,
+ struct radix_tree_node *node,
+ unsigned int tag, unsigned int offset)
+{
+ while (node) {
+ if (tag_get(node, tag, offset))
+ return;
+ tag_set(node, tag, offset);
+ offset = node->offset;
+ node = node->parent;
+ }
+
+ if (!root_tag_get(root, tag))
+ root_tag_set(root, tag);
+}
+
/**
* radix_tree_tag_clear - clear a tag on a radix tree node
* @root: radix tree root
@@ -1228,7 +1244,7 @@ unsigned long radix_tree_range_tag_if_tagged(struct radix_tree_root *root,
unsigned long nr_to_tag,
unsigned int iftag, unsigned int settag)
{
- struct radix_tree_node *parent, *node, *child;
+ struct radix_tree_node *node, *child;
unsigned long maxindex;
unsigned long tagged = 0;
unsigned long index = *first_indexp;
@@ -1263,22 +1279,8 @@ unsigned long radix_tree_range_tag_if_tagged(struct radix_tree_root *root,
continue;
}
- /* tag the leaf */
tagged++;
- tag_set(node, settag, offset);
-
- /* walk back up the path tagging interior nodes */
- parent = node;
- for (;;) {
- offset = parent->offset;
- parent = parent->parent;
- if (!parent)
- break;
- /* stop if we find a node with the tag already set */
- if (tag_get(parent, settag, offset))
- break;
- tag_set(parent, settag, offset);
- }
+ node_tag_set(root, node, settag, offset);
next:
/* Go to next entry in node */
index = ((index >> node->shift) + 1) << node->shift;
@@ -1300,12 +1302,7 @@ unsigned long radix_tree_range_tag_if_tagged(struct radix_tree_root *root,
if (tagged >= nr_to_tag)
break;
}
- /*
- * We need not to tag the root tag if there is no tag which is set with
- * settag within the range from *first_indexp to last_index.
- */
- if (tagged > 0)
- root_tag_set(root, settag);
+
*first_indexp = index;
return tagged;
--
2.10.2
[toc] | [prev] | [next] | [standalone]
| From | Matthew Wilcox <mawilcox@linuxonhyperv.com> |
|---|---|
| Date | 2016-11-28 21:10 +0100 |
| Subject | [PATCH v3 27/33] radix tree test suite: Check multiorder iteration |
| Message-ID | <sIApt-6LC-45@gated-at.bofh.it> |
| In reply to | #1531607 |
From: Matthew Wilcox <mawilcox@microsoft.com>
The random iteration test only inserts order-0 entries currently.
Update it to insert entries of order between 7 and 0. Also make the
maximum index configurable, make some variables static, make the test
duration variable, remove some useless spinning, and add a fifth thread
which calls tag_tagged_items().
Signed-off-by: Matthew Wilcox <mawilcox@microsoft.com>
---
tools/testing/radix-tree/iteration_check.c | 80 ++++++++++++++++++------------
tools/testing/radix-tree/main.c | 3 +-
tools/testing/radix-tree/multiorder.c | 23 +++++++++
tools/testing/radix-tree/test.h | 2 +-
4 files changed, 73 insertions(+), 35 deletions(-)
diff --git a/tools/testing/radix-tree/iteration_check.c b/tools/testing/radix-tree/iteration_check.c
index f328a66..7572b7e 100644
--- a/tools/testing/radix-tree/iteration_check.c
+++ b/tools/testing/radix-tree/iteration_check.c
@@ -16,26 +16,36 @@
#include <pthread.h>
#include "test.h"
-#define NUM_THREADS 4
-#define TAG 0
+#define NUM_THREADS 5
+#define MAX_IDX 100
+#define TAG 0
+#define NEW_TAG 1
+
static pthread_mutex_t tree_lock = PTHREAD_MUTEX_INITIALIZER;
static pthread_t threads[NUM_THREADS];
static unsigned int seeds[3];
-RADIX_TREE(tree, GFP_KERNEL);
-bool test_complete;
+static RADIX_TREE(tree, GFP_KERNEL);
+static bool test_complete;
+static int max_order;
/* relentlessly fill the tree with tagged entries */
static void *add_entries_fn(void *arg)
{
- int pgoff;
-
rcu_register_thread();
while (!test_complete) {
- for (pgoff = 0; pgoff < 100; pgoff++) {
+ unsigned long pgoff;
+ int order;
+
+ for (pgoff = 0; pgoff < MAX_IDX; pgoff++) {
pthread_mutex_lock(&tree_lock);
- if (item_insert(&tree, pgoff) == 0)
- item_tag_set(&tree, pgoff, TAG);
+ for (order = max_order; order >= 0; order--) {
+ if (item_insert_order(&tree, pgoff, order)
+ == 0) {
+ item_tag_set(&tree, pgoff, TAG);
+ break;
+ }
+ }
pthread_mutex_unlock(&tree_lock);
}
}
@@ -62,14 +72,7 @@ static void *tagged_iteration_fn(void *arg)
while (!test_complete) {
rcu_read_lock();
radix_tree_for_each_tagged(slot, &tree, &iter, 0, TAG) {
- void *entry;
- int i;
-
- /* busy wait to let removals happen */
- for (i = 0; i < 1000000; i++)
- ;
-
- entry = radix_tree_deref_slot(slot);
+ void *entry = radix_tree_deref_slot(slot);
if (unlikely(!entry))
continue;
@@ -110,14 +113,7 @@ static void *untagged_iteration_fn(void *arg)
while (!test_complete) {
rcu_read_lock();
radix_tree_for_each_slot(slot, &tree, &iter, 0) {
- void *entry;
- int i;
-
- /* busy wait to let removals happen */
- for (i = 0; i < 1000000; i++)
- ;
-
- entry = radix_tree_deref_slot(slot);
+ void *entry = radix_tree_deref_slot(slot);
if (unlikely(!entry))
continue;
@@ -152,7 +148,7 @@ static void *remove_entries_fn(void *arg)
while (!test_complete) {
int pgoff;
- pgoff = rand_r(&seeds[2]) % 100;
+ pgoff = rand_r(&seeds[2]) % MAX_IDX;
pthread_mutex_lock(&tree_lock);
item_delete(&tree, pgoff);
@@ -164,36 +160,54 @@ static void *remove_entries_fn(void *arg)
return NULL;
}
+static void *tag_entries_fn(void *arg)
+{
+ rcu_register_thread();
+
+ while (!test_complete) {
+ tag_tagged_items(&tree, &tree_lock, 0, MAX_IDX, 10, TAG,
+ NEW_TAG);
+ }
+ rcu_unregister_thread();
+ return NULL;
+}
+
/* This is a unit test for a bug found by the syzkaller tester */
-void iteration_test(void)
+void iteration_test(unsigned order, unsigned test_duration)
{
int i;
- printf("Running iteration tests for 10 seconds\n");
+ printf("Running %siteration tests for %d seconds\n",
+ order > 0 ? "multiorder " : "", test_duration);
+ max_order = order;
test_complete = false;
for (i = 0; i < 3; i++)
seeds[i] = rand();
if (pthread_create(&threads[0], NULL, tagged_iteration_fn, NULL)) {
- perror("pthread_create");
+ perror("create tagged iteration thread");
exit(1);
}
if (pthread_create(&threads[1], NULL, untagged_iteration_fn, NULL)) {
- perror("pthread_create");
+ perror("create untagged iteration thread");
exit(1);
}
if (pthread_create(&threads[2], NULL, add_entries_fn, NULL)) {
- perror("pthread_create");
+ perror("create add entry thread");
exit(1);
}
if (pthread_create(&threads[3], NULL, remove_entries_fn, NULL)) {
- perror("pthread_create");
+ perror("create remove entry thread");
+ exit(1);
+ }
+ if (pthread_create(&threads[4], NULL, tag_entries_fn, NULL)) {
+ perror("create tag entry thread");
exit(1);
}
- sleep(10);
+ sleep(test_duration);
test_complete = true;
for (i = 0; i < NUM_THREADS; i++) {
diff --git a/tools/testing/radix-tree/main.c b/tools/testing/radix-tree/main.c
index 170175c..f7e9801 100644
--- a/tools/testing/radix-tree/main.c
+++ b/tools/testing/radix-tree/main.c
@@ -350,7 +350,8 @@ int main(int argc, char **argv)
regression1_test();
regression2_test();
regression3_test();
- iteration_test();
+ iteration_test(0, 10);
+ iteration_test(7, 20);
single_thread_tests(long_run);
/* Free any remaining preallocated nodes */
diff --git a/tools/testing/radix-tree/multiorder.c b/tools/testing/radix-tree/multiorder.c
index 9757b89..08b4e16 100644
--- a/tools/testing/radix-tree/multiorder.c
+++ b/tools/testing/radix-tree/multiorder.c
@@ -75,8 +75,27 @@ static void __multiorder_tag_test(int index, int order)
item_kill_tree(&tree);
}
+static void __multiorder_tag_test2(unsigned order, unsigned long index2)
+{
+ RADIX_TREE(tree, GFP_KERNEL);
+ unsigned long index = (1 << order);
+ index2 += index;
+
+ assert(item_insert_order(&tree, 0, order) == 0);
+ assert(item_insert(&tree, index2) == 0);
+
+ assert(radix_tree_tag_set(&tree, 0, 0));
+ assert(radix_tree_tag_set(&tree, index2, 0));
+
+ assert(tag_tagged_items(&tree, NULL, 0, ~0UL, 10, 0, 1) == 2);
+
+ item_kill_tree(&tree);
+}
+
static void multiorder_tag_tests(void)
{
+ int i, j;
+
/* test multi-order entry for indices 0-7 with no sibling pointers */
__multiorder_tag_test(0, 3);
__multiorder_tag_test(5, 3);
@@ -116,6 +135,10 @@ static void multiorder_tag_tests(void)
__multiorder_tag_test(300, 8);
__multiorder_tag_test(0x12345678UL, 8);
+
+ for (i = 1; i < 10; i++)
+ for (j = 0; j < (10 << i); j++)
+ __multiorder_tag_test2(i, j);
}
static void multiorder_check(unsigned long index, int order)
diff --git a/tools/testing/radix-tree/test.h b/tools/testing/radix-tree/test.h
index 7c2611c..056a23b 100644
--- a/tools/testing/radix-tree/test.h
+++ b/tools/testing/radix-tree/test.h
@@ -32,7 +32,7 @@ unsigned long find_item(struct radix_tree_root *, void *item);
void tag_check(void);
void multiorder_checks(void);
-void iteration_test(void);
+void iteration_test(unsigned order, unsigned duration);
void benchmark(void);
struct item *
--
2.10.2
[toc] | [prev] | [next] | [standalone]
| From | Matthew Wilcox <mawilcox@linuxonhyperv.com> |
|---|---|
| Date | 2016-11-28 21:10 +0100 |
| Subject | [PATCH v3 21/33] radix-tree: Delete radix_tree_locate_item() |
| Message-ID | <sIApu-6LC-85@gated-at.bofh.it> |
| In reply to | #1531607 |
From: Matthew Wilcox <mawilcox@microsoft.com>
This rather complicated function can be better implemented as an iterator.
It has only one caller, so move the functionality to the only place that
needs it. Update the test suite to follow the same pattern.
Signed-off-by: Matthew Wilcox <mawilcox@microsoft.com>
Acked-by: Konstantin Khlebnikov <koct9i@gmail.com>
---
include/linux/radix-tree.h | 1 -
lib/radix-tree.c | 99 -----------------------------------------
mm/shmem.c | 26 ++++++++++-
tools/testing/radix-tree/main.c | 8 ++--
tools/testing/radix-tree/test.c | 22 +++++++++
tools/testing/radix-tree/test.h | 2 +
6 files changed, 53 insertions(+), 105 deletions(-)
diff --git a/include/linux/radix-tree.h b/include/linux/radix-tree.h
index 289d007..a13d3f7c6c 100644
--- a/include/linux/radix-tree.h
+++ b/include/linux/radix-tree.h
@@ -296,7 +296,6 @@ unsigned long radix_tree_range_tag_if_tagged(struct radix_tree_root *root,
unsigned long nr_to_tag,
unsigned int fromtag, unsigned int totag);
int radix_tree_tagged(struct radix_tree_root *root, unsigned int tag);
-unsigned long radix_tree_locate_item(struct radix_tree_root *root, void *item);
static inline void radix_tree_preload_end(void)
{
diff --git a/lib/radix-tree.c b/lib/radix-tree.c
index f8fab01..54ef055 100644
--- a/lib/radix-tree.c
+++ b/lib/radix-tree.c
@@ -1578,105 +1578,6 @@ radix_tree_gang_lookup_tag_slot(struct radix_tree_root *root, void ***results,
}
EXPORT_SYMBOL(radix_tree_gang_lookup_tag_slot);
-#if defined(CONFIG_SHMEM) && defined(CONFIG_SWAP)
-#include <linux/sched.h> /* for cond_resched() */
-
-struct locate_info {
- unsigned long found_index;
- bool stop;
-};
-
-/*
- * This linear search is at present only useful to shmem_unuse_inode().
- */
-static unsigned long __locate(struct radix_tree_node *slot, void *item,
- unsigned long index, struct locate_info *info)
-{
- unsigned long i;
-
- do {
- unsigned int shift = slot->shift;
-
- for (i = (index >> shift) & RADIX_TREE_MAP_MASK;
- i < RADIX_TREE_MAP_SIZE;
- i++, index += (1UL << shift)) {
- struct radix_tree_node *node =
- rcu_dereference_raw(slot->slots[i]);
- if (node == RADIX_TREE_RETRY)
- goto out;
- if (!radix_tree_is_internal_node(node)) {
- if (node == item) {
- info->found_index = index;
- info->stop = true;
- goto out;
- }
- continue;
- }
- node = entry_to_node(node);
- if (is_sibling_entry(slot, node))
- continue;
- slot = node;
- break;
- }
- } while (i < RADIX_TREE_MAP_SIZE);
-
-out:
- if ((index == 0) && (i == RADIX_TREE_MAP_SIZE))
- info->stop = true;
- return index;
-}
-
-/**
- * radix_tree_locate_item - search through radix tree for item
- * @root: radix tree root
- * @item: item to be found
- *
- * Returns index where item was found, or -1 if not found.
- * Caller must hold no lock (since this time-consuming function needs
- * to be preemptible), and must check afterwards if item is still there.
- */
-unsigned long radix_tree_locate_item(struct radix_tree_root *root, void *item)
-{
- struct radix_tree_node *node;
- unsigned long max_index;
- unsigned long cur_index = 0;
- struct locate_info info = {
- .found_index = -1,
- .stop = false,
- };
-
- do {
- rcu_read_lock();
- node = rcu_dereference_raw(root->rnode);
- if (!radix_tree_is_internal_node(node)) {
- rcu_read_unlock();
- if (node == item)
- info.found_index = 0;
- break;
- }
-
- node = entry_to_node(node);
-
- max_index = node_maxindex(node);
- if (cur_index > max_index) {
- rcu_read_unlock();
- break;
- }
-
- cur_index = __locate(node, item, cur_index, &info);
- rcu_read_unlock();
- cond_resched();
- } while (!info.stop && cur_index <= max_index);
-
- return info.found_index;
-}
-#else
-unsigned long radix_tree_locate_item(struct radix_tree_root *root, void *item)
-{
- return -1;
-}
-#endif /* CONFIG_SHMEM && CONFIG_SWAP */
-
/**
* __radix_tree_delete_node - try to free node after clearing a slot
* @root: radix tree root
diff --git a/mm/shmem.c b/mm/shmem.c
index b9a785e..0dd83bb 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -1049,6 +1049,30 @@ static void shmem_evict_inode(struct inode *inode)
clear_inode(inode);
}
+static unsigned long find_swap_entry(struct radix_tree_root *root, void *item)
+{
+ struct radix_tree_iter iter;
+ void **slot;
+ unsigned long found = -1;
+ unsigned int checked = 0;
+
+ rcu_read_lock();
+ radix_tree_for_each_slot(slot, root, &iter, 0) {
+ if (*slot == item) {
+ found = iter.index;
+ break;
+ }
+ checked++;
+ if ((checked % 4096) != 0)
+ continue;
+ slot = radix_tree_iter_resume(slot, &iter);
+ cond_resched_rcu();
+ }
+
+ rcu_read_unlock();
+ return found;
+}
+
/*
* If swap found in inode, free it and move page from swapcache to filecache.
*/
@@ -1062,7 +1086,7 @@ static int shmem_unuse_inode(struct shmem_inode_info *info,
int error = 0;
radswap = swp_to_radix_entry(swap);
- index = radix_tree_locate_item(&mapping->page_tree, radswap);
+ index = find_swap_entry(&mapping->page_tree, radswap);
if (index == -1)
return -EAGAIN; /* tell shmem_unuse we found nothing */
diff --git a/tools/testing/radix-tree/main.c b/tools/testing/radix-tree/main.c
index 76d9c95..a028dae 100644
--- a/tools/testing/radix-tree/main.c
+++ b/tools/testing/radix-tree/main.c
@@ -239,7 +239,7 @@ static void __locate_check(struct radix_tree_root *tree, unsigned long index,
item_insert_order(tree, index, order);
item = item_lookup(tree, index);
- index2 = radix_tree_locate_item(tree, item);
+ index2 = find_item(tree, item);
if (index != index2) {
printf("index %ld order %d inserted; found %ld\n",
index, order, index2);
@@ -273,17 +273,17 @@ static void locate_check(void)
index += (1UL << order)) {
__locate_check(&tree, index + offset, order);
}
- if (radix_tree_locate_item(&tree, &tree) != -1)
+ if (find_item(&tree, &tree) != -1)
abort();
item_kill_tree(&tree);
}
}
- if (radix_tree_locate_item(&tree, &tree) != -1)
+ if (find_item(&tree, &tree) != -1)
abort();
__locate_check(&tree, -1, 0);
- if (radix_tree_locate_item(&tree, &tree) != -1)
+ if (find_item(&tree, &tree) != -1)
abort();
item_kill_tree(&tree);
}
diff --git a/tools/testing/radix-tree/test.c b/tools/testing/radix-tree/test.c
index 0de5489..88bf57f 100644
--- a/tools/testing/radix-tree/test.c
+++ b/tools/testing/radix-tree/test.c
@@ -151,6 +151,28 @@ void item_full_scan(struct radix_tree_root *root, unsigned long start,
assert(nfound == 0);
}
+/* Use the same pattern as find_swap_entry() in mm/shmem.c */
+unsigned long find_item(struct radix_tree_root *root, void *item)
+{
+ struct radix_tree_iter iter;
+ void **slot;
+ unsigned long found = -1;
+ unsigned long checked = 0;
+
+ radix_tree_for_each_slot(slot, root, &iter, 0) {
+ if (*slot == item) {
+ found = iter.index;
+ break;
+ }
+ checked++;
+ if ((checked % 4) != 0)
+ continue;
+ slot = radix_tree_iter_resume(slot, &iter);
+ }
+
+ return found;
+}
+
static int verify_node(struct radix_tree_node *slot, unsigned int tag,
int tagged)
{
diff --git a/tools/testing/radix-tree/test.h b/tools/testing/radix-tree/test.h
index 617416e..3d9d1d3 100644
--- a/tools/testing/radix-tree/test.h
+++ b/tools/testing/radix-tree/test.h
@@ -25,6 +25,8 @@ void item_full_scan(struct radix_tree_root *root, unsigned long start,
unsigned long nr, int chunk);
void item_kill_tree(struct radix_tree_root *root);
+unsigned long find_item(struct radix_tree_root *, void *item);
+
void tag_check(void);
void multiorder_checks(void);
void iteration_test(void);
--
2.10.2
[toc] | [prev] | [next] | [standalone]
| From | Matthew Wilcox <mawilcox@linuxonhyperv.com> |
|---|---|
| Date | 2016-11-28 21:20 +0100 |
| Subject | [PATCH v3 08/33] radix tree test suite: benchmark for iterator |
| Message-ID | <sIAz8-6P8-9@gated-at.bofh.it> |
| In reply to | #1531607 |
From: Konstantin Khlebnikov <koct9i@gmail.com>
This adds simple benchmark for iterator similar to one I've used for
commit 78c1d78 ("radix-tree: introduce bit-optimized iterator")
Building with make BENCHMARK=1 set radix tree order to 6, this allows
to get performance comparable to in kernel performance.
Signed-off-by: Konstantin Khlebnikov <koct9i@gmail.com>
Signed-off-by: Matthew Wilcox <mawilcox@microsoft.com>
---
tools/testing/radix-tree/Makefile | 6 +-
tools/testing/radix-tree/benchmark.c | 98 +++++++++++++++++++++++++++++++++
tools/testing/radix-tree/linux/kernel.h | 4 ++
tools/testing/radix-tree/main.c | 2 +
tools/testing/radix-tree/test.h | 1 +
5 files changed, 110 insertions(+), 1 deletion(-)
create mode 100644 tools/testing/radix-tree/benchmark.c
diff --git a/tools/testing/radix-tree/Makefile b/tools/testing/radix-tree/Makefile
index 3c338dc..08283a8 100644
--- a/tools/testing/radix-tree/Makefile
+++ b/tools/testing/radix-tree/Makefile
@@ -4,7 +4,11 @@ LDFLAGS += -lpthread -lurcu
TARGETS = main
OFILES = main.o radix-tree.o linux.o test.o tag_check.o find_next_bit.o \
regression1.o regression2.o regression3.o multiorder.o \
- iteration_check.o
+ iteration_check.o benchmark.o
+
+ifdef BENCHMARK
+ CFLAGS += -DBENCHMARK=1
+endif
targets: $(TARGETS)
diff --git a/tools/testing/radix-tree/benchmark.c b/tools/testing/radix-tree/benchmark.c
new file mode 100644
index 0000000..215ca86
--- /dev/null
+++ b/tools/testing/radix-tree/benchmark.c
@@ -0,0 +1,98 @@
+/*
+ * benchmark.c:
+ * Author: Konstantin Khlebnikov <koct9i@gmail.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ */
+#include <linux/radix-tree.h>
+#include <linux/slab.h>
+#include <linux/errno.h>
+#include <time.h>
+#include "test.h"
+
+#define NSEC_PER_SEC 1000000000L
+
+static long long benchmark_iter(struct radix_tree_root *root, bool tagged)
+{
+ volatile unsigned long sink = 0;
+ struct radix_tree_iter iter;
+ struct timespec start, finish;
+ long long nsec;
+ int l, loops = 1;
+ void **slot;
+
+#ifdef BENCHMARK
+again:
+#endif
+ clock_gettime(CLOCK_MONOTONIC, &start);
+ for (l = 0; l < loops; l++) {
+ if (tagged) {
+ radix_tree_for_each_tagged(slot, root, &iter, 0, 0)
+ sink ^= (unsigned long)slot;
+ } else {
+ radix_tree_for_each_slot(slot, root, &iter, 0)
+ sink ^= (unsigned long)slot;
+ }
+ }
+ clock_gettime(CLOCK_MONOTONIC, &finish);
+
+ nsec = (finish.tv_sec - start.tv_sec) * NSEC_PER_SEC +
+ (finish.tv_nsec - start.tv_nsec);
+
+#ifdef BENCHMARK
+ if (loops == 1 && nsec * 5 < NSEC_PER_SEC) {
+ loops = NSEC_PER_SEC / nsec / 4 + 1;
+ goto again;
+ }
+#endif
+
+ nsec /= loops;
+ return nsec;
+}
+
+static void benchmark_size(unsigned long size, unsigned long step, int order)
+{
+ RADIX_TREE(tree, GFP_KERNEL);
+ long long normal, tagged;
+ unsigned long index;
+
+ for (index = 0 ; index < size ; index += step) {
+ item_insert_order(&tree, index, order);
+ radix_tree_tag_set(&tree, index, 0);
+ }
+
+ tagged = benchmark_iter(&tree, true);
+ normal = benchmark_iter(&tree, false);
+
+ printf("Size %ld, step %6ld, order %d tagged %10lld ns, normal %10lld ns\n",
+ size, step, order, tagged, normal);
+
+ item_kill_tree(&tree);
+ rcu_barrier();
+}
+
+void benchmark(void)
+{
+ unsigned long size[] = {1 << 10, 1 << 20, 0};
+ unsigned long step[] = {1, 2, 7, 15, 63, 64, 65,
+ 128, 256, 512, 12345, 0};
+ int c, s;
+
+ printf("starting benchmarks\n");
+ printf("RADIX_TREE_MAP_SHIFT = %d\n", RADIX_TREE_MAP_SHIFT);
+
+ for (c = 0; size[c]; c++)
+ for (s = 0; step[s]; s++)
+ benchmark_size(size[c], step[s], 0);
+
+ for (c = 0; size[c]; c++)
+ for (s = 0; step[s]; s++)
+ benchmark_size(size[c], step[s] << 9, 9);
+}
diff --git a/tools/testing/radix-tree/linux/kernel.h b/tools/testing/radix-tree/linux/kernel.h
index be98a47..dbe4b92 100644
--- a/tools/testing/radix-tree/linux/kernel.h
+++ b/tools/testing/radix-tree/linux/kernel.h
@@ -10,7 +10,11 @@
#include "../../include/linux/compiler.h"
#include "../../../include/linux/kconfig.h"
+#ifdef BENCHMARK
+#define RADIX_TREE_MAP_SHIFT 6
+#else
#define RADIX_TREE_MAP_SHIFT 3
+#endif
#ifndef NULL
#define NULL 0
diff --git a/tools/testing/radix-tree/main.c b/tools/testing/radix-tree/main.c
index 2eb6949..f1d1e3b 100644
--- a/tools/testing/radix-tree/main.c
+++ b/tools/testing/radix-tree/main.c
@@ -352,6 +352,8 @@ int main(int argc, char **argv)
/* Free any remaining preallocated nodes */
radix_tree_cpu_dead(0);
+ benchmark();
+
sleep(1);
printf("after sleep(1): %d allocated, preempt %d\n",
nr_allocated, preempt_count);
diff --git a/tools/testing/radix-tree/test.h b/tools/testing/radix-tree/test.h
index 5d2fad0..215ab77 100644
--- a/tools/testing/radix-tree/test.h
+++ b/tools/testing/radix-tree/test.h
@@ -28,6 +28,7 @@ void item_kill_tree(struct radix_tree_root *root);
void tag_check(void);
void multiorder_checks(void);
void iteration_test(void);
+void benchmark(void);
struct item *
item_tag_set(struct radix_tree_root *root, unsigned long index, int tag);
--
2.10.2
[toc] | [prev] | [next] | [standalone]
| From | Matthew Wilcox <mawilcox@linuxonhyperv.com> |
|---|---|
| Date | 2016-11-28 21:20 +0100 |
| Subject | [PATCH v3 22/33] radix-tree: Delete radix_tree_range_tag_if_tagged() |
| Message-ID | <sIAz8-6P8-5@gated-at.bofh.it> |
| In reply to | #1531607 |
From: Matthew Wilcox <mawilcox@microsoft.com>
This is an exceptionally complicated function with just one caller
(tag_pages_for_writeback). We devote a large portion of the runtime
of the test suite to testing this one function which has one caller.
By introducing the new function radix_tree_iter_tag_set(), we
can eliminate all of the complexity while keeping the performance.
The caller can now use a fairly standard radix_tree_for_each() loop,
and it doesn't need to worry about tricksy things like 'start' wrapping.
The test suite continues to spend a large amount of time investigating
this function, but now it's testing the underlying primitives such as
radix_tree_iter_resume() and the radix_tree_for_each_tagged() iterator
which are also used by other parts of the kernel.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
---
include/linux/radix-tree.h | 74 ++++++++++-----------
lib/radix-tree.c | 117 ++++++---------------------------
mm/page-writeback.c | 28 +++++---
tools/testing/radix-tree/main.c | 12 ++--
tools/testing/radix-tree/multiorder.c | 13 ++--
tools/testing/radix-tree/regression2.c | 3 +-
tools/testing/radix-tree/tag_check.c | 4 +-
tools/testing/radix-tree/test.c | 34 ++++++++++
tools/testing/radix-tree/test.h | 3 +
9 files changed, 125 insertions(+), 163 deletions(-)
diff --git a/include/linux/radix-tree.h b/include/linux/radix-tree.h
index a13d3f7c6c..7a8d251 100644
--- a/include/linux/radix-tree.h
+++ b/include/linux/radix-tree.h
@@ -121,6 +121,41 @@ static inline bool radix_tree_empty(struct radix_tree_root *root)
}
/**
+ * struct radix_tree_iter - radix tree iterator state
+ *
+ * @index: index of current slot
+ * @next_index: one beyond the last index for this chunk
+ * @tags: bit-mask for tag-iterating
+ * @node: node that contains current slot
+ * @shift: shift for the node that holds our slots
+ *
+ * This radix tree iterator works in terms of "chunks" of slots. A chunk is a
+ * subinterval of slots contained within one radix tree leaf node. It is
+ * described by a pointer to its first slot and a struct radix_tree_iter
+ * which holds the chunk's position in the tree and its size. For tagged
+ * iteration radix_tree_iter also holds the slots' bit-mask for one chosen
+ * radix tree tag.
+ */
+struct radix_tree_iter {
+ unsigned long index;
+ unsigned long next_index;
+ unsigned long tags;
+ struct radix_tree_node *node;
+#ifdef CONFIG_RADIX_TREE_MULTIORDER
+ unsigned int shift;
+#endif
+};
+
+static inline unsigned int iter_shift(const struct radix_tree_iter *iter)
+{
+#ifdef CONFIG_RADIX_TREE_MULTIORDER
+ return iter->shift;
+#else
+ return 0;
+#endif
+}
+
+/**
* Radix-tree synchronization
*
* The radix-tree API requires that users provide all synchronisation (with
@@ -283,6 +318,8 @@ void *radix_tree_tag_clear(struct radix_tree_root *root,
unsigned long index, unsigned int tag);
int radix_tree_tag_get(struct radix_tree_root *root,
unsigned long index, unsigned int tag);
+void radix_tree_iter_tag_set(struct radix_tree_root *root,
+ const struct radix_tree_iter *iter, unsigned int tag);
unsigned int
radix_tree_gang_lookup_tag(struct radix_tree_root *root, void **results,
unsigned long first_index, unsigned int max_items,
@@ -291,10 +328,6 @@ unsigned int
radix_tree_gang_lookup_tag_slot(struct radix_tree_root *root, void ***results,
unsigned long first_index, unsigned int max_items,
unsigned int tag);
-unsigned long radix_tree_range_tag_if_tagged(struct radix_tree_root *root,
- unsigned long *first_indexp, unsigned long last_index,
- unsigned long nr_to_tag,
- unsigned int fromtag, unsigned int totag);
int radix_tree_tagged(struct radix_tree_root *root, unsigned int tag);
static inline void radix_tree_preload_end(void)
@@ -302,39 +335,6 @@ static inline void radix_tree_preload_end(void)
preempt_enable();
}
-/**
- * struct radix_tree_iter - radix tree iterator state
- *
- * @index: index of current slot
- * @next_index: one beyond the last index for this chunk
- * @tags: bit-mask for tag-iterating
- * @shift: shift for the node that holds our slots
- *
- * This radix tree iterator works in terms of "chunks" of slots. A chunk is a
- * subinterval of slots contained within one radix tree leaf node. It is
- * described by a pointer to its first slot and a struct radix_tree_iter
- * which holds the chunk's position in the tree and its size. For tagged
- * iteration radix_tree_iter also holds the slots' bit-mask for one chosen
- * radix tree tag.
- */
-struct radix_tree_iter {
- unsigned long index;
- unsigned long next_index;
- unsigned long tags;
-#ifdef CONFIG_RADIX_TREE_MULTIORDER
- unsigned int shift;
-#endif
-};
-
-static inline unsigned int iter_shift(struct radix_tree_iter *iter)
-{
-#ifdef CONFIG_RADIX_TREE_MULTIORDER
- return iter->shift;
-#else
- return 0;
-#endif
-}
-
#define RADIX_TREE_ITER_TAG_MASK 0x00FF /* tag index in lower byte */
#define RADIX_TREE_ITER_TAGGED 0x0100 /* lookup tagged slots */
#define RADIX_TREE_ITER_CONTIG 0x0200 /* stop at first hole */
diff --git a/lib/radix-tree.c b/lib/radix-tree.c
index 54ef055..e1ed58b 100644
--- a/lib/radix-tree.c
+++ b/lib/radix-tree.c
@@ -218,6 +218,11 @@ radix_tree_find_next_bit(struct radix_tree_node *node, unsigned int tag,
return RADIX_TREE_MAP_SIZE;
}
+static unsigned int iter_offset(const struct radix_tree_iter *iter)
+{
+ return (iter->index >> iter_shift(iter)) & RADIX_TREE_MAP_MASK;
+}
+
/*
* The maximum index which can be stored in a radix tree
*/
@@ -1014,6 +1019,18 @@ static void node_tag_set(struct radix_tree_root *root,
}
/**
+ * radix_tree_iter_tag_set - set a tag on the current iterator entry
+ * @root: radix tree root
+ * @iter: iterator state
+ * @tag: tag to set
+ */
+void radix_tree_iter_tag_set(struct radix_tree_root *root,
+ const struct radix_tree_iter *iter, unsigned int tag)
+{
+ node_tag_set(root, iter->node, tag, iter_offset(iter));
+}
+
+/**
* radix_tree_tag_clear - clear a tag on a radix tree node
* @root: radix tree root
* @index: index key
@@ -1163,6 +1180,7 @@ void ** __radix_tree_next_slot(void **slot, struct radix_tree_iter *iter,
if (node == RADIX_TREE_RETRY)
return slot;
node = entry_to_node(node);
+ iter->node = node;
iter->shift = node->shift;
if (flags & RADIX_TREE_ITER_TAGGED) {
@@ -1265,6 +1283,7 @@ void **radix_tree_next_chunk(struct radix_tree_root *root,
iter->index = index;
iter->next_index = maxindex + 1;
iter->tags = 1;
+ iter->node = NULL;
__set_iter_shift(iter, 0);
return (void **)&root->rnode;
}
@@ -1307,6 +1326,7 @@ void **radix_tree_next_chunk(struct radix_tree_root *root,
/* Update the iterator state */
iter->index = (index &~ node_maxindex(node)) | (offset << node->shift);
iter->next_index = (index | node_maxindex(node)) + 1;
+ iter->node = node;
__set_iter_shift(iter, node->shift);
if (flags & RADIX_TREE_ITER_TAGGED)
@@ -1317,103 +1337,6 @@ void **radix_tree_next_chunk(struct radix_tree_root *root,
EXPORT_SYMBOL(radix_tree_next_chunk);
/**
- * radix_tree_range_tag_if_tagged - for each item in given range set given
- * tag if item has another tag set
- * @root: radix tree root
- * @first_indexp: pointer to a starting index of a range to scan
- * @last_index: last index of a range to scan
- * @nr_to_tag: maximum number items to tag
- * @iftag: tag index to test
- * @settag: tag index to set if tested tag is set
- *
- * This function scans range of radix tree from first_index to last_index
- * (inclusive). For each item in the range if iftag is set, the function sets
- * also settag. The function stops either after tagging nr_to_tag items or
- * after reaching last_index.
- *
- * The tags must be set from the leaf level only and propagated back up the
- * path to the root. We must do this so that we resolve the full path before
- * setting any tags on intermediate nodes. If we set tags as we descend, then
- * we can get to the leaf node and find that the index that has the iftag
- * set is outside the range we are scanning. This reults in dangling tags and
- * can lead to problems with later tag operations (e.g. livelocks on lookups).
- *
- * The function returns the number of leaves where the tag was set and sets
- * *first_indexp to the first unscanned index.
- * WARNING! *first_indexp can wrap if last_index is ULONG_MAX. Caller must
- * be prepared to handle that.
- */
-unsigned long radix_tree_range_tag_if_tagged(struct radix_tree_root *root,
- unsigned long *first_indexp, unsigned long last_index,
- unsigned long nr_to_tag,
- unsigned int iftag, unsigned int settag)
-{
- struct radix_tree_node *node, *child;
- unsigned long maxindex;
- unsigned long tagged = 0;
- unsigned long index = *first_indexp;
-
- radix_tree_load_root(root, &child, &maxindex);
- last_index = min(last_index, maxindex);
- if (index > last_index)
- return 0;
- if (!nr_to_tag)
- return 0;
- if (!root_tag_get(root, iftag)) {
- *first_indexp = last_index + 1;
- return 0;
- }
- if (!radix_tree_is_internal_node(child)) {
- *first_indexp = last_index + 1;
- root_tag_set(root, settag);
- return 1;
- }
-
- node = entry_to_node(child);
-
- for (;;) {
- unsigned offset = radix_tree_descend(node, &child, index);
- if (!child)
- goto next;
- if (!tag_get(node, iftag, offset))
- goto next;
- /* Sibling slots never have tags set on them */
- if (radix_tree_is_internal_node(child)) {
- node = entry_to_node(child);
- continue;
- }
-
- tagged++;
- node_tag_set(root, node, settag, offset);
- next:
- /* Go to next entry in node */
- index = ((index >> node->shift) + 1) << node->shift;
- /* Overflow can happen when last_index is ~0UL... */
- if (index > last_index || !index)
- break;
- offset = (index >> node->shift) & RADIX_TREE_MAP_MASK;
- while (offset == 0) {
- /*
- * We've fully scanned this node. Go up. Because
- * last_index is guaranteed to be in the tree, what
- * we do below cannot wander astray.
- */
- node = node->parent;
- offset = (index >> node->shift) & RADIX_TREE_MAP_MASK;
- }
- if (is_sibling_entry(node, node->slots[offset]))
- goto next;
- if (tagged >= nr_to_tag)
- break;
- }
-
- *first_indexp = index;
-
- return tagged;
-}
-EXPORT_SYMBOL(radix_tree_range_tag_if_tagged);
-
-/**
* radix_tree_gang_lookup - perform multiple lookup on a radix tree
* @root: radix tree root
* @results: where the results of the lookup are placed
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 52e2f8e..290e8b7 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -2106,18 +2106,26 @@ void tag_pages_for_writeback(struct address_space *mapping,
pgoff_t start, pgoff_t end)
{
#define WRITEBACK_TAG_BATCH 4096
- unsigned long tagged;
-
- do {
- spin_lock_irq(&mapping->tree_lock);
- tagged = radix_tree_range_tag_if_tagged(&mapping->page_tree,
- &start, end, WRITEBACK_TAG_BATCH,
- PAGECACHE_TAG_DIRTY, PAGECACHE_TAG_TOWRITE);
+ unsigned long tagged = 0;
+ struct radix_tree_iter iter;
+ void **slot;
+
+ spin_lock_irq(&mapping->tree_lock);
+ radix_tree_for_each_tagged(slot, &mapping->page_tree, &iter, start,
+ PAGECACHE_TAG_DIRTY) {
+ if (iter.index > end)
+ break;
+ radix_tree_iter_tag_set(&mapping->page_tree, &iter,
+ PAGECACHE_TAG_TOWRITE);
+ tagged++;
+ if ((tagged % WRITEBACK_TAG_BATCH) != 0)
+ continue;
+ slot = radix_tree_iter_resume(slot, &iter);
spin_unlock_irq(&mapping->tree_lock);
- WARN_ON_ONCE(tagged > WRITEBACK_TAG_BATCH);
cond_resched();
- /* We check 'start' to handle wrapping when end == ~0UL */
- } while (tagged >= WRITEBACK_TAG_BATCH && start);
+ spin_lock_irq(&mapping->tree_lock);
+ }
+ spin_unlock_irq(&mapping->tree_lock);
}
EXPORT_SYMBOL(tag_pages_for_writeback);
diff --git a/tools/testing/radix-tree/main.c b/tools/testing/radix-tree/main.c
index a028dae..170175c 100644
--- a/tools/testing/radix-tree/main.c
+++ b/tools/testing/radix-tree/main.c
@@ -205,8 +205,7 @@ void copy_tag_check(void)
}
// printf("\ncopying tags...\n");
- cur = start;
- tagged = radix_tree_range_tag_if_tagged(&tree, &cur, end, ITEMS, 0, 1);
+ tagged = tag_tagged_items(&tree, NULL, start, end, ITEMS, 0, 1);
// printf("checking copied tags\n");
assert(tagged == count);
@@ -214,16 +213,13 @@ void copy_tag_check(void)
/* Copy tags in several rounds */
// printf("\ncopying tags...\n");
- cur = start;
- do {
- tmp = rand() % (count/10+2);
- tagged = radix_tree_range_tag_if_tagged(&tree, &cur, end, tmp, 0, 2);
- } while (tmp == tagged);
+ tmp = rand() % (count / 10 + 2);
+ tagged = tag_tagged_items(&tree, NULL, start, end, tmp, 0, 2);
+ assert(tagged == count);
// printf("%lu %lu %lu\n", tagged, tmp, count);
// printf("checking copied tags\n");
check_copied_tags(&tree, start, end, idx, ITEMS, 0, 2);
- assert(tagged < tmp);
verify_tag_consistency(&tree, 0);
verify_tag_consistency(&tree, 1);
verify_tag_consistency(&tree, 2);
diff --git a/tools/testing/radix-tree/multiorder.c b/tools/testing/radix-tree/multiorder.c
index b9be885..86daf23 100644
--- a/tools/testing/radix-tree/multiorder.c
+++ b/tools/testing/radix-tree/multiorder.c
@@ -26,7 +26,6 @@ static void __multiorder_tag_test(int index, int order)
{
RADIX_TREE(tree, GFP_KERNEL);
int base, err, i;
- unsigned long first = 0;
/* our canonical entry */
base = index & ~((1 << order) - 1);
@@ -60,7 +59,7 @@ static void __multiorder_tag_test(int index, int order)
assert(!radix_tree_tag_get(&tree, i, 1));
}
- assert(radix_tree_range_tag_if_tagged(&tree, &first, ~0UL, 10, 0, 1) == 1);
+ assert(tag_tagged_items(&tree, NULL, 0, ~0UL, 10, 0, 1) == 1);
assert(radix_tree_tag_clear(&tree, index, 0));
for_each_index(i, base, order) {
@@ -251,7 +250,6 @@ void multiorder_tagged_iteration(void)
RADIX_TREE(tree, GFP_KERNEL);
struct radix_tree_iter iter;
void **slot;
- unsigned long first = 0;
int i, j;
printf("Multiorder tagged iteration test\n");
@@ -296,8 +294,8 @@ void multiorder_tagged_iteration(void)
}
}
- radix_tree_range_tag_if_tagged(&tree, &first, ~0UL,
- MT_NUM_ENTRIES, 1, 2);
+ assert(tag_tagged_items(&tree, NULL, 0, ~0UL, TAG_ENTRIES, 1, 2) ==
+ TAG_ENTRIES);
for (j = 0; j < 256; j++) {
int mask, k;
@@ -323,9 +321,8 @@ void multiorder_tagged_iteration(void)
}
}
- first = 1;
- radix_tree_range_tag_if_tagged(&tree, &first, ~0UL,
- MT_NUM_ENTRIES, 1, 0);
+ assert(tag_tagged_items(&tree, NULL, 1, ~0UL, MT_NUM_ENTRIES * 2, 1, 0)
+ == TAG_ENTRIES);
i = 0;
radix_tree_for_each_tagged(slot, &tree, &iter, 0, 0) {
assert(iter.index == tag_index[i]);
diff --git a/tools/testing/radix-tree/regression2.c b/tools/testing/radix-tree/regression2.c
index 63bf347..a41325d 100644
--- a/tools/testing/radix-tree/regression2.c
+++ b/tools/testing/radix-tree/regression2.c
@@ -50,6 +50,7 @@
#include <stdio.h>
#include "regression.h"
+#include "test.h"
#define PAGECACHE_TAG_DIRTY 0
#define PAGECACHE_TAG_WRITEBACK 1
@@ -90,7 +91,7 @@ void regression2_test(void)
/* 1. */
start = 0;
end = max_slots - 2;
- radix_tree_range_tag_if_tagged(&mt_tree, &start, end, 1,
+ tag_tagged_items(&mt_tree, NULL, start, end, 1,
PAGECACHE_TAG_DIRTY, PAGECACHE_TAG_TOWRITE);
/* 2. */
diff --git a/tools/testing/radix-tree/tag_check.c b/tools/testing/radix-tree/tag_check.c
index 186f6e4..ed5f87d 100644
--- a/tools/testing/radix-tree/tag_check.c
+++ b/tools/testing/radix-tree/tag_check.c
@@ -23,7 +23,7 @@ __simple_checks(struct radix_tree_root *tree, unsigned long index, int tag)
item_tag_set(tree, index, tag);
ret = item_tag_get(tree, index, tag);
assert(ret != 0);
- ret = radix_tree_range_tag_if_tagged(tree, &first, ~0UL, 10, tag, !tag);
+ ret = tag_tagged_items(tree, NULL, first, ~0UL, 10, tag, !tag);
assert(ret == 1);
ret = item_tag_get(tree, index, !tag);
assert(ret != 0);
@@ -320,7 +320,7 @@ static void single_check(void)
assert(ret == 0);
verify_tag_consistency(&tree, 0);
verify_tag_consistency(&tree, 1);
- ret = radix_tree_range_tag_if_tagged(&tree, &first, 10, 10, 0, 1);
+ ret = tag_tagged_items(&tree, NULL, first, 10, 10, 0, 1);
assert(ret == 1);
ret = radix_tree_gang_lookup_tag(&tree, (void **)items, 0, BATCH, 1);
assert(ret == 1);
diff --git a/tools/testing/radix-tree/test.c b/tools/testing/radix-tree/test.c
index 88bf57f..e5726e3 100644
--- a/tools/testing/radix-tree/test.c
+++ b/tools/testing/radix-tree/test.c
@@ -151,6 +151,40 @@ void item_full_scan(struct radix_tree_root *root, unsigned long start,
assert(nfound == 0);
}
+/* Use the same pattern as tag_pages_for_writeback() in mm/page-writeback.c */
+int tag_tagged_items(struct radix_tree_root *root, pthread_mutex_t *lock,
+ unsigned long start, unsigned long end, unsigned batch,
+ unsigned iftag, unsigned thentag)
+{
+ unsigned long tagged = 0;
+ struct radix_tree_iter iter;
+ void **slot;
+
+ if (batch == 0)
+ batch = 1;
+
+ if (lock)
+ pthread_mutex_lock(lock);
+ radix_tree_for_each_tagged(slot, root, &iter, start, iftag) {
+ if (iter.index > end)
+ break;
+ radix_tree_iter_tag_set(root, &iter, thentag);
+ tagged++;
+ if ((tagged % batch) != 0)
+ continue;
+ slot = radix_tree_iter_resume(slot, &iter);
+ if (lock) {
+ pthread_mutex_unlock(lock);
+ rcu_barrier();
+ pthread_mutex_lock(lock);
+ }
+ }
+ if (lock)
+ pthread_mutex_unlock(lock);
+
+ return tagged;
+}
+
/* Use the same pattern as find_swap_entry() in mm/shmem.c */
unsigned long find_item(struct radix_tree_root *root, void *item)
{
diff --git a/tools/testing/radix-tree/test.h b/tools/testing/radix-tree/test.h
index 3d9d1d3..e11e4d2 100644
--- a/tools/testing/radix-tree/test.h
+++ b/tools/testing/radix-tree/test.h
@@ -25,6 +25,9 @@ void item_full_scan(struct radix_tree_root *root, unsigned long start,
unsigned long nr, int chunk);
void item_kill_tree(struct radix_tree_root *root);
+int tag_tagged_items(struct radix_tree_root *, pthread_mutex_t *,
+ unsigned long start, unsigned long end, unsigned batch,
+ unsigned iftag, unsigned thentag);
unsigned long find_item(struct radix_tree_root *, void *item);
void tag_check(void);
--
2.10.2
[toc] | [prev] | [next] | [standalone]
| From | Matthew Wilcox <mawilcox@linuxonhyperv.com> |
|---|---|
| Date | 2016-11-28 21:20 +0100 |
| Subject | [PATCH v3 17/33] radix-tree: Make radix_tree_find_next_bit more useful |
| Message-ID | <sIAz8-6P8-11@gated-at.bofh.it> |
| In reply to | #1531607 |
From: Matthew Wilcox <willy@linux.intel.com>
Since this function is specialised to the radix tree, pass in the node
and tag to calculate the address of the bitmap in radix_tree_find_next_bit()
instead of the caller. Likewise, there is no need to pass in the size of
the bitmap.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
---
lib/radix-tree.c | 17 +++++++----------
1 file changed, 7 insertions(+), 10 deletions(-)
diff --git a/lib/radix-tree.c b/lib/radix-tree.c
index c72da89..01ed70b 100644
--- a/lib/radix-tree.c
+++ b/lib/radix-tree.c
@@ -190,13 +190,12 @@ static inline int any_tag_set(struct radix_tree_node *node, unsigned int tag)
* Returns next bit offset, or size if nothing found.
*/
static __always_inline unsigned long
-radix_tree_find_next_bit(const unsigned long *addr,
- unsigned long size, unsigned long offset)
+radix_tree_find_next_bit(struct radix_tree_node *node, unsigned int tag,
+ unsigned long offset)
{
- if (!__builtin_constant_p(size))
- return find_next_bit(addr, size, offset);
+ const unsigned long *addr = node->tags[tag];
- if (offset < size) {
+ if (offset < RADIX_TREE_MAP_SIZE) {
unsigned long tmp;
addr += offset / BITS_PER_LONG;
@@ -204,14 +203,14 @@ radix_tree_find_next_bit(const unsigned long *addr,
if (tmp)
return __ffs(tmp) + offset;
offset = (offset + BITS_PER_LONG) & ~(BITS_PER_LONG - 1);
- while (offset < size) {
+ while (offset < RADIX_TREE_MAP_SIZE) {
tmp = *++addr;
if (tmp)
return __ffs(tmp) + offset;
offset += BITS_PER_LONG;
}
}
- return size;
+ return RADIX_TREE_MAP_SIZE;
}
#ifndef __KERNEL__
@@ -1159,9 +1158,7 @@ void **radix_tree_next_chunk(struct radix_tree_root *root,
return NULL;
if (flags & RADIX_TREE_ITER_TAGGED)
- offset = radix_tree_find_next_bit(
- node->tags[tag],
- RADIX_TREE_MAP_SIZE,
+ offset = radix_tree_find_next_bit(node, tag,
offset + 1);
else
while (++offset < RADIX_TREE_MAP_SIZE) {
--
2.10.2
[toc] | [prev] | [next] | [standalone]
| From | Matthew Wilcox <mawilcox@linuxonhyperv.com> |
|---|---|
| Date | 2016-11-28 21:20 +0100 |
| Subject | [PATCH v3 23/33] radix-tree: Add radix_tree_join |
| Message-ID | <sIAz8-6P8-13@gated-at.bofh.it> |
| In reply to | #1531607 |
From: Matthew Wilcox <willy@linux.intel.com>
This new function allows for the replacement of many smaller entries in
the radix tree with one larger multiorder entry. From the point of view
of an RCU walker, they may see a mixture of the smaller entries and the
large entry during the same walk, but they will never see NULL for an
index which was populated before the join.
Signed-off-by: Matthew Wilcox <willy@linux.intel.com>
---
include/linux/radix-tree.h | 3 +
lib/radix-tree.c | 183 ++++++++++++++++++++++++++++------
tools/testing/radix-tree/multiorder.c | 58 +++++++++++
3 files changed, 213 insertions(+), 31 deletions(-)
diff --git a/include/linux/radix-tree.h b/include/linux/radix-tree.h
index 7a8d251..935293a 100644
--- a/include/linux/radix-tree.h
+++ b/include/linux/radix-tree.h
@@ -335,6 +335,9 @@ static inline void radix_tree_preload_end(void)
preempt_enable();
}
+int radix_tree_join(struct radix_tree_root *, unsigned long index,
+ unsigned new_order, void *);
+
#define RADIX_TREE_ITER_TAG_MASK 0x00FF /* tag index in lower byte */
#define RADIX_TREE_ITER_TAGGED 0x0100 /* lookup tagged slots */
#define RADIX_TREE_ITER_CONTIG 0x0200 /* stop at first hole */
diff --git a/lib/radix-tree.c b/lib/radix-tree.c
index e1ed58b..257ad9d 100644
--- a/lib/radix-tree.c
+++ b/lib/radix-tree.c
@@ -338,17 +338,14 @@ static void radix_tree_node_rcu_free(struct rcu_head *head)
{
struct radix_tree_node *node =
container_of(head, struct radix_tree_node, rcu_head);
- int i;
/*
- * must only free zeroed nodes into the slab. radix_tree_shrink
- * can leave us with a non-NULL entry in the first slot, so clear
- * that here to make sure.
+ * Must only free zeroed nodes into the slab. We can be left with
+ * non-NULL entries by radix_tree_free_nodes, so clear the entries
+ * and tags here.
*/
- for (i = 0; i < RADIX_TREE_MAX_TAGS; i++)
- tag_clear(node, i, 0);
-
- node->slots[0] = NULL;
+ memset(node->slots, 0, sizeof(node->slots));
+ memset(node->tags, 0, sizeof(node->tags));
INIT_LIST_HEAD(&node->private_list);
kmem_cache_free(radix_tree_node_cachep, node);
@@ -677,14 +674,14 @@ int __radix_tree_create(struct radix_tree_root *root, unsigned long index,
shift = radix_tree_load_root(root, &child, &maxindex);
/* Make sure the tree is high enough. */
+ if (order > 0 && max == ((1UL << order) - 1))
+ max++;
if (max > maxindex) {
int error = radix_tree_extend(root, max, shift);
if (error < 0)
return error;
shift = error;
child = root->rnode;
- if (order == shift)
- shift += RADIX_TREE_MAP_SHIFT;
}
while (shift > order) {
@@ -696,6 +693,8 @@ int __radix_tree_create(struct radix_tree_root *root, unsigned long index,
return -ENOMEM;
child->shift = shift;
child->offset = offset;
+ child->count = 0;
+ child->exceptional = 0;
child->parent = node;
rcu_assign_pointer(*slot, node_to_entry(child));
if (node)
@@ -709,31 +708,121 @@ int __radix_tree_create(struct radix_tree_root *root, unsigned long index,
slot = &node->slots[offset];
}
+ if (nodep)
+ *nodep = node;
+ if (slotp)
+ *slotp = slot;
+ return 0;
+}
+
#ifdef CONFIG_RADIX_TREE_MULTIORDER
- /* Insert pointers to the canonical entry */
- if (order > shift) {
- unsigned i, n = 1 << (order - shift);
+/*
+ * Free any nodes below this node. The tree is presumed to not need
+ * shrinking, and any user data in the tree is presumed to not need a
+ * destructor called on it. If we need to add a destructor, we can
+ * add that functionality later. Note that we may not clear tags or
+ * slots from the tree as an RCU walker may still have a pointer into
+ * this subtree. We could replace the entries with RADIX_TREE_RETRY,
+ * but we'll still have to clear those in rcu_free.
+ */
+static void radix_tree_free_nodes(struct radix_tree_node *node)
+{
+ unsigned offset = 0;
+ struct radix_tree_node *child = entry_to_node(node);
+
+ for (;;) {
+ void *entry = child->slots[offset];
+ if (radix_tree_is_internal_node(entry) &&
+ !is_sibling_entry(child, entry)) {
+ child = entry_to_node(entry);
+ offset = 0;
+ continue;
+ }
+ offset++;
+ while (offset == RADIX_TREE_MAP_SIZE) {
+ struct radix_tree_node *old = child;
+ offset = child->offset + 1;
+ child = child->parent;
+ radix_tree_node_free(old);
+ if (old == entry_to_node(node))
+ return;
+ }
+ }
+}
+
+static inline int insert_entries(struct radix_tree_node *node, void **slot,
+ void *item, unsigned order, bool replace)
+{
+ struct radix_tree_node *child;
+ unsigned i, n, tag, offset, tags = 0;
+
+ if (node) {
+ n = 1 << (order - node->shift);
+ offset = get_slot_offset(node, slot);
+ } else {
+ n = 1;
+ offset = 0;
+ }
+
+ if (n > 1) {
offset = offset & ~(n - 1);
slot = &node->slots[offset];
- child = node_to_entry(slot);
- for (i = 0; i < n; i++) {
- if (slot[i])
+ }
+ child = node_to_entry(slot);
+
+ for (i = 0; i < n; i++) {
+ if (slot[i]) {
+ if (replace) {
+ node->count--;
+ for (tag = 0; tag < RADIX_TREE_MAX_TAGS; tag++)
+ if (tag_get(node, tag, offset + i))
+ tags |= 1 << tag;
+ } else
return -EEXIST;
}
+ }
- for (i = 1; i < n; i++) {
+ for (i = 0; i < n; i++) {
+ struct radix_tree_node *old = slot[i];
+ if (i) {
rcu_assign_pointer(slot[i], child);
- node->count++;
+ for (tag = 0; tag < RADIX_TREE_MAX_TAGS; tag++)
+ if (tags & (1 << tag))
+ tag_clear(node, tag, offset + i);
+ } else {
+ rcu_assign_pointer(slot[i], item);
+ for (tag = 0; tag < RADIX_TREE_MAX_TAGS; tag++)
+ if (tags & (1 << tag))
+ tag_set(node, tag, offset);
}
+ if (radix_tree_is_internal_node(old) &&
+ !is_sibling_entry(node, old))
+ radix_tree_free_nodes(old);
+ if (radix_tree_exceptional_entry(old))
+ node->exceptional--;
}
-#endif
-
- if (nodep)
- *nodep = node;
- if (slotp)
- *slotp = slot;
- return 0;
+ if (node) {
+ node->count += n;
+ if (radix_tree_exceptional_entry(item))
+ node->exceptional += n;
+ }
+ return n;
}
+#else
+static inline int insert_entries(struct radix_tree_node *node, void **slot,
+ void *item, unsigned order, bool replace)
+{
+ if (*slot)
+ return -EEXIST;
+ rcu_assign_pointer(*slot, item);
+ if (node) {
+ node->count++;
+ if (radix_tree_exceptional_entry(item))
+ node->exceptional++;
+ }
+ return 1;
+}
+#endif
/**
* __radix_tree_insert - insert into a radix tree
@@ -756,15 +845,13 @@ int __radix_tree_insert(struct radix_tree_root *root, unsigned long index,
error = __radix_tree_create(root, index, order, &node, &slot);
if (error)
return error;
- if (*slot != NULL)
- return -EEXIST;
- rcu_assign_pointer(*slot, item);
+
+ error = insert_entries(node, slot, item, order, false);
+ if (error < 0)
+ return error;
if (node) {
unsigned offset = get_slot_offset(node, slot);
- node->count++;
- if (radix_tree_exceptional_entry(item))
- node->exceptional++;
BUG_ON(tag_get(node, 0, offset));
BUG_ON(tag_get(node, 1, offset));
BUG_ON(tag_get(node, 2, offset));
@@ -941,6 +1028,40 @@ void radix_tree_replace_slot(struct radix_tree_root *root,
replace_slot(root, NULL, slot, item, true);
}
+#ifdef CONFIG_RADIX_TREE_MULTIORDER
+/**
+ * radix_tree_join - replace multiple entries with one multiorder entry
+ * @root: radix tree root
+ * @index: an index inside the new entry
+ * @order: order of the new entry
+ * @item: new entry
+ *
+ * Call this function to replace several entries with one larger entry.
+ * The existing entries are presumed to not need freeing as a result of
+ * this call.
+ *
+ * The replacement entry will have all the tags set on it that were set
+ * on any of the entries it is replacing.
+ */
+int radix_tree_join(struct radix_tree_root *root, unsigned long index,
+ unsigned order, void *item)
+{
+ struct radix_tree_node *node;
+ void **slot;
+ int error;
+
+ BUG_ON(radix_tree_is_internal_node(item));
+
+ error = __radix_tree_create(root, index, order, &node, &slot);
+ if (!error)
+ error = insert_entries(node, slot, item, order, true);
+ if (error > 0)
+ error = 0;
+
+ return error;
+}
+#endif
+
/**
* radix_tree_tag_set - set a tag on a radix tree node
* @root: radix tree root
diff --git a/tools/testing/radix-tree/multiorder.c b/tools/testing/radix-tree/multiorder.c
index 86daf23..c9f656c 100644
--- a/tools/testing/radix-tree/multiorder.c
+++ b/tools/testing/radix-tree/multiorder.c
@@ -332,6 +332,63 @@ void multiorder_tagged_iteration(void)
item_kill_tree(&tree);
}
+static void __multiorder_join(unsigned long index,
+ unsigned order1, unsigned order2)
+{
+ unsigned long loc;
+ void *item, *item2 = item_create(index + 1, order1);
+ RADIX_TREE(tree, GFP_KERNEL);
+
+ item_insert_order(&tree, index, order2);
+ item = radix_tree_lookup(&tree, index);
+ radix_tree_join(&tree, index + 1, order1, item2);
+ loc = find_item(&tree, item);
+ if (loc == -1)
+ free(item);
+ item = radix_tree_lookup(&tree, index + 1);
+ assert(item == item2);
+ item_kill_tree(&tree);
+}
+
+static void __multiorder_join2(unsigned order1, unsigned order2)
+{
+ RADIX_TREE(tree, GFP_KERNEL);
+ struct radix_tree_node *node;
+ void *item1 = item_create(0, order1);
+ void *item2;
+
+ item_insert_order(&tree, 0, order2);
+ radix_tree_insert(&tree, 1 << order2, (void *)0x12UL);
+ item2 = __radix_tree_lookup(&tree, 1 << order2, &node, NULL);
+ assert(item2 == (void *)0x12UL);
+ assert(node->exceptional == 1);
+
+ radix_tree_join(&tree, 0, order1, item1);
+ item2 = __radix_tree_lookup(&tree, 1 << order2, &node, NULL);
+ assert(item2 == item1);
+ assert(node->exceptional == 0);
+ item_kill_tree(&tree);
+}
+
+static void multiorder_join(void)
+{
+ int i, j, idx;
+
+ for (idx = 0; idx < 1024; idx = idx * 2 + 3) {
+ for (i = 1; i < 15; i++) {
+ for (j = 0; j < i; j++) {
+ __multiorder_join(idx, i, j);
+ }
+ }
+ }
+
+ for (i = 1; i < 15; i++) {
+ for (j = 0; j < i; j++) {
+ __multiorder_join2(i, j);
+ }
+ }
+}
+
void multiorder_checks(void)
{
int i;
@@ -349,4 +406,5 @@ void multiorder_checks(void)
multiorder_tag_tests();
multiorder_iteration();
multiorder_tagged_iteration();
+ multiorder_join();
}
--
2.10.2
[toc] | [prev] | [next] | [standalone]
| From | Matthew Wilcox <mawilcox@linuxonhyperv.com> |
|---|---|
| Date | 2016-11-28 21:20 +0100 |
| Subject | [PATCH v3 18/33] radix-tree: Improve dump output |
| Message-ID | <sIAz8-6P8-15@gated-at.bofh.it> |
| In reply to | #1531607 |
From: Matthew Wilcox <willy@infradead.org>
Print the indices of the entries as unsigned (instead of signed) integers
and print the parent node of each entry to help navigate around larger
trees where the layout is not quite so obvious. Print the indices
covered by a node. Rearrange the order of fields printed so the indices
and parents line up for each type of entry.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
---
lib/radix-tree.c | 49 ++++++++++++++++++++++++++-----------------------
1 file changed, 26 insertions(+), 23 deletions(-)
diff --git a/lib/radix-tree.c b/lib/radix-tree.c
index 01ed70b..49b320e 100644
--- a/lib/radix-tree.c
+++ b/lib/radix-tree.c
@@ -213,15 +213,29 @@ radix_tree_find_next_bit(struct radix_tree_node *node, unsigned int tag,
return RADIX_TREE_MAP_SIZE;
}
+/*
+ * The maximum index which can be stored in a radix tree
+ */
+static inline unsigned long shift_maxindex(unsigned int shift)
+{
+ return (RADIX_TREE_MAP_SIZE << shift) - 1;
+}
+
+static inline unsigned long node_maxindex(struct radix_tree_node *node)
+{
+ return shift_maxindex(node->shift);
+}
+
#ifndef __KERNEL__
static void dump_node(struct radix_tree_node *node, unsigned long index)
{
unsigned long i;
- pr_debug("radix node: %p offset %d tags %lx %lx %lx shift %d count %d exceptional %d parent %p\n",
- node, node->offset,
+ pr_debug("radix node: %p offset %d indices %lu-%lu parent %p tags %lx %lx %lx shift %d count %d exceptional %d\n",
+ node, node->offset, index, index | node_maxindex(node),
+ node->parent,
node->tags[0][0], node->tags[1][0], node->tags[2][0],
- node->shift, node->count, node->exceptional, node->parent);
+ node->shift, node->count, node->exceptional);
for (i = 0; i < RADIX_TREE_MAP_SIZE; i++) {
unsigned long first = index | (i << node->shift);
@@ -229,14 +243,16 @@ static void dump_node(struct radix_tree_node *node, unsigned long index)
void *entry = node->slots[i];
if (!entry)
continue;
- if (is_sibling_entry(node, entry)) {
- pr_debug("radix sblng %p offset %ld val %p indices %ld-%ld\n",
- entry, i,
- *(void **)entry_to_node(entry),
- first, last);
+ if (entry == RADIX_TREE_RETRY) {
+ pr_debug("radix retry offset %ld indices %lu-%lu parent %p\n",
+ i, first, last, node);
} else if (!radix_tree_is_internal_node(entry)) {
- pr_debug("radix entry %p offset %ld indices %ld-%ld\n",
- entry, i, first, last);
+ pr_debug("radix entry %p offset %ld indices %lu-%lu parent %p\n",
+ entry, i, first, last, node);
+ } else if (is_sibling_entry(node, entry)) {
+ pr_debug("radix sblng %p offset %ld indices %lu-%lu parent %p val %p\n",
+ entry, i, first, last, node,
+ *(void **)entry_to_node(entry));
} else {
dump_node(entry_to_node(entry), first);
}
@@ -454,19 +470,6 @@ int radix_tree_maybe_preload_order(gfp_t gfp_mask, int order)
return __radix_tree_preload(gfp_mask, nr_nodes);
}
-/*
- * The maximum index which can be stored in a radix tree
- */
-static inline unsigned long shift_maxindex(unsigned int shift)
-{
- return (RADIX_TREE_MAP_SIZE << shift) - 1;
-}
-
-static inline unsigned long node_maxindex(struct radix_tree_node *node)
-{
- return shift_maxindex(node->shift);
-}
-
static unsigned radix_tree_load_root(struct radix_tree_root *root,
struct radix_tree_node **nodep, unsigned long *maxindex)
{
--
2.10.2
[toc] | [prev] | [next] | [standalone]
| From | Matthew Wilcox <mawilcox@linuxonhyperv.com> |
|---|---|
| Date | 2016-11-28 21:20 +0100 |
| Subject | [PATCH v3 05/33] radix tree test suite: Free preallocated nodes |
| Message-ID | <sIAz8-6P8-17@gated-at.bofh.it> |
| In reply to | #1531607 |
From: Matthew Wilcox <willy@infradead.org>
It can be a source of mild concern when the test suite shows that we're
leaking nodes. While poring over the source code looking for leaks
can lead to some fascinating bugs being discovered, sometimes the leak
is simply that these nodes were preallocated and are sitting on the
per-CPU list. Free them by calling the CPU dead callback.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
---
tools/testing/radix-tree/main.c | 3 +++
tools/testing/radix-tree/test.h | 1 +
2 files changed, 4 insertions(+)
diff --git a/tools/testing/radix-tree/main.c b/tools/testing/radix-tree/main.c
index 64ffe67..52ce1ea 100644
--- a/tools/testing/radix-tree/main.c
+++ b/tools/testing/radix-tree/main.c
@@ -344,6 +344,9 @@ int main(int argc, char **argv)
iteration_test();
single_thread_tests(long_run);
+ /* Free any remaining preallocated nodes */
+ radix_tree_cpu_dead(0);
+
sleep(1);
printf("after sleep(1): %d allocated, preempt %d\n",
nr_allocated, preempt_count);
diff --git a/tools/testing/radix-tree/test.h b/tools/testing/radix-tree/test.h
index 217fb24..5d2fad0 100644
--- a/tools/testing/radix-tree/test.h
+++ b/tools/testing/radix-tree/test.h
@@ -44,3 +44,4 @@ void radix_tree_dump(struct radix_tree_root *root);
int root_tag_get(struct radix_tree_root *root, unsigned int tag);
unsigned long node_maxindex(struct radix_tree_node *);
unsigned long shift_maxindex(unsigned int shift);
+int radix_tree_cpu_dead(unsigned int cpu);
--
2.10.2
[toc] | [prev] | [next] | [standalone]
Page 1 of 2 [1] 2 Next page →
Back to top | Article view | linux.kernel
csiph-web