Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1450355 > unrolled thread

[PATCH 0/3] new feature: monitoring page cache events

Back to article view | Back to linux.kernel

Contents

  [PATCH 0/3] new feature: monitoring page cache events George Amvrosiadis <gamvrosi@gmail.com> - 2016-07-26 06:10 +0200
    [PATCH 2/3] mm/duet: syscall wiring George Amvrosiadis <gamvrosi@gmail.com> - 2016-07-26 06:10 +0200
    [PATCH 1/3] mm: support for duet hooks George Amvrosiadis <gamvrosi@gmail.com> - 2016-07-26 06:10 +0200
    Re: [PATCH 0/3] new feature: monitoring page cache events Dave Hansen <dave.hansen@intel.com> - 2016-07-28 23:10 +0200
      Re: [PATCH 0/3] new feature: monitoring page cache events George Amvrosiadis <gamvrosi@gmail.com> - 2016-07-29 05:50 +0200
        Re: [PATCH 0/3] new feature: monitoring page cache events Dave Hansen <dave.hansen@intel.com> - 2016-07-29 17:40 +0200
          Re: [PATCH 0/3] new feature: monitoring page cache events George Amvrosiadis <gamvrosi@gmail.com> - 2016-07-30 19:40 +0200
            Re: [PATCH 0/3] new feature: monitoring page cache events Dave Hansen <dave.hansen@intel.com> - 2016-08-01 16:20 +0200

#1450355 — [PATCH 0/3] new feature: monitoring page cache events

I'm attaching a patch set implementing a mechanism we call Duet, which allows
applications to monitor events at the page cache level: page additions,
removals, dirtying, and flushing. Using such events, applications can identify
and prioritize processing of cached data, thereby reducing their I/O footprint.

One user of these events are maintenance tasks that scan large amounts of data
(e.g., backup, defrag, scrubbing). Knowing what is currently cached allows them
to piggy-back on each other and other applications running in the system. I've
managed to run up to 3 such applications together (backup, scrubbing, defrag)
and have them finish their work with 1/3rd of the I/O by using Duet. In this
case, the task that traversed the data the fastest (scrubber) allowed the rest
of the tasks to piggyback on the data brought into the cache. I.e., a file that
was read to be backed up was also picked up by the scrubber and defrag process.

I've found adapting applications to be straight-forward. Although I don't
include examples in this patch set, I've adapted btrfs scrubbing, btrfs send
(backup), btrfs defrag, rsync, and f2fs garbage collection in a few hundred
lines of code each (basically just had to add an event handler and wire it up
to the task's processing loop). You can read more about this in our full paper:
http://dl.acm.org/citation.cfm?id=2815424. I'd be happy to generate subsequent
patch sets for individual tasks if there's interest in this one. We've also
used Duet to speed up Hadoop and Spark by taking into account cache residency
of HDFS blocks across the cluster, when scheduling tasks, by up to 54%
depending on overlap on the data processed:
https://www.usenix.org/conference/hotstorage16/workshop-program/presentation/deslauriers


Syscall interface (and how it works): Duet uses hooks into the page cache (see
the "mm: support for duet hooks" patch). These hooks inform Duet of page events,
which are stored in a hash table. Only events that are of interest to running
tasks are stored, and only one copy of each event is stored for all interested
tasks. To register for events, the following syscalls are used (see the
"mm/duet: syscall wiring" patch for prototypes):

- sys_duet_init(char *taskname, u32 regmask, char *path): returns an fd that
  watches for events under PATH (e.g. '/home') and are also described in the
  REGMASK (e.g. DUET_PAGE_ADDED | DUET_PAGE_REMOVED). TASKNAME is an optional,
  human-readable name for the task.

- sys_duet_bmap(u16 flags, struct duet_uuid_arg *uuid): Duet allows applications
  to track processed items on an internal bitmap (which improves performance by
  being used to filter unnecessary events). The specified UUID is what read()
  returns on the fd created with sys_duet_init(), and uniquely identifies a
  file. FLAGS allow the bitmap to be set, reset, or have its state checked.

- sys_duet_get_path(struct duet_uuid_arg *uuid, char *buf, int bufsize):
  Applications running with Duet do not understand UUIDs, but pathnames. This
  syscall traverses the dentry cache and returns the corresponding path in BUF.

- sys_duet_status(u16 flags, struct duet_status_args *arg): Currently, the Duet
  framework can be turned on/off manually. This allows the admin to specify the
  number of max applications that will be registered concurrently, which allows
  us to size the internal hash table nodes appropriately (and limit performance
  or memory overhead). The syscall is also used for debugging purposes. I think
  this functionality should probably be exposed through ioctl()s to a device,
  and I'm open to suggestions on how to improve the current implementation.

The framework itself (a bit less than 2300 LoC) is currently placed under
mm/duet and the code is included in the "mm/duet: framework code" patch.


Application interface: Applications interface with Duet through a user library,
which is available at https://github.com/gamvrosi/duet-tools. In the same repo,
I have included a dummy_task application which provides an example of how Duet
can be used.


Changelog: The patches are based on Linus' v4.7 tag, and touch on the following
parts of the kernel:

- mm/filemap.c and include/linux/page-flags.h: hooks in the page cache to track
  page events on page addition, removal, dirtying, and flushing.

- arch/x86/*, include/linux/syscalls.h, kernel/sys_ni.h: wiring the 4 syscalls

- mm/duet/*: framework code



George Amvrosiadis (3):
  mm: support for duet hooks
  mm/duet: syscall wiring
  mm/duet: framework code

 arch/x86/entry/syscalls/syscall_32.tbl |   4 +
 arch/x86/entry/syscalls/syscall_64.tbl |   4 +
 include/linux/duet.h                   |  43 +++
 include/linux/page-flags.h             |  53 +++
 include/linux/syscalls.h               |   8 +
 include/uapi/asm-generic/unistd.h      |  12 +-
 init/Kconfig                           |   2 +
 kernel/sys_ni.c                        |   6 +
 mm/Makefile                            |   1 +
 mm/duet/Kconfig                        |  31 ++
 mm/duet/Makefile                       |   7 +
 mm/duet/bittree.c                      | 537 ++++++++++++++++++++++++++++++
 mm/duet/common.h                       | 211 ++++++++++++
 mm/duet/debug.c                        |  98 ++++++
 mm/duet/hash.c                         | 315 ++++++++++++++++++
 mm/duet/hook.c                         |  81 +++++
 mm/duet/init.c                         | 172 ++++++++++
 mm/duet/path.c                         | 184 +++++++++++
 mm/duet/syscall.h                      |  61 ++++
 mm/duet/task.c                         | 584 +++++++++++++++++++++++++++++++++
 mm/filemap.c                           |  11 +
 21 files changed, 2424 insertions(+), 1 deletion(-)
 create mode 100644 include/linux/duet.h
 create mode 100644 mm/duet/Kconfig
 create mode 100644 mm/duet/Makefile
 create mode 100644 mm/duet/bittree.c
 create mode 100644 mm/duet/common.h
 create mode 100644 mm/duet/debug.c
 create mode 100644 mm/duet/hash.c
 create mode 100644 mm/duet/hook.c
 create mode 100644 mm/duet/init.c
 create mode 100644 mm/duet/path.c
 create mode 100644 mm/duet/syscall.h
 create mode 100644 mm/duet/task.c

-- 
2.7.4

[toc] | [next] | [standalone]

#1450356 — [PATCH 2/3] mm/duet: syscall wiring

Usual syscall wiring for the four Duet syscalls.

Signed-off-by: George Amvrosiadis <gamvrosi@gmail.com>
---
 arch/x86/entry/syscalls/syscall_32.tbl |  4 ++++
 arch/x86/entry/syscalls/syscall_64.tbl |  4 ++++
 include/linux/syscalls.h               |  8 ++++++++
 include/uapi/asm-generic/unistd.h      | 12 +++++++++++-
 kernel/sys_ni.c                        |  6 ++++++
 5 files changed, 33 insertions(+), 1 deletion(-)

diff --git a/arch/x86/entry/syscalls/syscall_32.tbl b/arch/x86/entry/syscalls/syscall_32.tbl
index 4cddd17..f34ff94 100644
--- a/arch/x86/entry/syscalls/syscall_32.tbl
+++ b/arch/x86/entry/syscalls/syscall_32.tbl
@@ -386,3 +386,7 @@
 377	i386	copy_file_range		sys_copy_file_range
 378	i386	preadv2			sys_preadv2			compat_sys_preadv2
 379	i386	pwritev2		sys_pwritev2			compat_sys_pwritev2
+380	i386	duet_status		sys_duet_status
+381	i386	duet_init		sys_duet_init
+382	i386	duet_bmap		sys_duet_bmap
+383	i386	duet_get_path		sys_duet_get_path
diff --git a/arch/x86/entry/syscalls/syscall_64.tbl b/arch/x86/entry/syscalls/syscall_64.tbl
index 555263e..d04efaa 100644
--- a/arch/x86/entry/syscalls/syscall_64.tbl
+++ b/arch/x86/entry/syscalls/syscall_64.tbl
@@ -335,6 +335,10 @@
 326	common	copy_file_range		sys_copy_file_range
 327	64	preadv2			sys_preadv2
 328	64	pwritev2		sys_pwritev2
+329	common	duet_status		sys_duet_status
+330	common	duet_init		sys_duet_init
+331	common	duet_bmap		sys_duet_bmap
+332	common	duet_get_path		sys_duet_get_path
 
 #
 # x32-specific system call numbers start at 512 to avoid cache impact
diff --git a/include/linux/syscalls.h b/include/linux/syscalls.h
index d022390..da1049e 100644
--- a/include/linux/syscalls.h
+++ b/include/linux/syscalls.h
@@ -65,6 +65,8 @@ struct old_linux_dirent;
 struct perf_event_attr;
 struct file_handle;
 struct sigaltstack;
+struct duet_status_args;
+struct duet_uuid_arg;
 union bpf_attr;
 
 #include <linux/types.h>
@@ -898,4 +900,10 @@ asmlinkage long sys_copy_file_range(int fd_in, loff_t __user *off_in,
 
 asmlinkage long sys_mlock2(unsigned long start, size_t len, int flags);
 
+asmlinkage long sys_duet_status(u16 flags, struct duet_status_args __user *arg);
+asmlinkage long sys_duet_init(const char __user *taskname, u32 regmask,
+			      const char __user *pathname);
+asmlinkage long sys_duet_bmap(u16 flags, struct duet_uuid_arg __user *arg);
+asmlinkage long sys_duet_get_path(struct duet_uuid_arg __user *uarg,
+				  char __user *pathbuf, int pathbufsize);
 #endif
diff --git a/include/uapi/asm-generic/unistd.h b/include/uapi/asm-generic/unistd.h
index a26415b..7c287c0 100644
--- a/include/uapi/asm-generic/unistd.h
+++ b/include/uapi/asm-generic/unistd.h
@@ -725,8 +725,18 @@ __SC_COMP(__NR_preadv2, sys_preadv2, compat_sys_preadv2)
 #define __NR_pwritev2 287
 __SC_COMP(__NR_pwritev2, sys_pwritev2, compat_sys_pwritev2)
 
+/* mm/duet/syscall.c */
+#define __NR_duet_status 288
+__SYSCALL(__NR_duet_status, sys_duet_status)
+#define __NR_duet_init 289
+__SYSCALL(__NR_duet_init, sys_duet_init)
+#define __NR_duet_bmap 290
+__SYSCALL(__NR_duet_bmap, sys_duet_bmap)
+#define __NR_duet_get_path 291
+__SYSCALL(__NR_duet_get_path, sys_duet_get_path)
+
 #undef __NR_syscalls
-#define __NR_syscalls 288
+#define __NR_syscalls 292
 
 /*
  * All syscalls below here should go away really,
diff --git a/kernel/sys_ni.c b/kernel/sys_ni.c
index 2c5e3a8..3d4c53a 100644
--- a/kernel/sys_ni.c
+++ b/kernel/sys_ni.c
@@ -176,6 +176,12 @@ cond_syscall(sys_capget);
 cond_syscall(sys_capset);
 cond_syscall(sys_copy_file_range);
 
+/* Duet syscall entries */
+cond_syscall(sys_duet_status);
+cond_syscall(sys_duet_init);
+cond_syscall(sys_duet_bmap);
+cond_syscall(sys_duet_get_path);
+
 /* arch-specific weak syscall entries */
 cond_syscall(sys_pciconfig_read);
 cond_syscall(sys_pciconfig_write);
-- 
2.7.4

[toc] | [prev] | [next] | [standalone]

#1450358 — [PATCH 1/3] mm: support for duet hooks

Adds the Duet hooks in the page cache. In filemap.c, two hooks are added at the
time of addition and removal of a page descriptor. In page-flags.h, two more
hooks are added to track page dirtying and flushing.

The hooks are inactive while Duet is offline.

Signed-off-by: George Amvrosiadis <gamvrosi@gmail.com>
---
 include/linux/duet.h       | 43 +++++++++++++++++++++++++++++++++++++
 include/linux/page-flags.h | 53 ++++++++++++++++++++++++++++++++++++++++++++++
 mm/filemap.c               | 11 ++++++++++
 3 files changed, 107 insertions(+)
 create mode 100644 include/linux/duet.h

diff --git a/include/linux/duet.h b/include/linux/duet.h
new file mode 100644
index 0000000..80491e2
--- /dev/null
+++ b/include/linux/duet.h
@@ -0,0 +1,43 @@
+/*
+ * Defs necessary for Duet hooks
+ *
+ * Author: George Amvrosiadis <gamvrosi@gmail.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public
+ * License v2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ */
+#ifndef _DUET_H
+#define _DUET_H
+
+/*
+ * Duet hooks into the page cache to monitor four types of events:
+ *   ADDED:	a page __descriptor__ was inserted into the page cache
+ *   REMOVED:	a page __describptor__ was removed from the page cache
+ *   DIRTY:	page's dirty bit was set
+ *   FLUSHED:	page's dirty bit was cleared
+ */
+#define DUET_PAGE_ADDED		0x0001
+#define DUET_PAGE_REMOVED	0x0002
+#define DUET_PAGE_DIRTY		0x0004
+#define DUET_PAGE_FLUSHED	0x0008
+
+#define DUET_HOOK(funp, evt, data) \
+	do { \
+		rcu_read_lock(); \
+		funp = rcu_dereference(duet_hook_fp); \
+		if (funp) \
+			funp(evt, (void *)data); \
+		rcu_read_unlock(); \
+	} while (0)
+
+/* Hook function pointer initialized by the Duet framework */
+typedef void (duet_hook_t) (__u16, void *);
+extern duet_hook_t *duet_hook_fp;
+
+#endif /* _DUET_H */
diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index e5a3244..53be4a0 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -12,6 +12,9 @@
 #include <linux/mm_types.h>
 #include <generated/bounds.h>
 #endif /* !__GENERATING_BOUNDS_H */
+#ifdef CONFIG_DUET
+#include <linux/duet.h>
+#endif /* CONFIG_DUET */
 
 /*
  * Various page->flags bits:
@@ -254,8 +257,58 @@ PAGEFLAG(Error, error, PF_NO_COMPOUND) TESTCLEARFLAG(Error, error, PF_NO_COMPOUN
 PAGEFLAG(Referenced, referenced, PF_HEAD)
 	TESTCLEARFLAG(Referenced, referenced, PF_HEAD)
 	__SETPAGEFLAG(Referenced, referenced, PF_HEAD)
+#ifdef CONFIG_DUET
+TESTPAGEFLAG(Dirty, dirty, PF_HEAD)
+
+static inline void SetPageDirty(struct page *page)
+{
+	duet_hook_t *dhfp = NULL;
+
+	if (!test_and_set_bit(PG_dirty, &page->flags))
+		DUET_HOOK(dhfp, DUET_PAGE_DIRTY, page);
+}
+
+static inline void __ClearPageDirty(struct page *page)
+{
+	duet_hook_t *dhfp = NULL;
+
+	if (__test_and_clear_bit(PG_dirty, &page->flags))
+		DUET_HOOK(dhfp, DUET_PAGE_FLUSHED, page);
+}
+
+static inline void ClearPageDirty(struct page *page)
+{
+	duet_hook_t *dhfp = NULL;
+
+	if (test_and_clear_bit(PG_dirty, &page->flags))
+		DUET_HOOK(dhfp, DUET_PAGE_FLUSHED, page);
+}
+
+static inline int TestSetPageDirty(struct page *page)
+{
+	duet_hook_t *dhfp = NULL;
+
+	if (!test_and_set_bit(PG_dirty, &page->flags)) {
+		DUET_HOOK(dhfp, DUET_PAGE_DIRTY, page);
+		return 0;
+	}
+	return 1;
+}
+
+static inline int TestClearPageDirty(struct page *page)
+{
+	duet_hook_t *dhfp = NULL;
+
+	if (test_and_clear_bit(PG_dirty, &page->flags)) {
+		DUET_HOOK(dhfp, DUET_PAGE_FLUSHED, page);
+		return 1;
+	}
+	return 0;
+}
+#else
 PAGEFLAG(Dirty, dirty, PF_HEAD) TESTSCFLAG(Dirty, dirty, PF_HEAD)
 	__CLEARPAGEFLAG(Dirty, dirty, PF_HEAD)
+#endif /* CONFIG_DUET */
 PAGEFLAG(LRU, lru, PF_HEAD) __CLEARPAGEFLAG(LRU, lru, PF_HEAD)
 PAGEFLAG(Active, active, PF_HEAD) __CLEARPAGEFLAG(Active, active, PF_HEAD)
 	TESTCLEARFLAG(Active, active, PF_HEAD)
diff --git a/mm/filemap.c b/mm/filemap.c
index 20f3b1f..f06ebc0 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -166,6 +166,11 @@ static void page_cache_tree_delete(struct address_space *mapping,
 void __delete_from_page_cache(struct page *page, void *shadow)
 {
 	struct address_space *mapping = page->mapping;
+#ifdef CONFIG_DUET
+	duet_hook_t *dhfp = NULL;
+
+	DUET_HOOK(dhfp, DUET_PAGE_REMOVED, page);
+#endif /* CONFIG_DUET */
 
 	trace_mm_filemap_delete_from_page_cache(page);
 	/*
@@ -628,6 +633,9 @@ static int __add_to_page_cache_locked(struct page *page,
 	int huge = PageHuge(page);
 	struct mem_cgroup *memcg;
 	int error;
+#ifdef CONFIG_DUET
+	duet_hook_t *dhfp = NULL;
+#endif
 
 	VM_BUG_ON_PAGE(!PageLocked(page), page);
 	VM_BUG_ON_PAGE(PageSwapBacked(page), page);
@@ -663,6 +671,9 @@ static int __add_to_page_cache_locked(struct page *page,
 	if (!huge)
 		mem_cgroup_commit_charge(page, memcg, false, false);
 	trace_mm_filemap_add_to_page_cache(page);
+#ifdef CONFIG_DUET
+	DUET_HOOK(dhfp, DUET_PAGE_ADDED, page);
+#endif
 	return 0;
 err_insert:
 	page->mapping = NULL;
-- 
2.7.4

[toc] | [prev] | [next] | [standalone]

#1452103

On 07/25/2016 08:47 PM, George Amvrosiadis wrote:
>  21 files changed, 2424 insertions(+), 1 deletion(-)

I like the idea, but yikes, that's a lot of code.

Have you considered using or augmenting the kernel's existing tracing
mechanisms?  Have you considered using something like netlink for
transporting the data out of the kernel?

The PageDirty() hooks look simple but turn out to be horribly deep.
Where we used to have a plain old bit set, we now have new locks,
potentially long periods of irq disabling, and loops over all the tasks
doing duet, even path lookup!

Given a big system, I would imagine these locks slowing down
SetPageDirty() and things like write() pretty severely.  Have you done
an assessment of the performance impact of this change?   I can't
imagine this being used in any kind of performance or
scalability-sensitive environment.

The current tracing code has a model where the trace producers put data
in *one* place, then all the mulitple consumers pull it out of that
place.  Duet seems to have the model that the producer puts the data in
multiple places and consumers consume it from their own private copies.
 That seems a bit backwards and puts cost directly in to hot code paths.
 Even a single task watching a single file on the system makes everyone
go in and pay some of this cost for every SetPageDirty().

Let's say we had a big system with virtually everything sitting in the
page cache.  Does duet have a way to find things currently _in_ the
cache, or only when things move in/out of it?

Tasks seem to have a fixed 'struct path' ->regpath at duet_task_init()
time.  The code goes page->mapping->inode->i_dentry and then tries to
compare that with the originally recorded path.  Does this even work in
the face of things like bind mounts, mounts that change after
duet_task_init(), or mounting a fs with a different superblock
underneath a watched path?  It seems awfully fragile.

[toc] | [prev] | [next] | [standalone]

#1452238

On Thu, Jul 28, 2016 at 02:02:45PM -0700, Dave Hansen wrote:
> On 07/25/2016 08:47 PM, George Amvrosiadis wrote:
> >  21 files changed, 2424 insertions(+), 1 deletion(-)
> 
> I like the idea, but yikes, that's a lot of code.
> 
> Have you considered using or augmenting the kernel's existing tracing
> mechanisms?  Have you considered using something like netlink for
> transporting the data out of the kernel?
>

We contemplated a couple other solutions. One was extending existing debugging
mechanisms. E.g., there are already tracepoints at __add_to_page_cache_locked()
and __delete_from_page_cache(). A consistent reaction I got for doing that,
however, was that of exposing an unruly interface to user applications, which
is certainly not an elegant solution. After that experience, and reading up on
issues of the auditing subsystem (https://lwn.net/Articles/600568/) I decided
to avoid going further down that path.

> The PageDirty() hooks look simple but turn out to be horribly deep.
> Where we used to have a plain old bit set, we now have new locks,
> potentially long periods of irq disabling, and loops over all the tasks
> doing duet, even path lookup!
> 

It's true that the hooks are deep, but they are fully exercised only once per
inode, per task. The reasoning behind the 'struct bmap_rbnode'->seen bitmap is
to remember whether an inode was seen before by a given task. During that first
access is when we do the path lookup to decide whether this inode is relevant
to the task and mark 'struct bmap_rbnode'->relv accordingly. If it is not, we
ignore future events from it. Tasks can also use the 'structu bmap_rbnode'->done
bitmap to indicate that they are done with a specific inode, which also stops
those events from passing that task loop.

> Given a big system, I would imagine these locks slowing down
> SetPageDirty() and things like write() pretty severely.  Have you done
> an assessment of the performance impact of this change?   I can't
> imagine this being used in any kind of performance or
> scalability-sensitive environment.
> 

I have used filebench to saturate an HDD and an SSD, registered a task to be
notified about every file in the filesystem, and measured no difference in I/O
throughput. To measure the CPU utilization of Duet, I tried an extreme case
where I booted using only one core and again saturated an HDD using filebench.
There was a 1-1.5% increase in CPU utilization. There is a description of this
result in the paper. I have also tuned filebench to hit the cache often in my
experiments (more than 60-70% of accesses going to less than 10% of the data),
but the results were similar. For the Hadoop and Spark experiments we used a
24-node cluster and these overhead numbers didn't seem to affect performance.

> The current tracing code has a model where the trace producers put data
> in *one* place, then all the mulitple consumers pull it out of that
> place.  Duet seems to have the model that the producer puts the data in
> multiple places and consumers consume it from their own private copies.
>  That seems a bit backwards and puts cost directly in to hot code paths.
>  Even a single task watching a single file on the system makes everyone
> go in and pay some of this cost for every SetPageDirty().
> 

Duet operates in a similar way. There is one large global hash table to avoid
collisions, so that on average a single lookup is sufficient to place a page in
it. Due to its global nature, if a page is of interest to multiple tasks, only
one entry is used to hold the events for that page across all tasks. And to
avoid walking that hash table for relevant events on a read(), each task
maintains a separate bitmap of the hash table's buckets that tells it which
buckets to look into. (In the past I've also tried a work queue approach on the
hot code path, but the overhead was almost double as a result of allocating the
work queue items.)

Having said all the above, Dave, I've seen your work at the 2013 Linux Plumbers
Conference on scalability issues, so if you think I'm missing something in my
replies, please call me out on that. I'm definitely open to improving this code.

> Let's say we had a big system with virtually everything sitting in the
> page cache.  Does duet have a way to find things currently _in_ the
> cache, or only when things move in/out of it?
> 

At task registration time we grab the superblock for the filesystem of the
registered path, and then scan_page_cache() traverses the list of inodes
currently in memory. We enqueue ADDED and DIRTY events for relevant inodes
as needed.

> Tasks seem to have a fixed 'struct path' ->regpath at duet_task_init()
> time.  The code goes page->mapping->inode->i_dentry and then tries to
> compare that with the originally recorded path.  Does this even work in
> the face of things like bind mounts, mounts that change after
> duet_task_init(), or mounting a fs with a different superblock
> underneath a watched path?  It seems awfully fragile.

This is an excellent point. Currently any events that occur on inodes of a
different superblock would get filtered at duet_hook() for those tasks that
haven't registered with that superblock. One solution is to do a duet_init()
once per file system, and have the user application use select() on all those
fds. This could potentially be done under the covers by the userlevel library.

I'm not sure how to handle mounts that change, however. At the very least I
would like to be able to somehow inform Duet at unmount to close any
outstanding fds. Any ideas/thoughts on this would be really appreciated,
obviously.

[toc] | [prev] | [next] | [standalone]

#1452452

On 07/28/2016 08:47 PM, George Amvrosiadis wrote:
> On Thu, Jul 28, 2016 at 02:02:45PM -0700, Dave Hansen wrote:
>> On 07/25/2016 08:47 PM, George Amvrosiadis wrote:
>>>  21 files changed, 2424 insertions(+), 1 deletion(-)
>>
>> I like the idea, but yikes, that's a lot of code.
>>
>> Have you considered using or augmenting the kernel's existing tracing
>> mechanisms?  Have you considered using something like netlink for
>> transporting the data out of the kernel?
> 
> We contemplated a couple other solutions. One was extending existing debugging
> mechanisms. E.g., there are already tracepoints at __add_to_page_cache_locked()
> and __delete_from_page_cache(). A consistent reaction I got for doing that,
> however, was that of exposing an unruly interface to user applications, which
> is certainly not an elegant solution.

What's to stop you from using tracing to gather and transport data out
of the kernel and then aggregate and present it to apps in an "elegant"
way of your choosing?

>> The PageDirty() hooks look simple but turn out to be horribly deep.
>> Where we used to have a plain old bit set, we now have new locks,
>> potentially long periods of irq disabling, and loops over all the tasks
>> doing duet, even path lookup!
> 
> It's true that the hooks are deep, but they are fully exercised only once per
> inode, per task. The reasoning behind the 'struct bmap_rbnode'->seen bitmap is
> to remember whether an inode was seen before by a given task. During that first
> access is when we do the path lookup to decide whether this inode is relevant
> to the task and mark 'struct bmap_rbnode'->relv accordingly. If it is not, we
> ignore future events from it. Tasks can also use the 'structu bmap_rbnode'->done
> bitmap to indicate that they are done with a specific inode, which also stops
> those events from passing that task loop.

OK, but it still disables interrupts and takes a spinlock for each
bitmap it checks.  That spinlock becomes essentially a global lock in a
hot path, which can't be good.

>> Given a big system, I would imagine these locks slowing down
>> SetPageDirty() and things like write() pretty severely.  Have you done
>> an assessment of the performance impact of this change?   I can't
>> imagine this being used in any kind of performance or
>> scalability-sensitive environment.
> 
> I have used filebench to saturate an HDD and an SSD, registered a task to be
> notified about every file in the filesystem, and measured no difference in I/O
> throughput. To measure the CPU utilization of Duet, I tried an extreme case
> where I booted using only one core and again saturated an HDD using filebench.
> There was a 1-1.5% increase in CPU utilization. There is a description of this
> result in the paper. I have also tuned filebench to hit the cache often in my
> experiments (more than 60-70% of accesses going to less than 10% of the data),
> but the results were similar. For the Hadoop and Spark experiments we used a
> 24-node cluster and these overhead numbers didn't seem to affect performance.

I'd say testing with _more_ cores is important, not less.  How about
trying to watch a single file, then have one process per core writing to
a 1MB file in a loop.  What does that do?  Or, heck, just compile a
kernel on a modern 2-socket system.

In any case, I still can't see the current duet _model_ ever working
out, much less the implementation posted here.

>> The current tracing code has a model where the trace producers put data
>> in *one* place, then all the mulitple consumers pull it out of that
>> place.  Duet seems to have the model that the producer puts the data in
>> multiple places and consumers consume it from their own private copies.
>>  That seems a bit backwards and puts cost directly in to hot code paths.
>>  Even a single task watching a single file on the system makes everyone
>> go in and pay some of this cost for every SetPageDirty().
> 
> Duet operates in a similar way. There is one large global hash table to avoid
> collisions, so that on average a single lookup is sufficient to place a page in
> it. Due to its global nature, if a page is of interest to multiple tasks, only
> one entry is used to hold the events for that page across all tasks. And to
> avoid walking that hash table for relevant events on a read(), each task
> maintains a separate bitmap of the hash table's buckets that tells it which
> buckets to look into. (In the past I've also tried a work queue approach on the
> hot code path, but the overhead was almost double as a result of allocating the
> work queue items.)

I don't think Duet operates in a similar way.  Asserting that it does
makes me wary that you've understood and actually considered how tracing
works.

Duet takes global locks in hot paths.  The tracing code doesn't do that.
 It uses percpu buffers that don't require shared locks when adding
records.  It makes the reader of the data do all the hard work.

>> Tasks seem to have a fixed 'struct path' ->regpath at duet_task_init()
>> time.  The code goes page->mapping->inode->i_dentry and then tries to
>> compare that with the originally recorded path.  Does this even work in
>> the face of things like bind mounts, mounts that change after
>> duet_task_init(), or mounting a fs with a different superblock
>> underneath a watched path?  It seems awfully fragile.
> 
> This is an excellent point. Currently any events that occur on inodes of a
> different superblock would get filtered at duet_hook() for those tasks that
> haven't registered with that superblock. One solution is to do a duet_init()
> once per file system, and have the user application use select() on all those
> fds. This could potentially be done under the covers by the userlevel library.
> 
> I'm not sure how to handle mounts that change, however. At the very least I
> would like to be able to somehow inform Duet at unmount to close any
> outstanding fds. Any ideas/thoughts on this would be really appreciated,
> obviously.

It's complicated.  You can't simply toss things at unmount because it
might have been a bind mount and still be mounted somewhere else.

I don't think it's really even worth having an in-depth discussion of
how to modify duet.  I can't imagine that this would get merged as-is,
or even anything resembling the current design.  If you want to see
duet-like functionality in the kernel, I think it needs to be integrated
better and enhance or take advantage of existing mechanisms.

You've identified a real problem and a real solution, and it is in an
area where Linux is weak (monitoring the page cache).  If you are really
interested in seeing a solution that folks can use, I think you need to
find some way to leverage existing kernel functionality (ftrace,
fanotify, netlink, etc...), or come up with a much more compelling story
about why you can't use them.

[toc] | [prev] | [next] | [standalone]

#1452731

On Fri, Jul 29, 2016 at 08:33:34AM -0700, Dave Hansen wrote:
> What's to stop you from using tracing to gather and transport data out
> of the kernel and then aggregate and present it to apps in an "elegant"
> way of your choosing?
> 
> I don't think it's really even worth having an in-depth discussion of
> how to modify duet.  I can't imagine that this would get merged as-is,
> or even anything resembling the current design.  If you want to see
> duet-like functionality in the kernel, I think it needs to be integrated
> better and enhance or take advantage of existing mechanisms.
> 
> You've identified a real problem and a real solution, and it is in an
> area where Linux is weak (monitoring the page cache).  If you are really
> interested in seeing a solution that folks can use, I think you need to
> find some way to leverage existing kernel functionality (ftrace,
> fanotify, netlink, etc...), or come up with a much more compelling story
> about why you can't use them.

I took a few measurements of the ftrace overhead, and if limited to the page
cache functions we're interested in, it's very reasonable. Duet does depend
on exporting some data with each event, however, and tracepoints seem to be
the most efficient way to do this. There are two issues, however:

(a) There are no tracepoints for page dirtying and flushing. Those would have
to be added at the same place as the Duet hooks I submitted (unwrapping the
page-flags.h macros) to catch those cases where pages are locked and the dirty
bit is set manually.

(b) The page cache tracepoints are currently not exported symbols. If I can
export those four tracepoints for page addition, removal, dirtying, and
flushing, then the rest of the work (exporting the information to userspace)
can be carried out within a module. In the future, once we reach a point of
maturity where we are confident about the stability of the exporting interface
and performance, we could engage in another conversation about potentially
mainlining some of that code.

Dave, I can produce a patch that adds the extra two tracepoints and exports
all four tracepoint symbols. This would be a short patch that would just
extend existing tracing functionality. What do you think?

[toc] | [prev] | [next] | [standalone]

#1453236

On 07/30/2016 10:31 AM, George Amvrosiadis wrote:
> Dave, I can produce a patch that adds the extra two tracepoints and exports
> all four tracepoint symbols. This would be a short patch that would just
> extend existing tracing functionality. What do you think?

Adding those tracepoints is probably useful.  It's probably something we
need to have anyway as long as they don't cause too much code bloat or a
noticeable performance impact when they're off.

As for exporting symbols, that's not done until something is merged.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web