Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1351295 > unrolled thread
| Started by | Wang Nan <wangnan0@huawei.com> |
|---|---|
| First post | 2016-03-07 05:00 +0100 |
| Last post | 2016-03-08 22:10 +0100 |
| Articles | 20 on this page of 36 — 6 participants |
Back to article view | Back to linux.kernel
[RESEND PATCH 0/5] perf core: Support overwrite ring buffer Wang Nan <wangnan0@huawei.com> - 2016-03-07 05:00 +0100
[RESEND PATCH 5/5] perf core: Reduce perf event output overhead by new overflow handler Wang Nan <wangnan0@huawei.com> - 2016-03-07 05:00 +0100
[RESEND PATCH 4/5] perf core: Add backward attribute to perf event Wang Nan <wangnan0@huawei.com> - 2016-03-07 05:00 +0100
[RESEND PATCH 3/5] perf core: Prepare writing into ring buffer from end Wang Nan <wangnan0@huawei.com> - 2016-03-07 05:00 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Peter Zijlstra <peterz@infradead.org> - 2016-03-08 14:50 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Ingo Molnar <mingo@kernel.org> - 2016-03-08 14:50 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Peter Zijlstra <peterz@infradead.org> - 2016-03-08 15:00 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Ingo Molnar <mingo@kernel.org> - 2016-03-08 16:30 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Dmitry Vyukov <dvyukov@google.com> - 2016-03-08 16:40 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Ingo Molnar <mingo@kernel.org> - 2016-03-08 17:00 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Dmitry Vyukov <dvyukov@google.com> - 2016-03-08 17:20 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Ingo Molnar <mingo@kernel.org> - 2016-03-08 17:30 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Dmitry Vyukov <dvyukov@google.com> - 2016-03-08 17:40 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Peter Zijlstra <peterz@infradead.org> - 2016-03-08 17:40 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Ingo Molnar <mingo@kernel.org> - 2016-03-08 17:50 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Ingo Molnar <mingo@kernel.org> - 2016-03-08 17:50 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Dmitry Vyukov <dvyukov@google.com> - 2016-03-08 18:10 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Ingo Molnar <mingo@kernel.org> - 2016-03-08 18:30 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Dmitry Vyukov <dvyukov@google.com> - 2016-03-08 18:30 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Ingo Molnar <mingo@kernel.org> - 2016-03-08 18:40 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Dmitry Vyukov <dvyukov@google.com> - 2016-03-08 18:50 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Ingo Molnar <mingo@kernel.org> - 2016-03-08 18:50 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Peter Zijlstra <peterz@infradead.org> - 2016-03-08 19:00 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Ingo Molnar <mingo@kernel.org> - 2016-03-08 19:00 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Ingo Molnar <mingo@kernel.org> - 2016-03-08 19:10 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Ingo Molnar <mingo@kernel.org> - 2016-03-08 19:30 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Ingo Molnar <mingo@kernel.org> - 2016-03-08 19:40 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Dmitry Vyukov <dvyukov@google.com> - 2016-03-08 19:00 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Peter Zijlstra <peterz@infradead.org> - 2016-03-08 19:00 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Peter Zijlstra <peterz@infradead.org> - 2016-03-08 17:40 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Borislav Petkov <bp@alien8.de> - 2016-03-09 12:00 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Dmitry Vyukov <dvyukov@google.com> - 2016-03-09 12:30 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Jiri Olsa <jolsa@redhat.com> - 2016-03-08 21:00 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Peter Zijlstra <peterz@infradead.org> - 2016-03-08 21:10 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Jiri Olsa <jolsa@redhat.com> - 2016-03-08 21:50 +0100
Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Peter Zijlstra <peterz@infradead.org> - 2016-03-08 22:10 +0100
Page 1 of 2 [1] 2 Next page →
| From | Wang Nan <wangnan0@huawei.com> |
|---|---|
| Date | 2016-03-07 05:00 +0100 |
| Subject | [RESEND PATCH 0/5] perf core: Support overwrite ring buffer |
| Message-ID | <r9UuS-5Xo-5@gated-at.bofh.it> |
This patch set has been posted multiple times (with and without corresponding 'perf tool' patches), and doesn't receive further comment. I think it should be okay to merge them into mainline. There are many perf's improvement depend on it. However, Peter is not responsive after I fixed some problems he pointed out. Introduces 'write_backward' into perf_event_attr, allows kernel writing the ring buffer from the end of it. This feature allows extracting data from overwritable ring buffer. Wang Nan (5): perf core: Introduce new ioctl options to pause and resume ring buffer perf core: Set event's default overflow_handler perf core: Prepare writing into ring buffer from end perf core: Add backward attribute to perf event perf core: Reduce perf event output overhead by new overflow handler include/linux/perf_event.h | 22 +++++++++++-- include/uapi/linux/perf_event.h | 4 ++- kernel/events/core.c | 73 +++++++++++++++++++++++++++++++++++------ kernel/events/internal.h | 11 +++++++ kernel/events/ring_buffer.c | 63 +++++++++++++++++++++++++++++++---- 5 files changed, 153 insertions(+), 20 deletions(-) Cc: He Kuang <hekuang@huawei.com> Cc: Alexei Starovoitov <ast@kernel.org> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Brendan Gregg <brendan.d.gregg@gmail.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Zefan Li <lizefan@huawei.com> Cc: pi3orama@163.com -- 1.8.3.4
[toc] | [next] | [standalone]
| From | Wang Nan <wangnan0@huawei.com> |
|---|---|
| Date | 2016-03-07 05:00 +0100 |
| Subject | [RESEND PATCH 5/5] perf core: Reduce perf event output overhead by new overflow handler |
| Message-ID | <r9UuS-5Xo-17@gated-at.bofh.it> |
| In reply to | #1351295 |
By creating onward and backward specific overflow handlers and setting
them according to event's backward setting, normal sampling events
don't need checking backward setting of an event any more.
This is the last patch of backward writing patchset. After this patch,
there's no extra overhead introduced to the fast path of sampling
output.
Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: He Kuang <hekuang@huawei.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Brendan Gregg <brendan.d.gregg@gmail.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
---
include/linux/perf_event.h | 17 +++++++++++++++--
kernel/events/core.c | 41 ++++++++++++++++++++++++++++++++++++-----
kernel/events/ring_buffer.c | 12 ++++++++++++
3 files changed, 63 insertions(+), 7 deletions(-)
diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
index 88e2b47..bbe913f 100644
--- a/include/linux/perf_event.h
+++ b/include/linux/perf_event.h
@@ -829,9 +829,15 @@ extern int perf_event_overflow(struct perf_event *event,
struct perf_sample_data *data,
struct pt_regs *regs);
+extern void perf_event_output_onward(struct perf_event *event,
+ struct perf_sample_data *data,
+ struct pt_regs *regs);
+extern void perf_event_output_backward(struct perf_event *event,
+ struct perf_sample_data *data,
+ struct pt_regs *regs);
extern void perf_event_output(struct perf_event *event,
- struct perf_sample_data *data,
- struct pt_regs *regs);
+ struct perf_sample_data *data,
+ struct pt_regs *regs);
extern void
perf_event_header__init_id(struct perf_event_header *header,
@@ -1038,6 +1044,13 @@ static inline bool is_write_backward(struct perf_event *event)
extern int perf_output_begin(struct perf_output_handle *handle,
struct perf_event *event, unsigned int size);
+extern int perf_output_begin_onward(struct perf_output_handle *handle,
+ struct perf_event *event,
+ unsigned int size);
+extern int perf_output_begin_backward(struct perf_output_handle *handle,
+ struct perf_event *event,
+ unsigned int size);
+
extern void perf_output_end(struct perf_output_handle *handle);
extern unsigned int perf_output_copy(struct perf_output_handle *handle,
const void *buf, unsigned int len);
diff --git a/kernel/events/core.c b/kernel/events/core.c
index 513b556..969c43e 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -5606,9 +5606,13 @@ void perf_prepare_sample(struct perf_event_header *header,
}
}
-void perf_event_output(struct perf_event *event,
- struct perf_sample_data *data,
- struct pt_regs *regs)
+static void __always_inline
+__perf_event_output(struct perf_event *event,
+ struct perf_sample_data *data,
+ struct pt_regs *regs,
+ int (*output_begin)(struct perf_output_handle *,
+ struct perf_event *,
+ unsigned int))
{
struct perf_output_handle handle;
struct perf_event_header header;
@@ -5618,7 +5622,7 @@ void perf_event_output(struct perf_event *event,
perf_prepare_sample(&header, data, event, regs);
- if (perf_output_begin(&handle, event, header.size))
+ if (output_begin(&handle, event, header.size))
goto exit;
perf_output_sample(&handle, &header, data, event);
@@ -5629,6 +5633,30 @@ exit:
rcu_read_unlock();
}
+void
+perf_event_output_onward(struct perf_event *event,
+ struct perf_sample_data *data,
+ struct pt_regs *regs)
+{
+ __perf_event_output(event, data, regs, perf_output_begin_onward);
+}
+
+void
+perf_event_output_backward(struct perf_event *event,
+ struct perf_sample_data *data,
+ struct pt_regs *regs)
+{
+ __perf_event_output(event, data, regs, perf_output_begin_backward);
+}
+
+void
+perf_event_output(struct perf_event *event,
+ struct perf_sample_data *data,
+ struct pt_regs *regs)
+{
+ __perf_event_output(event, data, regs, perf_output_begin);
+}
+
/*
* read event_id
*/
@@ -7963,8 +7991,11 @@ perf_event_alloc(struct perf_event_attr *attr, int cpu,
if (overflow_handler) {
event->overflow_handler = overflow_handler;
event->overflow_handler_context = context;
+ } else if (is_write_backward(event)){
+ event->overflow_handler = perf_event_output_backward;
+ event->overflow_handler_context = NULL;
} else {
- event->overflow_handler = perf_event_output;
+ event->overflow_handler = perf_event_output_onward;
event->overflow_handler_context = NULL;
}
diff --git a/kernel/events/ring_buffer.c b/kernel/events/ring_buffer.c
index 80b1fa7..7e30e012 100644
--- a/kernel/events/ring_buffer.c
+++ b/kernel/events/ring_buffer.c
@@ -230,6 +230,18 @@ out:
return -ENOSPC;
}
+int perf_output_begin_onward(struct perf_output_handle *handle,
+ struct perf_event *event, unsigned int size)
+{
+ return __perf_output_begin(handle, event, size, false);
+}
+
+int perf_output_begin_backward(struct perf_output_handle *handle,
+ struct perf_event *event, unsigned int size)
+{
+ return __perf_output_begin(handle, event, size, true);
+}
+
int perf_output_begin(struct perf_output_handle *handle,
struct perf_event *event, unsigned int size)
{
--
1.8.3.4
[toc] | [prev] | [next] | [standalone]
| From | Wang Nan <wangnan0@huawei.com> |
|---|---|
| Date | 2016-03-07 05:00 +0100 |
| Subject | [RESEND PATCH 4/5] perf core: Add backward attribute to perf event |
| Message-ID | <r9UuS-5Xo-19@gated-at.bofh.it> |
| In reply to | #1351295 |
In perf_event_attr a new bit 'write_backward' is appended to indicate
this event should write ring buffer from its end to beginning.
In perf_output_begin(), prepare ring buffer according this bit.
This patch introduces small overhead into perf_output_begin():
an extra memory read and a conditional branch. Further patch can remove
this overhead by using custom output handler.
Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: He Kuang <hekuang@huawei.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Brendan Gregg <brendan.d.gregg@gmail.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
---
include/linux/perf_event.h | 5 +++++
include/uapi/linux/perf_event.h | 3 ++-
kernel/events/core.c | 7 +++++++
kernel/events/ring_buffer.c | 2 ++
4 files changed, 16 insertions(+), 1 deletion(-)
diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
index a9d8cab..88e2b47 100644
--- a/include/linux/perf_event.h
+++ b/include/linux/perf_event.h
@@ -1031,6 +1031,11 @@ static inline bool has_aux(struct perf_event *event)
return event->pmu->setup_aux;
}
+static inline bool is_write_backward(struct perf_event *event)
+{
+ return !!event->attr.write_backward;
+}
+
extern int perf_output_begin(struct perf_output_handle *handle,
struct perf_event *event, unsigned int size);
extern void perf_output_end(struct perf_output_handle *handle);
diff --git a/include/uapi/linux/perf_event.h b/include/uapi/linux/perf_event.h
index a3c1903..43fc8d2 100644
--- a/include/uapi/linux/perf_event.h
+++ b/include/uapi/linux/perf_event.h
@@ -340,7 +340,8 @@ struct perf_event_attr {
comm_exec : 1, /* flag comm events that are due to an exec */
use_clockid : 1, /* use @clockid for time fields */
context_switch : 1, /* context switch data */
- __reserved_1 : 37;
+ write_backward : 1, /* Write ring buffer from end to beginning */
+ __reserved_1 : 36;
union {
__u32 wakeup_events; /* wakeup every n events */
diff --git a/kernel/events/core.c b/kernel/events/core.c
index ed94c91..513b556 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -8196,6 +8196,13 @@ perf_event_set_output(struct perf_event *event, struct perf_event *output_event)
goto out;
/*
+ * Either writing ring buffer from beginning or from end.
+ * Mixing is not allowed.
+ */
+ if (is_write_backward(output_event) != is_write_backward(event))
+ goto out;
+
+ /*
* If both events generate aux data, they must be on the same PMU
*/
if (has_aux(event) && has_aux(output_event) &&
diff --git a/kernel/events/ring_buffer.c b/kernel/events/ring_buffer.c
index 37c11c6..80b1fa7 100644
--- a/kernel/events/ring_buffer.c
+++ b/kernel/events/ring_buffer.c
@@ -233,6 +233,8 @@ out:
int perf_output_begin(struct perf_output_handle *handle,
struct perf_event *event, unsigned int size)
{
+ if (unlikely(is_write_backward(event)))
+ return __perf_output_begin(handle, event, size, true);
return __perf_output_begin(handle, event, size, false);
}
--
1.8.3.4
[toc] | [prev] | [next] | [standalone]
| From | Wang Nan <wangnan0@huawei.com> |
|---|---|
| Date | 2016-03-07 05:00 +0100 |
| Subject | [RESEND PATCH 3/5] perf core: Prepare writing into ring buffer from end |
| Message-ID | <r9UuS-5Xo-15@gated-at.bofh.it> |
| In reply to | #1351295 |
Convert perf_output_begin to __perf_output_begin and make the later
function able to write records from the end of the ring buffer.
Following commits will utilize the 'backward' flag.
This patch doesn't introduce any extra performance overhead since we
use always_inline.
Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: He Kuang <hekuang@huawei.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Brendan Gregg <brendan.d.gregg@gmail.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
---
kernel/events/ring_buffer.c | 42 ++++++++++++++++++++++++++++++++++++------
1 file changed, 36 insertions(+), 6 deletions(-)
diff --git a/kernel/events/ring_buffer.c b/kernel/events/ring_buffer.c
index 22e1a47..37c11c6 100644
--- a/kernel/events/ring_buffer.c
+++ b/kernel/events/ring_buffer.c
@@ -102,8 +102,21 @@ out:
preempt_enable();
}
-int perf_output_begin(struct perf_output_handle *handle,
- struct perf_event *event, unsigned int size)
+static bool __always_inline
+ring_buffer_has_space(unsigned long head, unsigned long tail,
+ unsigned long data_size, unsigned int size,
+ bool backward)
+{
+ if (!backward)
+ return CIRC_SPACE(head, tail, data_size) >= size;
+ else
+ return CIRC_SPACE(tail, head, data_size) >= size;
+}
+
+static int __always_inline
+__perf_output_begin(struct perf_output_handle *handle,
+ struct perf_event *event, unsigned int size,
+ bool backward)
{
struct ring_buffer *rb;
unsigned long tail, offset, head;
@@ -146,9 +159,12 @@ int perf_output_begin(struct perf_output_handle *handle,
do {
tail = READ_ONCE(rb->user_page->data_tail);
offset = head = local_read(&rb->head);
- if (!rb->overwrite &&
- unlikely(CIRC_SPACE(head, tail, perf_data_size(rb)) < size))
- goto fail;
+ if (!rb->overwrite) {
+ if (unlikely(!ring_buffer_has_space(head, tail,
+ perf_data_size(rb),
+ size, backward)))
+ goto fail;
+ }
/*
* The above forms a control dependency barrier separating the
@@ -162,9 +178,17 @@ int perf_output_begin(struct perf_output_handle *handle,
* See perf_output_put_handle().
*/
- head += size;
+ if (!backward)
+ head += size;
+ else
+ head -= size;
} while (local_cmpxchg(&rb->head, offset, head) != offset);
+ if (backward) {
+ offset = head;
+ head = (u64)(-head);
+ }
+
/*
* We rely on the implied barrier() by local_cmpxchg() to ensure
* none of the data stores below can be lifted up by the compiler.
@@ -206,6 +230,12 @@ out:
return -ENOSPC;
}
+int perf_output_begin(struct perf_output_handle *handle,
+ struct perf_event *event, unsigned int size)
+{
+ return __perf_output_begin(handle, event, size, false);
+}
+
unsigned int perf_output_copy(struct perf_output_handle *handle,
const void *buf, unsigned int len)
{
--
1.8.3.4
[toc] | [prev] | [next] | [standalone]
| From | Peter Zijlstra <peterz@infradead.org> |
|---|---|
| Date | 2016-03-08 14:50 +0100 |
| Message-ID | <raqbn-1xM-1@gated-at.bofh.it> |
| In reply to | #1351295 |
On Mon, Mar 07, 2016 at 03:50:14AM +0000, Wang Nan wrote: > This patch set has been posted multiple times (with and without > corresponding 'perf tool' patches), and doesn't receive further > comment. I think it should be okay to merge them into mainline. > There are many perf's improvement depend on it. However, Peter > is not responsive after I fixed some problems he pointed out. > > Introduces 'write_backward' into perf_event_attr, allows kernel > writing the ring buffer from the end of it. This feature allows > extracting data from overwritable ring buffer. > > Wang Nan (5): > perf core: Introduce new ioctl options to pause and resume ring buffer > perf core: Set event's default overflow_handler > perf core: Prepare writing into ring buffer from end > perf core: Add backward attribute to perf event > perf core: Reduce perf event output overhead by new overflow handler perf kernel features are currently on hold until I can manage to run a fuzzer for more than a few minutes without my machine having a seizure.
[toc] | [prev] | [next] | [standalone]
| From | Ingo Molnar <mingo@kernel.org> |
|---|---|
| Date | 2016-03-08 14:50 +0100 |
| Message-ID | <raqbp-1xM-47@gated-at.bofh.it> |
| In reply to | #1353046 |
* Peter Zijlstra <peterz@infradead.org> wrote: > On Mon, Mar 07, 2016 at 03:50:14AM +0000, Wang Nan wrote: > > This patch set has been posted multiple times (with and without > > corresponding 'perf tool' patches), and doesn't receive further > > comment. I think it should be okay to merge them into mainline. > > There are many perf's improvement depend on it. However, Peter > > is not responsive after I fixed some problems he pointed out. > > > > Introduces 'write_backward' into perf_event_attr, allows kernel > > writing the ring buffer from the end of it. This feature allows > > extracting data from overwritable ring buffer. > > > > Wang Nan (5): > > perf core: Introduce new ioctl options to pause and resume ring buffer > > perf core: Set event's default overflow_handler > > perf core: Prepare writing into ring buffer from end > > perf core: Add backward attribute to perf event > > perf core: Reduce perf event output overhead by new overflow handler > > perf kernel features are currently on hold until I can manage to run a > fuzzer for more than a few minutes without my machine having a seizure. Btw., could you describe exactly what commands you are running, with what configuration options (if that matters), so that people who'd like our feature freeze to be lifted can help out? Thanks, Ingo
[toc] | [prev] | [next] | [standalone]
| From | Peter Zijlstra <peterz@infradead.org> |
|---|---|
| Date | 2016-03-08 15:00 +0100 |
| Message-ID | <raql5-1Bi-33@gated-at.bofh.it> |
| In reply to | #1353063 |
On Tue, Mar 08, 2016 at 02:49:01PM +0100, Ingo Molnar wrote:
>
> * Peter Zijlstra <peterz@infradead.org> wrote:
>
> > On Mon, Mar 07, 2016 at 03:50:14AM +0000, Wang Nan wrote:
> > > This patch set has been posted multiple times (with and without
> > > corresponding 'perf tool' patches), and doesn't receive further
> > > comment. I think it should be okay to merge them into mainline.
> > > There are many perf's improvement depend on it. However, Peter
> > > is not responsive after I fixed some problems he pointed out.
> > >
> > > Introduces 'write_backward' into perf_event_attr, allows kernel
> > > writing the ring buffer from the end of it. This feature allows
> > > extracting data from overwritable ring buffer.
> > >
> > > Wang Nan (5):
> > > perf core: Introduce new ioctl options to pause and resume ring buffer
> > > perf core: Set event's default overflow_handler
> > > perf core: Prepare writing into ring buffer from end
> > > perf core: Add backward attribute to perf event
> > > perf core: Reduce perf event output overhead by new overflow handler
> >
> > perf kernel features are currently on hold until I can manage to run a
> > fuzzer for more than a few minutes without my machine having a seizure.
>
> Btw., could you describe exactly what commands you are running, with what
> configuration options (if that matters), so that people who'd like our feature
> freeze to be lifted can help out?
Mostly syz-kaller, but also Vince's perf-fuzzer and your perf-stress
script, which I'm not sure is publicly available.
perf_fuzzer lives at:
https://github.com/deater/perf_event_tests.git
Here's a thread on syz-kaller:
lkml.kernel.org/r/CACT4Y+Ym0TZLkmRrM0ZGgLpu8kqS-YjoWTMrvaLz=tx2tnyO3w@mail.gmail.com
If things have shifted again I'm sure Dmitry is willing to help.
I run the thing natively on actual real hardware, which ensure I get to
test the PMU drivers too.
# cat go-fuzz.sh
#!/bin/bash
echo 1 > /proc/sys/kernel/traceoff_on_warning
echo 1 > /debug/tracing/options/stacktrace
echo 1 > /debug/tracing/events/sched/enable
cd gopath/src/github.com/google/syzkaller/
./bin/syz-manager -config perf.cfg
# cat gopath/src/github.com/google/syzkaller/perf.cfg
{
"http": "localhost:50000",
"workdir": "/root/gopath/src/github.com/google/syzkaller/workdir",
"syzkaller": "/root/gopath/src/github.com/google/syzkaller",
"vmlinux": "-",
"type": "local",
"count": 1,
"procs": 160,
"cover": false,
"dropprivs": false,
"enable_syscalls": [
"getpid",
"gettid",
"perf_event_open",
"ioctl$PERF*",
"prctl$void",
"bpf$*",
"sched_yield"
]
}
[toc] | [prev] | [next] | [standalone]
| From | Ingo Molnar <mingo@kernel.org> |
|---|---|
| Date | 2016-03-08 16:30 +0100 |
| Message-ID | <rarKa-2CK-17@gated-at.bofh.it> |
| In reply to | #1353069 |
* Peter Zijlstra <peterz@infradead.org> wrote: > Here's a thread on syz-kaller: > > lkml.kernel.org/r/CACT4Y+Ym0TZLkmRrM0ZGgLpu8kqS-YjoWTMrvaLz=tx2tnyO3w@mail.gmail.com > > If things have shifted again I'm sure Dmitry is willing to help. So I tried to install 'go' but it's a _really_ unintuitive tool I have to say. I installed golang-go on Ubuntu, which gave me a 'go' command: triton:~> go version go version go1.5.1 linux/amd64 that was the only step that worked. It's a dead end from that point on: triton:~> go get github.com/google/syzkaller package github.com/google/syzkaller: cannot download, $GOPATH not set. For more details see: go help gopath ok, so lets set GOPATH to the distro src directory: triton:~> export GOPATH=/usr/lib/go/src/ triton:~> go get github.com/google/syzkaller package github.com/google/syzkaller: mkdir /usr/lib/go/src/src: permission denied btw., this is what the directory contains: triton:~> ls /usr/lib/go/src/ all.bash bootstrap.bash bytes compress debug flag html io make.bat nacltest.bash race.bash run.bash strconv testing unsafe all.bat bufio clean.bash container encoding fmt image iostest.bash Make.dist net race.bat run.bat strings text androidtest.bash buildall.bash clean.bat crypto errors go index log math os reflect runtime sync time archive builtin cmd database expvar hash internal make.bash mime path regexp sort syscall unicode so, according to the error message it wants a writable directory. Lets try it that way: triton:~> mkdir go triton:~> triton:~> export GOPATH=/home/mingo/go/ triton:~> go get github.com/google/syzkaller can't load package: package github.com/google/syzkaller: no buildable Go source files in /home/mingo/go/src/github.com/google/syzkaller looks like someone wants 'Go' to be used as little as possible! ;-) Thanks, Ingo
[toc] | [prev] | [next] | [standalone]
| From | Dmitry Vyukov <dvyukov@google.com> |
|---|---|
| Date | 2016-03-08 16:40 +0100 |
| Message-ID | <rarTQ-2Ho-3@gated-at.bofh.it> |
| In reply to | #1353142 |
On Tue, Mar 8, 2016 at 4:29 PM, Ingo Molnar <mingo@kernel.org> wrote: > > * Peter Zijlstra <peterz@infradead.org> wrote: > >> Here's a thread on syz-kaller: >> >> lkml.kernel.org/r/CACT4Y+Ym0TZLkmRrM0ZGgLpu8kqS-YjoWTMrvaLz=tx2tnyO3w@mail.gmail.com >> >> If things have shifted again I'm sure Dmitry is willing to help. > > So I tried to install 'go' but it's a _really_ unintuitive tool I have to say. > > I installed golang-go on Ubuntu, which gave me a 'go' command: > > triton:~> go version > go version go1.5.1 linux/amd64 > > that was the only step that worked. It's a dead end from that point on: > > triton:~> go get github.com/google/syzkaller > package github.com/google/syzkaller: cannot download, $GOPATH not set. For more details see: go help gopath > > ok, so lets set GOPATH to the distro src directory: > > triton:~> export GOPATH=/usr/lib/go/src/ > triton:~> go get github.com/google/syzkaller > package github.com/google/syzkaller: mkdir /usr/lib/go/src/src: permission denied > > btw., this is what the directory contains: > > triton:~> ls /usr/lib/go/src/ > all.bash bootstrap.bash bytes compress debug flag html io make.bat nacltest.bash race.bash run.bash strconv testing unsafe > all.bat bufio clean.bash container encoding fmt image iostest.bash Make.dist net race.bat run.bat strings text > androidtest.bash buildall.bash clean.bat crypto errors go index log math os reflect runtime sync time > archive builtin cmd database expvar hash internal make.bash mime path regexp sort syscall unicode > > so, according to the error message it wants a writable directory. Lets try it that > way: > > triton:~> mkdir go > triton:~> > triton:~> export GOPATH=/home/mingo/go/ > triton:~> go get github.com/google/syzkaller > can't load package: package github.com/google/syzkaller: no buildable Go source files in /home/mingo/go/src/github.com/google/syzkaller Yes, GOPATH needs to be set a writable dir. You can ignore "can't load package" error. The goal of that step is checkout syzkaller with all dependencies into correct dirs under GOPATH. That's already done by now. Or you can do (/... at the end): $ go get github.com/google/syzkaller/... That will checkout and build. Either way you can continue with the make step. > looks like someone wants 'Go' to be used as little as possible! ;-) You probably did not observe recently a noobie trying to build a C project with sufficiently-complicate-build-system and a bunch of dependencies that needs to be on specific, unknown revisions, and that still does not compile with you compiler and does not link with your linker, and then of course target machine has a wrong glibc version :)
[toc] | [prev] | [next] | [standalone]
| From | Ingo Molnar <mingo@kernel.org> |
|---|---|
| Date | 2016-03-08 17:00 +0100 |
| Message-ID | <rasdc-2OP-21@gated-at.bofh.it> |
| In reply to | #1353144 |
* Dmitry Vyukov <dvyukov@google.com> wrote:
> > so, according to the error message it wants a writable directory. Lets try it that
> > way:
> >
> > triton:~> mkdir go
> > triton:~>
> > triton:~> export GOPATH=/home/mingo/go/
> > triton:~> go get github.com/google/syzkaller
> > can't load package: package github.com/google/syzkaller: no buildable Go source files in /home/mingo/go/src/github.com/google/syzkaller
>
> Yes, GOPATH needs to be set a writable dir.
> You can ignore "can't load package" error. The goal of that step is
> checkout syzkaller with all dependencies into correct dirs under
> GOPATH. That's already done by now.
> Or you can do (/... at the end):
> $ go get github.com/google/syzkaller/...
> That will checkout and build.
> Either way you can continue with the make step.
Cool, the '/...' trick works.
> > looks like someone wants 'Go' to be used as little as possible! ;-)
>
> You probably did not observe recently a noobie trying to build a C
> project with sufficiently-complicate-build-system and a bunch of
> dependencies that needs to be on specific, unknown revisions, and that
> still does not compile with you compiler and does not link with your
> linker, and then of course target machine has a wrong glibc version :)
Nah, old Linux tools very much suck, we know that and we suffer from it.
But new tools should not suck! :-)
So, going from the description at:
http://lkml.kernel.org/r/CACT4Y+Ym0TZLkmRrM0ZGgLpu8kqS-YjoWTMrvaLz=tx2tnyO3w@mail.gmail.com
I now have it built, and created $GOPATH/src/github.com/google/syzkaller/perf.cfg
with:
triton:~/go/src/github.com/google/syzkaller> cat perf.cfg
{
"http": "localhost:50000",
"workdir": "/home/mingo/go/src/github.com/google/syzkaller/workdir",
"syzkaller": "/home/mingo/go/src/github.com/google/syzkaller",
"vmlinux": "-",
"type": "local",
"count": 1,
"procs": 16,
"nocover": true,
"nodropprivs": true,
"enable_syscalls": [
"getpid",
"perf_event_open",
"ioctl$PERF*",
"prctl$void",
"bpf$*",
"sched_yield"
]
}
Then I tried to do:
$ bin/syz-manager -config perf.cfg
in ~/go/src/github.com/google/syzkaller, but that doesn't work because there's no
'bin' directory:
triton:~/go/src/github.com/google/syzkaller> bin/syz-manager -config perf.cfg
bash: bin/syz-manager: No such file or directory
So that should really read something like:
cd ~/go
bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
next roadblock:
fomalhaut:~/go> bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
2016/03/08 16:53:44 bad config syzkaller param: can't find bin/syz-fuzzer
so how do I proceed from here?
Thanks,
Ingo
[toc] | [prev] | [next] | [standalone]
| From | Dmitry Vyukov <dvyukov@google.com> |
|---|---|
| Date | 2016-03-08 17:20 +0100 |
| Message-ID | <raswy-3bK-21@gated-at.bofh.it> |
| In reply to | #1353172 |
On Tue, Mar 8, 2016 at 4:54 PM, Ingo Molnar <mingo@kernel.org> wrote:
>
> * Dmitry Vyukov <dvyukov@google.com> wrote:
>
>> > so, according to the error message it wants a writable directory. Lets try it that
>> > way:
>> >
>> > triton:~> mkdir go
>> > triton:~>
>> > triton:~> export GOPATH=/home/mingo/go/
>> > triton:~> go get github.com/google/syzkaller
>> > can't load package: package github.com/google/syzkaller: no buildable Go source files in /home/mingo/go/src/github.com/google/syzkaller
>>
>> Yes, GOPATH needs to be set a writable dir.
>> You can ignore "can't load package" error. The goal of that step is
>> checkout syzkaller with all dependencies into correct dirs under
>> GOPATH. That's already done by now.
>> Or you can do (/... at the end):
>> $ go get github.com/google/syzkaller/...
>> That will checkout and build.
>> Either way you can continue with the make step.
>
> Cool, the '/...' trick works.
>
>
>> > looks like someone wants 'Go' to be used as little as possible! ;-)
>>
>> You probably did not observe recently a noobie trying to build a C
>> project with sufficiently-complicate-build-system and a bunch of
>> dependencies that needs to be on specific, unknown revisions, and that
>> still does not compile with you compiler and does not link with your
>> linker, and then of course target machine has a wrong glibc version :)
>
> Nah, old Linux tools very much suck, we know that and we suffer from it.
>
> But new tools should not suck! :-)
>
> So, going from the description at:
>
> http://lkml.kernel.org/r/CACT4Y+Ym0TZLkmRrM0ZGgLpu8kqS-YjoWTMrvaLz=tx2tnyO3w@mail.gmail.com
>
> I now have it built, and created $GOPATH/src/github.com/google/syzkaller/perf.cfg
> with:
>
>
> triton:~/go/src/github.com/google/syzkaller> cat perf.cfg
> {
> "http": "localhost:50000",
> "workdir": "/home/mingo/go/src/github.com/google/syzkaller/workdir",
> "syzkaller": "/home/mingo/go/src/github.com/google/syzkaller",
> "vmlinux": "-",
> "type": "local",
> "count": 1,
> "procs": 16,
> "nocover": true,
> "nodropprivs": true,
> "enable_syscalls": [
> "getpid",
> "perf_event_open",
> "ioctl$PERF*",
> "prctl$void",
> "bpf$*",
> "sched_yield"
> ]
> }
>
> Then I tried to do:
>
> $ bin/syz-manager -config perf.cfg
>
> in ~/go/src/github.com/google/syzkaller, but that doesn't work because there's no
> 'bin' directory:
>
> triton:~/go/src/github.com/google/syzkaller> bin/syz-manager -config perf.cfg
> bash: bin/syz-manager: No such file or directory
>
> So that should really read something like:
>
> cd ~/go
> bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
>
> next roadblock:
>
> fomalhaut:~/go> bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
> 2016/03/08 16:53:44 bad config syzkaller param: can't find bin/syz-fuzzer
>
> so how do I proceed from here?
You need to call make in syzkaller dir, it will create
syzkaller/bin/syz-manager. I.e.
$ cd /home/mingo/go/src/github.com/google/syzkaller
$ make
$ bin/syz-manager -config perf.cfg
[toc] | [prev] | [next] | [standalone]
| From | Ingo Molnar <mingo@kernel.org> |
|---|---|
| Date | 2016-03-08 17:30 +0100 |
| Message-ID | <rasGe-3hq-7@gated-at.bofh.it> |
| In reply to | #1353189 |
* Dmitry Vyukov <dvyukov@google.com> wrote:
> On Tue, Mar 8, 2016 at 4:54 PM, Ingo Molnar <mingo@kernel.org> wrote:
> >
> > * Dmitry Vyukov <dvyukov@google.com> wrote:
> >
> >> > so, according to the error message it wants a writable directory. Lets try it that
> >> > way:
> >> >
> >> > triton:~> mkdir go
> >> > triton:~>
> >> > triton:~> export GOPATH=/home/mingo/go/
> >> > triton:~> go get github.com/google/syzkaller
> >> > can't load package: package github.com/google/syzkaller: no buildable Go source files in /home/mingo/go/src/github.com/google/syzkaller
> >>
> >> Yes, GOPATH needs to be set a writable dir.
> >> You can ignore "can't load package" error. The goal of that step is
> >> checkout syzkaller with all dependencies into correct dirs under
> >> GOPATH. That's already done by now.
> >> Or you can do (/... at the end):
> >> $ go get github.com/google/syzkaller/...
> >> That will checkout and build.
> >> Either way you can continue with the make step.
> >
> > Cool, the '/...' trick works.
> >
> >
> >> > looks like someone wants 'Go' to be used as little as possible! ;-)
> >>
> >> You probably did not observe recently a noobie trying to build a C
> >> project with sufficiently-complicate-build-system and a bunch of
> >> dependencies that needs to be on specific, unknown revisions, and that
> >> still does not compile with you compiler and does not link with your
> >> linker, and then of course target machine has a wrong glibc version :)
> >
> > Nah, old Linux tools very much suck, we know that and we suffer from it.
> >
> > But new tools should not suck! :-)
> >
> > So, going from the description at:
> >
> > http://lkml.kernel.org/r/CACT4Y+Ym0TZLkmRrM0ZGgLpu8kqS-YjoWTMrvaLz=tx2tnyO3w@mail.gmail.com
> >
> > I now have it built, and created $GOPATH/src/github.com/google/syzkaller/perf.cfg
> > with:
> >
> >
> > triton:~/go/src/github.com/google/syzkaller> cat perf.cfg
> > {
> > "http": "localhost:50000",
> > "workdir": "/home/mingo/go/src/github.com/google/syzkaller/workdir",
> > "syzkaller": "/home/mingo/go/src/github.com/google/syzkaller",
> > "vmlinux": "-",
> > "type": "local",
> > "count": 1,
> > "procs": 16,
> > "nocover": true,
> > "nodropprivs": true,
> > "enable_syscalls": [
> > "getpid",
> > "perf_event_open",
> > "ioctl$PERF*",
> > "prctl$void",
> > "bpf$*",
> > "sched_yield"
> > ]
> > }
> >
> > Then I tried to do:
> >
> > $ bin/syz-manager -config perf.cfg
> >
> > in ~/go/src/github.com/google/syzkaller, but that doesn't work because there's no
> > 'bin' directory:
> >
> > triton:~/go/src/github.com/google/syzkaller> bin/syz-manager -config perf.cfg
> > bash: bin/syz-manager: No such file or directory
> >
> > So that should really read something like:
> >
> > cd ~/go
> > bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
> >
> > next roadblock:
> >
> > fomalhaut:~/go> bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
> > 2016/03/08 16:53:44 bad config syzkaller param: can't find bin/syz-fuzzer
> >
> > so how do I proceed from here?
>
>
> You need to call make in syzkaller dir, it will create
> syzkaller/bin/syz-manager. I.e.
>
> $ cd /home/mingo/go/src/github.com/google/syzkaller
> $ make
Ok, cool, this got me further.
> $ bin/syz-manager -config perf.cfg
I now get periodic output of:
fomalhaut:~/go/src/github.com/google/syzkaller> bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
2016/03/08 17:24:07 failed to read config file: open src/github.com/google/syzkaller/perf.cfg: no such file or directory
fomalhaut:~/go/src/github.com/google/syzkaller> bin/syz-manager -config perf.cfg
2016/03/08 17:24:19 loading corpus...
2016/03/08 17:24:19 loaded 0 programs
2016/03/08 17:24:19 serving http on http://localhost:50000
2016/03/08 17:24:19 serving rpc on tcp://127.0.0.1:37006
2016/03/08 17:24:34 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454274467286949
2016/03/08 17:24:34 local-0: lost connection: exit status 1
2016/03/08 17:24:34 local-0: saving crash 'lost connection' to crash-local-0-1457454274467603509
2016/03/08 17:24:49 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454289719432704
2016/03/08 17:24:49 local-0: lost connection: exit status 1
2016/03/08 17:24:49 local-0: saving crash 'lost connection' to crash-local-0-1457454289719774031
2016/03/08 17:25:04 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454304992856310
2016/03/08 17:25:04 local-0: lost connection: exit status 1
2016/03/08 17:25:04 local-0: saving crash 'lost connection' to crash-local-0-1457454304993224299
2016/03/08 17:25:20 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454320280216980
2016/03/08 17:25:20 local-0: lost connection: exit status 1
2016/03/08 17:25:20 local-0: saving crash 'lost connection' to crash-local-0-1457454320280581459
2016/03/08 17:25:35 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454335572633035
2016/03/08 17:25:35 local-0: lost connection: exit status 1
2016/03/08 17:25:35 local-0: saving crash 'lost connection' to crash-local-0-1457454335572967343
2016/03/08 17:25:50 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454350865099485
2016/03/08 17:25:50 local-0: lost connection: exit status 1
2016/03/08 17:25:50 local-0: saving crash 'lost connection' to crash-local-0-1457454350865429049
is CONFIG_KCOV=y a must-have feature? There's no KCOV support upstream that I can
see.
Thanks,
Ingo
[toc] | [prev] | [next] | [standalone]
| From | Dmitry Vyukov <dvyukov@google.com> |
|---|---|
| Date | 2016-03-08 17:40 +0100 |
| Message-ID | <rasPT-3kP-1@gated-at.bofh.it> |
| In reply to | #1353192 |
On Tue, Mar 8, 2016 at 5:27 PM, Ingo Molnar <mingo@kernel.org> wrote:
>
> * Dmitry Vyukov <dvyukov@google.com> wrote:
>
>> On Tue, Mar 8, 2016 at 4:54 PM, Ingo Molnar <mingo@kernel.org> wrote:
>> >
>> > * Dmitry Vyukov <dvyukov@google.com> wrote:
>> >
>> >> > so, according to the error message it wants a writable directory. Lets try it that
>> >> > way:
>> >> >
>> >> > triton:~> mkdir go
>> >> > triton:~>
>> >> > triton:~> export GOPATH=/home/mingo/go/
>> >> > triton:~> go get github.com/google/syzkaller
>> >> > can't load package: package github.com/google/syzkaller: no buildable Go source files in /home/mingo/go/src/github.com/google/syzkaller
>> >>
>> >> Yes, GOPATH needs to be set a writable dir.
>> >> You can ignore "can't load package" error. The goal of that step is
>> >> checkout syzkaller with all dependencies into correct dirs under
>> >> GOPATH. That's already done by now.
>> >> Or you can do (/... at the end):
>> >> $ go get github.com/google/syzkaller/...
>> >> That will checkout and build.
>> >> Either way you can continue with the make step.
>> >
>> > Cool, the '/...' trick works.
>> >
>> >
>> >> > looks like someone wants 'Go' to be used as little as possible! ;-)
>> >>
>> >> You probably did not observe recently a noobie trying to build a C
>> >> project with sufficiently-complicate-build-system and a bunch of
>> >> dependencies that needs to be on specific, unknown revisions, and that
>> >> still does not compile with you compiler and does not link with your
>> >> linker, and then of course target machine has a wrong glibc version :)
>> >
>> > Nah, old Linux tools very much suck, we know that and we suffer from it.
>> >
>> > But new tools should not suck! :-)
>> >
>> > So, going from the description at:
>> >
>> > http://lkml.kernel.org/r/CACT4Y+Ym0TZLkmRrM0ZGgLpu8kqS-YjoWTMrvaLz=tx2tnyO3w@mail.gmail.com
>> >
>> > I now have it built, and created $GOPATH/src/github.com/google/syzkaller/perf.cfg
>> > with:
>> >
>> >
>> > triton:~/go/src/github.com/google/syzkaller> cat perf.cfg
>> > {
>> > "http": "localhost:50000",
>> > "workdir": "/home/mingo/go/src/github.com/google/syzkaller/workdir",
>> > "syzkaller": "/home/mingo/go/src/github.com/google/syzkaller",
>> > "vmlinux": "-",
>> > "type": "local",
>> > "count": 1,
>> > "procs": 16,
>> > "nocover": true,
>> > "nodropprivs": true,
>> > "enable_syscalls": [
>> > "getpid",
>> > "perf_event_open",
>> > "ioctl$PERF*",
>> > "prctl$void",
>> > "bpf$*",
>> > "sched_yield"
>> > ]
>> > }
>> >
>> > Then I tried to do:
>> >
>> > $ bin/syz-manager -config perf.cfg
>> >
>> > in ~/go/src/github.com/google/syzkaller, but that doesn't work because there's no
>> > 'bin' directory:
>> >
>> > triton:~/go/src/github.com/google/syzkaller> bin/syz-manager -config perf.cfg
>> > bash: bin/syz-manager: No such file or directory
>> >
>> > So that should really read something like:
>> >
>> > cd ~/go
>> > bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
>> >
>> > next roadblock:
>> >
>> > fomalhaut:~/go> bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
>> > 2016/03/08 16:53:44 bad config syzkaller param: can't find bin/syz-fuzzer
>> >
>> > so how do I proceed from here?
>>
>>
>> You need to call make in syzkaller dir, it will create
>> syzkaller/bin/syz-manager. I.e.
>>
>> $ cd /home/mingo/go/src/github.com/google/syzkaller
>> $ make
>
> Ok, cool, this got me further.
>
>> $ bin/syz-manager -config perf.cfg
>
> I now get periodic output of:
>
> fomalhaut:~/go/src/github.com/google/syzkaller> bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
> 2016/03/08 17:24:07 failed to read config file: open src/github.com/google/syzkaller/perf.cfg: no such file or directory
> fomalhaut:~/go/src/github.com/google/syzkaller> bin/syz-manager -config perf.cfg
> 2016/03/08 17:24:19 loading corpus...
> 2016/03/08 17:24:19 loaded 0 programs
> 2016/03/08 17:24:19 serving http on http://localhost:50000
> 2016/03/08 17:24:19 serving rpc on tcp://127.0.0.1:37006
> 2016/03/08 17:24:34 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454274467286949
> 2016/03/08 17:24:34 local-0: lost connection: exit status 1
> 2016/03/08 17:24:34 local-0: saving crash 'lost connection' to crash-local-0-1457454274467603509
> 2016/03/08 17:24:49 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454289719432704
> 2016/03/08 17:24:49 local-0: lost connection: exit status 1
> 2016/03/08 17:24:49 local-0: saving crash 'lost connection' to crash-local-0-1457454289719774031
> 2016/03/08 17:25:04 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454304992856310
> 2016/03/08 17:25:04 local-0: lost connection: exit status 1
> 2016/03/08 17:25:04 local-0: saving crash 'lost connection' to crash-local-0-1457454304993224299
> 2016/03/08 17:25:20 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454320280216980
> 2016/03/08 17:25:20 local-0: lost connection: exit status 1
> 2016/03/08 17:25:20 local-0: saving crash 'lost connection' to crash-local-0-1457454320280581459
> 2016/03/08 17:25:35 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454335572633035
> 2016/03/08 17:25:35 local-0: lost connection: exit status 1
> 2016/03/08 17:25:35 local-0: saving crash 'lost connection' to crash-local-0-1457454335572967343
> 2016/03/08 17:25:50 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454350865099485
> 2016/03/08 17:25:50 local-0: lost connection: exit status 1
> 2016/03/08 17:25:50 local-0: saving crash 'lost connection' to crash-local-0-1457454350865429049
>
> is CONFIG_KCOV=y a must-have feature? There's no KCOV support upstream that I can
> see.
Change:
"nocover": true,
to:
"cover": false,
in the config file.
KCOV will increase efficiency of the fuzzer, but it is not necessary.
As far as I understand Peter tested without KCOV.
[toc] | [prev] | [next] | [standalone]
| From | Peter Zijlstra <peterz@infradead.org> |
|---|---|
| Date | 2016-03-08 17:40 +0100 |
| Message-ID | <rasPU-3kP-17@gated-at.bofh.it> |
| In reply to | #1353198 |
On Tue, Mar 08, 2016 at 05:29:50PM +0100, Dmitry Vyukov wrote: > KCOV will increase efficiency of the fuzzer, but it is not necessary. > As far as I understand Peter tested without KCOV. Correct, I never quite got around to enabling it. Still on the TODO list somewhere.
[toc] | [prev] | [next] | [standalone]
| From | Ingo Molnar <mingo@kernel.org> |
|---|---|
| Date | 2016-03-08 17:50 +0100 |
| Message-ID | <rasZA-3on-9@gated-at.bofh.it> |
| In reply to | #1353198 |
* Dmitry Vyukov <dvyukov@google.com> wrote:
> On Tue, Mar 8, 2016 at 5:27 PM, Ingo Molnar <mingo@kernel.org> wrote:
> >
> > * Dmitry Vyukov <dvyukov@google.com> wrote:
> >
> >> On Tue, Mar 8, 2016 at 4:54 PM, Ingo Molnar <mingo@kernel.org> wrote:
> >> >
> >> > * Dmitry Vyukov <dvyukov@google.com> wrote:
> >> >
> >> >> > so, according to the error message it wants a writable directory. Lets try it that
> >> >> > way:
> >> >> >
> >> >> > triton:~> mkdir go
> >> >> > triton:~>
> >> >> > triton:~> export GOPATH=/home/mingo/go/
> >> >> > triton:~> go get github.com/google/syzkaller
> >> >> > can't load package: package github.com/google/syzkaller: no buildable Go source files in /home/mingo/go/src/github.com/google/syzkaller
> >> >>
> >> >> Yes, GOPATH needs to be set a writable dir.
> >> >> You can ignore "can't load package" error. The goal of that step is
> >> >> checkout syzkaller with all dependencies into correct dirs under
> >> >> GOPATH. That's already done by now.
> >> >> Or you can do (/... at the end):
> >> >> $ go get github.com/google/syzkaller/...
> >> >> That will checkout and build.
> >> >> Either way you can continue with the make step.
> >> >
> >> > Cool, the '/...' trick works.
> >> >
> >> >
> >> >> > looks like someone wants 'Go' to be used as little as possible! ;-)
> >> >>
> >> >> You probably did not observe recently a noobie trying to build a C
> >> >> project with sufficiently-complicate-build-system and a bunch of
> >> >> dependencies that needs to be on specific, unknown revisions, and that
> >> >> still does not compile with you compiler and does not link with your
> >> >> linker, and then of course target machine has a wrong glibc version :)
> >> >
> >> > Nah, old Linux tools very much suck, we know that and we suffer from it.
> >> >
> >> > But new tools should not suck! :-)
> >> >
> >> > So, going from the description at:
> >> >
> >> > http://lkml.kernel.org/r/CACT4Y+Ym0TZLkmRrM0ZGgLpu8kqS-YjoWTMrvaLz=tx2tnyO3w@mail.gmail.com
> >> >
> >> > I now have it built, and created $GOPATH/src/github.com/google/syzkaller/perf.cfg
> >> > with:
> >> >
> >> >
> >> > triton:~/go/src/github.com/google/syzkaller> cat perf.cfg
> >> > {
> >> > "http": "localhost:50000",
> >> > "workdir": "/home/mingo/go/src/github.com/google/syzkaller/workdir",
> >> > "syzkaller": "/home/mingo/go/src/github.com/google/syzkaller",
> >> > "vmlinux": "-",
> >> > "type": "local",
> >> > "count": 1,
> >> > "procs": 16,
> >> > "nocover": true,
> >> > "nodropprivs": true,
> >> > "enable_syscalls": [
> >> > "getpid",
> >> > "perf_event_open",
> >> > "ioctl$PERF*",
> >> > "prctl$void",
> >> > "bpf$*",
> >> > "sched_yield"
> >> > ]
> >> > }
> >> >
> >> > Then I tried to do:
> >> >
> >> > $ bin/syz-manager -config perf.cfg
> >> >
> >> > in ~/go/src/github.com/google/syzkaller, but that doesn't work because there's no
> >> > 'bin' directory:
> >> >
> >> > triton:~/go/src/github.com/google/syzkaller> bin/syz-manager -config perf.cfg
> >> > bash: bin/syz-manager: No such file or directory
> >> >
> >> > So that should really read something like:
> >> >
> >> > cd ~/go
> >> > bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
> >> >
> >> > next roadblock:
> >> >
> >> > fomalhaut:~/go> bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
> >> > 2016/03/08 16:53:44 bad config syzkaller param: can't find bin/syz-fuzzer
> >> >
> >> > so how do I proceed from here?
> >>
> >>
> >> You need to call make in syzkaller dir, it will create
> >> syzkaller/bin/syz-manager. I.e.
> >>
> >> $ cd /home/mingo/go/src/github.com/google/syzkaller
> >> $ make
> >
> > Ok, cool, this got me further.
> >
> >> $ bin/syz-manager -config perf.cfg
> >
> > I now get periodic output of:
> >
> > fomalhaut:~/go/src/github.com/google/syzkaller> bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
> > 2016/03/08 17:24:07 failed to read config file: open src/github.com/google/syzkaller/perf.cfg: no such file or directory
> > fomalhaut:~/go/src/github.com/google/syzkaller> bin/syz-manager -config perf.cfg
> > 2016/03/08 17:24:19 loading corpus...
> > 2016/03/08 17:24:19 loaded 0 programs
> > 2016/03/08 17:24:19 serving http on http://localhost:50000
> > 2016/03/08 17:24:19 serving rpc on tcp://127.0.0.1:37006
> > 2016/03/08 17:24:34 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454274467286949
> > 2016/03/08 17:24:34 local-0: lost connection: exit status 1
> > 2016/03/08 17:24:34 local-0: saving crash 'lost connection' to crash-local-0-1457454274467603509
> > 2016/03/08 17:24:49 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454289719432704
> > 2016/03/08 17:24:49 local-0: lost connection: exit status 1
> > 2016/03/08 17:24:49 local-0: saving crash 'lost connection' to crash-local-0-1457454289719774031
> > 2016/03/08 17:25:04 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454304992856310
> > 2016/03/08 17:25:04 local-0: lost connection: exit status 1
> > 2016/03/08 17:25:04 local-0: saving crash 'lost connection' to crash-local-0-1457454304993224299
> > 2016/03/08 17:25:20 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454320280216980
> > 2016/03/08 17:25:20 local-0: lost connection: exit status 1
> > 2016/03/08 17:25:20 local-0: saving crash 'lost connection' to crash-local-0-1457454320280581459
> > 2016/03/08 17:25:35 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454335572633035
> > 2016/03/08 17:25:35 local-0: lost connection: exit status 1
> > 2016/03/08 17:25:35 local-0: saving crash 'lost connection' to crash-local-0-1457454335572967343
> > 2016/03/08 17:25:50 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454350865099485
> > 2016/03/08 17:25:50 local-0: lost connection: exit status 1
> > 2016/03/08 17:25:50 local-0: saving crash 'lost connection' to crash-local-0-1457454350865429049
> >
> > is CONFIG_KCOV=y a must-have feature? There's no KCOV support upstream that I can
> > see.
>
> Change:
> "nocover": true,
> to:
> "cover": false,
> in the config file.
>
> KCOV will increase efficiency of the fuzzer, but it is not necessary.
> As far as I understand Peter tested without KCOV.
Ok, so now I get:
fomalhaut:~/go/src/github.com/google/syzkaller> bin/syz-manager -config perf.cfg
2016/03/08 17:39:25 loading corpus...
2016/03/08 17:39:25 loaded 0 programs
2016/03/08 17:39:25 serving http on http://localhost:50000
2016/03/08 17:39:25 serving rpc on tcp://127.0.0.1:33809
| |-sshd(49552)---sshd(49555)---bash(49562)---syz-manager(49652)-+-syz-fuzzer(49661)-+-syz-executor(49935)
| | | |-syz-executor(49936)
| | | |-syz-executor(49937)
| | | |-syz-executor(49938)
| | | |-syz-executor(49939)
| | | |-syz-executor(49940)
| | | |-syz-executor(49941)
| | | |-syz-executor(49942)
| | | |-syz-executor(49943)
| | | |-syz-executor(49944)
| | | |-syz-executor(49945)
| | | |-syz-executor(49946)
| | | |-syz-executor(49947)
| | | |-syz-executor(49948)
| | | |-syz-executor(49949)
| | | |-syz-executor(49950)
| | | |-{syz-fuzzer}(49662)
| | | |-{syz-fuzzer}(49663)
| | | |-{syz-fuzzer}(49664)
| | | |-{syz-fuzzer}(49665)
| | | |-{syz-fuzzer}(49666)
| | | |-{syz-fuzzer}(49688)
| | | |-{syz-fuzzer}(49689)
| | | |-{syz-fuzzer}(49690)
| | | |-{syz-fuzzer}(49691)
| | | |-{syz-fuzzer}(49692)
| | | |-{syz-fuzzer}(49693)
| | | |-{syz-fuzzer}(49694)
| | | |-{syz-fuzzer}(49695)
| | | |-{syz-fuzzer}(49696)
| | | |-{syz-fuzzer}(49697)
| | | |-{syz-fuzzer}(49698)
| | | |-{syz-fuzzer}(49699)
| | | |-{syz-fuzzer}(49700)
| | | |-{syz-fuzzer}(49701)
| | | |-{syz-fuzzer}(49704)
| | | |-{syz-fuzzer}(49705)
| | | |-{syz-fuzzer}(49706)
| | | |-{syz-fuzzer}(49710)
| | | |-{syz-fuzzer}(49711)
| | | |-{syz-fuzzer}(49737)
| | | |-{syz-fuzzer}(49739)
| | | |-{syz-fuzzer}(49762)
| | | |-{syz-fuzzer}(49764)
| | | |-{syz-fuzzer}(49787)
| | | |-{syz-fuzzer}(49789)
| | | |-{syz-fuzzer}(49802)
| | | |-{syz-fuzzer}(49803)
| | | |-{syz-fuzzer}(49804)
| | | |-{syz-fuzzer}(49818)
| | | |-{syz-fuzzer}(49822)
| | | |-{syz-fuzzer}(49846)
| | | |-{syz-fuzzer}(49847)
| | | |-{syz-fuzzer}(49848)
| | | |-{syz-fuzzer}(49849)
| | | |-{syz-fuzzer}(49850)
| | | |-{syz-fuzzer}(49851)
| | | |-{syz-fuzzer}(49852)
| | | |-{syz-fuzzer}(49893)
| | | |-{syz-fuzzer}(49894)
| | | |-{syz-fuzzer}(49895)
| | | |-{syz-fuzzer}(49896)
| | | |-{syz-fuzzer}(49933)
| | | `-{syz-fuzzer}(49934)
| | |-{syz-manager}(49653)
| | |-{syz-manager}(49654)
| | |-{syz-manager}(49655)
| | |-{syz-manager}(49656)
| | |-{syz-manager}(49657)
| | |-{syz-manager}(49658)
| | |-{syz-manager}(49659)
| | |-{syz-manager}(49667)
| | |-{syz-manager}(49668)
| | |-{syz-manager}(49669)
| | |-{syz-manager}(49670)
| | |-{syz-manager}(49671)
| | |-{syz-manager}(49672)
| | |-{syz-manager}(49673)
| | |-{syz-manager}(49674)
| | |-{syz-manager}(49675)
| | |-{syz-manager}(49676)
| | |-{syz-manager}(49677)
| | |-{syz-manager}(49678)
| | |-{syz-manager}(49679)
| | |-{syz-manager}(49680)
| | |-{syz-manager}(49681)
| | |-{syz-manager}(49682)
| | |-{syz-manager}(49683)
| | |-{syz-manager}(49684)
| | |-{syz-manager}(49685)
| | |-{syz-manager}(49686)
| | |-{syz-manager}(49687)
| | |-{syz-manager}(49913)
| | `-{syz-manager}(49914)
I'm seeing some zombies:
49935 pts/2 Z+ 0:00 [syz-executor] <defunct>
49936 pts/2 Z+ 0:00 [syz-executor] <defunct>
49937 pts/2 Z+ 0:00 [syz-executor] <defunct>
49938 pts/2 Z+ 0:00 [syz-executor] <defunct>
49939 pts/2 Z+ 0:00 [syz-executor] <defunct>
49940 pts/2 Z+ 0:00 [syz-executor] <defunct>
49941 pts/2 Z+ 0:00 [syz-executor] <defunct>
49942 pts/2 Z+ 0:00 [syz-executor] <defunct>
49943 pts/2 Z+ 0:00 [syz-executor] <defunct>
49944 pts/2 Z+ 0:00 [syz-executor] <defunct>
49945 pts/2 Z+ 0:00 [syz-executor] <defunct>
49946 pts/2 Z+ 0:00 [syz-executor] <defunct>
49947 pts/2 Z+ 0:00 [syz-executor] <defunct>
49948 pts/2 Z+ 0:00 [syz-executor] <defunct>
49949 pts/2 Z+ 0:00 [syz-executor] <defunct>
49950 pts/2 Z+ 0:00 [syz-executor] <defun
does it mean it found a kernel bug already?
It only had a couple of seconds of runtime:
49652 mingo 20 0 1434276 52144 11344 S 0.0 0.0 0:00.54 syz-manager
49661 mingo 20 0 2196672 43948 10448 S 0.0 0.0 0:05.59 syz-fuzzer
Thanks,
Ingo
[toc] | [prev] | [next] | [standalone]
| From | Ingo Molnar <mingo@kernel.org> |
|---|---|
| Date | 2016-03-08 17:50 +0100 |
| Message-ID | <rasZA-3on-7@gated-at.bofh.it> |
| In reply to | #1353205 |
* Ingo Molnar <mingo@kernel.org> wrote: > It only had a couple of seconds of runtime: > > 49652 mingo 20 0 1434276 52144 11344 S 0.0 0.0 0:00.54 syz-manager > 49661 mingo 20 0 2196672 43948 10448 S 0.0 0.0 0:05.59 syz-fuzzer Ah, so it appears to making some progress: 49652 mingo 20 0 1581740 47600 11344 S 0.0 0.0 0:00.58 syz-manager 49661 mingo 20 0 2204868 43720 10448 S 0.0 0.0 0:07.49 syz-fuzzer 49652 mingo 20 0 1598132 31512 11344 S 0.0 0.0 0:00.61 syz-manager 49661 mingo 20 0 2204868 44252 10448 S 0.0 0.0 0:09.09 syz-fuzzer but only about +1 second runtime added every minute or so. Is that expected? There's no progress mark anywhere suggesting that the tool thinks it is going fine. You might want to emit periodic (once a minute or so) 'I am still OK!' messages or so. Thanks, Ingo
[toc] | [prev] | [next] | [standalone]
| From | Dmitry Vyukov <dvyukov@google.com> |
|---|---|
| Date | 2016-03-08 18:10 +0100 |
| Message-ID | <ratiV-3LD-7@gated-at.bofh.it> |
| In reply to | #1353206 |
On Tue, Mar 8, 2016 at 5:48 PM, Ingo Molnar <mingo@kernel.org> wrote: > > * Ingo Molnar <mingo@kernel.org> wrote: > >> It only had a couple of seconds of runtime: >> >> 49652 mingo 20 0 1434276 52144 11344 S 0.0 0.0 0:00.54 syz-manager >> 49661 mingo 20 0 2196672 43948 10448 S 0.0 0.0 0:05.59 syz-fuzzer > > Ah, so it appears to making some progress: > > 49652 mingo 20 0 1581740 47600 11344 S 0.0 0.0 0:00.58 syz-manager > 49661 mingo 20 0 2204868 43720 10448 S 0.0 0.0 0:07.49 syz-fuzzer > > 49652 mingo 20 0 1598132 31512 11344 S 0.0 0.0 0:00.61 syz-manager > 49661 mingo 20 0 2204868 44252 10448 S 0.0 0.0 0:09.09 syz-fuzzer > > but only about +1 second runtime added every minute or so. Is that expected? The main work is done by child syz-executor processes. syz-manager/syz-fuzzer only guide the process. You can set "procs" param in config to higher value to increase CPU utilization. To get more bugs you want to saturate all CPUs to trigger more unusual thread interleavings. > There's no progress mark anywhere suggesting that the tool thinks it is going > fine. You might want to emit periodic (once a minute or so) 'I am still OK!' > messages or so. Will do. Regarding the zombie processes, it may or may not be OK. If they hang for minutes and you can't kill them, then it is a kernel bug. If they hang for minutes and you can kill them, then it is either kernel bug of my bug. If they are recycled eventually, then it is OK. The first thing I do in such cases is: $ cat /proc/$PID/task/**/stack If there is a second unfinished thread hanging on a kernel spinlock or mutex, then it's definitely bad. It also helps to enable CONFIG_RCU_STALL_COMMON=y, CONFIG_DEBUG_ATOMIC_SLEEP=y, CONFIG_WQ_WATCHDOG=y and spinlock/mutex debugging. These can detect various stalls.
[toc] | [prev] | [next] | [standalone]
| From | Ingo Molnar <mingo@kernel.org> |
|---|---|
| Date | 2016-03-08 18:30 +0100 |
| Message-ID | <ratCi-3TL-11@gated-at.bofh.it> |
| In reply to | #1353244 |
* Dmitry Vyukov <dvyukov@google.com> wrote: > On Tue, Mar 8, 2016 at 5:48 PM, Ingo Molnar <mingo@kernel.org> wrote: > > > > * Ingo Molnar <mingo@kernel.org> wrote: > > > >> It only had a couple of seconds of runtime: > >> > >> 49652 mingo 20 0 1434276 52144 11344 S 0.0 0.0 0:00.54 syz-manager > >> 49661 mingo 20 0 2196672 43948 10448 S 0.0 0.0 0:05.59 syz-fuzzer > > > > Ah, so it appears to making some progress: > > > > 49652 mingo 20 0 1581740 47600 11344 S 0.0 0.0 0:00.58 syz-manager > > 49661 mingo 20 0 2204868 43720 10448 S 0.0 0.0 0:07.49 syz-fuzzer > > > > 49652 mingo 20 0 1598132 31512 11344 S 0.0 0.0 0:00.61 syz-manager > > 49661 mingo 20 0 2204868 44252 10448 S 0.0 0.0 0:09.09 syz-fuzzer > > > > but only about +1 second runtime added every minute or so. Is that expected? > > The main work is done by child syz-executor processes. Hm, they don't seem to be doing anything: fomalhaut:~> ps aux | grep syz-executor mingo 41506 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> mingo 41509 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> mingo 41513 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> mingo 41515 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> mingo 41523 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> mingo 41601 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> mingo 41608 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> mingo 41662 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> mingo 41764 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> mingo 41966 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> mingo 42029 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> mingo 42084 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> mingo 42145 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> mingo 42149 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> mingo 42166 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> mingo 42175 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> mingo 57627 2.2 0.0 1860540 44884 pts/2 Sl+ 18:16 0:04 /home/mingo/go/src/github.com/google/syzkaller/workdir/instance-0/syz-fuzzer -executor /home/mingo/go/src/github.com/google/syzkaller/workdir/instance-0/syz-executor -name local-0 -manager 127.0.0.1:33809 -output=none -procs 16 -leak=false -cover=false -nobody=true -v 0 ... because they are recycling: fomalhaut:~> ps aux | grep syz-executor mingo 57627 1.6 0.0 1942468 44624 pts/2 Sl+ 18:16 0:05 /home/mingo/go/src/github.com/google/syzkaller/workdir/instance-0/syz-fuzzer -executor /home/mingo/go/src/github.com/google/syzkaller/workdir/instance-0/syz-executor -name local-0 -manager 127.0.0.1:33809 -output=none -procs 16 -leak=false -cover=false -nobody=true -v 0 mingo 98448 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> mingo 98454 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> mingo 98468 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> mingo 98472 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> mingo 98476 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> mingo 98522 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> mingo 98525 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> mingo 98548 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> mingo 98568 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> mingo 98596 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> mingo 98618 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> mingo 98644 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> mingo 98695 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> mingo 98708 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> mingo 98737 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> mingo 98756 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> I'm not seeing anything happen in 'top' - only a mostly idle system. > syz-manager/syz-fuzzer only guide the process. > You can set "procs" param in config to higher value to increase CPU > utilization. To get more bugs you want to saturate all CPUs to trigger > more unusual thread interleavings. So right now it doesn't seem to saturate 16 CPUs - not even close to it. > If there is a second unfinished thread hanging on a kernel spinlock or > mutex, then it's definitely bad. > It also helps to enable CONFIG_RCU_STALL_COMMON=y, > CONFIG_DEBUG_ATOMIC_SLEEP=y, CONFIG_WQ_WATCHDOG=y and spinlock/mutex > debugging. These can detect various stalls. I can just Ctrl-C the manager and it shuts down within a few seconds: 2016/03/08 17:39:25 serving rpc on tcp://127.0.0.1:33809 2016/03/08 17:51:45 local-0: lost connection: exit status 2 2016/03/08 17:51:45 local-0: saving crash 'lost connection' to crash-local-0-1457455905403390570 2016/03/08 18:04:04 local-0: lost connection: exit status 2 2016/03/08 18:04:04 local-0: saving crash 'lost connection' to crash-local-0-1457456644779165131 2016/03/08 18:16:24 local-0: lost connection: exit status 2 2016/03/08 18:16:24 local-0: saving crash 'lost connection' to crash-local-0-1457457384707190124 ^C2016/03/08 18:22:53 shutting down... with nothing hanging around: fomalhaut:~/go/src/github.com/google/syzkaller> ps aux | grep -i syz mingo 1374 0.0 0.0 118476 2376 pts/2 S+ 18:23 0:00 grep --color=auto -i syz and with no kernel messages in dmesg - and with a fully functional system. I'm running the 16-task load on a 120 CPU system - should I increase it to 120? Does the code expect to saturate the system? Thanks, Ingo
[toc] | [prev] | [next] | [standalone]
| From | Dmitry Vyukov <dvyukov@google.com> |
|---|---|
| Date | 2016-03-08 18:30 +0100 |
| Message-ID | <ratCj-3TL-29@gated-at.bofh.it> |
| In reply to | #1353255 |
On Tue, Mar 8, 2016 at 6:24 PM, Ingo Molnar <mingo@kernel.org> wrote: > > * Dmitry Vyukov <dvyukov@google.com> wrote: > >> On Tue, Mar 8, 2016 at 5:48 PM, Ingo Molnar <mingo@kernel.org> wrote: >> > >> > * Ingo Molnar <mingo@kernel.org> wrote: >> > >> >> It only had a couple of seconds of runtime: >> >> >> >> 49652 mingo 20 0 1434276 52144 11344 S 0.0 0.0 0:00.54 syz-manager >> >> 49661 mingo 20 0 2196672 43948 10448 S 0.0 0.0 0:05.59 syz-fuzzer >> > >> > Ah, so it appears to making some progress: >> > >> > 49652 mingo 20 0 1581740 47600 11344 S 0.0 0.0 0:00.58 syz-manager >> > 49661 mingo 20 0 2204868 43720 10448 S 0.0 0.0 0:07.49 syz-fuzzer >> > >> > 49652 mingo 20 0 1598132 31512 11344 S 0.0 0.0 0:00.61 syz-manager >> > 49661 mingo 20 0 2204868 44252 10448 S 0.0 0.0 0:09.09 syz-fuzzer >> > >> > but only about +1 second runtime added every minute or so. Is that expected? >> >> The main work is done by child syz-executor processes. > > Hm, they don't seem to be doing anything: > > fomalhaut:~> ps aux | grep syz-executor > mingo 41506 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> > mingo 41509 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> > mingo 41513 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> > mingo 41515 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> > mingo 41523 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> > mingo 41601 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> > mingo 41608 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> > mingo 41662 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> > mingo 41764 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> > mingo 41966 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> > mingo 42029 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> > mingo 42084 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> > mingo 42145 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> > mingo 42149 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> > mingo 42166 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> > mingo 42175 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct> > mingo 57627 2.2 0.0 1860540 44884 pts/2 Sl+ 18:16 0:04 /home/mingo/go/src/github.com/google/syzkaller/workdir/instance-0/syz-fuzzer -executor /home/mingo/go/src/github.com/google/syzkaller/workdir/instance-0/syz-executor -name local-0 -manager 127.0.0.1:33809 -output=none -procs 16 -leak=false -cover=false -nobody=true -v 0 > > ... because they are recycling: > > fomalhaut:~> ps aux | grep syz-executor > mingo 57627 1.6 0.0 1942468 44624 pts/2 Sl+ 18:16 0:05 /home/mingo/go/src/github.com/google/syzkaller/workdir/instance-0/syz-fuzzer -executor /home/mingo/go/src/github.com/google/syzkaller/workdir/instance-0/syz-executor -name local-0 -manager 127.0.0.1:33809 -output=none -procs 16 -leak=false -cover=false -nobody=true -v 0 > mingo 98448 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> > mingo 98454 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> > mingo 98468 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> > mingo 98472 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> > mingo 98476 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> > mingo 98522 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> > mingo 98525 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> > mingo 98548 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> > mingo 98568 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> > mingo 98596 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> > mingo 98618 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> > mingo 98644 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> > mingo 98695 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> > mingo 98708 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> > mingo 98737 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> > mingo 98756 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct> > > I'm not seeing anything happen in 'top' - only a mostly idle system. > >> syz-manager/syz-fuzzer only guide the process. >> You can set "procs" param in config to higher value to increase CPU >> utilization. To get more bugs you want to saturate all CPUs to trigger >> more unusual thread interleavings. > > So right now it doesn't seem to saturate 16 CPUs - not even close to it. > >> If there is a second unfinished thread hanging on a kernel spinlock or >> mutex, then it's definitely bad. >> It also helps to enable CONFIG_RCU_STALL_COMMON=y, >> CONFIG_DEBUG_ATOMIC_SLEEP=y, CONFIG_WQ_WATCHDOG=y and spinlock/mutex >> debugging. These can detect various stalls. > > I can just Ctrl-C the manager and it shuts down within a few seconds: > > 2016/03/08 17:39:25 serving rpc on tcp://127.0.0.1:33809 > 2016/03/08 17:51:45 local-0: lost connection: exit status 2 > 2016/03/08 17:51:45 local-0: saving crash 'lost connection' to crash-local-0-1457455905403390570 > 2016/03/08 18:04:04 local-0: lost connection: exit status 2 > 2016/03/08 18:04:04 local-0: saving crash 'lost connection' to crash-local-0-1457456644779165131 > 2016/03/08 18:16:24 local-0: lost connection: exit status 2 > 2016/03/08 18:16:24 local-0: saving crash 'lost connection' to crash-local-0-1457457384707190124 > ^C2016/03/08 18:22:53 shutting down... > > with nothing hanging around: OK, this all looks good. > fomalhaut:~/go/src/github.com/google/syzkaller> ps aux | grep -i syz > mingo 1374 0.0 0.0 118476 2376 pts/2 S+ 18:23 0:00 grep --color=auto -i syz > > and with no kernel messages in dmesg - and with a fully functional system. > > I'm running the 16-task load on a 120 CPU system - should I increase it to 120? > Does the code expect to saturate the system? No, it does not expect to saturate the system. Set "procs" to 480, or something like that.
[toc] | [prev] | [next] | [standalone]
| From | Ingo Molnar <mingo@kernel.org> |
|---|---|
| Date | 2016-03-08 18:40 +0100 |
| Message-ID | <ratLZ-40W-37@gated-at.bofh.it> |
| In reply to | #1353256 |
* Dmitry Vyukov <dvyukov@google.com> wrote: > > fomalhaut:~/go/src/github.com/google/syzkaller> ps aux | grep -i syz > > mingo 1374 0.0 0.0 118476 2376 pts/2 S+ 18:23 0:00 grep --color=auto -i syz > > > > and with no kernel messages in dmesg - and with a fully functional system. > > > > I'm running the 16-task load on a 120 CPU system - should I increase it to 120? > > Does the code expect to saturate the system? > > No, it does not expect to saturate the system. Set "procs" to 480, or > something like that. Does not seem to help much: fomalhaut:~> vmstat 10 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 0 257465904 219940 4736092 0 0 0 102 16022 4396 0 1 99 0 0 2 0 0 257452144 220496 4755052 0 0 2 3649 14286 4627 0 1 99 0 0 2 0 0 257473408 221188 4770824 0 0 15 1898 17175 4474 0 1 99 0 0 Only around 1% system utilization. Should I go for 1,000 or more? :) Peter, do you experience with running syz-kaller on larger CPU count Intel systems? Thanks, Ingo
[toc] | [prev] | [next] | [standalone]
Page 1 of 2 [1] 2 Next page →
Back to top | Article view | linux.kernel
csiph-web