Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1570002 > unrolled thread
| Started by | Jiri Olsa <jolsa@redhat.com> |
|---|---|
| First post | 2017-01-30 19:50 +0100 |
| Last post | 2017-02-02 12:40 +0100 |
| Articles | 9 — 2 participants |
Back to article view | Back to linux.kernel
This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by
below is the oldest one visible, not the original post.
Re: [PATCH] perf: fix topology test on systems with sparse CPUs Jiri Olsa <jolsa@redhat.com> - 2017-01-30 19:50 +0100
Re: [PATCH] perf: fix topology test on systems with sparse CPUs Jan Stancek <jstancek@redhat.com> - 2017-01-30 20:30 +0100
Re: [PATCH] perf: fix topology test on systems with sparse CPUs Jan Stancek <jstancek@redhat.com> - 2017-01-31 17:10 +0100
Re: [PATCH] perf: fix topology test on systems with sparse CPUs Jiri Olsa <jolsa@redhat.com> - 2017-02-02 12:30 +0100
Re: [PATCH] perf: fix topology test on systems with sparse CPUs Jan Stancek <jstancek@redhat.com> - 2017-02-02 13:10 +0100
Re: [PATCH] perf: fix topology test on systems with sparse CPUs Jiri Olsa <jolsa@redhat.com> - 2017-02-02 14:10 +0100
[PATCH v2 1/3] perf: add cpu__max_present_cpu() Jan Stancek <jstancek@redhat.com> - 2017-02-13 16:40 +0100
[PATCH v2 2/3] perf: make build_cpu_topology skip offline/absent CPUs Jan Stancek <jstancek@redhat.com> - 2017-02-13 16:40 +0100
Re: [PATCH] perf: fix topology test on systems with sparse CPUs Jiri Olsa <jolsa@redhat.com> - 2017-02-02 12:40 +0100
| From | Jiri Olsa <jolsa@redhat.com> |
|---|---|
| Date | 2017-01-30 19:50 +0100 |
| Subject | Re: [PATCH] perf: fix topology test on systems with sparse CPUs |
| Message-ID | <t5pbz-7p2-7@gated-at.bofh.it> |
On Mon, Jan 30, 2017 at 05:53:34PM +0100, Jan Stancek wrote:
SNIP
> + ret = build_cpu_topo(tp, cpu);
> if (ret < 0)
> break;
> }
> +
> +out_free:
> + cpu_map__put(map);
> if (ret) {
> free_cpu_topo(tp);
> tp = NULL;
> }
> +out:
> return tp;
> }
>
> @@ -575,7 +579,7 @@ static int write_cpu_topology(int fd, struct perf_header *h __maybe_unused,
> if (ret < 0)
> goto done;
>
> - for (j = 0; j < perf_env.nr_cpus_avail; j++) {
> + for (j = 0; j < perf_env.nr_cpus_online; j++) {
so basically we're changing from avail to online cpus
have you checked all the users of this FEATURE
if such change is ok?
I can't find any after quick search, but it would be
good to be sure and mention that in changelog
thanks,
jirka
[toc] | [next] | [standalone]
| From | Jan Stancek <jstancek@redhat.com> |
|---|---|
| Date | 2017-01-30 20:30 +0100 |
| Message-ID | <t5pOj-7RS-21@gated-at.bofh.it> |
| In reply to | #1570002 |
----- Original Message -----
> From: "Jiri Olsa" <jolsa@redhat.com>
> To: "Jan Stancek" <jstancek@redhat.com>
> Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, mingo@redhat.com, acme@kernel.org, "alexander shishkin"
> <alexander.shishkin@linux.intel.com>, jolsa@kernel.org, mhiramat@kernel.org, "rui teng"
> <rui.teng@linux.vnet.ibm.com>, sukadev@linux.vnet.ibm.com
> Sent: Monday, 30 January, 2017 7:49:08 PM
> Subject: Re: [PATCH] perf: fix topology test on systems with sparse CPUs
>
> On Mon, Jan 30, 2017 at 05:53:34PM +0100, Jan Stancek wrote:
>
> SNIP
>
> > + ret = build_cpu_topo(tp, cpu);
> > if (ret < 0)
> > break;
> > }
> > +
> > +out_free:
> > + cpu_map__put(map);
> > if (ret) {
> > free_cpu_topo(tp);
> > tp = NULL;
> > }
> > +out:
> > return tp;
> > }
> >
> > @@ -575,7 +579,7 @@ static int write_cpu_topology(int fd, struct
> > perf_header *h __maybe_unused,
> > if (ret < 0)
> > goto done;
> >
> > - for (j = 0; j < perf_env.nr_cpus_avail; j++) {
> > + for (j = 0; j < perf_env.nr_cpus_online; j++) {
>
> so basically we're changing from avail to online cpus
>
> have you checked all the users of this FEATURE
> if such change is ok?
You're right, I missed some. Looking again, I see at least
perf_env__get_core() could break.
Regards,
Jan
[toc] | [prev] | [next] | [standalone]
| From | Jan Stancek <jstancek@redhat.com> |
|---|---|
| Date | 2017-01-31 17:10 +0100 |
| Message-ID | <t5Jai-2Mv-15@gated-at.bofh.it> |
| In reply to | #1570002 |
[Multipart message — attachments visible in raw view] — view raw
On 01/30/2017 07:49 PM, Jiri Olsa wrote:
> so basically we're changing from avail to online cpus
>
> have you checked all the users of this FEATURE
> if such change is ok?
Jiri,
It wasn't OK as there are other users who index cpu_topology_map by CPU id.
I decided to give the alternative a try (attached): keep cpu_topology_map
indexed by CPU id, but extend it to fit max present CPU.
So for a system like this one:
_SC_NPROCESSORS_CONF == 16
available: 2 nodes (0-1)
node 0 cpus: 0 6 8 10 16 22 24 26
node 0 size: 12004 MB
node 0 free: 9470 MB
node 1 cpus: 1 7 9 11 23 25 27
node 1 size: 12093 MB
node 1 free: 9406 MB
node distances:
node 0 1
0: 10 20
1: 20 10
HEADER_NRCPUS before:
nr_cpus_online = 15
nr_cpus_available = 16
HEADER_CPU_TOPOLOGY before:
core_sib_nr: 2
core_siblings: 0,6,8,10,16,22,24,26
core_siblings: 1,7,9,11,23,25,27
thread_sib_nr: 8
thread_siblings: 0,16
thread_siblings: 1
thread_siblings: 6,22
thread_siblings: 7,23
thread_siblings: 8,24
thread_siblings: 9,25
thread_siblings: 10,26
thread_siblings: 11,27
core_id: 0, socket_id: 0
core_id: 0, socket_id: 1
core_id: -1, socket_id: -1
core_id: -1, socket_id: -1
core_id: -1, socket_id: -1
core_id: -1, socket_id: -1
core_id: 10, socket_id: 0
core_id: 10, socket_id: 1
core_id: 1, socket_id: 0
core_id: 1, socket_id: 1
core_id: 9, socket_id: 0
core_id: 9, socket_id: 1
core_id: -1, socket_id: -1
core_id: -1, socket_id: -1
core_id: -1, socket_id: -1
core_id: -1, socket_id: -1
HEADER_NRCPUS after:
nr_cpus_online = 15
nr_cpus_available = 28
HEADER_CPU_TOPOLOGY after:
core_sib_nr: 2
core_siblings: 0,6,8,10,16,22,24,26
core_siblings: 1,7,9,11,23,25,27
thread_sib_nr: 8
thread_siblings: 0,16
thread_siblings: 1
thread_siblings: 6,22
thread_siblings: 7,23
thread_siblings: 8,24
thread_siblings: 9,25
thread_siblings: 10,26
thread_siblings: 11,27
core_id: 0, socket_id: 0
core_id: 0, socket_id: 1
core_id: -1, socket_id: -1
core_id: -1, socket_id: -1
core_id: -1, socket_id: -1
core_id: -1, socket_id: -1
core_id: 10, socket_id: 0
core_id: 10, socket_id: 1
core_id: 1, socket_id: 0
core_id: 1, socket_id: 1
core_id: 9, socket_id: 0
core_id: 9, socket_id: 1
core_id: -1, socket_id: -1
core_id: -1, socket_id: -1
core_id: -1, socket_id: -1
core_id: -1, socket_id: -1
core_id: 0, socket_id: 0
core_id: -1, socket_id: -1
core_id: -1, socket_id: -1
core_id: -1, socket_id: -1
core_id: -1, socket_id: -1
core_id: -1, socket_id: -1
core_id: 10, socket_id: 0
core_id: 10, socket_id: 1
core_id: 1, socket_id: 0
core_id: 1, socket_id: 1
core_id: 9, socket_id: 0
core_id: 9, socket_id: 1
Regards,
Jan
[toc] | [prev] | [next] | [standalone]
| From | Jiri Olsa <jolsa@redhat.com> |
|---|---|
| Date | 2017-02-02 12:30 +0100 |
| Message-ID | <t6nKp-3ij-3@gated-at.bofh.it> |
| In reply to | #1570854 |
On Tue, Jan 31, 2017 at 05:03:51PM +0100, Jan Stancek wrote:
> On 01/30/2017 07:49 PM, Jiri Olsa wrote:
> > so basically we're changing from avail to online cpus
> >
> > have you checked all the users of this FEATURE
> > if such change is ok?
>
> Jiri,
>
> It wasn't OK as there are other users who index cpu_topology_map by CPU id.
> I decided to give the alternative a try (attached): keep cpu_topology_map
> indexed by CPU id, but extend it to fit max present CPU.
please send this next time as a standard patchset,
it's hard to discuss over attachments
SNIP
> When build_cpu_topo() encounters offline/absent CPUs,
> it fails to find any sysfs entries and returns failure.
> This leads to build_cpu_topology() and write_cpu_topology()
> failing as well.
>
> Because HEADER_CPU_TOPOLOGY has not been written, read leaves
> cpu_topology_map NULL and we get NULL ptr deref at:
>
> ...
> cmd_test
> __cmd_test
> test_and_print
> run_test
> test_session_topology
> check_cpu_topology
So IIUIC that's the key issue here.. write_cpu_topology that fails
to write the TOPO data and following readers crashing on processing
uncomplete data? if thats the case write_cpu_topology needs to
be fixed, instead of doing workarounds
SNIP
> u32 nr, i;
> size_t sz;
> long ncpus;
> - int ret = -1;
> + int ret = 0;
> + struct cpu_map *map;
>
> ncpus = sysconf(_SC_NPROCESSORS_CONF);
> if (ncpus < 0)
> - return NULL;
> + goto out;
can just return NULL
> +
> + /* build online CPU map */
> + map = cpu_map__new(NULL);
> + if (map == NULL) {
> + pr_debug("failed to get system cpumap\n");
> + goto out;
> + }
>
> nr = (u32)(ncpus & UINT_MAX);
>
> sz = nr * sizeof(char *);
> -
> addr = calloc(1, sizeof(*tp) + 2 * sz);
> if (!addr)
> - return NULL;
> + goto out_free;
>
> tp = addr;
> tp->cpu_nr = nr;
> @@ -530,14 +537,21 @@ static struct cpu_topo *build_cpu_topology(void)
> tp->thread_siblings = addr;
>
> for (i = 0; i < nr; i++) {
> + if (!cpu_map__has(map, i))
> + continue;
> +
so this prevents build_cpu_topo to fail due to missing topology
info because cpu is offline.. can it fail for other reasons?
> ret = build_cpu_topo(tp, i);
> if (ret < 0)
> break;
SNIP
[toc] | [prev] | [next] | [standalone]
| From | Jan Stancek <jstancek@redhat.com> |
|---|---|
| Date | 2017-02-02 13:10 +0100 |
| Message-ID | <t6on8-3Ln-5@gated-at.bofh.it> |
| In reply to | #1572287 |
>
> > When build_cpu_topo() encounters offline/absent CPUs,
> > it fails to find any sysfs entries and returns failure.
> > This leads to build_cpu_topology() and write_cpu_topology()
> > failing as well.
> >
> > Because HEADER_CPU_TOPOLOGY has not been written, read leaves
> > cpu_topology_map NULL and we get NULL ptr deref at:
> >
> > ...
> > cmd_test
> > __cmd_test
> > test_and_print
> > run_test
> > test_session_topology
> > check_cpu_topology
>
> So IIUIC that's the key issue here.. write_cpu_topology that fails
> to write the TOPO data and following readers crashing on processing
> uncomplete data? if thats the case write_cpu_topology needs to
> be fixed, instead of doing workarounds
It's already late when you are in write_cpu_topology(), because
build_cpu_topology() returned you NULL - there's nothing to write.
That's why patch aims to fix this in build_cpu_topology().
>
> SNIP
>
> > u32 nr, i;
> > size_t sz;
> > long ncpus;
> > - int ret = -1;
> > + int ret = 0;
> > + struct cpu_map *map;
> >
> > ncpus = sysconf(_SC_NPROCESSORS_CONF);
> > if (ncpus < 0)
> > - return NULL;
> > + goto out;
>
> can just return NULL
>
> > +
> > + /* build online CPU map */
> > + map = cpu_map__new(NULL);
> > + if (map == NULL) {
> > + pr_debug("failed to get system cpumap\n");
> > + goto out;
> > + }
> >
> > nr = (u32)(ncpus & UINT_MAX);
> >
> > sz = nr * sizeof(char *);
> > -
> > addr = calloc(1, sizeof(*tp) + 2 * sz);
> > if (!addr)
> > - return NULL;
> > + goto out_free;
> >
> > tp = addr;
> > tp->cpu_nr = nr;
> > @@ -530,14 +537,21 @@ static struct cpu_topo *build_cpu_topology(void)
> > tp->thread_siblings = addr;
> >
> > for (i = 0; i < nr; i++) {
> > + if (!cpu_map__has(map, i))
> > + continue;
> > +
>
> so this prevents build_cpu_topo to fail due to missing topology
> info because cpu is offline.. can it fail for other reasons?
It's unlikely, though I suppose if you couldn't open and read something
from sysfs (say sysfs is not mounted) it can fail for online CPU too.
>
>
> > ret = build_cpu_topo(tp, i);
> > if (ret < 0)
> > break;
>
SNIP
> For example:
> _SC_NPROCESSORS_CONF == 16
> available: 2 nodes (0-1)
> node 0 cpus: 0 6 8 10 16 22 24 26
> node 0 size: 12004 MB
> node 0 free: 9470 MB
> node 1 cpus: 1 7 9 11 23 25 27
> node 1 size: 12093 MB
> node 1 free: 9406 MB
> node distances:
> node 0 1
> 0: 10 20
> 1: 20 10
> so what's max_present_cpu in this example?
It's 28, which is the number of core_id/socket_id entries,
for CPUs 0 up to 27.
Regards,
Jan
[toc] | [prev] | [next] | [standalone]
| From | Jiri Olsa <jolsa@redhat.com> |
|---|---|
| Date | 2017-02-02 14:10 +0100 |
| Message-ID | <t6pjc-4nr-21@gated-at.bofh.it> |
| In reply to | #1572310 |
On Thu, Feb 02, 2017 at 07:06:43AM -0500, Jan Stancek wrote: > > > > > When build_cpu_topo() encounters offline/absent CPUs, > > > it fails to find any sysfs entries and returns failure. > > > This leads to build_cpu_topology() and write_cpu_topology() > > > failing as well. > > > > > > Because HEADER_CPU_TOPOLOGY has not been written, read leaves > > > cpu_topology_map NULL and we get NULL ptr deref at: > > > > > > ... > > > cmd_test > > > __cmd_test > > > test_and_print > > > run_test > > > test_session_topology > > > check_cpu_topology > > > > So IIUIC that's the key issue here.. write_cpu_topology that fails > > to write the TOPO data and following readers crashing on processing > > uncomplete data? if thats the case write_cpu_topology needs to > > be fixed, instead of doing workarounds > > It's already late when you are in write_cpu_topology(), because > build_cpu_topology() returned you NULL - there's nothing to write. > That's why patch aims to fix this in build_cpu_topology(). ok, then we need to make sure we can't fail in write_cpu_topology might be another patch scope though.. we can go with your fix so far SNIP > > > For example: > > _SC_NPROCESSORS_CONF == 16 > > available: 2 nodes (0-1) > > node 0 cpus: 0 6 8 10 16 22 24 26 > > node 0 size: 12004 MB > > node 0 free: 9470 MB > > node 1 cpus: 1 7 9 11 23 25 27 > > node 1 size: 12093 MB > > node 1 free: 9406 MB > > node distances: > > node 0 1 > > 0: 10 20 > > 1: 20 10 > > so what's max_present_cpu in this example? > > It's 28, which is the number of core_id/socket_id entries, > for CPUs 0 up to 27. ok, good jirka
[toc] | [prev] | [next] | [standalone]
| From | Jan Stancek <jstancek@redhat.com> |
|---|---|
| Date | 2017-02-13 16:40 +0100 |
| Subject | [PATCH v2 1/3] perf: add cpu__max_present_cpu() |
| Message-ID | <taqTn-5xK-17@gated-at.bofh.it> |
| In reply to | #1572345 |
Similar to cpu__max_cpu() (which returns max possible CPU),
returns max present CPU.
Signed-off-by: Jan Stancek <jstancek@redhat.com>
---
tools/perf/util/cpumap.c | 22 ++++++++++++++++++++++
tools/perf/util/cpumap.h | 1 +
2 files changed, 23 insertions(+)
diff --git a/tools/perf/util/cpumap.c b/tools/perf/util/cpumap.c
index 2c0b52264a46..8c7504939113 100644
--- a/tools/perf/util/cpumap.c
+++ b/tools/perf/util/cpumap.c
@@ -9,6 +9,7 @@
#include "asm/bug.h"
static int max_cpu_num;
+static int max_present_cpu_num;
static int max_node_num;
static int *cpunode_map;
@@ -442,6 +443,7 @@ static void set_max_cpu_num(void)
/* set up default */
max_cpu_num = 4096;
+ max_present_cpu_num = 4096;
mnt = sysfs__mountpoint();
if (!mnt)
@@ -455,6 +457,17 @@ static void set_max_cpu_num(void)
}
ret = get_max_num(path, &max_cpu_num);
+ if (ret)
+ goto out;
+
+ /* get the highest present cpu number for a sparse allocation */
+ ret = snprintf(path, PATH_MAX, "%s/devices/system/cpu/present", mnt);
+ if (ret == PATH_MAX) {
+ pr_err("sysfs path crossed PATH_MAX(%d) size\n", PATH_MAX);
+ goto out;
+ }
+
+ ret = get_max_num(path, &max_present_cpu_num);
out:
if (ret)
@@ -505,6 +518,15 @@ int cpu__max_cpu(void)
return max_cpu_num;
}
+int cpu__max_present_cpu(void)
+{
+ if (unlikely(!max_present_cpu_num))
+ set_max_cpu_num();
+
+ return max_present_cpu_num;
+}
+
+
int cpu__get_node(int cpu)
{
if (unlikely(cpunode_map == NULL)) {
diff --git a/tools/perf/util/cpumap.h b/tools/perf/util/cpumap.h
index 06bd689f5989..1a0549af8f5c 100644
--- a/tools/perf/util/cpumap.h
+++ b/tools/perf/util/cpumap.h
@@ -62,6 +62,7 @@ static inline bool cpu_map__empty(const struct cpu_map *map)
int cpu__max_node(void);
int cpu__max_cpu(void);
+int cpu__max_present_cpu(void);
int cpu__get_node(int cpu);
int cpu_map__build_map(struct cpu_map *cpus, struct cpu_map **res,
--
1.8.3.1
[toc] | [prev] | [next] | [standalone]
| From | Jan Stancek <jstancek@redhat.com> |
|---|---|
| Date | 2017-02-13 16:40 +0100 |
| Subject | [PATCH v2 2/3] perf: make build_cpu_topology skip offline/absent CPUs |
| Message-ID | <taqTq-5xK-75@gated-at.bofh.it> |
| In reply to | #1579872 |
When build_cpu_topo() encounters offline/absent CPUs,
it fails to find any sysfs entries and returns failure.
This leads to build_cpu_topology() and write_cpu_topology()
failing as well.
Because HEADER_CPU_TOPOLOGY has not been written, read leaves
cpu_topology_map NULL and we get NULL ptr deref at:
...
cmd_test
__cmd_test
test_and_print
run_test
test_session_topology
check_cpu_topology
36: Session topology :
--- start ---
test child forked, pid 14902
templ file: /tmp/perf-test-4CKocW
failed to write feature HEADER_CPU_TOPOLOGY
perf: Segmentation fault
Obtained 9 stack frames.
./perf(sighandler_dump_stack+0x41) [0x5095f1]
/lib64/libc.so.6(+0x35250) [0x7f4b7c3c9250]
./perf(test_session_topology+0x1db) [0x490ceb]
./perf() [0x475b68]
./perf(cmd_test+0x5b9) [0x4763c9]
./perf() [0x4945a3]
./perf(main+0x69f) [0x427e8f]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f4b7c3b5b35]
./perf() [0x427fb9]
test child interrupted
---- end ----
Session topology: FAILED!
This patch makes build_cpu_topology() skip offline/absent CPUs,
by checking their presence against cpu_map built from online CPUs.
Signed-off-by: Jan Stancek <jstancek@redhat.com>
---
tools/perf/util/header.c | 21 +++++++++++++++++----
1 file changed, 17 insertions(+), 4 deletions(-)
Changes in v2:
- drop out label, use return NULL where possible
diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index d89c9c7ef4e5..4b0ea4e92e9d 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -503,24 +503,31 @@ static void free_cpu_topo(struct cpu_topo *tp)
static struct cpu_topo *build_cpu_topology(void)
{
- struct cpu_topo *tp;
+ struct cpu_topo *tp = NULL;
void *addr;
u32 nr, i;
size_t sz;
long ncpus;
- int ret = -1;
+ int ret = 0;
+ struct cpu_map *map;
ncpus = sysconf(_SC_NPROCESSORS_CONF);
if (ncpus < 0)
return NULL;
+ /* build online CPU map */
+ map = cpu_map__new(NULL);
+ if (map == NULL) {
+ pr_debug("failed to get system cpumap\n");
+ return NULL;
+ }
+
nr = (u32)(ncpus & UINT_MAX);
sz = nr * sizeof(char *);
-
addr = calloc(1, sizeof(*tp) + 2 * sz);
if (!addr)
- return NULL;
+ goto out_free;
tp = addr;
tp->cpu_nr = nr;
@@ -530,10 +537,16 @@ static struct cpu_topo *build_cpu_topology(void)
tp->thread_siblings = addr;
for (i = 0; i < nr; i++) {
+ if (!cpu_map__has(map, i))
+ continue;
+
ret = build_cpu_topo(tp, i);
if (ret < 0)
break;
}
+
+out_free:
+ cpu_map__put(map);
if (ret) {
free_cpu_topo(tp);
tp = NULL;
--
1.8.3.1
[toc] | [prev] | [next] | [standalone]
| From | Jiri Olsa <jolsa@redhat.com> |
|---|---|
| Date | 2017-02-02 12:40 +0100 |
| Message-ID | <t6nU5-3lz-25@gated-at.bofh.it> |
| In reply to | #1570854 |
On Tue, Jan 31, 2017 at 05:03:51PM +0100, Jan Stancek wrote: SNIP > > There are 2 problems wrt. cpu_topology_map on systems with sparse CPUs: > > 1. offline/absent CPUs will have their socket_id and core_id set to -1 > which triggers: > "socket_id number is too big.You may need to upgrade the perf tool." > > 2. size of cpu_topology_map (perf_env.cpu[]) is allocated based on > _SC_NPROCESSORS_CONF, but can be indexed with CPU ids going above. > Users of perf_env.cpu[] are using CPU id as index. This can lead > to read beyond what was allocated: > ==19991== Invalid read of size 4 > ==19991== at 0x490CEB: check_cpu_topology (topology.c:69) > ==19991== by 0x490CEB: test_session_topology (topology.c:106) > ... > > For example: > _SC_NPROCESSORS_CONF == 16 > available: 2 nodes (0-1) > node 0 cpus: 0 6 8 10 16 22 24 26 > node 0 size: 12004 MB > node 0 free: 9470 MB > node 1 cpus: 1 7 9 11 23 25 27 > node 1 size: 12093 MB > node 1 free: 9406 MB > node distances: > node 0 1 > 0: 10 20 > 1: 20 10 so what's max_present_cpu in this example? jirka
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web