Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1478857 > unrolled thread

Re: [PATCH 6/6] x86/dumpstack: allow preemption in show_stack_log_lvl() and dump_trace()

Started byIngo Molnar <mingo@kernel.org>
First post2016-09-08 09:10 +0200
Last post2016-09-09 08:20 +0200
Articles 7 — 3 participants

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH 6/6] x86/dumpstack: allow preemption in  show_stack_log_lvl() and dump_trace() Ingo Molnar <mingo@kernel.org> - 2016-09-08 09:10 +0200
    Re: [PATCH 6/6] x86/dumpstack: allow preemption in  show_stack_log_lvl() and dump_trace() Josh Poimboeuf <jpoimboe@redhat.com> - 2016-09-08 23:50 +0200
      [PATCH v2] x86/dumpstack: allow preemption in show_stack_log_lvl() and dump_trace() Josh Poimboeuf <jpoimboe@redhat.com> - 2016-09-09 00:00 +0200
        Re: [PATCH v2] x86/dumpstack: allow preemption in  show_stack_log_lvl() and dump_trace() Ingo Molnar <mingo@kernel.org> - 2016-09-13 20:40 +0200
          Re: [PATCH v2] x86/dumpstack: allow preemption in  show_stack_log_lvl() and dump_trace() Josh Poimboeuf <jpoimboe@redhat.com> - 2016-09-13 21:40 +0200
        [tip:x86/asm] x86/dumpstack: Allow preemption in  show_stack_log_lvl() and dump_trace() tip-bot for Josh Poimboeuf <tipbot@zytor.com> - 2016-09-14 19:50 +0200
      Re: [PATCH 6/6] x86/dumpstack: allow preemption in  show_stack_log_lvl() and dump_trace() Ingo Molnar <mingo@kernel.org> - 2016-09-09 08:20 +0200

#1478857 — Re: [PATCH 6/6] x86/dumpstack: allow preemption in show_stack_log_lvl() and dump_trace()

FromIngo Molnar <mingo@kernel.org>
Date2016-09-08 09:10 +0200
SubjectRe: [PATCH 6/6] x86/dumpstack: allow preemption in show_stack_log_lvl() and dump_trace()
Message-ID<sf1Dd-2F3-73@gated-at.bofh.it>
* Josh Poimboeuf <jpoimboe@redhat.com> wrote:

>  arch/x86/kernel/dumpstack_32.c | 14 ++++++--------

> --- a/arch/x86/kernel/dumpstack_32.c
> +++ b/arch/x86/kernel/dumpstack_32.c

> +static void *is_softirq_stack(unsigned long *stack);
>  {

FYI, this bit clearly wasn't build tested on 32-bit even once - I'm skipping this 
patch.

Thanks,

	Ingo

[toc] | [next] | [standalone]

#1479536

FromJosh Poimboeuf <jpoimboe@redhat.com>
Date2016-09-08 23:50 +0200
Message-ID<sffmN-2IM-9@gated-at.bofh.it>
In reply to#1478857
On Thu, Sep 08, 2016 at 09:04:01AM +0200, Ingo Molnar wrote:
> 
> * Josh Poimboeuf <jpoimboe@redhat.com> wrote:
> 
> >  arch/x86/kernel/dumpstack_32.c | 14 ++++++--------
> 
> > --- a/arch/x86/kernel/dumpstack_32.c
> > +++ b/arch/x86/kernel/dumpstack_32.c
> 
> > +static void *is_softirq_stack(unsigned long *stack);
> >  {
> 
> FYI, this bit clearly wasn't build tested on 32-bit even once - I'm skipping this 
> patch.

Argh, sorry about that.  I did a lot of testing on 32-bit with this
patch before.  I guess I goofed somehow when I split up the patches.

-- 
Josh

[toc] | [prev] | [next] | [standalone]

#1479541 — [PATCH v2] x86/dumpstack: allow preemption in show_stack_log_lvl() and dump_trace()

FromJosh Poimboeuf <jpoimboe@redhat.com>
Date2016-09-09 00:00 +0200
Subject[PATCH v2] x86/dumpstack: allow preemption in show_stack_log_lvl() and dump_trace()
Message-ID<sffwu-2LV-17@gated-at.bofh.it>
In reply to#1479536
show_stack_log_lvl() and dump_trace() are already preemption safe:

- If they're running in irq or exception context, preemption is already
  disabled and the percpu stack pointers can be trusted.

- If they're running with preemption enabled, they must be running on
  the task stack anyway, so it doesn't matter if they're comparing the
  stack pointer against a percpu stack pointer from this CPU or another
  one: either way it won't match.
---
 arch/x86/kernel/dumpstack_32.c | 14 ++++++--------
 arch/x86/kernel/dumpstack_64.c | 26 +++++++++-----------------
 2 files changed, 15 insertions(+), 25 deletions(-)

diff --git a/arch/x86/kernel/dumpstack_32.c b/arch/x86/kernel/dumpstack_32.c
index c533b8b..da5cd62 100644
--- a/arch/x86/kernel/dumpstack_32.c
+++ b/arch/x86/kernel/dumpstack_32.c
@@ -24,16 +24,16 @@ static void *is_irq_stack(void *p, void *irq)
 }
 
 
-static void *is_hardirq_stack(unsigned long *stack, int cpu)
+static void *is_hardirq_stack(unsigned long *stack)
 {
-	void *irq = per_cpu(hardirq_stack, cpu);
+	void *irq = this_cpu_read(hardirq_stack);
 
 	return is_irq_stack(stack, irq);
 }
 
-static void *is_softirq_stack(unsigned long *stack, int cpu)
+static void *is_softirq_stack(unsigned long *stack)
 {
-	void *irq = per_cpu(softirq_stack, cpu);
+	void *irq = this_cpu_read(softirq_stack);
 
 	return is_irq_stack(stack, irq);
 }
@@ -42,7 +42,6 @@ void dump_trace(struct task_struct *task, struct pt_regs *regs,
 		unsigned long *stack, unsigned long bp,
 		const struct stacktrace_ops *ops, void *data)
 {
-	const unsigned cpu = get_cpu();
 	int graph = 0;
 	u32 *prev_esp;
 
@@ -53,9 +52,9 @@ void dump_trace(struct task_struct *task, struct pt_regs *regs,
 	for (;;) {
 		void *end_stack;
 
-		end_stack = is_hardirq_stack(stack, cpu);
+		end_stack = is_hardirq_stack(stack);
 		if (!end_stack)
-			end_stack = is_softirq_stack(stack, cpu);
+			end_stack = is_softirq_stack(stack);
 
 		bp = ops->walk_stack(task, stack, bp, ops, data,
 				     end_stack, &graph);
@@ -74,7 +73,6 @@ void dump_trace(struct task_struct *task, struct pt_regs *regs,
 			break;
 		touch_nmi_watchdog();
 	}
-	put_cpu();
 }
 EXPORT_SYMBOL(dump_trace);
 
diff --git a/arch/x86/kernel/dumpstack_64.c b/arch/x86/kernel/dumpstack_64.c
index b243352..07373be 100644
--- a/arch/x86/kernel/dumpstack_64.c
+++ b/arch/x86/kernel/dumpstack_64.c
@@ -31,8 +31,8 @@ static char x86_stack_ids[][8] = {
 #endif
 };
 
-static unsigned long *in_exception_stack(unsigned cpu, unsigned long stack,
-					 unsigned *usedp, char **idp)
+static unsigned long *in_exception_stack(unsigned long stack, unsigned *usedp,
+					 char **idp)
 {
 	unsigned k;
 
@@ -41,7 +41,7 @@ static unsigned long *in_exception_stack(unsigned cpu, unsigned long stack,
 	 * 'stack' is in one of them:
 	 */
 	for (k = 0; k < N_EXCEPTION_STACKS; k++) {
-		unsigned long end = per_cpu(orig_ist, cpu).ist[k];
+		unsigned long end = raw_cpu_ptr(&orig_ist)->ist[k];
 		/*
 		 * Is 'stack' above this exception frame's end?
 		 * If yes then skip to the next frame.
@@ -111,7 +111,7 @@ enum stack_type {
 };
 
 static enum stack_type
-analyze_stack(int cpu, struct task_struct *task, unsigned long *stack,
+analyze_stack(struct task_struct *task, unsigned long *stack,
 	      unsigned long **stack_end, unsigned long *irq_stack,
 	      unsigned *used, char **id)
 {
@@ -121,8 +121,7 @@ analyze_stack(int cpu, struct task_struct *task, unsigned long *stack,
 	if ((unsigned long)task_stack_page(task) == addr)
 		return STACK_IS_NORMAL;
 
-	*stack_end = in_exception_stack(cpu, (unsigned long)stack,
-					used, id);
+	*stack_end = in_exception_stack((unsigned long)stack, used, id);
 	if (*stack_end)
 		return STACK_IS_EXCEPTION;
 
@@ -149,8 +148,7 @@ void dump_trace(struct task_struct *task, struct pt_regs *regs,
 		unsigned long *stack, unsigned long bp,
 		const struct stacktrace_ops *ops, void *data)
 {
-	const unsigned cpu = get_cpu();
-	unsigned long *irq_stack = (unsigned long *)per_cpu(irq_stack_ptr, cpu);
+	unsigned long *irq_stack = (unsigned long *)this_cpu_read(irq_stack_ptr);
 	unsigned used = 0;
 	int graph = 0;
 	int done = 0;
@@ -169,8 +167,8 @@ void dump_trace(struct task_struct *task, struct pt_regs *regs,
 		enum stack_type stype;
 		char *id;
 
-		stype = analyze_stack(cpu, task, stack, &stack_end,
-				      irq_stack, &used, &id);
+		stype = analyze_stack(task, stack, &stack_end, irq_stack, &used,
+				      &id);
 
 		/* Default finish unless specified to continue */
 		done = 1;
@@ -225,7 +223,6 @@ void dump_trace(struct task_struct *task, struct pt_regs *regs,
 	 * This handles the process stack:
 	 */
 	bp = ops->walk_stack(task, stack, bp, ops, data, NULL, &graph);
-	put_cpu();
 }
 EXPORT_SYMBOL(dump_trace);
 
@@ -236,13 +233,9 @@ show_stack_log_lvl(struct task_struct *task, struct pt_regs *regs,
 	unsigned long *irq_stack_end;
 	unsigned long *irq_stack;
 	unsigned long *stack;
-	int cpu;
 	int i;
 
-	preempt_disable();
-	cpu = smp_processor_id();
-
-	irq_stack_end = (unsigned long *)(per_cpu(irq_stack_ptr, cpu));
+	irq_stack_end = (unsigned long *)this_cpu_read(irq_stack_ptr);
 	irq_stack     = irq_stack_end - (IRQ_STACK_SIZE / sizeof(long));
 
 	sp = sp ? : get_stack_pointer(task, regs);
@@ -274,7 +267,6 @@ show_stack_log_lvl(struct task_struct *task, struct pt_regs *regs,
 		stack++;
 		touch_nmi_watchdog();
 	}
-	preempt_enable();
 
 	pr_cont("\n");
 	show_trace_log_lvl(task, regs, sp, bp, log_lvl);
-- 
2.7.4

[toc] | [prev] | [next] | [standalone]

#1482701 — Re: [PATCH v2] x86/dumpstack: allow preemption in show_stack_log_lvl() and dump_trace()

FromIngo Molnar <mingo@kernel.org>
Date2016-09-13 20:40 +0200
SubjectRe: [PATCH v2] x86/dumpstack: allow preemption in show_stack_log_lvl() and dump_trace()
Message-ID<sh0MF-5AJ-17@gated-at.bofh.it>
In reply to#1479541
* Josh Poimboeuf <jpoimboe@redhat.com> wrote:

> show_stack_log_lvl() and dump_trace() are already preemption safe:
> 
> - If they're running in irq or exception context, preemption is already
>   disabled and the percpu stack pointers can be trusted.
> 
> - If they're running with preemption enabled, they must be running on
>   the task stack anyway, so it doesn't matter if they're comparing the
>   stack pointer against a percpu stack pointer from this CPU or another
>   one: either way it won't match.

Yeah, so I'm having second thoughts about this patch. My worry here is: what if we 
get preempted in this sequence?

If the kernel is borked real bad then we could get technically correct but really, 
really weird looking stack traces if for example the task stack is getting 
corrupted or something like that.

Dunno. How long does the worst-case processing here take on a typical x86 system, 
does it really matter to scheduling latency?

Thanks,

	Ingo

[toc] | [prev] | [next] | [standalone]

#1482728 — Re: [PATCH v2] x86/dumpstack: allow preemption in show_stack_log_lvl() and dump_trace()

FromJosh Poimboeuf <jpoimboe@redhat.com>
Date2016-09-13 21:40 +0200
SubjectRe: [PATCH v2] x86/dumpstack: allow preemption in show_stack_log_lvl() and dump_trace()
Message-ID<sh1IJ-6cg-15@gated-at.bofh.it>
In reply to#1482701
On Tue, Sep 13, 2016 at 08:29:57PM +0200, Ingo Molnar wrote:
> 
> * Josh Poimboeuf <jpoimboe@redhat.com> wrote:
> 
> > show_stack_log_lvl() and dump_trace() are already preemption safe:
> > 
> > - If they're running in irq or exception context, preemption is already
> >   disabled and the percpu stack pointers can be trusted.
> > 
> > - If they're running with preemption enabled, they must be running on
> >   the task stack anyway, so it doesn't matter if they're comparing the
> >   stack pointer against a percpu stack pointer from this CPU or another
> >   one: either way it won't match.
> 
> Yeah, so I'm having second thoughts about this patch. My worry here is: what if we 
> get preempted in this sequence?
> 
> If the kernel is borked real bad then we could get technically correct but really, 
> really weird looking stack traces if for example the task stack is getting 
> corrupted or something like that.

If it's in the oops or BUG path, there can't be preemption anyway
because oops_begin() disables interrupts.

It does look like the WARN path could get preempted.  Not to mention all
the other callers of show_regs(), dump_stack(), show_stack_log_lvl(),
etc.  In those cases, if the stack dump got preempted in the middle, and
then another task dumped its stack, the two dumps could be interspersed
a bit which would indeed be a little confusing.

But that would be quite rare.  And anyway, we already have the same
issue today when two CPUs are dumping the stack at the same time.  So I
don't think it's much of an issue.

> Dunno. How long does the worst-case processing here take on a typical x86 system, 
> does it really matter to scheduling latency?

I haven't heard any complaints about latency.  The goal was just to try
to simplify the code a bit.

-- 
Josh

[toc] | [prev] | [next] | [standalone]

#1483508 — [tip:x86/asm] x86/dumpstack: Allow preemption in show_stack_log_lvl() and dump_trace()

Fromtip-bot for Josh Poimboeuf <tipbot@zytor.com>
Date2016-09-14 19:50 +0200
Subject[tip:x86/asm] x86/dumpstack: Allow preemption in show_stack_log_lvl() and dump_trace()
Message-ID<shmtP-3OB-5@gated-at.bofh.it>
In reply to#1479541
Commit-ID:  cfeeed279dc2fa83a00fbe4856ebd231d56201ab
Gitweb:     http://git.kernel.org/tip/cfeeed279dc2fa83a00fbe4856ebd231d56201ab
Author:     Josh Poimboeuf <jpoimboe@redhat.com>
AuthorDate: Thu, 8 Sep 2016 16:49:20 -0500
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Wed, 14 Sep 2016 17:23:30 +0200

x86/dumpstack: Allow preemption in show_stack_log_lvl() and dump_trace()

show_stack_log_lvl() and dump_trace() are already preemption safe:

- If they're running in irq or exception context, preemption is already
  disabled and the percpu stack pointers can be trusted.

- If they're running with preemption enabled, they must be running on
  the task stack anyway, so it doesn't matter if they're comparing the
  stack pointer against a percpu stack pointer from this CPU or another
  one: either way it won't match.

Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Byungchul Park <byungchul.park@lge.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Kees Cook <keescook@chromium.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Nilay Vaish <nilayvaish@gmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/a0ca0b1044eca97d4f0ec7c1619cf80b3b65560d.1473371307.git.jpoimboe@redhat.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 arch/x86/kernel/dumpstack_32.c | 14 ++++++--------
 arch/x86/kernel/dumpstack_64.c | 26 +++++++++-----------------
 2 files changed, 15 insertions(+), 25 deletions(-)

diff --git a/arch/x86/kernel/dumpstack_32.c b/arch/x86/kernel/dumpstack_32.c
index c533b8b..da5cd62 100644
--- a/arch/x86/kernel/dumpstack_32.c
+++ b/arch/x86/kernel/dumpstack_32.c
@@ -24,16 +24,16 @@ static void *is_irq_stack(void *p, void *irq)
 }
 
 
-static void *is_hardirq_stack(unsigned long *stack, int cpu)
+static void *is_hardirq_stack(unsigned long *stack)
 {
-	void *irq = per_cpu(hardirq_stack, cpu);
+	void *irq = this_cpu_read(hardirq_stack);
 
 	return is_irq_stack(stack, irq);
 }
 
-static void *is_softirq_stack(unsigned long *stack, int cpu)
+static void *is_softirq_stack(unsigned long *stack)
 {
-	void *irq = per_cpu(softirq_stack, cpu);
+	void *irq = this_cpu_read(softirq_stack);
 
 	return is_irq_stack(stack, irq);
 }
@@ -42,7 +42,6 @@ void dump_trace(struct task_struct *task, struct pt_regs *regs,
 		unsigned long *stack, unsigned long bp,
 		const struct stacktrace_ops *ops, void *data)
 {
-	const unsigned cpu = get_cpu();
 	int graph = 0;
 	u32 *prev_esp;
 
@@ -53,9 +52,9 @@ void dump_trace(struct task_struct *task, struct pt_regs *regs,
 	for (;;) {
 		void *end_stack;
 
-		end_stack = is_hardirq_stack(stack, cpu);
+		end_stack = is_hardirq_stack(stack);
 		if (!end_stack)
-			end_stack = is_softirq_stack(stack, cpu);
+			end_stack = is_softirq_stack(stack);
 
 		bp = ops->walk_stack(task, stack, bp, ops, data,
 				     end_stack, &graph);
@@ -74,7 +73,6 @@ void dump_trace(struct task_struct *task, struct pt_regs *regs,
 			break;
 		touch_nmi_watchdog();
 	}
-	put_cpu();
 }
 EXPORT_SYMBOL(dump_trace);
 
diff --git a/arch/x86/kernel/dumpstack_64.c b/arch/x86/kernel/dumpstack_64.c
index b243352..07373be 100644
--- a/arch/x86/kernel/dumpstack_64.c
+++ b/arch/x86/kernel/dumpstack_64.c
@@ -31,8 +31,8 @@ static char x86_stack_ids[][8] = {
 #endif
 };
 
-static unsigned long *in_exception_stack(unsigned cpu, unsigned long stack,
-					 unsigned *usedp, char **idp)
+static unsigned long *in_exception_stack(unsigned long stack, unsigned *usedp,
+					 char **idp)
 {
 	unsigned k;
 
@@ -41,7 +41,7 @@ static unsigned long *in_exception_stack(unsigned cpu, unsigned long stack,
 	 * 'stack' is in one of them:
 	 */
 	for (k = 0; k < N_EXCEPTION_STACKS; k++) {
-		unsigned long end = per_cpu(orig_ist, cpu).ist[k];
+		unsigned long end = raw_cpu_ptr(&orig_ist)->ist[k];
 		/*
 		 * Is 'stack' above this exception frame's end?
 		 * If yes then skip to the next frame.
@@ -111,7 +111,7 @@ enum stack_type {
 };
 
 static enum stack_type
-analyze_stack(int cpu, struct task_struct *task, unsigned long *stack,
+analyze_stack(struct task_struct *task, unsigned long *stack,
 	      unsigned long **stack_end, unsigned long *irq_stack,
 	      unsigned *used, char **id)
 {
@@ -121,8 +121,7 @@ analyze_stack(int cpu, struct task_struct *task, unsigned long *stack,
 	if ((unsigned long)task_stack_page(task) == addr)
 		return STACK_IS_NORMAL;
 
-	*stack_end = in_exception_stack(cpu, (unsigned long)stack,
-					used, id);
+	*stack_end = in_exception_stack((unsigned long)stack, used, id);
 	if (*stack_end)
 		return STACK_IS_EXCEPTION;
 
@@ -149,8 +148,7 @@ void dump_trace(struct task_struct *task, struct pt_regs *regs,
 		unsigned long *stack, unsigned long bp,
 		const struct stacktrace_ops *ops, void *data)
 {
-	const unsigned cpu = get_cpu();
-	unsigned long *irq_stack = (unsigned long *)per_cpu(irq_stack_ptr, cpu);
+	unsigned long *irq_stack = (unsigned long *)this_cpu_read(irq_stack_ptr);
 	unsigned used = 0;
 	int graph = 0;
 	int done = 0;
@@ -169,8 +167,8 @@ void dump_trace(struct task_struct *task, struct pt_regs *regs,
 		enum stack_type stype;
 		char *id;
 
-		stype = analyze_stack(cpu, task, stack, &stack_end,
-				      irq_stack, &used, &id);
+		stype = analyze_stack(task, stack, &stack_end, irq_stack, &used,
+				      &id);
 
 		/* Default finish unless specified to continue */
 		done = 1;
@@ -225,7 +223,6 @@ void dump_trace(struct task_struct *task, struct pt_regs *regs,
 	 * This handles the process stack:
 	 */
 	bp = ops->walk_stack(task, stack, bp, ops, data, NULL, &graph);
-	put_cpu();
 }
 EXPORT_SYMBOL(dump_trace);
 
@@ -236,13 +233,9 @@ show_stack_log_lvl(struct task_struct *task, struct pt_regs *regs,
 	unsigned long *irq_stack_end;
 	unsigned long *irq_stack;
 	unsigned long *stack;
-	int cpu;
 	int i;
 
-	preempt_disable();
-	cpu = smp_processor_id();
-
-	irq_stack_end = (unsigned long *)(per_cpu(irq_stack_ptr, cpu));
+	irq_stack_end = (unsigned long *)this_cpu_read(irq_stack_ptr);
 	irq_stack     = irq_stack_end - (IRQ_STACK_SIZE / sizeof(long));
 
 	sp = sp ? : get_stack_pointer(task, regs);
@@ -274,7 +267,6 @@ show_stack_log_lvl(struct task_struct *task, struct pt_regs *regs,
 		stack++;
 		touch_nmi_watchdog();
 	}
-	preempt_enable();
 
 	pr_cont("\n");
 	show_trace_log_lvl(task, regs, sp, bp, log_lvl);

[toc] | [prev] | [next] | [standalone]

#1479674

FromIngo Molnar <mingo@kernel.org>
Date2016-09-09 08:20 +0200
Message-ID<sfnkl-7LY-3@gated-at.bofh.it>
In reply to#1479536
* Josh Poimboeuf <jpoimboe@redhat.com> wrote:

> On Thu, Sep 08, 2016 at 09:04:01AM +0200, Ingo Molnar wrote:
> > 
> > * Josh Poimboeuf <jpoimboe@redhat.com> wrote:
> > 
> > >  arch/x86/kernel/dumpstack_32.c | 14 ++++++--------
> > 
> > > --- a/arch/x86/kernel/dumpstack_32.c
> > > +++ b/arch/x86/kernel/dumpstack_32.c
> > 
> > > +static void *is_softirq_stack(unsigned long *stack);
> > >  {
> > 
> > FYI, this bit clearly wasn't build tested on 32-bit even once - I'm skipping this 
> > patch.
> 
> Argh, sorry about that.  I did a lot of testing on 32-bit with this
> patch before.  I guess I goofed somehow when I split up the patches.

No problem!

Thanks,

	Ingo

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web