Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1695095 > unrolled thread

Re: [PATCH 02/37] binder: use group leader instead of open thread

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH 02/37] binder: use group leader instead of open thread John Stultz <john.stultz@linaro.org> - 2017-07-24 23:10 +0200
    Re: [PATCH 02/37] binder: use group leader instead of open thread John Stultz <john.stultz@linaro.org> - 2017-07-24 23:10 +0200
      Re: [PATCH 02/37] binder: use group leader instead of open thread Martijn Coenen <maco@google.com> - 2017-07-25 11:20 +0200
        Re: [PATCH 02/37] binder: use group leader instead of open thread Amit Pundir <amit.pundir@linaro.org> - 2017-07-27 11:10 +0200
          Re: [PATCH 02/37] binder: use group leader instead of open thread Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-27 15:30 +0200
            Re: [PATCH 02/37] binder: use group leader instead of open thread Martijn Coenen <maco@google.com> - 2017-07-27 15:50 +0200
            Re: [PATCH 02/37] binder: use group leader instead of open thread Amit Pundir <amit.pundir@linaro.org> - 2017-07-27 15:50 +0200
              Re: [PATCH 02/37] binder: use group leader instead of open thread Martijn Coenen <maco@google.com> - 2017-07-28 14:00 +0200
    Re: [PATCH 02/37] binder: use group leader instead of open thread Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2017-07-24 23:30 +0200
      Re: [PATCH 02/37] binder: use group leader instead of open thread John Stultz <john.stultz@linaro.org> - 2017-07-25 00:00 +0200

#1695095 — Re: [PATCH 02/37] binder: use group leader instead of open thread

On Thu, Jun 29, 2017 at 12:01 PM, Todd Kjos <tkjos@android.com> wrote:
> The binder allocator assumes that the thread that
> called binder_open will never die for the lifetime of
> that proc. That thread is normally the group_leader,
> however it may not be. Use the group_leader instead
> of current.
>
> Signed-off-by: Todd Kjos <tkjos@google.com>
> ---
>  drivers/android/binder.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/android/binder.c b/drivers/android/binder.c
> index 157bd3e49ff4..9393924ae8e8 100644
> --- a/drivers/android/binder.c
> +++ b/drivers/android/binder.c
> @@ -3460,8 +3460,8 @@ static int binder_open(struct inode *nodp, struct file *filp)
>         proc = kzalloc(sizeof(*proc), GFP_KERNEL);
>         if (proc == NULL)
>                 return -ENOMEM;
> -       get_task_struct(current);
> -       proc->tsk = current;
> +       get_task_struct(current->group_leader);
> +       proc->tsk = current->group_leader;
>         INIT_LIST_HEAD(&proc->todo);
>         init_waitqueue_head(&proc->wait);
>         proc->default_priority = task_nice(current);
> --

So this patch landed in 4.13-rc2 (c4ea41ba195d), and seems to be
causing a regression for me w/ HiKey. With it, I'm getting crashes
with the bluetooth and wifi HALs.  Reverting this patch seems to
resolve the issue

I suspect some other dependency from the original patchset is missing?

thanks
-john

[toc] | [next] | [standalone]

#1695102

On Mon, Jul 24, 2017 at 2:00 PM, John Stultz <john.stultz@linaro.org> wrote:
> On Thu, Jun 29, 2017 at 12:01 PM, Todd Kjos <tkjos@android.com> wrote:
>> The binder allocator assumes that the thread that
>> called binder_open will never die for the lifetime of
>> that proc. That thread is normally the group_leader,
>> however it may not be. Use the group_leader instead
>> of current.
>>
>> Signed-off-by: Todd Kjos <tkjos@google.com>
>> ---
>>  drivers/android/binder.c | 4 ++--
>>  1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/drivers/android/binder.c b/drivers/android/binder.c
>> index 157bd3e49ff4..9393924ae8e8 100644
>> --- a/drivers/android/binder.c
>> +++ b/drivers/android/binder.c
>> @@ -3460,8 +3460,8 @@ static int binder_open(struct inode *nodp, struct file *filp)
>>         proc = kzalloc(sizeof(*proc), GFP_KERNEL);
>>         if (proc == NULL)
>>                 return -ENOMEM;
>> -       get_task_struct(current);
>> -       proc->tsk = current;
>> +       get_task_struct(current->group_leader);
>> +       proc->tsk = current->group_leader;
>>         INIT_LIST_HEAD(&proc->todo);
>>         init_waitqueue_head(&proc->wait);
>>         proc->default_priority = task_nice(current);
>> --
>
> So this patch landed in 4.13-rc2 (c4ea41ba195d), and seems to be
> causing a regression for me w/ HiKey. With it, I'm getting crashes
> with the bluetooth and wifi HALs.  Reverting this patch seems to
> resolve the issue
>
> I suspect some other dependency from the original patchset is missing?

As for the crash, the logcat details show:

12-31 16:00:36.632  2518  2584 E hw-ProcessState: Using /dev/hwbinder
failed: unable to mmap transaction memory.
12-31 16:00:36.632  2518  2566 D bt_hci  : hci_module_start_up
starting async portion
12-31 16:00:36.632  2518  2584 E
android.hardware.bluetooth@1.0::BluetoothHci: getService:
defaultServiceManager()->getTransport returns
Status(EX_TRANSACTION_FAILED): '-9 Bad file descriptor: '
12-31 16:00:36.633  2518  2584 F         :
[1231/160036:FATAL:hci_layer_android.cc(109)] Check failed: btHci !=
nullptr.
12-31 16:00:36.634  2518  2584 F libc    : Fatal signal 6 (SIGABRT),
code -6 in tid 2584 (hci_thread)
...
12-31 16:00:38.027  2009  2061 E SupplicantStaIfaceHal: Exception
while trying to register a listener for ISupplicant service:
android.os.RemoteException: HwBinder Error: (-2147483648)
12-31 16:00:38.027  2009  2061 E WifiMonitor: startMonitoring(wlan0) failed!
12-31 16:00:38.028  2009  2061 E SupplicantStaIfaceHal: Can't call
setDebugParams, ISupplicant is null
12-31 16:00:38.030  2009  2061 D WifiConfigStore: Reading from stores
completed in 2 ms.
12-31 16:00:38.034  2009  2061 D WIFI    : Registering NetworkFactory
12-31 16:00:38.035  2009  2061 D WIFI_UT : Registering NetworkFactory
12-31 16:00:38.035  2009  2065 D ConnectivityService: Got
NetworkFactory Messenger for WIFI
12-31 16:00:38.035  2009  2065 D ConnectivityService: Got
NetworkFactory Messenger for WIFI_UT
12-31 16:00:38.037  2009  2061 D WifiConfigStore: Reading from user
store completed in 2 ms.
12-31 16:00:38.055  2009  2061 D WifiConfigStore: Writing to stores
completed in 17 ms.
12-31 16:00:38.055  2009  2061 E WifiStateMachine: Failed to setup
control channel, restart supplicant
...
etc.

thanks
-john

[toc] | [prev] | [next] | [standalone]

#1695537

Hi John,

On Mon, Jul 24, 2017 at 11:07 PM, John Stultz <john.stultz@linaro.org> wrote:
>
> 12-31 16:00:36.632  2518  2584 E hw-ProcessState: Using /dev/hwbinder
> failed: unable to mmap transaction memory.

This doesn't look right. Is there anything in the kernel log?

> 12-31 16:00:36.632  2518  2566 D bt_hci  : hci_module_start_up
> starting async portion
> 12-31 16:00:36.632  2518  2584 E
> android.hardware.bluetooth@1.0::BluetoothHci: getService:
> defaultServiceManager()->getTransport returns
> Status(EX_TRANSACTION_FAILED): '-9 Bad file descriptor: '
> 12-31 16:00:36.633  2518  2584 F         :
> [1231/160036:FATAL:hci_layer_android.cc(109)] Check failed: btHci !=
> nullptr.
> 12-31 16:00:36.634  2518  2584 F libc    : Fatal signal 6 (SIGABRT),
> code -6 in tid 2584 (hci_thread)
> ...
> 12-31 16:00:38.027  2009  2061 E SupplicantStaIfaceHal: Exception
> while trying to register a listener for ISupplicant service:
> android.os.RemoteException: HwBinder Error: (-2147483648)
> 12-31 16:00:38.027  2009  2061 E WifiMonitor: startMonitoring(wlan0) failed!
> 12-31 16:00:38.028  2009  2061 E SupplicantStaIfaceHal: Can't call
> setDebugParams, ISupplicant is null
> 12-31 16:00:38.030  2009  2061 D WifiConfigStore: Reading from stores
> completed in 2 ms.
> 12-31 16:00:38.034  2009  2061 D WIFI    : Registering NetworkFactory
> 12-31 16:00:38.035  2009  2061 D WIFI_UT : Registering NetworkFactory
> 12-31 16:00:38.035  2009  2065 D ConnectivityService: Got
> NetworkFactory Messenger for WIFI
> 12-31 16:00:38.035  2009  2065 D ConnectivityService: Got
> NetworkFactory Messenger for WIFI_UT
> 12-31 16:00:38.037  2009  2061 D WifiConfigStore: Reading from user
> store completed in 2 ms.
> 12-31 16:00:38.055  2009  2061 D WifiConfigStore: Writing to stores
> completed in 17 ms.
> 12-31 16:00:38.055  2009  2061 E WifiStateMachine: Failed to setup
> control channel, restart supplicant
> ...
> etc.
>
> thanks
> -john

[toc] | [prev] | [next] | [standalone]

#1697819

Hi,

On 25 July 2017 at 14:43, Martijn Coenen <maco@google.com> wrote:
> Hi John,
>
> On Mon, Jul 24, 2017 at 11:07 PM, John Stultz <john.stultz@linaro.org> wrote:
>>
>> 12-31 16:00:36.632  2518  2584 E hw-ProcessState: Using /dev/hwbinder
>> failed: unable to mmap transaction memory.
>
> This doesn't look right. Is there anything in the kernel log?

There is a slight difference in this patch and the one that got pushed
in android-4.9[1]. I cherry-picked those changes and got BT/WiFi
working again on Hikey with mainline tracking (4.13-rc2) tree.

Regards,
Amit Pundir
[1] https://android.googlesource.com/kernel/common/+/872c26eb0776ef160447d8703779e2bce0b7230a%5E%21/#F0

>
>> 12-31 16:00:36.632  2518  2566 D bt_hci  : hci_module_start_up
>> starting async portion
>> 12-31 16:00:36.632  2518  2584 E
>> android.hardware.bluetooth@1.0::BluetoothHci: getService:
>> defaultServiceManager()->getTransport returns
>> Status(EX_TRANSACTION_FAILED): '-9 Bad file descriptor: '
>> 12-31 16:00:36.633  2518  2584 F         :
>> [1231/160036:FATAL:hci_layer_android.cc(109)] Check failed: btHci !=
>> nullptr.
>> 12-31 16:00:36.634  2518  2584 F libc    : Fatal signal 6 (SIGABRT),
>> code -6 in tid 2584 (hci_thread)
>> ...
>> 12-31 16:00:38.027  2009  2061 E SupplicantStaIfaceHal: Exception
>> while trying to register a listener for ISupplicant service:
>> android.os.RemoteException: HwBinder Error: (-2147483648)
>> 12-31 16:00:38.027  2009  2061 E WifiMonitor: startMonitoring(wlan0) failed!
>> 12-31 16:00:38.028  2009  2061 E SupplicantStaIfaceHal: Can't call
>> setDebugParams, ISupplicant is null
>> 12-31 16:00:38.030  2009  2061 D WifiConfigStore: Reading from stores
>> completed in 2 ms.
>> 12-31 16:00:38.034  2009  2061 D WIFI    : Registering NetworkFactory
>> 12-31 16:00:38.035  2009  2061 D WIFI_UT : Registering NetworkFactory
>> 12-31 16:00:38.035  2009  2065 D ConnectivityService: Got
>> NetworkFactory Messenger for WIFI
>> 12-31 16:00:38.035  2009  2065 D ConnectivityService: Got
>> NetworkFactory Messenger for WIFI_UT
>> 12-31 16:00:38.037  2009  2061 D WifiConfigStore: Reading from user
>> store completed in 2 ms.
>> 12-31 16:00:38.055  2009  2061 D WifiConfigStore: Writing to stores
>> completed in 17 ms.
>> 12-31 16:00:38.055  2009  2061 E WifiStateMachine: Failed to setup
>> control channel, restart supplicant
>> ...
>> etc.
>>
>> thanks
>> -john

[toc] | [prev] | [next] | [standalone]

#1697985

On Thu, Jul 27, 2017 at 02:38:30PM +0530, Amit Pundir wrote:
> Hi,
> 
> On 25 July 2017 at 14:43, Martijn Coenen <maco@google.com> wrote:
> > Hi John,
> >
> > On Mon, Jul 24, 2017 at 11:07 PM, John Stultz <john.stultz@linaro.org> wrote:
> >>
> >> 12-31 16:00:36.632  2518  2584 E hw-ProcessState: Using /dev/hwbinder
> >> failed: unable to mmap transaction memory.
> >
> > This doesn't look right. Is there anything in the kernel log?
> 
> There is a slight difference in this patch and the one that got pushed
> in android-4.9[1]. I cherry-picked those changes and got BT/WiFi
> working again on Hikey with mainline tracking (4.13-rc2) tree.
> 
> Regards,
> Amit Pundir
> [1] https://android.googlesource.com/kernel/common/+/872c26eb0776ef160447d8703779e2bce0b7230a%5E%21/#F0

And what is that difference?

[toc] | [prev] | [next] | [standalone]

#1697996

Looks like this assignment somehow went missing in the upstream version:

proc->vma_vm_mm = current->group_leader->mm;

which probably causes us to bail out here in
binder_update_page_range() because proc->vma_vm_mm is NULL:

if (vma && mm != proc->vma_vm_mm) {
                        pr_err("%d: vma mm and task mm mismatch\n",
                                proc->pid);


I'll prep a patch to fix this ASAP.

On Thu, Jul 27, 2017 at 3:23 PM, Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
> On Thu, Jul 27, 2017 at 02:38:30PM +0530, Amit Pundir wrote:
>> Hi,
>>
>> On 25 July 2017 at 14:43, Martijn Coenen <maco@google.com> wrote:
>> > Hi John,
>> >
>> > On Mon, Jul 24, 2017 at 11:07 PM, John Stultz <john.stultz@linaro.org> wrote:
>> >>
>> >> 12-31 16:00:36.632  2518  2584 E hw-ProcessState: Using /dev/hwbinder
>> >> failed: unable to mmap transaction memory.
>> >
>> > This doesn't look right. Is there anything in the kernel log?
>>
>> There is a slight difference in this patch and the one that got pushed
>> in android-4.9[1]. I cherry-picked those changes and got BT/WiFi
>> working again on Hikey with mainline tracking (4.13-rc2) tree.
>>
>> Regards,
>> Amit Pundir
>> [1] https://android.googlesource.com/kernel/common/+/872c26eb0776ef160447d8703779e2bce0b7230a%5E%21/#F0
>
> And what is that difference?

[toc] | [prev] | [next] | [standalone]

#1698000

On 27 July 2017 at 18:53, Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote:
> On Thu, Jul 27, 2017 at 02:38:30PM +0530, Amit Pundir wrote:
>> Hi,
>>
>> On 25 July 2017 at 14:43, Martijn Coenen <maco@google.com> wrote:
>> > Hi John,
>> >
>> > On Mon, Jul 24, 2017 at 11:07 PM, John Stultz <john.stultz@linaro.org> wrote:
>> >>
>> >> 12-31 16:00:36.632  2518  2584 E hw-ProcessState: Using /dev/hwbinder
>> >> failed: unable to mmap transaction memory.
>> >
>> > This doesn't look right. Is there anything in the kernel log?
>>
>> There is a slight difference in this patch and the one that got pushed
>> in android-4.9[1]. I cherry-picked those changes and got BT/WiFi
>> working again on Hikey with mainline tracking (4.13-rc2) tree.
>>
>> Regards,
>> Amit Pundir
>> [1] https://android.googlesource.com/kernel/common/+/872c26eb0776ef160447d8703779e2bce0b7230a%5E%21/#F0
>
> And what is that difference?

These couple of lines of change:

diff --git a/drivers/android/binder.c b/drivers/android/binder.c
index f7665c31feca..d7291a5a13e1 100644
--- a/drivers/android/binder.c
+++ b/drivers/android/binder.c
@@ -3362,7 +3362,7 @@ static int binder_mmap(struct file *filp, struct
vm_area_struct *vma)
        const char *failure_string;
        struct binder_buffer *buffer;

-       if (proc->tsk != current)
+       if (proc->tsk != current->group_leader)
                return -EINVAL;

        if ((vma->vm_end - vma->vm_start) > SZ_4M)
@@ -3466,6 +3466,7 @@ static int binder_open(struct inode *nodp,
struct file *filp)
                return -ENOMEM;
        get_task_struct(current->group_leader);
        proc->tsk = current->group_leader;
+       proc->vma_vm_mm = current->group_leader->mm;
        INIT_LIST_HEAD(&proc->todo);
        init_waitqueue_head(&proc->wait);
        proc->default_priority = task_nice(current);

John reported hw_binder mmap failure in his logcat. So I'm assuming
that the check in binder_mmap is where the binder failed and returned.

Regards,
Amit Pundir

[toc] | [prev] | [next] | [standalone]

#1698717

I took the time to look at what else is different between common and
upstream, and just sent 3 patches to LKML to reconcile. One of the 3
fixes this particular problem.

On Thu, Jul 27, 2017 at 3:42 PM, Amit Pundir <amit.pundir@linaro.org> wrote:
> On 27 July 2017 at 18:53, Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote:
>> On Thu, Jul 27, 2017 at 02:38:30PM +0530, Amit Pundir wrote:
>>> Hi,
>>>
>>> On 25 July 2017 at 14:43, Martijn Coenen <maco@google.com> wrote:
>>> > Hi John,
>>> >
>>> > On Mon, Jul 24, 2017 at 11:07 PM, John Stultz <john.stultz@linaro.org> wrote:
>>> >>
>>> >> 12-31 16:00:36.632  2518  2584 E hw-ProcessState: Using /dev/hwbinder
>>> >> failed: unable to mmap transaction memory.
>>> >
>>> > This doesn't look right. Is there anything in the kernel log?
>>>
>>> There is a slight difference in this patch and the one that got pushed
>>> in android-4.9[1]. I cherry-picked those changes and got BT/WiFi
>>> working again on Hikey with mainline tracking (4.13-rc2) tree.
>>>
>>> Regards,
>>> Amit Pundir
>>> [1] https://android.googlesource.com/kernel/common/+/872c26eb0776ef160447d8703779e2bce0b7230a%5E%21/#F0
>>
>> And what is that difference?
>
> These couple of lines of change:
>
> diff --git a/drivers/android/binder.c b/drivers/android/binder.c
> index f7665c31feca..d7291a5a13e1 100644
> --- a/drivers/android/binder.c
> +++ b/drivers/android/binder.c
> @@ -3362,7 +3362,7 @@ static int binder_mmap(struct file *filp, struct
> vm_area_struct *vma)
>         const char *failure_string;
>         struct binder_buffer *buffer;
>
> -       if (proc->tsk != current)
> +       if (proc->tsk != current->group_leader)
>                 return -EINVAL;
>
>         if ((vma->vm_end - vma->vm_start) > SZ_4M)
> @@ -3466,6 +3466,7 @@ static int binder_open(struct inode *nodp,
> struct file *filp)
>                 return -ENOMEM;
>         get_task_struct(current->group_leader);
>         proc->tsk = current->group_leader;
> +       proc->vma_vm_mm = current->group_leader->mm;
>         INIT_LIST_HEAD(&proc->todo);
>         init_waitqueue_head(&proc->wait);
>         proc->default_priority = task_nice(current);
>
> John reported hw_binder mmap failure in his logcat. So I'm assuming
> that the check in binder_mmap is where the binder failed and returned.
>
> Regards,
> Amit Pundir

[toc] | [prev] | [next] | [standalone]

#1695114

On Mon, Jul 24, 2017 at 02:00:45PM -0700, John Stultz wrote:
> On Thu, Jun 29, 2017 at 12:01 PM, Todd Kjos <tkjos@android.com> wrote:
> > The binder allocator assumes that the thread that
> > called binder_open will never die for the lifetime of
> > that proc. That thread is normally the group_leader,
> > however it may not be. Use the group_leader instead
> > of current.
> >
> > Signed-off-by: Todd Kjos <tkjos@google.com>
> > ---
> >  drivers/android/binder.c | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/drivers/android/binder.c b/drivers/android/binder.c
> > index 157bd3e49ff4..9393924ae8e8 100644
> > --- a/drivers/android/binder.c
> > +++ b/drivers/android/binder.c
> > @@ -3460,8 +3460,8 @@ static int binder_open(struct inode *nodp, struct file *filp)
> >         proc = kzalloc(sizeof(*proc), GFP_KERNEL);
> >         if (proc == NULL)
> >                 return -ENOMEM;
> > -       get_task_struct(current);
> > -       proc->tsk = current;
> > +       get_task_struct(current->group_leader);
> > +       proc->tsk = current->group_leader;
> >         INIT_LIST_HEAD(&proc->todo);
> >         init_waitqueue_head(&proc->wait);
> >         proc->default_priority = task_nice(current);
> > --
> 
> So this patch landed in 4.13-rc2 (c4ea41ba195d), and seems to be
> causing a regression for me w/ HiKey. With it, I'm getting crashes
> with the bluetooth and wifi HALs.  Reverting this patch seems to
> resolve the issue
> 
> I suspect some other dependency from the original patchset is missing?

If you use linux-next (or all of the patches in this series), does the
problem go away?

thanks,

greg k-h

[toc] | [prev] | [next] | [standalone]

#1695166

On Mon, Jul 24, 2017 at 2:23 PM, Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
> On Mon, Jul 24, 2017 at 02:00:45PM -0700, John Stultz wrote:
>> On Thu, Jun 29, 2017 at 12:01 PM, Todd Kjos <tkjos@android.com> wrote:
>> > The binder allocator assumes that the thread that
>> > called binder_open will never die for the lifetime of
>> > that proc. That thread is normally the group_leader,
>> > however it may not be. Use the group_leader instead
>> > of current.
>> >
>> > Signed-off-by: Todd Kjos <tkjos@google.com>
>> > ---
>> >  drivers/android/binder.c | 4 ++--
>> >  1 file changed, 2 insertions(+), 2 deletions(-)
>> >
>> > diff --git a/drivers/android/binder.c b/drivers/android/binder.c
>> > index 157bd3e49ff4..9393924ae8e8 100644
>> > --- a/drivers/android/binder.c
>> > +++ b/drivers/android/binder.c
>> > @@ -3460,8 +3460,8 @@ static int binder_open(struct inode *nodp, struct file *filp)
>> >         proc = kzalloc(sizeof(*proc), GFP_KERNEL);
>> >         if (proc == NULL)
>> >                 return -ENOMEM;
>> > -       get_task_struct(current);
>> > -       proc->tsk = current;
>> > +       get_task_struct(current->group_leader);
>> > +       proc->tsk = current->group_leader;
>> >         INIT_LIST_HEAD(&proc->todo);
>> >         init_waitqueue_head(&proc->wait);
>> >         proc->default_priority = task_nice(current);
>> > --
>>
>> So this patch landed in 4.13-rc2 (c4ea41ba195d), and seems to be
>> causing a regression for me w/ HiKey. With it, I'm getting crashes
>> with the bluetooth and wifi HALs.  Reverting this patch seems to
>> resolve the issue
>>
>> I suspect some other dependency from the original patchset is missing?
>
> If you use linux-next (or all of the patches in this series), does the
> problem go away?

I had tested awhile back the entire set from Todd, and didn't see this issue.

I'll try to find some time to give -next a spin, but it might not be today.

thanks
-john

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web