Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.bugs.dist > #1148058 > unrolled thread

Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat

Back to article view | Back to linux.debian.bugs.dist

Contents

  Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat Otto Kekäläinen <otto@kekalainen.net> - 2023-05-27 04:00 +0200
    Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat Helge Deller <deller@gmx.de> - 2023-05-27 09:40 +0200
      Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat Salvatore Bonaccorso <carnil@debian.org> - 2023-05-27 12:00 +0200
        Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat Salvatore Bonaccorso <carnil@debian.org> - 2023-05-27 12:00 +0200
          Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat Salvatore Bonaccorso <carnil@debian.org> - 2023-05-27 20:40 +0200
            Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat Mathias Gibbens <gibmat@debian.org> - 2023-05-29 02:20 +0200
              Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat Paul Gevers <elbrus@debian.org> - 2023-05-29 07:40 +0200
                Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat Pierre-Elliott Bécue <peb@debian.org> - 2023-05-29 14:40 +0200
                  Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat Mathias Gibbens <gibmat@debian.org> - 2023-05-29 16:10 +0200
    Bug#1036818: [pkg-lxc-devel] Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat Mathias Gibbens <gibmat@debian.org> - 2023-06-01 13:40 +0200
      Bug#1036818: [pkg-lxc-devel] Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat Pierre-Elliott Bécue <peb@debian.org> - 2023-06-01 19:10 +0200
        Bug#1036818: [pkg-lxc-devel] Bug#1036818: Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat Mathias Gibbens <gibmat@debian.org> - 2023-06-06 04:10 +0200
          Bug#1036818: [pkg-lxc-devel] Bug#1036818: Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat Paul Gevers <elbrus@debian.org> - 2023-06-06 20:30 +0200
          Bug#1036818: [pkg-lxc-devel] Bug#1036818: Bug#1036818: Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat Pierre-Elliott Bécue <peb@debian.org> - 2023-06-07 11:10 +0200

#1148058 — Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat

Package: linux
Version: 6.1.0

Hi!

I noticed that the autopkgtests on Debian on armhf and armel that run
the mariadb-test-run have been failing since the Linux kernel was
updated from 5.10.0 to 6.1.0. The failure is due to a Perl module not
being able to get from /proc/cpu the number of processors:

Last passing one:
2023-04-28 https://ci.debian.net/data/autopkgtest/unstable/armel/m/mariadb/33218554/log.gz

kernel: Linux 5.10.0-21-arm64 #1 SMP Debian 5.10.162-1 (2023-01-21)
perl 5.36.0-7
libdbi-perl armel 1.643-4
libconfig-inifiles-perl all 3.000003-2

First failing one:
2023-05-05 https://ci.debian.net/data/autopkgtest/unstable/armel/m/mariadb/33379866/log.gz

kernel: Linux 6.1.0-7-arm64 #1 SMP Debian 6.1.20-2 (2023-04-08)
perl 5.36.0-7
libdbi-perl armhf 1.643-4
libconfig-inifiles-perl all 3.000003-2

Error:

starting mysql-test-tun.pl...
Logging: ./mysql-test-run.pl  --force --testcase-timeout=120
--suite-timeout=540 --retry=3 --
...
Collecting tests...
Installing system database...
Can't use an undefined value as an ARRAY reference at
lib/My/SysInfo.pm line 166.

This line 166 in src:mariadb/mysql-test/lib/My/SysInfo.pm has:

# Return the number of cpus found
sub num_cpus {
  my ($self)= @_;
  return int(@{$self->{cpus}}) or
    confess "INTERNAL ERROR: No cpus in list";
}

The cpus is initialized to be an empty list on the line 119:

118   my $self= bless {
119                    cpus => (),
120                   }, $class;

Then it tries to fill it from /proc/cpuinfo (line 67) and `kstat`
(line 95). If nothing worked it'll create one dummy cpu:

145   push(@{$self->{cpus}},
146      {
147       bogomips => DEFAULT_BOGO_MIPS,
148       model_name => "unknown",
149      });

See more discussion from MariaDB devs:
https://lists.launchpad.net/maria-developers/msg13356.html

Thus the primary suspect here is the kernel upgrade. Perl versions
have not changed. This only happens on armel/armhf, other archs are
fine.

Reproducing the environment on ci.debian.net / ci-worker-arm??-?? to
study how /proc/cpu etc looks like, so filing this against the Linux
package is somewhat of a guess, but at least we get a Bug# to
reference for further research.

[toc] | [next] | [standalone]

#1148067

Just wondering / guessing:

Are the ARM machines on ci.debian.net (ci-worker-arm??-??)
physical machines, or are they running on qemu-user VMs?

If they run qemu, this bug report
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1024653
might be similiar.

If so, then qemu probably needs fixing of the output of /proc/cpuinfo
for ARM, e.g. like this:
https://gitlab.com/qemu-project/qemu/-/commit/e0174afeea23e56765db56fbbe465ed1fcbdd07a

Helge

[toc] | [prev] | [next] | [standalone]

#1148073

Hi Helge, hi Otto,

On Sat, May 27, 2023 at 09:26:06AM +0200, Helge Deller wrote:
> Just wondering / guessing:
> 
> Are the ARM machines on ci.debian.net (ci-worker-arm??-??)
> physical machines, or are they running on qemu-user VMs?
> 
> If they run qemu, this bug report
>  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1024653
> might be similiar.
> 
> If so, then qemu probably needs fixing of the output of /proc/cpuinfo
> for ARM, e.g. like this:
> https://gitlab.com/qemu-project/qemu/-/commit/e0174afeea23e56765db56fbbe465ed1fcbdd07a

The suspect is that /proc/cpuinfo is empty or not readable, and this
seems to be a problem with lxcfs after mentioning the issue today to
Paul and Jochen.

Jochen, understanding you correctly there is already an upstream fix
which is supposed to addres the issue?

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#1148074

Hi,

On Sat, May 27, 2023 at 11:50:06AM +0200, Salvatore Bonaccorso wrote:
> Hi Helge, hi Otto,
> 
> On Sat, May 27, 2023 at 09:26:06AM +0200, Helge Deller wrote:
> > Just wondering / guessing:
> > 
> > Are the ARM machines on ci.debian.net (ci-worker-arm??-??)
> > physical machines, or are they running on qemu-user VMs?
> > 
> > If they run qemu, this bug report
> >  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1024653
> > might be similiar.
> > 
> > If so, then qemu probably needs fixing of the output of /proc/cpuinfo
> > for ARM, e.g. like this:
> > https://gitlab.com/qemu-project/qemu/-/commit/e0174afeea23e56765db56fbbe465ed1fcbdd07a
> 
> The suspect is that /proc/cpuinfo is empty or not readable, and this
> seems to be a problem with lxcfs after mentioning the issue today to
> Paul and Jochen.
> 
> Jochen, understanding you correctly there is already an upstream fix
> which is supposed to addres the issue?

The upstream issue should be: https://github.com/lxc/lxcfs/issues/553

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#1148100

Control: reassign -1 src:lxcfs 5.0.3-1
Control: forwarded -1 https://github.com/lxc/lxcfs/issues/553
Control: affects -1 src:mariadb

Hi,

On Sat, May 27, 2023 at 11:51:26AM +0200, Salvatore Bonaccorso wrote:
> Hi,
> 
> On Sat, May 27, 2023 at 11:50:06AM +0200, Salvatore Bonaccorso wrote:
> > Hi Helge, hi Otto,
> > 
> > On Sat, May 27, 2023 at 09:26:06AM +0200, Helge Deller wrote:
> > > Just wondering / guessing:
> > > 
> > > Are the ARM machines on ci.debian.net (ci-worker-arm??-??)
> > > physical machines, or are they running on qemu-user VMs?
> > > 
> > > If they run qemu, this bug report
> > >  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1024653
> > > might be similiar.
> > > 
> > > If so, then qemu probably needs fixing of the output of /proc/cpuinfo
> > > for ARM, e.g. like this:
> > > https://gitlab.com/qemu-project/qemu/-/commit/e0174afeea23e56765db56fbbe465ed1fcbdd07a
> > 
> > The suspect is that /proc/cpuinfo is empty or not readable, and this
> > seems to be a problem with lxcfs after mentioning the issue today to
> > Paul and Jochen.
> > 
> > Jochen, understanding you correctly there is already an upstream fix
> > which is supposed to addres the issue?
> 
> The upstream issue should be: https://github.com/lxc/lxcfs/issues/553

Now reassigning to the lxcfs package. lxcfs maintainers, can you
please adjust the severity as needed. It affects at least mariadb's
autopkgtests.

Otto, spaking of the issue, I guess Paul will agree, that you can
ignore it for now for the mariadb upload to unstable.

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#1148240

[Multipart message — attachments visible in raw view] — view raw

  What version of lxcfs is currently installed on the ci.debian.net
machines? I'm guessing from the kernel version they've been upgraded to
bookworm, so lxcfs should be at 5.0.3-1, but I'd like to clarify that.

  lxcfs 5.0.3-1 does indeed include the referenced fix for upstream
issue #553, and has been in testing since 2023-01-22. If the CI
machines have that version installed, then I'd lean towards a related
but distinct issue than the one identified at the moment. If not, let's
upgrade lxcfs and see if that fixes the issue.

Mathias

[toc] | [prev] | [next] | [standalone]

#1148251

[Multipart message — attachments visible in raw view] — view raw

Hi,

[reducing the people in CC, hope I didn't drop too many and those still 
there are interested]

On Mon, 29 May 2023 00:14:10 +0000 Mathias Gibbens <gibmat@debian.org> 
wrote:
>   What version of lxcfs is currently installed on the ci.debian.net
> machines? I'm guessing from the kernel version they've been upgraded to
> bookworm, so lxcfs should be at 5.0.3-1, but I'd like to clarify that.

I just ran $(apt-cache show lxcfs | grep Version) on all the worker 
hosts and indeed they all run 5.0.3-1.

>   lxcfs 5.0.3-1 does indeed include the referenced fix for upstream
> issue #553, and has been in testing since 2023-01-22. If the CI
> machines have that version installed, then I'd lean towards a related
> but distinct issue than the one identified at the moment.

Is there more information I can get for you from one of the effected 
architectures?

Paul
PS: I missed former messages due to a minor mistake in my address, but 
I'm now subscribed.

[toc] | [prev] | [next] | [standalone]

#1148285

De : Paul Gevers <elbrus@debian.org>
À : 1036818@bugs.debian.org
Cc : Evgeni Golov <evgeni@debian.org>; Pierre-Elliott Bécue <peb@debian.org>; Jochen Sprickerhof <jspricke@debian.org>
Date : 29 mai 2023 07:41:30
Objet : Re: Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat

> Hi,
> 
> [reducing the people in CC, hope I didn't drop too many and those still there are interested]
> 
> On Mon, 29 May 2023 00:14:10 +0000 Mathias Gibbens <gibmat@debian.org> wrote:
>>   What version of lxcfs is currently installed on the ci.debian.net
>> machines? I'm guessing from the kernel version they've been upgraded to
>> bookworm, so lxcfs should be at 5.0.3-1, but I'd like to clarify that.
> 
> I just ran $(apt-cache show lxcfs | grep Version) on all the worker hosts and indeed they all run 5.0.3-1.
> 
>>   lxcfs 5.0.3-1 does indeed include the referenced fix for upstream
>> issue #553, and has been in testing since 2023-01-22. If the CI
>> machines have that version installed, then I'd lean towards a related
>> but distinct issue than the one identified at the moment.
> 
> Is there more information I can get for you from one of the effected architectures?
> 
> Paul
> PS: I missed former messages due to a minor mistake in my address, but I'm now subscribed.
I think you should keep gibmat in the Cc list. :)

-- 
Pierre-Elliott Bécue

[toc] | [prev] | [next] | [standalone]

#1148298

[Multipart message — attachments visible in raw view] — view raw

Control: notforwarded -1

On Mon, 2023-05-29 at 14:26 +0200, Pierre-Elliott Bécue wrote:
> On Mon, 2023-05-29 at 07:37 +0200, Paul Gevers wrote:
> > Hi,
> > 
> > [reducing the people in CC, hope I didn't drop too many and those
> > still there are interested]
> > 
> > On Mon, 29 May 2023 00:14:10 +0000 Mathias Gibbens
> > <gibmat@debian.org> wrote:
> > >   What version of lxcfs is currently installed on the
> > > ci.debian.net machines? I'm guessing from the kernel version
> > > they've been upgraded to bookworm, so lxcfs should be at 5.0.3-1,
> > > but I'd like to clarify that.
> > 
> > I just ran $(apt-cache show lxcfs | grep Version) on all the worker
> > hosts and indeed they all run 5.0.3-1.

  Thanks for confirming that -- for the time being I've unforwarded
this bug from upstream issue #553 as it looks like this is a new
problem.

> > 
> > >   lxcfs 5.0.3-1 does indeed include the referenced fix for
> > > upstream issue #553, and has been in testing since 2023-01-22. If
> > > the CI machines have that version installed, then I'd lean
> > > towards a related but distinct issue than the one identified at
> > > the moment.
> > 
> > Is there more information I can get for you from one of the
> > effected architectures?

  Can you grab /proc/cpuinfo from a physical CI host as well as from
within a container for the armel, armhf, and arm64 architectures? That
will let us see what difference(s) lxcfs is presenting and might give a
clue for the root issue. Since only the 32bit arm architectures appear
to be affected, I'm also curious to see what the arm64 cpuinfo is.

> > 
> > Paul
> > PS: I missed former messages due to a minor mistake in my address,
> > but I'm now subscribed.
> I think you should keep gibmat in the Cc list. :)

  +1 :)

Mathias

[toc] | [prev] | [next] | [standalone]

#1148520 — Bug#1036818: [pkg-lxc-devel] Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat

[Multipart message — attachments visible in raw view] — view raw

On Mon, 2023-05-29 at 20:03 +0200, Paul Gevers wrote:
> We have three layers here. The bare metal is arm64. It hosts several 
> arm* QEMU VM. Inside the VM we run the lxc. I put the output of all 
> three at the bottom. *Maybe* it's relevant that the bare metal still 
> runs bullseye while the VM's run bookworm. I'll also share the
> content for an arm64 host (which is a VM where I don't have access to
> the bare metal.)

[snip]

> # generating the container and showing inside
> debian@ci-worker-armhf-01:~$ sudo lxc-copy -N elbrus 
> autopkgtest-unstable-armhf && sudo lxc-start elbrus && sudo lxc-
> attach elbrus
> root@elbrus:/# cat /proc/cpuinfo
> root@elbrus:/# ls -al /proc/cpuinfo
> -r--r--r-- 1 root root 3878 May 29 17:48 /proc/cpuinfo

  Yeah, that's definitely not right! I don't currently have an
armel/armhf system to test on, but did try using the abel.d.o
porterbox. lxcfs requires elevated permissions to start, which I don't
have on that box.

  Some other things we can look at -- are there any errors/warnings in
the lxcfs service journal and/or the system's dmesg that is running
lxcfs? It might also be useful to start lxcfs with debugging (`-d`) if
there's nothing being logged about populating /proc/cpuinfo.

  I should have time this weekend when I can spin up a qemu vm to test
in, if we're not able to get a root cause identified before then.

Mathias

[toc] | [prev] | [next] | [standalone]

#1148544 — Bug#1036818: [pkg-lxc-devel] Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat

De : Mathias Gibbens <gibmat@debian.org>
À : Paul Gevers <elbrus@debian.org>; 1036818@bugs.debian.org
Cc : Pierre-Elliott Bécue <peb@debian.org>; Evgeni Golov <evgeni@debian.org>; Jochen Sprickerhof <jspricke@debian.org>
Date : 1 juin 2023 13:33:33
Objet : Re: [pkg-lxc-devel] Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat

> On Mon, 2023-05-29 at 20:03 +0200, Paul Gevers wrote:
>> We have three layers here. The bare metal is arm64. It hosts several
>> arm* QEMU VM. Inside the VM we run the lxc. I put the output of all
>> three at the bottom. *Maybe* it's relevant that the bare metal still
>> runs bullseye while the VM's run bookworm. I'll also share the
>> content for an arm64 host (which is a VM where I don't have access to
>> the bare metal.)
> 
> [snip]
> 
>> # generating the container and showing inside
>> debian@ci-worker-armhf-01:~$ sudo lxc-copy -N elbrus
>> autopkgtest-unstable-armhf && sudo lxc-start elbrus && sudo lxc-
>> attach elbrus
>> root@elbrus:/# cat /proc/cpuinfo
>> root@elbrus:/# ls -al /proc/cpuinfo
>> -r--r--r-- 1 root root 3878 May 29 17:48 /proc/cpuinfo
> 
>   Yeah, that's definitely not right! I don't currently have an
> armel/armhf system to test on, but did try using the abel.d.o
> porterbox. lxcfs requires elevated permissions to start, which I don't
> have on that box.
> 
>   Some other things we can look at -- are there any errors/warnings in
> the lxcfs service journal and/or the system's dmesg that is running
> lxcfs? It might also be useful to start lxcfs with debugging (`-d`) if
> there's nothing being logged about populating /proc/cpuinfo.
> 
>   I should have time this weekend when I can spin up a qemu vm to test
> in, if we're not able to get a root cause identified before then.
> 
> Mathias
I can probably grant you privileges on abel as soon as I get an ack from my fellow DSA mates.

-- 
Pierre-Elliott Bécue

[toc] | [prev] | [next] | [standalone]

#1148826 — Bug#1036818: [pkg-lxc-devel] Bug#1036818: Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat

[Multipart message — attachments visible in raw view] — view raw

On Thu, 2023-06-01 at 18:58 +0200, Pierre-Elliott Bécue wrote:
> >   I should have time this weekend when I can spin up a qemu vm to
> > test in, if we're not able to get a root cause identified before
> > then.

  I did try to reproduce the issue over the weekend with qemu. Using
qemu-user-static and systemd-nspawn was insufficient due to lxcfs
needing proper access to the fuse kernel api. After trying and failing
to bootstrap an armhf instance by hand, I grabbed a raspberry pi 2
bookworm image from raspi.d.n, and got it running under qemu-system-
arm. Within that environment, lxcfs appeared to work correctly
(/var/lib/lxcfs/proc/cpuinfo was correct, no obvious errors or warnings
noticed). I didn't spin up a full lxc container instance within that
environment.

> I can probably grant you privileges on abel as soon as I get an ack
> from my fellow DSA mates.

  If it's possible to get temporary permissions on abel, that would be
useful.

  One other thing to double check on ci-worker-armhf-01 would be the
contents of /var/lib/lxcfs/proc/cpuinfo, so we can see what lxcfs is
doing from the host side.

Mathias

[toc] | [prev] | [next] | [standalone]

#1148884 — Bug#1036818: [pkg-lxc-devel] Bug#1036818: Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat

[Multipart message — attachments visible in raw view] — view raw

Hi,

On 01-06-2023 13:28, Mathias Gibbens wrote:
>    Some other things we can look at -- are there any errors/warnings in
> the lxcfs service journal and/or the system's dmesg that is running
> lxcfs? It might also be useful to start lxcfs with debugging (`-d`) if
> there's nothing being logged about populating /proc/cpuinfo.

This maybe?

Jun 06 17:35:55 ci-worker-armhf-01 lxcfs[686]: ../src/proc_cpuview.c:
1055: proc_cpuinfo_read: Write to cache was truncated

On 06-06-2023 04:06, Mathias Gibbens wrote:
> One other thing to double check on ci-worker-armhf-01 would be the 
> contents of /var/lib/lxcfs/proc/cpuinfo, so we can see what lxcfs is
>  doing from the host side.

debian@ci-worker-armhf-01:~$ cat /var/lib/lxcfs/proc/cpuinfo
processor       : 0
BogoMIPS        : 50.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics
fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x3
CPU part        : 0xd0c
CPU revision    : 1

processor       : 1
BogoMIPS        : 50.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics
fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x3
CPU part        : 0xd0c
CPU revision    : 1

[ etc ]

Paul

[toc] | [prev] | [next] | [standalone]

#1148938 — Bug#1036818: [pkg-lxc-devel] Bug#1036818: Bug#1036818: Bug#1036818: linux on armel/armhf: Perl library unable to access get CPU info from /proc/cpu or kstat

Mathias Gibbens <gibmat@debian.org> wrote on 06/06/2023 at 04:06:14+0200:

> [[PGP Signed Part:No public key for 29EEE2D6ECF442F9 created at 2023-06-06T04:06:14+0200 using RSA]]
> On Thu, 2023-06-01 at 18:58 +0200, Pierre-Elliott Bécue wrote:
>> >   I should have time this weekend when I can spin up a qemu vm to
>> > test in, if we're not able to get a root cause identified before
>> > then.
>
>   I did try to reproduce the issue over the weekend with qemu. Using
> qemu-user-static and systemd-nspawn was insufficient due to lxcfs
> needing proper access to the fuse kernel api. After trying and failing
> to bootstrap an armhf instance by hand, I grabbed a raspberry pi 2
> bookworm image from raspi.d.n, and got it running under qemu-system-
> arm. Within that environment, lxcfs appeared to work correctly
> (/var/lib/lxcfs/proc/cpuinfo was correct, no obvious errors or warnings
> noticed). I didn't spin up a full lxc container instance within that
> environment.
>
>> I can probably grant you privileges on abel as soon as I get an ack
>> from my fellow DSA mates.
>
>   If it's possible to get temporary permissions on abel, that would be
> useful.
>
>   One other thing to double check on ci-worker-armhf-01 would be the
> contents of /var/lib/lxcfs/proc/cpuinfo, so we can see what lxcfs is
> doing from the host side.

Unfortunately my teammates have some concerns and I don't have the time
for the debate now.

What I can do is run any required tests myself.

Could you tell me what you wanted to try so that I can try it out for
you and report back with the results?

-- 
PEB

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.bugs.dist

csiph-web