Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > muc.lists.freebsd.hardware > #1975 > unrolled thread

Re: watchdog timer programming

Back to article view | Back to muc.lists.freebsd.hardware

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: watchdog timer programming Stephane Rochoy <stephane.rochoy@stormshield.eu> - 2024-09-24 08:46 +0200
    Re: watchdog timer programming Stephane Rochoy <stephane.rochoy@stormshield.eu> - 2024-09-25 09:13 +0200
      Re: watchdog timer programming Stephane Rochoy <stephane.rochoy@stormshield.eu> - 2024-09-27 08:09 +0200
      Re: watchdog timer programming Stephane Rochoy <stephane.rochoy@stormshield.eu> - 2024-09-27 15:55 +0200
        Re: watchdog timer programming Stephane Rochoy <stephane.rochoy@stormshield.eu> - 2024-09-30 09:18 +0200
        Re: watchdog timer programming Stephane Rochoy <stephane.rochoy@stormshield.eu> - 2024-10-01 08:07 +0200
    Re: watchdog timer programming (progress) Stephane Rochoy <stephane.rochoy@stormshield.eu> - 2024-10-02 11:13 +0200

#1975 — Re: watchdog timer programming

mike tancsa <mike@sentex.net> writes:

> I am trying to get
>
> superio0: <ITE IT8613 SuperIO (revision 0x0c)> at port 0x2e-0x2f 
> on isa0
> itwd0: <Watchdog Timer on ITE SuperIO> at WDT ldn 0x07 on 
> superio0
> itwd0: Configured for system reset on timeout
>
> working on FreeBSD.  The driver seems to load / attach fine, but 
> it does
> not want to reboot the box.  Adding a #define DIAGNOSTICS 1
>
> shows
>
> itwd0: <Watchdog Timer on ITE SuperIO> at WDT ldn 0x07 on 
> superio0
> itwd0: Configured for system reset on timeout
> itwd0: setting timeout to 4
> itwd0: setting timeout to 4
> itwd0: setting timeout to 4
> itwd0: setting timeout to 4
> itwd0: setting timeout to 4
> itwd0: setting timeout to 4
>
> when I do
>
>  watchdogd -t 3
>  killall -9 watchdogd
>
> but never a reboot :(
>
> Any idea how to get this hardware working ?

Do you know if, at least, the pre-timeout is working?

Glancing at the code, this chip can be configured to use a 
specific IRQ (its
default one is 0x40) and have an "NMI mode" (relying on an
explicitly configured IRQ). Maybe playing with the dev.itwd.irq 
and .nwi
tunables could get us some hints.

Also this code from itwd's wd_func puzzle me a bit:

    superio_write(dev, 0x73, val);
    if (superio_read(dev, 0x73) != val)
      superio_write(dev, 0x73, val);

It let me think that we either are writing twice to the same I/O
port by mistake or the corresponding chip's register enforce a
specific protocol to modify it. If the later, the code don't seems
to be able to detect a "protocol error".

By the way, do you have the datasheet of the ITE chip? (I know I
must be very optimistic to ask such a question ;))

Regards,
-- 
Stéphane Rochoy
O: Stormshield


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [next] | [standalone]

#1976

mike tancsa <mike@sentex.net> writes:

> On 9/24/2024 2:46 AM, Stephane Rochoy wrote:
>>
>> mike tancsa <mike@sentex.net> writes:
>>
>>> I am trying to get
>>>
>>> superio0: <ITE IT8613 SuperIO (revision 0x0c)> at port 
>>> 0x2e-0x2f on isa0
>>> itwd0: <Watchdog Timer on ITE SuperIO> at WDT ldn 0x07 on 
>>> superio0
>>> itwd0: Configured for system reset on timeout
>>>
>>> working on FreeBSD.  The driver seems to load / attach fine, 
>>> but it does
>>> not want to reboot the box.  Adding a #define DIAGNOSTICS 1
>>>
>>> shows
>>>
>>> itwd0: <Watchdog Timer on ITE SuperIO> at WDT ldn 0x07 on 
>>> superio0
>>> itwd0: Configured for system reset on timeout
>>> itwd0: setting timeout to 4
>>> itwd0: setting timeout to 4
>>> itwd0: setting timeout to 4
>>> itwd0: setting timeout to 4
>>> itwd0: setting timeout to 4
>>> itwd0: setting timeout to 4
>>>
>>> when I do
>>>
>>>  watchdogd -t 3
>>>  killall -9 watchdogd
>>>
>>> but never a reboot :(
>>>
>>> Any idea how to get this hardware working ?
>>
>> Do you know if, at least, the pre-timeout is working?
>
>     How do I check that ?

Using --pretimeout and --pretimeout-action. See watchdogd(8) for
more details.

>> By the way, do you have the datasheet of the ITE chip? (I know 
>> I
>> must be very optimistic to ask such a question ;))
>
> I dont have the datasheet sadly. However, they do seem to be at
> …
> and are publicly available according to the
> Linux driver.
>
> I did load up ubuntu on the box, and the watchdog does seem to 
> work
> correctly.   sysctl doesnt seem to allow for setting those 
> values

The .nmi and .irq are FreeBSD-specific I guess. And they have to
be set via loader.conf(5). Note sysctl != tunable but tunable is,
most of the time, made available as sysctl.

> sysctl -A dev.itwd
> dev.itwd.0.%parent: superio0
> dev.itwd.0.%pnpinfo: type=WDT
> dev.itwd.0.%location: ldn=0x07
> dev.itwd.0.%driver: itwd
> dev.itwd.0.%desc: Watchdog Timer on ITE SuperIO
> dev.itwd.%parent:
>
> On linux,
> modprobe it87_wdt
> ] it87_wdt: Chip IT8613 revision 12 initialized. timeout=60 sec
> (nowayout=0 testmode=0)
>
> wdctl
> Device:        /dev/watchdog0
> Identity:      IT87 WDT [version 1]
> Timeout:       60 seconds
> Pre-timeout:    0 seconds
> FLAG           DESCRIPTION               STATUS BOOT-STATUS
> KEEPALIVEPING  Keep alive ping reply          1           0
> MAGICCLOSE     Supports magic close char      0           0
> SETTIMEOUT     Set timeout (in seconds)       0           0
>
> The code is at
> https://github.com/torvalds/linux/blob/master/drivers/watchdog/it87_wdt.c
> if that helps shed any light.

So chip is IT8613.

> i tried defining the irq and playing with the nmi at 1 or 0, and 
> still
> no luck. e.g. setting it to 0x05 and nmi 0, kldload (with boot 
> verbose)
> and then starting up watchdogd -t 8... wait 9 seconds,  and then 
> killall
> -9 watchdogd, still does not reboot the box
>
> itwd0: <Watchdog Timer on ITE SuperIO> at WDT ldn 0x07 on 
> superio0
> ioapic0: routing intpin 5 (ISA IRQ 5) to lapic 0 vector 56
> itwd0: Using IRQ5 to signal timeout
> itwd0: setting timeout to 8
> itwd0: setting timeout to 8
> itwd0: setting timeout to 8
> itwd0: setting timeout to 8
> itwd0: setting timeout to 8
> itwd0: setting timeout to 8
> itwd0: got interrupt, wdt status = 1
>
> Same with irq at 0x0f with and without nmi=0 or 1.
>
> itwd0: <Watchdog Timer on ITE SuperIO> at WDT ldn 0x07 on 
> superio0
> itwd0: Using IRQ15 to signal timeout
> itwd0: setting timeout to 16
> itwd0: setting timeout to 16
> itwd0: setting timeout to 16
> itwd0: setting timeout to 16
> itwd0: setting timeout to 4
> itwd0: setting timeout to 4
> itwd0: setting timeout to 4
> itwd0: setting timeout to 4
> itwd0: setting timeout to 4
> itwd0: setting timeout to 4
> itwd0: setting timeout to 4
> itwd0: setting timeout to 4

OK no luck. The Linux driver don't seems to set IRQ either.

Regards,
-- 
Stéphane Rochoy
O: Stormshield


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#1977

mike tancsa <mike@sentex.net> writes:

> On 9/25/2024 3:13 AM, Stephane Rochoy wrote:
>>>>>
>>>>> Any idea how to get this hardware working ?
>>>>
>>>> Do you know if, at least, the pre-timeout is working?
>>>
>>>     How do I check that ?
>>
>> Using --pretimeout and --pretimeout-action. See watchdogd(8) 
>> for
>> more details.
>>
> Reverting to the code thats in the tree, I started it up with
>
> watchdogd --pretimeout 10 --pretimeout-action log -t 15
>
> Doesnt seem to work or at least it doesnt make sense to me. 
> After I
> start that up, I see in the logs about 25 seconds after it 
> starts, but
> not always.
>
> Sep 26 11:31:20 mpki2024 kernel: watchdog pre-timeout, 
> WD_SOFT_LOG
> Sep 26 11:31:20 mpki2024 kernel: Sep 26 11:31:20 pki2024 kernel:
> watchdog pre-timeout, WD_SOFT_LOG
>
> and then doing a kill -9 does nothing :(

AFAIK, the pre-timeout should trigger reliably. Weird.
-- 
Stéphane Rochoy
O: Stormshield


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#1978

mike tancsa <mike@sentex.net> writes:

> I also started to play around with the non hardware based 
> watchdog
> timer.  For some reason, I cant get the box to reboot. It just 
> prints a
> stack trace and continues
>
> This is with no driver loaded.
>
> 0{t}# watchdogd --softtimeout-action panic -t 5
> 0{t}# ps -auxww | grep dog
> root     1487   0.4  0.2   12820  12920  -  S<s  09:03 0:00.01 
> watchdogd
> --softtimeout-action panic -t 5
> root     1489   0.0  0.0   12808   2636 u0  S+   09:03 
> 0:00.00
> grep dog
> 0{t}# kill -9 1487
> 0{t}# KDB: stack backtrace:
> #0 0xffffffff80b7fefd at kdb_backtrace+0x5d
> #1 0xffffffff80abec93 at hardclock+0x103
> #2 0xffffffff80abfe8b at handleevents+0xab
> #3 0xffffffff80ac0b7c at timercb+0x24c
> #4 0xffffffff810d0ebb at lapic_handle_timer+0xab
> #5 0xffffffff80fd8a71 at Xtimerint+0xb1
> #6 0xffffffff804b3685 at acpi_cpu_idle+0x2c5
> #7 0xffffffff80fc48f6 at cpu_idle_acpi+0x46
> #8 0xffffffff80fc49ad at cpu_idle+0x9d
> #9 0xffffffff80b67bb6 at sched_idletd+0x576
> #10 0xffffffff80aecf7f at fork_exit+0x7f
> #11 0xffffffff80fd7dae at fork_trampoline+0xe
>
> 0{t}#
>
> Should it not reboot at this point ?

I don't think so. `kdb_backtrace` being involved let me think the
kernel was compiled with KDB. You should play with ddb(8) to set a
default action like reset when entering kdb.

Regards,
-- 
Stéphane Rochoy
O: Stormshield


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#1979

mike tancsa <mike@sentex.net> writes:

> Do you know off hand how to set the system to just reboot ? The 
> ddb man
> page seems to imply I need options DDB as well, which is not in 
> GENERIC
> in order to set script actions.

I would try the following:

  ddb script kdb.enter.default=reset

Regards,
-- 
Stéphane Rochoy
O: Stormshield


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#1980

mike tancsa <mike@sentex.net> writes:

> WARNING: This e-mail comes from someone outside your 
> organisation. Do not click
> on links or open attachments if you do not know the sender and 
> are not sure that
> the content is safe.
>
> On 9/30/2024 3:18 AM, Stephane Rochoy wrote:
>>
>> mike tancsa <mike@sentex.net> writes:
>>
>>> Do you know off hand how to set the system to just reboot ? 
>>> The ddb man
>>> page seems to imply I need options DDB as well, which is not 
>>> in GENERIC
>>> in order to set script actions.
>>
>> I would try the following:
>>
>>  ddb script kdb.enter.default=reset
>>
> If I build a custom kernel then that will work. But with GENERIC 
> (I am
> tracking project via freebsd-update), it fails
>
> # ddb script kdb.enter.default=reset
> ddb: sysctl: debug.ddb.scripting.scripts: No such file or 
> directory
>
> With a customer kernel, adding
>
> options DDB
>
> it works perfectly.
>
> Is there any way to get this to work without having ddb custom
> compiled in ?

I don't understand what's happening here. AFAIK, the code
corresponding to the soft watchdog being triggered is the
following:

  static void
  wd_timeout_cb(void *arg)
  {
    const char *type = arg;

  #ifdef DDB
    if ((wd_pretimeout_act & WD_SOFT_DDB)) {
      char kdb_why[80];
      snprintf(kdb_why, sizeof(kdb_why), "watchdog %s-timeout", 
      type);
      kdb_backtrace();
      kdb_enter(KDB_WHY_WATCHDOG, kdb_why);
    }
  #endif
    if ((wd_pretimeout_act & WD_SOFT_LOG))
      log(LOG_EMERG, "watchdog %s-timeout, WD_SOFT_LOG\n", type);
    if ((wd_pretimeout_act & WD_SOFT_PRINTF))
      printf("watchdog %s-timeout, WD_SOFT_PRINTF\n", type);
    if ((wd_pretimeout_act & WD_SOFT_PANIC))
      panic("watchdog %s-timeout, WD_SOFT_PANIC set", type);
  }

So without DDB, it should call panic. But in your case, it
called kdb_backtrace. So initial hypothesis was wrong. What I
missed is that panic was natively able to kdb_backtrace if gently
asked to do so:

  #ifdef KDB
    if ((newpanic || trace_all_panics) && trace_on_panic)
      kdb_backtrace();
    if (debugger_on_panic)
      kdb_enter(KDB_WHY_PANIC, "panic");
    else if (!newpanic && debugger_on_recursive_panic)
      kdb_enter(KDB_WHY_PANIC, "re-panic");
  #endif
    /*thread_lock(td); */
    td->td_flags |= TDF_INPANIC;
    /* thread_unlock(td); */
    if (!sync_on_panic)
      bootopt |= RB_NOSYNC;
    if (poweroff_on_panic)
      bootopt |= RB_POWEROFF;
    if (powercycle_on_panic)
      bootopt |= RB_POWERCYCLE;
    kern_reboot(bootopt);

So it definitely should reboot but as it don't, maybe playing with
kern.powercycle_on_panic would help?

Regards,
--  
Stéphane Rochoy
O: Stormshield


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#1981 — Re: watchdog timer programming (progress)

mike tancsa <mike@sentex.net> writes:

> On 10/1/2024 5:02 PM, mike tancsa wrote:
>> On 10/1/2024 4:03 PM, mike tancsa wrote:
>>> On 10/1/2024 2:07 AM, Stephane Rochoy wrote:
>>>>
>>>> mike tancsa <mike@sentex.net> writes:
>>>>
>>>>> WARNING: This e-mail comes from someone outside your 
>>>>> organisation.
>>>>> Do not click
>>>>> on links or open attachments if you do not know the sender 
>>>>> and are
>>>>> not sure that
>>>>> the content is safe.
>>>>>
>>>>> On 9/30/2024 3:18 AM, Stephane Rochoy wrote:
>>>>>>
>>>>>> mike tancsa <mike@sentex.net> writes:
>>>>>>
>>>>>>> Do you know off hand how to set the system to just reboot 
>>>>>>> ? The
>>>>>>> ddb man
>>>>>>> page seems to imply I need options DDB as well, which is 
>>>>>>> not in
>>>>>>> GENERIC
>>>>>>> in order to set script actions.
>>>>>>
>>>>>> I would try the following:
>>>>>>
>>>>>>  ddb script kdb.enter.default=reset
>>>>>>
>>>>> If I build a custom kernel then that will work. But with 
>>>>> GENERIC (I am
>>>>> tracking project via freebsd-update), it fails
>>>>>
>>>>> # ddb script kdb.enter.default=reset
>>>>> ddb: sysctl: debug.ddb.scripting.scripts: No such file or 
>>>>> directory
>>>>>
>>>>> With a customer kernel, adding
>>>>>
>>>>> options DDB
>>>>>
>>>>> it works perfectly.
>>>>>
>>>>> Is there any way to get this to work without having ddb 
>>>>> custom
>>>>> compiled in ?
>>>>
>>>> I don't understand what's happening here. AFAIK, the code
>>>> corresponding to the soft watchdog being triggered is the
>>>> following:
>>>>
>>>>  static void
>>>>  wd_timeout_cb(void *arg)
>>>>  {
>>>>    const char *type = arg;
>>>>
>>>>  #ifdef DDB
>>>>    if ((wd_pretimeout_act & WD_SOFT_DDB)) {
>>>>      char kdb_why[80];
>>>>      snprintf(kdb_why, sizeof(kdb_why), "watchdog 
>>>>      %s-timeout",
>>>> type);
>>>>      kdb_backtrace();
>>>>      kdb_enter(KDB_WHY_WATCHDOG, kdb_why);
>>>>    }
>>>>  #endif
>>>>    if ((wd_pretimeout_act & WD_SOFT_LOG))
>>>>      log(LOG_EMERG, "watchdog %s-timeout, WD_SOFT_LOG\n", 
>>>>      type);
>>>>    if ((wd_pretimeout_act & WD_SOFT_PRINTF))
>>>>      printf("watchdog %s-timeout, WD_SOFT_PRINTF\n", type);
>>>>    if ((wd_pretimeout_act & WD_SOFT_PANIC))
>>>>      panic("watchdog %s-timeout, WD_SOFT_PANIC set", type);
>>>>  }
>>>>
>>>> So without DDB, it should call panic. But in your case, it
>>>> called kdb_backtrace. So initial hypothesis was wrong. What I
>>>> missed is that panic was natively able to kdb_backtrace if 
>>>> gently
>>>> asked to do so:
>>>>
>>>>  #ifdef KDB
>>>>    if ((newpanic || trace_all_panics) && trace_on_panic)
>>>>      kdb_backtrace();
>>>>    if (debugger_on_panic)
>>>>      kdb_enter(KDB_WHY_PANIC, "panic");
>>>>    else if (!newpanic && debugger_on_recursive_panic)
>>>>      kdb_enter(KDB_WHY_PANIC, "re-panic");
>>>>  #endif
>>>>    /*thread_lock(td); */
>>>>    td->td_flags |= TDF_INPANIC;
>>>>    /* thread_unlock(td); */
>>>>    if (!sync_on_panic)
>>>>      bootopt |= RB_NOSYNC;
>>>>    if (poweroff_on_panic)
>>>>      bootopt |= RB_POWEROFF;
>>>>    if (powercycle_on_panic)
>>>>      bootopt |= RB_POWERCYCLE;
>>>>    kern_reboot(bootopt);
>>>>
>>>> So it definitely should reboot but as it don't, maybe playing 
>>>> with
>>>> kern.powercycle_on_panic would help?
>>>>
>>>>
>>>
>>> Thank you for your continued help on this. Still no luck with 
>>> the
>>> GENERIC kernel
>>>
>>> 0{p9999}# sysctl -w kern.powercycle_on_panic=1
>>> kern.powercycle_on_panic: 0 -> 1
>>> 0{p9999}# ps -auxwww | grep dog
>>> root     4752   0.0  0.2   12820  12916  -  S<s  15:38 0:00.01
>>> watchdogd --softtimeout-action panic -t 10
>>> root     4792   0.0  0.0   12808   2644 u0  S+   15:39 0:00.00 
>>> grep dog
>>> 0{p9999}# kill -9 4752
>>> 0{p9999}# KDB: stack backtrace:
>>> #0 0xffffffff80b7fefd at kdb_backtrace+0x5d
>>> #1 0xffffffff80abec93 at hardclock+0x103
>>> #2 0xffffffff80abfe8b at handleevents+0xab
>>> #3 0xffffffff80ac0b7c at timercb+0x24c
>>> #4 0xffffffff810d0ebb at lapic_handle_timer+0xab
>>> #5 0xffffffff80fd8a71 at Xtimerint+0xb1
>>> #6 0xffffffff804b3685 at acpi_cpu_idle+0x2c5
>>> #7 0xffffffff80fc48f6 at cpu_idle_acpi+0x46
>>> #8 0xffffffff80fc49ad at cpu_idle+0x9d
>>> #9 0xffffffff80b67bb6 at sched_idletd+0x576
>>> #10 0xffffffff80aecf7f at fork_exit+0x7f
>>> #11 0xffffffff80fd7dae at fork_trampoline+0xe
>>>
>>> 0{p9999}#
>>>
>>> Where would be the best place to hack in something like this 
>>> in the
>>> driver ?
>>>  sysctl -w debug.kdb.panic_str="Watchdog Panic"
>>>
>>> which actually does panic the box
>>>
>>>
>>
>> One other datapoint. It seems starting
>>
>> watchdogd --softtimeout-action panic --softtimeout -t 10
>>
>> After kill -9
>> it eventually prints out
>>
>> watchdog soft-timeout, WD_SOFT_LOG
>>
>> to dmesg.  But after that, I cannot start a new watchdogd with 
>> just
>>
>> watchdogd --softtimeout-action panic -t 10
>>
>> I get
>>
>> watchdogd: setting WDIOC_SETSOFT 1: Invalid argument
>> watchdogd: patting the dog: Invalid argument
>
>
> I made these 2 changes to the driver
>
> --- watchdog.c  2024-10-01 20:37:28.667869000 -0400
> +++ /tmp/watchdog.c     2024-10-01 20:36:59.764330000 -0400
> @@ -61,7 +61,8 @@
>  static struct callout wd_softtimeo_handle;
>  static int wd_softtimer;       /* true = use softtimer instead 
>  of hardware
>                                    watchdog */
> -static int wd_softtimeout_act = WD_SOFT_LOG;   /* action for 
> the
> software timeout */
> +// static int wd_softtimeout_act = WD_SOFT_LOG;        /* 
> action for
> the software timeout */
> +static int wd_softtimeout_act = WD_SOFT_PANIC; /* action for 
> the
> software timeout */
>
>  static struct cdev *wd_dev;
>  static volatile u_int wd_last_u;    /* last timeout value set 
>  by
> kern_do_pat */
> @@ -241,6 +242,7 @@
>  wd_timeout_cb(void *arg)
>  {
>         const char *type = arg;
> +       panic("mdt watchdog %s-timeout, WD_SOFT_PANIC set", 
> type);
>
>  #ifdef DDB
>         if ((wd_pretimeout_act & WD_SOFT_DDB)) {
>
>
> and it works now
>
> KDB: stack backtrace:
> #0 0xffffffff80b8943d at kdb_backtrace+0x5d
> #1 0xffffffff80b3bfd1 at vpanic+0x131
> #2 0xffffffff80b3be93 at panic+0x43
> #3 0xffffffff8098b585 at wd_timeout_cb+0x15
> #4 0xffffffff80b59fcc at softclock_call_cc+0x12c
> #5 0xffffffff80b5b815 at softclock_thread+0xe5
> #6 0xffffffff80af61df at fork_exit+0x7f
> #7 0xffffffff80ff76ce at fork_trampoline+0xe
> Uptime: 1m13s
>
> it seems the soft timeout value action is never overridden for 
> some reason.
>
> This kinda feels like a bug / pr ?

Well, honestly I'm puzzled:
- in one hand, watchdog.c don't seems to use wd_softtimeout_act
- and on the other hand hardclock seems to directly call
  watchdog_fire which just kdb_enter or panic.

Note that wd_timeout_cb seems to be about both pretimeout and
timeout handling.

Regards,
-- 
Stéphane Rochoy
O: Stormshield


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [standalone]

Back to top | Article view | muc.lists.freebsd.hardware

csiph-web