Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #264685 > unrolled thread

raid10 is killing me, and applications that aren't willing to wait for it to respond

Back to article view | Back to linux.debian.user

Contents

  raid10 is killing me, and applications that aren't willing to wait  for it to respond gene heskett <gheskett@shentel.net> - 2023-12-13 16:30 +0100
    Re: raid10 is killing me, and applications that aren't willing to  wait for it to respond Tom Furie <tom@furie.org.uk> - 2023-12-13 16:50 +0100
      Re: raid10 is killing me, and applications that aren't willing towait  for it to respond gene heskett <gheskett@shentel.net> - 2023-12-13 19:20 +0100
        Re: raid10 is killing me, and applications that aren't willing  towait for it to respond Tom Furie <tom@furie.org.uk> - 2023-12-13 20:00 +0100
          Re: raid10 is killing me, and applications that aren't willingtowait  for it to respond gene heskett <gheskett@shentel.net> - 2023-12-13 20:30 +0100
    Re: raid10 is killing me, and applications that aren't willing to  wait for it to respond Pocket <pocket@columbus.rr.com> - 2023-12-13 18:00 +0100
      Re: raid10 is killing me, and applications that aren't willing towait  for it to respond gene heskett <gheskett@shentel.net> - 2023-12-13 19:30 +0100
        Re: raid10 is killing me, and applications that aren't willing  towait for it to respond Nicolas George <george@nsup.org> - 2023-12-13 19:50 +0100
          Re: raid10 is killing me, and applications that aren't willing towait  for it to respond Pocket <pocket@columbus.rr.com> - 2023-12-13 20:10 +0100
        Re: raid10 is killing me, and applications that aren't willing towait  for it to respond Pocket <pocket@columbus.rr.com> - 2023-12-13 19:50 +0100
          Re: raid10 is killing me, and applications that aren't willing  towait for it to respond Dan Ritter <dsr@randomstring.org> - 2023-12-13 20:10 +0100
            Re: raid10 is killing me, and applications that aren't willing towait  for it to respond Pocket <pocket@columbus.rr.com> - 2023-12-13 20:20 +0100
            Re: raid10 is killing me, and applications that aren't willingtowait  for it to respond gene heskett <gheskett@shentel.net> - 2023-12-13 20:30 +0100
    Re: raid10 is killing me, and applications that aren't willing to  wait for it to respond "Andrew M.A. Cater" <amacater@einval.com> - 2023-12-13 19:30 +0100
      Re: raid10 is killing me, and applications that aren't willing  towait for it to respond "Andrew M.A. Cater" <amacater@einval.com> - 2023-12-13 23:00 +0100
        Re: raid10 is killing me, and applications that aren't willingtowait  for it to respond gene heskett <gheskett@shentel.net> - 2023-12-14 01:10 +0100
    Re: raid10 is killing me, and applications that aren't willing to  wait for it to respond Andy Smith <andy@strugglers.net> - 2023-12-13 22:40 +0100
      Re: raid10 is killing me, and applications that aren't willing to  wait for it to respond Andy Smith <andy@strugglers.net> - 2023-12-14 00:10 +0100
      Re: raid10 is killing me, and applications that aren't willing towait  for it to respond gene heskett <gheskett@shentel.net> - 2023-12-14 00:40 +0100
        Re: raid10 is killing me, and applications that aren't willing towait  for it to respond David Christensen <dpchrist@holgerdanske.com> - 2023-12-14 04:20 +0100
    Re: raid10 is killing me, and applications that aren't willing to  wait for it to respond <tomas@tuxteam.de> - 2023-12-14 06:50 +0100
      Re: raid10 is killing me, and applications that aren't willing to  wait for it to respond Nicolas George <george@nsup.org> - 2023-12-14 10:20 +0100
        Re: raid10 is killing me, and applications that aren't willing to  wait for it to respond <tomas@tuxteam.de> - 2023-12-14 10:40 +0100
        Re: raid10 is killing me, and applications that aren't willing towait  for it to respond gene heskett <gheskett@shentel.net> - 2023-12-14 13:10 +0100
      Re: raid10 is killing me, and applications that aren't willing towait  for it to respond gene heskett <gheskett@shentel.net> - 2023-12-14 12:00 +0100
        Re: raid10 is killing me, and applications that aren't willing towait for it to respond Anssi Saari <anssi.saari@debian-user.mail.kapsi.fi> - 2023-12-14 22:40 +0100
          Re: raid10 is killing me, and applications that aren't willing towait  for it to respond gene heskett <gheskett@shentel.net> - 2023-12-15 03:40 +0100
            Re: raid10 is killing me, and applications that aren't willing towait  for it to respond David Christensen <dpchrist@holgerdanske.com> - 2023-12-15 09:10 +0100
              Re: raid10 is killing me, and applications that aren't willing  towait for it to respond Andy Smith <andy@strugglers.net> - 2023-12-15 13:30 +0100
                Re: raid10 is killing me, and applications that aren't willing towait for it to respond Stefan Monnier <monnier@iro.umontreal.ca> - 2023-12-15 16:00 +0100
              Re: raid10 is killing me, and applications that aren't willing  towaitfor it to respond gene heskett <gheskett@shentel.net> - 2023-12-16 03:30 +0100
                Re: raid10 is killing me, and applications that aren't willing  towaitfor it to respond David Christensen <dpchrist@holgerdanske.com> - 2023-12-16 05:00 +0100
                  Re: raid10 is killing me, and applications that aren't  willingtowaitfor it to respond gene heskett <gheskett@shentel.net> - 2023-12-16 16:10 +0100
                    Re: raid10 is killing me, and applications that aren't  willingtowaitfor it to respond David Christensen <dpchrist@holgerdanske.com> - 2023-12-16 21:30 +0100
                Re: raid10 is killing me, and applications that aren't willing  towaitfor it to respond David Christensen <dpchrist@holgerdanske.com> - 2023-12-16 05:10 +0100
                  Re: raid10 is killing me, and applications that aren't willing  towaitfor it to respond Matt <matt@jmgresham.xyz> - 2023-12-16 05:20 +0100
                    Re: raid10 is killing me, and applications that aren't willing  towaitfor it to respond Greg Wooledge <greg@wooledge.org> - 2023-12-16 05:30 +0100
                      Re: raid10 is killing me, and applications that aren't willing  towaitfor it to respond Matt <matt@jmgresham.xyz> - 2023-12-16 05:30 +0100
                      Re: raid10 is killing me, and applications that aren't willing towaitfor it to respond Anssi Saari <anssi.saari@debian-user.mail.kapsi.fi> - 2023-12-16 09:30 +0100
                        Re: raid10 is killing me, and applications that aren't willing  towaitfor it to respond David Christensen <dpchrist@holgerdanske.com> - 2023-12-16 20:20 +0100
            Re: raid10 is killing me, and applications that aren't willing towait for it to respond Anssi Saari <anssi.saari@debian-user.mail.kapsi.fi> - 2023-12-16 09:50 +0100

Page 1 of 3  [1] 2 3  Next page →

#264685 — raid10 is killing me, and applications that aren't willing to wait for it to respond

Greetings all;

I thought I was doing things right a year back when I built a raid10 for 
my /home partition. but I'm tired of fighting with it for access. 
Anything that wants to open a file on it, is subjected to a freeze of at 
least 30 seconds BEFORE the file requester is drawn on screen.  Once it 
has done the screen draw and the path is established, read/writes then 
proceed at multi-gigabyte speeds just like it should, but some 
applications refuse to wait that long, so digiKam cannot import from my 
camera for example one, QIDISlicer is another that get plumb upset and 
declares a segfault, core dumped, but it can't write the core dump for 
the same reason it declared a segfault.  Here is a copy/paste of the 
last attempt to select the "device" tab in QIDISlicer:
-----------------------------------------------
Error creating proxy: Error calling StartServiceByName for 
org.gtk.vfs.GPhoto2VolumeMonitor: Timeout was reached (g-io-error-quark, 24)

** (qidi-slicer:389574): CRITICAL **: 04:55:46.975: Cannot register URI 
scheme wxfs more than once

** (qidi-slicer:389574): CRITICAL **: 04:55:46.975: Cannot register URI 
scheme memory more than once

(qidi-slicer:389574): Gtk-CRITICAL **: 04:55:47.084: 
gtk_box_gadget_distribute: assertion 'size >= 0' failed in GtkScrollbar
[2023-12-13 05:10:27.325222] [0x00007f77e6ffd6c0] [error]   Socket 
created. Multicast: 255.255.255.255. Interface: 192.168.71.3
Unhandled unknown exception; terminating the application.
Segmentation fault (core dumped)
-----------------------------------------------------
This where it was attempting to open the cache buffers if needed to 
remember what moonraker, a web server driver which is part of the 
klipper install on the printer, addressed at 192.168.71.110: with an 
odd, high numbered port above 10,000.

I've been here several times with this problem without any constructive 
responses other than strace, which of course does NOT work for network 
stuff, and would if my past history with it is any indication, generate 
several terabytes of output, but it fails for the same reason, no place 
to put its output because I assume, it can't write to the raid10 in a 
timely manner.

So one more time: Why can't I use my software raid10 on 4 1T SSD's ?????

Cheers, Gene Heskett.
-- 
"There are four boxes to be used in defense of liberty:
  soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
  - Louis D. Brandeis

[toc] | [next] | [standalone]

#264687 — Re: raid10 is killing me, and applications that aren't willing to wait for it to respond

gene heskett <gheskett@shentel.net> writes:

> I thought I was doing things right a year back when I built a raid10
> for my /home partition. but I'm tired of fighting with it for
> access. Anything that wants to open a file on it, is subjected to a
> freeze of at least 30 seconds BEFORE the file requester is drawn on
> screen.  Once it has done the screen draw and the path is established,

Where is the raid10 located and how is it interfaced to the device
you're accessing it from? That delay, along with other things you
mentioned suggests (but this is only a guess without other relevant
information) a DNS timeout.

Cheers,
Tom

[toc] | [prev] | [next] | [standalone]

#264691 — Re: raid10 is killing me, and applications that aren't willing towait for it to respond

On 12/13/23 10:41, Tom Furie wrote:
> gene heskett <gheskett@shentel.net> writes:
> 
>> I thought I was doing things right a year back when I built a raid10
>> for my /home partition. but I'm tired of fighting with it for
>> access. Anything that wants to open a file on it, is subjected to a
>> freeze of at least 30 seconds BEFORE the file requester is drawn on
>> screen.  Once it has done the screen draw and the path is established,
> 
> Where is the raid10 located and how is it interfaced to the device
> you're accessing it from? That delay, along with other things you
> mentioned suggests (but this is only a guess without other relevant
> information) a DNS timeout.
> 
It is a separate 6 port sata controller because the mobo is out of 
ports.  There is no obvious lag during bios post or grub booting it.
/etc/fstab:
gene@coyote:/etc/lvm/profile$ cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# systemd generates mount units based on this file, see systemd.mount(5).
# Please run 'systemctl daemon-reload' after making changes here.
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/sda1 during installation
UUID=f295334b-fdcb-4428-bed3-cb9e9e129be6 /               ext4 
errors=remount-ro 0       1
# /tmp was on /dev/sda3 during installation
UUID=518cb65d-21f0-493f-8bb5-a5f435796991 /tmp            ext4 
defaults        0       2
# swap was on /dev/sda2 during installation
UUID=422b50db-9913-4ed3-92c3-dc18be72cc61 none            swap    sw 
          0       0
/dev/sr0        /media/cdrom0   udf,iso9660 user,noauto     0       0
UUID=bc6135de-0578-4e3b-b2c0-5c4687abd9bd /home         ext4 
errors=remount-ro  0       2
UUID=d24c3a99-9f40-4b71-92d4-916804553cb5 none          swap    sw 
0       0
-----------------------------
 From df:
gene@coyote:/etc/lvm/profile$ sudo df
[sudo] password for gene:
Filesystem      1K-blocks      Used  Available Use% Mounted on
udev             16328024         0   16328024   0% /dev
tmpfs             3272676      1876    3270800   1% /run
/dev/sda1       863983352  18784928  801236776   3% /
tmpfs            16363376        12   16363364   1% /dev/shm
tmpfs                5120         8       5112   1% /run/lock
/dev/sda3        47749868    222628   45069232   1% /tmp
/dev/md0p1     1796382580 330887008 1374170596  20% /home
tmpfs             3272672     45868    3226804   2% /run/user/1000
-----------------------------
A dhcpd has been installed, but is limited to issuing a single fixed 
address to a 3d printer plugged into my network, as the printer doesn't 
seem to know what to do with a hosts file that runs the rest of my home 
network. Anything else you want, just ask.

> Cheers,
> Tom
> 
> .
Thank you Tom.

Cheers, Gene Heskett.
-- 
"There are four boxes to be used in defense of liberty:
  soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
  - Louis D. Brandeis

[toc] | [prev] | [next] | [standalone]

#264696 — Re: raid10 is killing me, and applications that aren't willing towait for it to respond

gene heskett <gheskett@shentel.net> writes:

> It is a separate 6 port sata controller because the mobo is out of
> ports.  There is no obvious lag during bios post or grub booting it.

That *should* rule out DNS then, unless something really strange is
going on. What does mdadm tell you about the raid device, and its
component devices?

Is the filesystem on the raid healthy?

Cheers,
Tom

[toc] | [prev] | [next] | [standalone]

#264701 — Re: raid10 is killing me, and applications that aren't willingtowait for it to respond

On 12/13/23 13:56, Tom Furie wrote:
> gene heskett <gheskett@shentel.net> writes:
> 
>> It is a separate 6 port sata controller because the mobo is out of
>> ports.  There is no obvious lag during bios post or grub booting it.
> 
> That *should* rule out DNS then, unless something really strange is
> going on. What does mdadm tell you about the raid device, and its
> component devices?
> 
> Is the filesystem on the raid healthy?

According to the tools, it was a few days ago. This has been a pita from 
the gitgo of installing bookworm over a year ago with the original 
net-installer.
> 
> Cheers,
> Tom
> 

Thanks Tom.

Cheers, Gene Heskett.
-- 
"There are four boxes to be used in defense of liberty:
  soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
  - Louis D. Brandeis

[toc] | [prev] | [next] | [standalone]

#264689 — Re: raid10 is killing me, and applications that aren't willing to wait for it to respond

On 12/13/23 10:26, gene heskett wrote:
> Greetings all;
>
> I thought I was doing things right a year back when I built a raid10 
> for my /home partition. but I'm tired of fighting with it for access. 
> Anything that wants to open a file on it, is subjected to a freeze of 
> at least 30 seconds BEFORE the file requester is drawn on screen.  
> Once it has done the screen draw and the path is established, 
> read/writes then proceed at multi-gigabyte speeds just like it should, 
> but some applications refuse to wait that long, so digiKam cannot 
> import from my camera for example one, QIDISlicer is another that get 
> plumb upset and declares a segfault, core dumped, but it can't write 
> the core dump for the same reason it declared a segfault.  Here is a 
> copy/paste of the last attempt to select the "device" tab in QIDISlicer:
> -----------------------------------------------
> Error creating proxy: Error calling StartServiceByName for 
> org.gtk.vfs.GPhoto2VolumeMonitor: Timeout was reached 
> (g-io-error-quark, 24)
>
> ** (qidi-slicer:389574): CRITICAL **: 04:55:46.975: Cannot register 
> URI scheme wxfs more than once
>
> ** (qidi-slicer:389574): CRITICAL **: 04:55:46.975: Cannot register 
> URI scheme memory more than once
>
> (qidi-slicer:389574): Gtk-CRITICAL **: 04:55:47.084: 
> gtk_box_gadget_distribute: assertion 'size >= 0' failed in GtkScrollbar
> [2023-12-13 05:10:27.325222] [0x00007f77e6ffd6c0] [error]   Socket 
> created. Multicast: 255.255.255.255. Interface: 192.168.71.3
> Unhandled unknown exception; terminating the application.
> Segmentation fault (core dumped)
> -----------------------------------------------------
> This where it was attempting to open the cache buffers if needed to 
> remember what moonraker, a web server driver which is part of the 
> klipper install on the printer, addressed at 192.168.71.110: with an 
> odd, high numbered port above 10,000.
>
> I've been here several times with this problem without any 
> constructive responses other than strace, which of course does NOT 
> work for network stuff, and would if my past history with it is any 
> indication, generate several terabytes of output, but it fails for the 
> same reason, no place to put its output because I assume, it can't 
> write to the raid10 in a timely manner.
>
> So one more time: Why can't I use my software raid10 on 4 1T SSD's ?????
>
> Cheers, Gene Heskett.


I gave up using raid many years ago and I used the extra drives as backups.

Wrote a script to rsync  /home to the backup drives.

-- 
It's not easy to be me

[toc] | [prev] | [next] | [standalone]

#264692 — Re: raid10 is killing me, and applications that aren't willing towait for it to respond

On 12/13/23 11:51, Pocket wrote:
> 
> On 12/13/23 10:26, gene heskett wrote:
>> Greetings all;
>>
>> I thought I was doing things right a year back when I built a raid10 
>> for my /home partition. but I'm tired of fighting with it for access. 
>> Anything that wants to open a file on it, is subjected to a freeze of 
>> at least 30 seconds BEFORE the file requester is drawn on screen. Once 
>> it has done the screen draw and the path is established, read/writes 
>> then proceed at multi-gigabyte speeds just like it should, but some 
>> applications refuse to wait that long, so digiKam cannot import from 
>> my camera for example one, QIDISlicer is another that get plumb upset 
>> and declares a segfault, core dumped, but it can't write the core dump 
>> for the same reason it declared a segfault.  Here is a copy/paste of 
>> the last attempt to select the "device" tab in QIDISlicer:
>> -----------------------------------------------
>> Error creating proxy: Error calling StartServiceByName for 
>> org.gtk.vfs.GPhoto2VolumeMonitor: Timeout was reached 
>> (g-io-error-quark, 24)
>>
>> ** (qidi-slicer:389574): CRITICAL **: 04:55:46.975: Cannot register 
>> URI scheme wxfs more than once
>>
>> ** (qidi-slicer:389574): CRITICAL **: 04:55:46.975: Cannot register 
>> URI scheme memory more than once
>>
>> (qidi-slicer:389574): Gtk-CRITICAL **: 04:55:47.084: 
>> gtk_box_gadget_distribute: assertion 'size >= 0' failed in GtkScrollbar
>> [2023-12-13 05:10:27.325222] [0x00007f77e6ffd6c0] [error]   Socket 
>> created. Multicast: 255.255.255.255. Interface: 192.168.71.3
>> Unhandled unknown exception; terminating the application.
>> Segmentation fault (core dumped)
>> -----------------------------------------------------
>> This where it was attempting to open the cache buffers if needed to 
>> remember what moonraker, a web server driver which is part of the 
>> klipper install on the printer, addressed at 192.168.71.110: with an 
>> odd, high numbered port above 10,000.
>>
>> I've been here several times with this problem without any 
>> constructive responses other than strace, which of course does NOT 
>> work for network stuff, and would if my past history with it is any 
>> indication, generate several terabytes of output, but it fails for the 
>> same reason, no place to put its output because I assume, it can't 
>> write to the raid10 in a timely manner.
>>
>> So one more time: Why can't I use my software raid10 on 4 1T SSD's ?????
>>
>> Cheers, Gene Heskett.
> 
> 
> I gave up using raid many years ago and I used the extra drives as backups.
> 
So why did you give up? Must have been a reason.

> Wrote a script to rsync  /home to the backup drives.
> 

Cheers, Gene Heskett.
-- 
"There are four boxes to be used in defense of liberty:
  soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
  - Louis D. Brandeis

[toc] | [prev] | [next] | [standalone]

#264694 — Re: raid10 is killing me, and applications that aren't willing towait for it to respond

[Multipart message — attachments visible in raw view] — view raw

Pocket (12023-12-13):
> If the RAID controller

Then use software RAID with a Libre implementation.

> I found it is better to just have my data on several backup disks

Yeah, backups and RAID are not meant to protect against the same issues,
so if you think one replaces the other…

> After removing raid, I completely redesigned my network to be more inline
> with the howtos and other information.

You know that RAID has nothing to do with the setup of your network,
right?

-- 
  Nicolas George

[toc] | [prev] | [next] | [standalone]

#264697 — Re: raid10 is killing me, and applications that aren't willing towait for it to respond

On 12/13/23 13:47, Nicolas George wrote:
> Pocket (12023-12-13):
>> If the RAID controller
> Then use software RAID with a Libre implementation.


Nope been there done that and I ain't doing that


>
>> I found it is better to just have my data on several backup disks
> Yeah, backups and RAID are not meant to protect against the same issues,
> so if you think one replaces the other…
>
>> After removing raid, I completely redesigned my network to be more inline
>> with the howtos and other information.
> You know that RAID has nothing to do with the setup of your network,
> right?


Not saying it did


-- 
It's not easy to be me

[toc] | [prev] | [next] | [standalone]

#264695 — Re: raid10 is killing me, and applications that aren't willing towait for it to respond

On 12/13/23 13:20, gene heskett wrote:
> On 12/13/23 11:51, Pocket wrote:
>>
>> On 12/13/23 10:26, gene heskett wrote:
>>> Greetings all;
>>>
>>> I thought I was doing things right a year back when I built a raid10 
>>> for my /home partition. but I'm tired of fighting with it for 
>>> access. Anything that wants to open a file on it, is subjected to a 
>>> freeze of at least 30 seconds BEFORE the file requester is drawn on 
>>> screen. Once it has done the screen draw and the path is 
>>> established, read/writes then proceed at multi-gigabyte speeds just 
>>> like it should, but some applications refuse to wait that long, so 
>>> digiKam cannot import from my camera for example one, QIDISlicer is 
>>> another that get plumb upset and declares a segfault, core dumped, 
>>> but it can't write the core dump for the same reason it declared a 
>>> segfault.  Here is a copy/paste of the last attempt to select the 
>>> "device" tab in QIDISlicer:
>>> -----------------------------------------------
>>> Error creating proxy: Error calling StartServiceByName for 
>>> org.gtk.vfs.GPhoto2VolumeMonitor: Timeout was reached 
>>> (g-io-error-quark, 24)
>>>
>>> ** (qidi-slicer:389574): CRITICAL **: 04:55:46.975: Cannot register 
>>> URI scheme wxfs more than once
>>>
>>> ** (qidi-slicer:389574): CRITICAL **: 04:55:46.975: Cannot register 
>>> URI scheme memory more than once
>>>
>>> (qidi-slicer:389574): Gtk-CRITICAL **: 04:55:47.084: 
>>> gtk_box_gadget_distribute: assertion 'size >= 0' failed in GtkScrollbar
>>> [2023-12-13 05:10:27.325222] [0x00007f77e6ffd6c0] [error] Socket 
>>> created. Multicast: 255.255.255.255. Interface: 192.168.71.3
>>> Unhandled unknown exception; terminating the application.
>>> Segmentation fault (core dumped)
>>> -----------------------------------------------------
>>> This where it was attempting to open the cache buffers if needed to 
>>> remember what moonraker, a web server driver which is part of the 
>>> klipper install on the printer, addressed at 192.168.71.110: with an 
>>> odd, high numbered port above 10,000.
>>>
>>> I've been here several times with this problem without any 
>>> constructive responses other than strace, which of course does NOT 
>>> work for network stuff, and would if my past history with it is any 
>>> indication, generate several terabytes of output, but it fails for 
>>> the same reason, no place to put its output because I assume, it 
>>> can't write to the raid10 in a timely manner.
>>>
>>> So one more time: Why can't I use my software raid10 on 4 1T SSD's 
>>> ?????
>>>
>>> Cheers, Gene Heskett.
>>
>>
>> I gave up using raid many years ago and I used the extra drives as 
>> backups.
>>
> So why did you give up? Must have been a reason.

Many reasons........

No real benefit (companies excepted), and issues like you have been posting.

If the RAID controller bites the bullet you are usually toast unless you 
have another RAID controller (same manufacturer and type) as a spare.

I have zero luck replacing one companies raid controller with another 
and ditto on raid built into the motherboard.

I really don't need any help losing my data/files as I do a good job of 
that all by myself ;)

I found it is better to just have my data on several backup disks, that 
way if one fails I get another disk and copy all the data to the newly 
purchased disk.

After removing raid, I completely redesigned my network to be more 
inline with the howtos and other information.

I have little to nothing on the client system I use daily, everything is 
on networks systems and they have certain things they do.

I have a "git" server that has all my setup/custom/building scripts and 
all my programming and solidworks projects.

I have DELPHI build apps going back to about 1995.

It all backed up to a backup server(master and slave) and also a 4TB 
offline external hard drive.  I have not "lost" any information since.

I also found that DHCP and NetworkManager is your friend.

Maybe you should review your network setup as you seem to have a lot is 
issues with it?

>
>> Wrote a script to rsync  /home to the backup drives.
>>
>
> Cheers, Gene Heskett.

-- 
It's not easy to be me

[toc] | [prev] | [next] | [standalone]

#264698 — Re: raid10 is killing me, and applications that aren't willing towait for it to respond

Pocket wrote: 
> 
> Many reasons........
> 
> If the RAID controller bites the bullet you are usually toast unless you
> have another RAID controller (same manufacturer and type) as a spare.

mdadm, zfs and btrfs all lack this problem.

> I have zero luck replacing one companies raid controller with another and
> ditto on raid built into the motherboard.

As above.
 
> I really don't need any help losing my data/files as I do a good job of that
> all by myself ;)

btrfs and zfs have snapshots which really help avoiding losing
data. On other machines, rsnapshot is often suitable.


> I found it is better to just have my data on several backup disks, that way
> if one fails I get another disk and copy all the data to the newly purchased
> disk.

RAID isn't a backup solution, it's a way of keeping things going
until you have time to restore. (And also a way of improving
performance and/or manageability.)

If you don't need or want it, you shouldn't use it. Same as any
tool.

-dsr-

[toc] | [prev] | [next] | [standalone]

#264699 — Re: raid10 is killing me, and applications that aren't willing towait for it to respond

On 12/13/23 13:50, Dan Ritter wrote:
> Pocket wrote:
>> Many reasons........
>>
>> If the RAID controller bites the bullet you are usually toast unless you
>> have another RAID controller (same manufacturer and type) as a spare.
> mdadm, zfs and btrfs all lack this problem.

Not for me as I am not going down that worm hole


>
>> I have zero luck replacing one companies raid controller with another and
>> ditto on raid built into the motherboard.
> As above.


As above


>   
>> I really don't need any help losing my data/files as I do a good job of that
>> all by myself ;)
> btrfs and zfs have snapshots which really help avoiding losing
> data. On other machines, rsnapshot is often suitable.


I am exploring rdiff-backup


>> I found it is better to just have my data on several backup disks, that way
>> if one fails I get another disk and copy all the data to the newly purchased
>> disk.
> RAID isn't a backup solution, it's a way of keeping things going
> until you have time to restore. (And also a way of improving
> performance and/or manageability.)
>
> If you don't need or want it, you shouldn't use it. Same as any
> tool.

I don't need the expense or trouble.

Raspberry pi(s) and USB drives equate to "just works"

-- 

It's not easy to be me

[toc] | [prev] | [next] | [standalone]

#264700 — Re: raid10 is killing me, and applications that aren't willingtowait for it to respond

On 12/13/23 14:08, Dan Ritter wrote:
> Pocket wrote:
>>
>> Many reasons........
>>
>> If the RAID controller bites the bullet you are usually toast unless you
>> have another RAID controller (same manufacturer and type) as a spare.
> 
None of these controllers are self contained raids, it is all by mdadm 
and friends.

> mdadm, zfs and btrfs all lack this problem.
> 
>> I have zero luck replacing one companies raid controller with another and
>> ditto on raid built into the motherboard.
> 
> As above.
>   
>> I really don't need any help losing my data/files as I do a good job of that
>> all by myself ;)
> 
> btrfs and zfs have snapshots which really help avoiding losing
> data. On other machines, rsnapshot is often suitable.
> 
> 
>> I found it is better to just have my data on several backup disks, that way
>> if one fails I get another disk and copy all the data to the newly purchased
>> disk.
> 
> RAID isn't a backup solution, it's a way of keeping things going
> until you have time to restore. (And also a way of improving
> performance and/or manageability.)
> 
> If you don't need or want it, you shouldn't use it. Same as any
> tool.
> 
> -dsr-
> 
> .

Cheers, Gene Heskett.
-- 
"There are four boxes to be used in defense of liberty:
  soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
  - Louis D. Brandeis

[toc] | [prev] | [next] | [standalone]

#264693 — Re: raid10 is killing me, and applications that aren't willing to wait for it to respond

On Wed, Dec 13, 2023 at 10:26:19AM -0500, gene heskett wrote:
> Greetings all;
> 
> I thought I was doing things right a year back when I built a raid10 for my
> /home partition. but I'm tired of fighting with it for access. Anything that
> wants to open a file on it, is subjected to a freeze of at least 30 seconds
> BEFORE the file requester is drawn on screen.  Once it has done the screen
> draw and the path is established, read/writes then proceed at multi-gigabyte
> speeds just like it should, but some applications refuse to wait that long,
> so digiKam cannot import from my camera for example one, QIDISlicer is
> another that get plumb upset and declares a segfault, core dumped, but it
> can't write the core dump for the same reason it declared a segfault.  Here
> is a copy/paste of the last attempt to select the "device" tab in
> QIDISlicer:
> -----------------------------------------------
> Error creating proxy: Error calling StartServiceByName for
> org.gtk.vfs.GPhoto2VolumeMonitor: Timeout was reached (g-io-error-quark, 24)
> 
> ** (qidi-slicer:389574): CRITICAL **: 04:55:46.975: Cannot register URI
> scheme wxfs more than once
> 
> ** (qidi-slicer:389574): CRITICAL **: 04:55:46.975: Cannot register URI
> scheme memory more than once
> 
> (qidi-slicer:389574): Gtk-CRITICAL **: 04:55:47.084:
> gtk_box_gadget_distribute: assertion 'size >= 0' failed in GtkScrollbar
> [2023-12-13 05:10:27.325222] [0x00007f77e6ffd6c0] [error]   Socket created.
> Multicast: 255.255.255.255. Interface: 192.168.71.3
> Unhandled unknown exception; terminating the application.
> Segmentation fault (core dumped)
> -----------------------------------------------------
> This where it was attempting to open the cache buffers if needed to remember
> what moonraker, a web server driver which is part of the klipper install on
> the printer, addressed at 192.168.71.110: with an odd, high numbered port
> above 10,000.
> 
> I've been here several times with this problem without any constructive
> responses other than strace, which of course does NOT work for network
> stuff, and would if my past history with it is any indication, generate
> several terabytes of output, but it fails for the same reason, no place to
> put its output because I assume, it can't write to the raid10 in a timely
> manner.
> 

Hi Gene,

Respectfully, if I were you, I might consider tearing down one machine
and rebuilding the data on it bit by bit.

Questions to answer first:

1. Are all the disks the same size?

2. Are all the disks the same manufacturer?

3. Are they all connected to the same controller if this is an add-in card?

If not an add in card:

4. Are they all connected to the SATA sockets on the motherboard?
motherboard?

4. If to the motherboard, are they the only devices connected to the SATA
sockets there?

5. What is the primary device that has / on it - NVME / SSD / spinning rust?

> So one more time: Why can't I use my software raid10 on 4 1T SSD's ?????
> 

_How did you set the RAID 10 up?

Would you be willing to scrap the data in /home and start again?

All best, as ever,

Andy

(amacater@debian.org)

> Cheers, Gene Heskett.
> -- 
> "There are four boxes to be used in defense of liberty:
>  soap, ballot, jury, and ammo. Please use in that order."
> -Ed Howdershelt (Author, 1940)
> If we desire respect for the law, we must first make the law respectable.
>  - Louis D. Brandeis
> 

[toc] | [prev] | [next] | [standalone]

#264703 — Re: raid10 is killing me, and applications that aren't willing towait for it to respond

On Wed, Dec 13, 2023 at 02:19:07PM -0500, gene heskett wrote:
> On 12/13/23 13:24, Andrew M.A. Cater wrote:
> > On Wed, Dec 13, 2023 at 10:26:19AM -0500, gene heskett wrote:
> > > Greetings all;
> > > 
> > 
> > Hi Gene,
> > 
> > Respectfully, if I were you, I might consider tearing down one machine
> > and rebuilding the data on it bit by bit.
> > 
> > Questions to answer first:
> > 
> > 1. Are all the disks the same size?
> > 
> yes
> > 2. Are all the disks the same manufacturer?
> All 1T Samsung 870's
> > 3. Are they all connected to the same controller if this is an add-in card?
> > 
> yes. add in card.
> 
> > If not an add in card:
> > 
> > 4. Are they all connected to the SATA sockets on the motherboard?
> > motherboard?
> > 
> No. All connected to a 6 port board, in port order.
> 
> > 4. If to the motherboard, are they the only devices connected to the SATA
> > sockets there?
> No, main board is Asus Prime Z370-A II but it only has 6 ports, all busy.
> > 
> > 5. What is the primary device that has / on it - NVME / SSD / spinning rust?
> 
> SSD, another 1T samsung.
> 
> > > So one more time: Why can't I use my software raid10 on 4 1T SSD's ?????
> > > 
> > 
> > _How did you set the RAID 10 up?
> > 
> > Would you be willing to scrap the data in /home and start again?
> No, I have a lot of work I'd be the rest of my life rebuilding.
> Howevr, in preparation to restarting amanda, I've just installed a 2nd sata
> add on card, this one with 16 ports, 4 of which are already loaded with 2T
> gigastones so I do have the means to rsync /home to 1 or more of those.
> 

Copy /home to another drive - then disconnect power and drive cables to it.
You seem to like adding many disks to one machine: I'd honestly suggest
grabbing another machine to put half these disks into.

If you've got NVME - put that in as your boot drive, maybe.
Maybe use LVM and guided partitioning with all files in one partition.

Then use the four 1T disks and the four way card and mdadm to set up the
mirrored RAID with LVM on top for /home and add that to your fstab.

Rsync the data back from your one drive that you put the original /home
onto and you're done with that disk.

Do all this with a brand new bookworm disk and linuxcnc and you're done

Simplify, simplify, simplify :)

Andy
(amacater@debian.org)

[toc] | [prev] | [next] | [standalone]

#264706 — Re: raid10 is killing me, and applications that aren't willingtowait for it to respond

On 12/13/23 16:55, Andrew M.A. Cater wrote:
> On Wed, Dec 13, 2023 at 02:19:07PM -0500, gene heskett wrote:
>> On 12/13/23 13:24, Andrew M.A. Cater wrote:
>>> On Wed, Dec 13, 2023 at 10:26:19AM -0500, gene heskett wrote:
>>>> Greetings all;
>>>>
>>>
>>> Hi Gene,
>>>
>>> Respectfully, if I were you, I might consider tearing down one machine
>>> and rebuilding the data on it bit by bit.
>>>
>>> Questions to answer first:
>>>
>>> 1. Are all the disks the same size?
>>>
>> yes
>>> 2. Are all the disks the same manufacturer?
>> All 1T Samsung 870's
>>> 3. Are they all connected to the same controller if this is an add-in card?
>>>
>> yes. add in card.
>>
>>> If not an add in card:
>>>
>>> 4. Are they all connected to the SATA sockets on the motherboard?
>>> motherboard?
>>>
>> No. All connected to a 6 port board, in port order.
>>
>>> 4. If to the motherboard, are they the only devices connected to the SATA
>>> sockets there?
>> No, main board is Asus Prime Z370-A II but it only has 6 ports, all busy.
>>>
>>> 5. What is the primary device that has / on it - NVME / SSD / spinning rust?
>>
>> SSD, another 1T samsung.
>>
>>>> So one more time: Why can't I use my software raid10 on 4 1T SSD's ?????
>>>>
>>>
>>> _How did you set the RAID 10 up?
>>>
>>> Would you be willing to scrap the data in /home and start again?
>> No, I have a lot of work I'd be the rest of my life rebuilding.
>> Howevr, in preparation to restarting amanda, I've just installed a 2nd sata
>> add on card, this one with 16 ports, 4 of which are already loaded with 2T
>> gigastones so I do have the means to rsync /home to 1 or more of those.
>>
> 
> Copy /home to another drive - then disconnect power and drive cables to it.
> You seem to like adding many disks to one machine: I'd honestly suggest
> grabbing another machine to put half these disks into.
> 
> If you've got NVME - put that in as your boot drive, maybe.
> Maybe use LVM and guided partitioning with all files in one partition.

I've never had NVMe before, and installing what I have is a teardown to 
get to where it goes.  Other stuff has to come out first then pi back 
in.  What do I expect to see it do that's new?

> Then use the four 1T disks and the four way card and mdadm to set up the
> mirrored RAID with LVM on top for /home and add that to your fstab.
> 
> Rsync the data back from your one drive that you put the original /home
> onto and you're done with that disk.
> 
> Do all this with a brand new bookworm disk and linuxcnc and you're done
> 
> Simplify, simplify, simplify :)

Even more simplifed Andy. this machine does not have a realtime kernel 
as it will never actually command a machine so the only linuxcnc 
exercise is running a simulator to check the gcode I write for 
subtractive manufacture. But even that's fairly rare because my machines 
are not one size fits all basic, they know tricks unique to each machine 
that the simulators know nothing about.  So generally, I write the gcode 
for that machine on that machine where I can single step and stop it 
before the crash that might wreck $20 to $200 worth of tooling. :o(

> Andy
> (amacater@debian.org)
> 
> .

Cheers, Gene Heskett.
-- 
"There are four boxes to be used in defense of liberty:
  soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
  - Louis D. Brandeis

[toc] | [prev] | [next] | [standalone]

#264702 — Re: raid10 is killing me, and applications that aren't willing to wait for it to respond

Hello,

On Wed, Dec 13, 2023 at 10:26:19AM -0500, gene heskett wrote:
> I thought I was doing things right a year back when I built a raid10 for my
> /home partition. but I'm tired of fighting with it for access. Anything that
> wants to open a file on it, is subjected to a freeze of at least 30 seconds
> BEFORE the file requester is drawn on screen.

I haven't chimed in to any of the multiple times you've brought this
to the list, because it's just so bizarre. I've about 20 years'
experience of using mdadm and have never seen anything like what
you're reporting, so I just don't know what the problem could be or
how to find it.

The only times I've seen anything remotely like it have been when
there's been hardware problems with failing writes, but I know
you've been through this with the list several time sand no such low
level issues were ever uncovered.

Would it be correct to say that you only experience these IO delays
from GUI applications? Like, if you do a simple:

$ time echo "test" > ~/foo

does that complete in a normal time?

And if you did a bigger write, again from the command line?

$ dd if=/dev/zero of=/path/to/your/home/dir/zero bs=1m count=100
00+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.0783268 s, 1.3 GB/s

and with sync:

$ dd if=/dev/zero of=/path/to/your/home/dir/zero bs=1m count=100 oflag=sync
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.463356 s, 226 MB/s

But then when you try to use some GUI application to save something
to a file, the initial save file dialog takers ages to appear and
everything seems frozen?

If so then I feel like this may actually be some sort of problem
with your desktop environment, but then I've no idea how to narrow
that down.

Thanks,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

[toc] | [prev] | [next] | [standalone]

#264704 — Re: raid10 is killing me, and applications that aren't willing to wait for it to respond

On Wed, Dec 13, 2023 at 09:29:44PM +0000, Andy Smith wrote:
> $ dd if=/dev/zero of=/path/to/your/home/dir/zero bs=1m count=100

The above and the other dd invocation should have 'bs=1M'.

Thanks,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

[toc] | [prev] | [next] | [standalone]

#264705 — Re: raid10 is killing me, and applications that aren't willing towait for it to respond

On 12/13/23 16:30, Andy Smith wrote:
> Hello,
> 
> On Wed, Dec 13, 2023 at 10:26:19AM -0500, gene heskett wrote:
>> I thought I was doing things right a year back when I built a raid10 for my
>> /home partition. but I'm tired of fighting with it for access. Anything that
>> wants to open a file on it, is subjected to a freeze of at least 30 seconds
>> BEFORE the file requester is drawn on screen.
> 
> I haven't chimed in to any of the multiple times you've brought this
> to the list, because it's just so bizarre. I've about 20 years'
> experience of using mdadm and have never seen anything like what
> you're reporting, so I just don't know what the problem could be or
> how to find it.
> 
> The only times I've seen anything remotely like it have been when
> there's been hardware problems with failing writes, but I know
> you've been through this with the list several time sand no such low
> level issues were ever uncovered.
> 
> Would it be correct to say that you only experience these IO delays
> from GUI applications? Like, if you do a simple:
> 
> $ time echo "test" > ~/foo
> 
> does that complete in a normal time?
The lshw >lshw.txt would be in the "guiless" category, and it worked 
even quicker than if I left it to come out on the cli. Another item that 
may be of interest is that the gui is xfce4.

That was a bit over a 40k write
> 
> And if you did a bigger write, again from the command line?
> 
> $ dd if=/dev/zero of=/path/to/your/home/dir/zero bs=1m count=100
> 00+0 records in
> 100+0 records out
> 104857600 bytes (105 MB, 100 MiB) copied, 0.0783268 s, 1.3 GB/s

I had to use an uppercase M in the bs=, to get:
gene@coyote:~$ time dd if=/dev/zero of=/home/gene/zero bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.0829926 s, 1.3 GB/s

real    0m0.085s
user    0m0.005s
sys     0m0.080s
I'd have to say thats realtime. no lag.
> and with sync:
> 
> $ dd if=/dev/zero of=/path/to/your/home/dir/zero bs=1m count=100 oflag=sync
> 100+0 records in
> 100+0 records out
> 104857600 bytes (105 MB, 100 MiB) copied, 0.463356 s, 226 MB/s
> 
gene@coyote:~$ time dd if=/dev/zero of=/home/gene/zero bs=1M count=100 
oflag=sync
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.935655 s, 112 MB/s

real    0m0.940s
user    0m0.000s
sys     0m0.254s
A lot slower but still no lag, about an even second.

> But then when you try to use some GUI application to save something
> to a file, the initial save file dialog takers ages to appear and
> everything seems frozen?
Correct...

> 
> If so then I feel like this may actually be some sort of problem
> with your desktop environment, but then I've no idea how to narrow
> that down.
I think we have made progress, Andy, having narrowed it down to the gui. 
  To me that is progress.  Now I need a gui expert, which I am for sure 
not.  Never have been, never will be.
> 
> Thanks,
> Andy
> 
Thanks a bunch Andy, I think your logic was quite helpful in narrowing 
down the problem area.
Take care, stay warm and well.

Cheers, Gene Heskett.
-- 
"There are four boxes to be used in defense of liberty:
  soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
  - Louis D. Brandeis

[toc] | [prev] | [next] | [standalone]

#264717 — Re: raid10 is killing me, and applications that aren't willing towait for it to respond

On 12/13/23 15:33, gene heskett wrote:
> gene@coyote:~$ time dd if=/dev/zero of=/home/gene/zero bs=1M count=100 
> oflag=sync
> 100+0 records in
> 100+0 records out
> 104857600 bytes (105 MB, 100 MiB) copied, 0.935655 s, 112 MB/s
> 
> real    0m0.940s
> user    0m0.000s
> sys     0m0.254s


Thank you for providing a console session that confirms the issue is not 
md RAID.


For completeness, I suggest that you do both write and read benchmarks:

2023-12-13 17:56:58 root@taz ~
# smartctl -i /dev/sda | grep "Device Model"
Device Model:     INTEL SSDSC2CW060A3

2023-12-13 17:57:12 root@taz ~
# dd if=/dev/zero of=/home/dpchrist/100mb.zero bs=1M count=100 oflag=sync
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 1.6329 s, 64.2 MB/s

2023-12-13 17:57:57 root@taz ~
# free && sync && echo 3 > /proc/sys/vm/drop_caches && free
                total        used        free      shared  buff/cache 
available
Mem:        32698252     1941032    29428472      761160     1328748 
29606508
Swap:         976892           0      976892
                total        used        free      shared  buff/cache 
available
Mem:        32698252     1941516    29580404      717360     1176332 
29650836
Swap:         976892           0      976892

2023-12-13 17:58:03 root@taz ~
# dd of=/dev/null if=/home/dpchrist/100mb.zero bs=1M count=100 oflag=sync
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.263723 s, 398 MB/s


I have found that as I run computers, there is an accumulation of cruft 
over time.  The more I mess with a computer, the sooner it becomes 
unstable.   Eventually, "finding the needle in the haystack" and 
"putting Humpty Dumpty back together again" do not work -- the computer 
requires a backup-wipe-install-restore cycle.  Your posts indicate that 
your computer is overdue.


And, I suspect a deeper issue -- you have one computer that is your 
workstation, your file server, and your backup server.  This 
over-complicates everything and creates a strong disincentive to 
backup-wipe-install-restore.  I have been there, done that, lost 
service, and lost data.  Now I have several laptops/ desktops/ 
workstations, a dedicated file server, and a dedicated backup server. 
Life is good.  :-)


Again -- I suggest that you build a backup server, then build a file 
server, then rebuild the workstation.  I am confident you will be 
rewarded with simpler administration and improved reliability.


David

[toc] | [prev] | [next] | [standalone]

Page 1 of 3  [1] 2 3  Next page →

Back to top | Article view | linux.debian.user

csiph-web