Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.debian.user > #266119 > unrolled thread
| Started by | Default User <hunguponcontent@gmail.com> |
|---|---|
| First post | 2024-01-17 17:30 +0100 |
| Last post | 2024-01-19 08:10 +0100 |
| Articles | 13 on this page of 73 — 25 participants |
Back to article view | Back to linux.debian.user
rsync --delete vs rsync --delete-after Default User <hunguponcontent@gmail.com> - 2024-01-17 17:30 +0100
Re: rsync --delete vs rsync --delete-after David Christensen <dpchrist@holgerdanske.com> - 2024-01-17 18:30 +0100
Re: rsync --delete vs rsync --delete-after Keith Bainbridgge <keithrbaugroups@gmail.com> - 2024-01-18 00:10 +0100
Re: rsync --delete vs rsync --delete-after Default User <hunguponcontent@gmail.com> - 2024-01-18 02:30 +0100
Re: rsync --delete vs rsync --delete-after David Christensen <dpchrist@holgerdanske.com> - 2024-01-18 08:40 +0100
Re: rsync --delete vs rsync --delete-after Michael Kjörling <2695bd53d63c@ewoof.net> - 2024-01-18 12:10 +0100
Re: rsync --delete vs rsync --delete-after <tomas@tuxteam.de> - 2024-01-18 12:20 +0100
Re: rsync --delete vs rsync --delete-after Michael Kjörling <2695bd53d63c@ewoof.net> - 2024-01-18 12:50 +0100
Re: rsync --delete vs rsync --delete-after Michel Verdier <mv524@free.fr> - 2024-01-18 14:00 +0100
Re: rsync --delete vs rsync --delete-after Kushal Kumaran <kushal@locationd.net> - 2024-01-17 19:30 +0100
Re: rsync --delete vs rsync --delete-after Default User <hunguponcontent@gmail.com> - 2024-01-17 21:00 +0100
Re: rsync --delete vs rsync --delete-after Michael Kjörling <2695bd53d63c@ewoof.net> - 2024-01-17 21:20 +0100
Re: rsync --delete vs rsync --delete-after Andy Smith <andy@strugglers.net> - 2024-01-17 22:00 +0100
Re: rsync --delete vs rsync --delete-after Michel Verdier <mv524@free.fr> - 2024-01-18 02:40 +0100
Re: rsync --delete vs rsync --delete-after hw <hw@adminart.net> - 2024-01-18 12:20 +0100
Re: rsync --delete vs rsync --delete-after Ralph Aichinger <ra@h5.or.at> - 2024-01-18 13:50 +0100
Re: rsync --delete vs rsync --delete-after Michael Kjörling <2695bd53d63c@ewoof.net> - 2024-01-18 14:10 +0100
Re: rsync --delete vs rsync --delete-after Ralph Aichinger <ra@h5.or.at> - 2024-01-18 15:30 +0100
Re: rsync --delete vs rsync --delete-after hw <hw@adminart.net> - 2024-01-26 16:40 +0100
Re: rsync --delete vs rsync --delete-after Michael Kjörling <2695bd53d63c@ewoof.net> - 2024-01-26 17:30 +0100
Re: rsync --delete vs rsync --delete-after hw <hw@adminart.net> - 2024-01-28 18:00 +0100
Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after) Andy Smith <andy@strugglers.net> - 2024-01-26 16:20 +0100
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after) fxkl47BF@protonmail.com - 2024-01-26 16:30 +0100
Re: Home UPS recommendations Michael Kjörling <2695bd53d63c@ewoof.net> - 2024-01-26 16:40 +0100
Re: Home UPS recommendations Tixy <tixy@yxit.co.uk> - 2024-01-26 17:20 +0100
Re: Home UPS recommendations "James H. H. Lampert" <jamesl@touchtonecorp.com> - 2024-01-26 17:20 +0100
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after) Roger Price <debian@rogerprice.org> - 2024-01-26 19:10 +0100
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after) David Wright <deblis@lionunicorn.co.uk> - 2024-01-26 20:50 +0100
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after) Roger Price <debian@rogerprice.org> - 2024-01-27 11:00 +0100
Re: Home UPS recommendations Felix Miata <mrmazda@earthlink.net> - 2024-01-26 20:10 +0100
Re: Home UPS recommendations ghe2001 <ghe2001@protonmail.com> - 2024-01-26 20:30 +0100
Re: Home UPS recommendations debian-user@howorth.org.uk - 2024-01-26 21:50 +0100
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after) hw <hw@adminart.net> - 2024-01-28 19:00 +0100
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after) Andy Smith <andy@strugglers.net> - 2024-01-28 20:00 +0100
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync--delete-after) gene heskett <gheskett@shentel.net> - 2024-01-29 07:00 +0100
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after) Andy Smith <andy@strugglers.net> - 2024-02-08 16:30 +0100
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after) Charles Curley <charlescurley@charlescurley.com> - 2024-02-08 17:30 +0100
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after) Curt <curty@free.fr> - 2024-02-08 17:40 +0100
Re: Home UPS recommendations Michael Kjörling <2695bd53d63c@ewoof.net> - 2024-02-08 18:00 +0100
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after) hw <hw@adminart.net> - 2024-02-09 12:10 +0100
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after) Dan Ritter <dsr@randomstring.org> - 2024-02-09 13:10 +0100
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after) hw <hw@adminart.net> - 2024-02-09 22:40 +0100
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after) Dan Ritter <dsr@randomstring.org> - 2024-02-09 23:30 +0100
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after) "Roy J. Tellason, Sr." <roy@rtellason.com> - 2024-02-09 17:40 +0100
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after) Stefan Monnier <monnier@iro.umontreal.ca> - 2024-02-09 18:20 +0100
Re: Home UPS recommendations Felix Miata <mrmazda@earthlink.net> - 2024-02-10 01:10 +0100
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after) hw <hw@adminart.net> - 2024-02-09 22:50 +0100
Re: Home UPS recommendations (Was Re: rsync --delete vs rsync --delete-after) "Roy J. Tellason, Sr." <roy@rtellason.com> - 2024-02-11 19:10 +0100
Re: Home UPS recommendations Felix Miata <mrmazda@earthlink.net> - 2024-02-09 18:20 +0100
Re: Home UPS recommendations debian-user@howorth.org.uk - 2024-02-09 21:40 +0100
Re: Home UPS recommendations hw <hw@adminart.net> - 2024-02-09 22:50 +0100
Re: Home UPS recommendations Felix Miata <mrmazda@earthlink.net> - 2024-02-10 01:00 +0100
Re: Home UPS recommendations hw <hw@adminart.net> - 2024-02-10 03:20 +0100
Re: Home UPS recommendations Felix Miata <mrmazda@earthlink.net> - 2024-02-10 04:30 +0100
Re: Home UPS recommendations hw <hw@adminart.net> - 2024-02-10 11:10 +0100
Re: Home UPS recommendations Felix Miata <mrmazda@earthlink.net> - 2024-02-10 15:00 +0100
Re: Home UPS recommendations hw <hw@adminart.net> - 2024-02-10 16:50 +0100
Re: Home UPS recommendations Joe <joe@jretrading.com> - 2024-02-10 19:50 +0100
Re: Home UPS recommendations gene heskett <gheskett@shentel.net> - 2024-02-10 22:50 +0100
Re: Home UPS recommendations hw <hw@adminart.net> - 2024-02-11 02:00 +0100
Re: rsync --delete vs rsync --delete-after hw <hw@adminart.net> - 2024-01-26 16:20 +0100
Re: Data and hardware protection measures; was: rsync --delete vs rsync --delete-after Michael Kjörling <2695bd53d63c@ewoof.net> - 2024-01-26 17:00 +0100
Re: Data and hardware protection measures; was: rsync --delete vs rsync --delete-after hw <hw@adminart.net> - 2024-01-28 19:30 +0100
Re: Data and hardware protection measures Michael Kjörling <2695bd53d63c@ewoof.net> - 2024-01-28 20:30 +0100
Re: Data and hardware protection measures Felix Miata <mrmazda@earthlink.net> - 2024-01-28 21:50 +0100
Re: rsync --delete vs rsync --delete-after Ralph Aichinger <ra@h5.or.at> - 2024-01-27 14:10 +0100
Re: rsync --delete vs rsync --delete-after Stefan Monnier <monnier@iro.umontreal.ca> - 2024-01-18 05:20 +0100
Re: rsync --delete vs rsync --delete-after Andy Smith <andy@strugglers.net> - 2024-01-18 15:40 +0100
Re: rsync --delete vs rsync --delete-after Stefan Monnier <monnier@iro.umontreal.ca> - 2024-01-18 16:10 +0100
Re: rsync --delete vs rsync --delete-after Michel Verdier <mv524@free.fr> - 2024-01-18 22:10 +0100
Re: rsync --delete vs rsync --delete-after Andy Smith <andy@strugglers.net> - 2024-01-18 22:50 +0100
Re: rsync --delete vs rsync --delete-after Default User <hunguponcontent@gmail.com> - 2024-01-19 03:40 +0100
Re: rsync --delete vs rsync --delete-after Michel Verdier <mv524@free.fr> - 2024-01-19 08:10 +0100
Page 4 of 4 — ← Prev page 1 2 3 [4]
| From | hw <hw@adminart.net> |
|---|---|
| Date | 2024-01-26 16:20 +0100 |
| Message-ID | <I0w9P-5IWo-5@gated-at.bofh.it> |
| In reply to | #266191 |
On Thu, 2024-01-18 at 13:26 +0100, Ralph Aichinger wrote: > Hello fellow Debian users, > > On Thu, 2024-01-18 at 12:18 +0100, hw wrote: > > > Always use an UPS. > > > Here I have a somewhat contrarian view, I hope not to offend too much: It's not offending, you merely have a different opinion. > For countries with stable electricity supplies (like Austria where I > live) having a small UPS might actually lead to more problems instead > of less, unless you are putting a lot of effort into it. Very often > have I had problems with UPSes, e.g. batteries dying, the UPS going > into some self test mode and inadvertedly shutting down, etc. I've never had issues with any UPS due to self tests. The batteries need to be replaced when they are worn out. How often that is required depends on the UPS and the conditions it is working in, usually every 3--5 years. I rather spend the money on new batteries (EUR 40 last time after 5 years) every couple years rather than spending thousands on replacing the hardware when a power surge damages it which could have been prevented by the UPS, and it's better to have the machines shut down properly rather taking risks with potential data loss, regardless of file systems and RAID setups in use. The hardware is usually extremely difficult --- and may be impossible --- to replace. On top of that, noone pays for the time and effort lost, and even someone did, I'd rather keep my hardware instead of going to all the lengths required to replace it. Loosing hours of work due to power outages can also be an issue, and nobody pays for that, either. > I've had no external power outage in the last 5 or 10 years, but a UPS > often needs at least one battery replacement during that time. Outages are (still) rare here, but it suffices to trigger a fuse or the main switch when some device shorts out, or someone working on the solar power systems some of the neighbours have, causing crazy voltage fluctuations, or a lightning strike somewhere in the vinicity or whatever reason for an UPS to be required. Way too many times than I could count my UPSs saved me and the hardware from power supply problems, so I can't recommend to go without one. It doesn't matter how stable you think the power grid in your country is. > [...] > Here I also doubt if this is a wise suggestion for the typical home > or small office user. RAID leads to lots and lots of complexity, that > is often not needed in a home setup. RAID isn't as complicated as you think. Hardware RAID is most simple, followed by btrfs, followed by mdadm. With hardware RAID I can instruct someone who has no idea what they're doing to replace a failed disk remotely. Same goes for btrfs and mdadm, though it better be someone who isn't entirely clueless. More importantly, the hassle involved in trying to recover from a failed disk is ridiculously enormous without RAID and can get expensive when hours of work were lost. With RAID, you don't even notice unless you keep an eye on it, and when a disk has failed, you simply order a replacement and plug it in. It's not like you could go to a hardware store around the corner and get a new disk same or next day. Even if you have a store around, they will need to order the disk, and that can, these days, take weeks or months or longer if it's a small store. The only way to get a replacement is ordering online, which requires that your computer still works and that you have access to your passwords. > I'd rather have a working backup setup with many independent copies > before I even start thinking about RAID. Yes, disks can fail, but > data loss often is due to user error and malware. That is simply wrong. RAID doesn't protect you from malware, and nothing protects you from user error. If you have data losses from malware and/or user error more often than from failed disks, you're doing something majorly wrong. > RAID helps very little with the latter two causes of data loss. And > all too often have I seen people mess up their complicated RAID > setups, because they pulled the wrong disk when another one broke, > or because they misinterpreted complicated error messages, creating > unnecessary data loss out of user error by themselves. This shows that you have no experience with RAID and is not an argument. Making backups is way more complicated than RAID. You can way more easily overwrite the wrong backup or misinterpret error messages of your backup solution than you can pull the wrong disk from a RAID or misinterpret error messages from your RAID. How exactly would you pull the wrong disk from a RAID and thus cause data loss? Before you pull one, you make a backup. When the disk has been pulled, its contents remain unchanged and when you put it back in, your data is still there --- plus you have the backup. Sure it can sometimes be difficult to tell which disk you need to replace, and it's not an issue because you can always figure out which one you need to replace. You can always tell with a good hardware RAID because it will indicate on the trays which disk has failed and the controller tells you. > As a home/SOHO user, I'd rather have a working backup every few hours > or every day than some RAID10 wonder that makes me lose more time on > reading RAID documentation, and ordering spare drives (you've got > one of those spares for each array, do you?) than is actually lost by > not being able to restore to the exact last minute before a hard disk > died. No, I generally don't have spares, and I don't leave my backup server running all the time to make backups every few hours or every day because electricity is way too expensive, plus it's somewhat loud and gives off quite a bit of heat. How often do you verify that you can actually restore everything from your backups, and how do you do that? Yes, my setup is far from ideal when it comes to backups in that I should make backups more frequently. That doesn't mean I shouldn't have good backups and that UPSs and RAID were not required.
[toc] | [prev] | [next] | [standalone]
| From | Michael Kjörling <2695bd53d63c@ewoof.net> |
|---|---|
| Date | 2024-01-26 17:00 +0100 |
| Subject | Re: Data and hardware protection measures; was: rsync --delete vs rsync --delete-after |
| Message-ID | <I0wMx-5Jaf-19@gated-at.bofh.it> |
| In reply to | #266627 |
On 26 Jan 2024 16:11 +0100, from hw@adminart.net (hw): > I rather spend the money on new batteries (EUR 40 last time after 5 > years) every couple years [...] > > The hardware is usually extremely difficult --- and may be impossible > --- to replace. And let's not forget that you can _plan_ to perform the battery replacement for whenever that is convenient. Which is quite the contrast to a lightning strike blowing out even _just_ the PSU and it needing replacement before you can even use the computer again (and you _hope_ that nothing more took a hit, which it probably did even if the computer _seems_ to be working fine). >> I've had no external power outage in the last 5 or 10 years, but a UPS >> often needs at least one battery replacement during that time. > > Outages are (still) rare here, but it suffices to trigger a fuse or > the main switch when some device shorts out, or someone working on the > solar power systems some of the neighbours have, causing crazy voltage > fluctuations, or a lightning strike somewhere in the vinicity or > whatever reason for an UPS to be required. It's also worth talking to your local electrician about installing an incoming-mains overvoltage protection for lightning protection. I won't quote prices because I had mine installed a good while ago and also did it together with some other electrical work, but I was surprised at how low the cost for that was, and I _know_ that it has saved me on at least one occasion. It won't do power conditioning or power loss protection of course, but it _does_ greatly increase the odds that your home wiring survives a lightning-related voltage surge. (Nothing will realistically protect you against a _direct_ lightning strike; in that case the very best you can hope for is damage containment.) > More importantly, the hassle involved in trying to recover from a > failed disk is ridiculously enormous without RAID and can get > expensive when hours of work were lost. With RAID, you don't even > notice unless you keep an eye on it, and when a disk has failed, you > simply order a replacement and plug it in. Indeed; the point of RAID is uptime. > You can always tell with a good hardware RAID because it > will indicate on the trays which disk has failed and the controller > tells you. Or you can label the physical disks. Whenever I replace a disk, I print a label with the WWN of the new disk and place it so that it is readable without removing any disks or cabling; then I use the WWN to identify the disk in software; in both cases because the WWN is a stable identifier that I can fully expect will never change throughout the disk's lifetime. So when the system tells me that wwn-0x123456789abcdef0 is having issues, I can quickly and accurately identify the exact physical device that needs replacement once I have a replacement on hand. And if the kernel logs are telling me that, say, sdg is having issues, I can map that back to whatever WWN happens to map to that identifier at that particular time. (In practice, I'm more likely to get useful error details through ZFS status monitoring tools, where I already use the WWN, so I likely won't need to go that somewhat circuitous route.) > Yes, my setup is far from ideal when it comes to backups in that I > should make backups more frequently. That doesn't mean I shouldn't > have good backups and that UPSs and RAID were not required. Or that, again, they solve different problems. -- Michael Kjörling 🔗 https://michael.kjorling.se “Remember when, on the Internet, nobody cared that you were a dog?”
[toc] | [prev] | [next] | [standalone]
| From | hw <hw@adminart.net> |
|---|---|
| Date | 2024-01-28 19:30 +0100 |
| Subject | Re: Data and hardware protection measures; was: rsync --delete vs rsync --delete-after |
| Message-ID | <I1i4N-6ckc-7@gated-at.bofh.it> |
| In reply to | #266633 |
On Fri, 2024-01-26 at 15:56 +0000, Michael Kjörling wrote: > On 26 Jan 2024 16:11 +0100, from hw@adminart.net (hw): > > I rather spend the money on new batteries (EUR 40 last time after 5 > > years) every couple years [...] To comment myself, I think was 3 years, not 5, sorry. > > The hardware is usually extremely difficult --- and may be impossible > > --- to replace. > > And let's not forget that you can _plan_ to perform the battery > replacement for whenever that is convenient. How do you know in advance when the battery will have failed? > Which is quite the contrast to a lightning strike blowing out even > _just_ the PSU and it needing replacement before you can even use > the computer again (and you _hope_ that nothing more took a hit, > which it probably did even if the computer _seems_ to be working > fine). It would also hit the display(s), the switches and through that everything that's connected to the network, the server(s) ... That adds up to a lot of money. > [...] > It's also worth talking to your local electrician about installing an > incoming-mains overvoltage protection for lightning protection. I > won't quote prices because I had mine installed a good while ago and > also did it together with some other electrical work, but I was > surprised at how low the cost for that was, and I _know_ that it has > saved me on at least one occasion. Hm I thought it's expensive. I'll ask when I get a chance. > [...] > > You can always tell with a good hardware RAID because it > > will indicate on the trays which disk has failed and the controller > > tells you. > > Or you can label the physical disks. Whenever I replace a disk, I > print a label with the WWN of the new disk and place it so that it is > readable without removing any disks or cabling; That doesn't exactly help when the failed disk has disappeared altogether, as if it had been removed ;) But then, you can go by the numbers of the disks you can still see. And beware of SSDs; when they fail, they're usually entirely inaccessible whereas you may be still able to resuce (some) data from a spinning disk after it failed. It's probably really bad with mainbaords that use M2 storage since apparently, they seem to support only one (of the some type at least) rather than two. So you can't use those at all. What's the point of that? ZFS cache maybe?
[toc] | [prev] | [next] | [standalone]
| From | Michael Kjörling <2695bd53d63c@ewoof.net> |
|---|---|
| Date | 2024-01-28 20:30 +0100 |
| Subject | Re: Data and hardware protection measures |
| Message-ID | <I1j0R-6cSH-3@gated-at.bofh.it> |
| In reply to | #266684 |
On 28 Jan 2024 19:19 +0100, from hw@adminart.net (hw): > On Fri, 2024-01-26 at 15:56 +0000, Michael Kjörling wrote: >> On 26 Jan 2024 16:11 +0100, from hw@adminart.net (hw): >>> I rather spend the money on new batteries (EUR 40 last time after 5 >>> years) every couple years [...] > > To comment myself, I think was 3 years, not 5, sorry. > >>> The hardware is usually extremely difficult --- and may be impossible >>> --- to replace. >> >> And let's not forget that you can _plan_ to perform the battery >> replacement for whenever that is convenient. > > How do you know in advance when the battery will have failed? You replace the battery before it fails completely. Most batteries don't go from perfectly fine to completely dead within one charge cycle. If the battery drains completely during a power outage before the UPS has a chance to respond to the battery's loss of capacity, that becomes a (hopefully clean) power cut, which _still_ is _a lot_ better than equipment which isn't designed to deal with a significant overvoltage condition taking the brunt of a lightning strike. I'm assuming, of course, that you replace the battery with one of the same chemistry. The UPS will probably assume some discharge characteristic depending on what battery type the OEM uses (lead acid, NiCd, NiMH, LiIon, ...); of course if you give the UPS a battery using some other chemistry, that'll immediately wreak havoc with lots of things. >> Which is quite the contrast to a lightning strike blowing out even >> _just_ the PSU and it needing replacement before you can even use >> the computer again (and you _hope_ that nothing more took a hit, >> which it probably did even if the computer _seems_ to be working >> fine). > > It would also hit the display(s), the switches and through that > everything that's connected to the network, the server(s) ... That > adds up to a lot of money. Which is why I said "even _just_ the PSU", emphasis original. >> It's also worth talking to your local electrician about installing an >> incoming-mains overvoltage protection for lightning protection. > > Hm I thought it's expensive. So did I until I actually asked someone who could give me a quote for actually installing it. > That doesn't exactly help when the failed disk has disappeared > altogether, as if it had been removed ;) If that happens, I'd get output along the lines of: # zpool status pool: tank state: DEGRADED scan: scrub repaired <n>B in <amount of time> with <n> errors on <date and time> config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 wwn-0x0000000000000001-crypt ONLINE 0 0 0 8446744073709551616 UNAVAIL 0 0 0 was /dev/mapper/wwn-0x1111111111111113-crypt wwn-0x2222222222222225-crypt ONLINE 0 0 0 wwn-0x3333333333333337-crypt ONLINE 0 0 0 wwn-0x4444444444444449-crypt ONLINE 0 0 0 wwn-0x555555555555555b-crypt ONLINE 0 0 0 clearly identifying the problem. And also most likely a lot of event notifications telling me that wwn-0x1111111111111113-crypt is having issues within the "tank" pool, plus any applicable kernel logs for the device disconnection and perhaps lower-level I/O errors. Similarly, if a storage device suddenly starts returning garbage, that will show up likely as CKSUM errors and the device will eventually get kicked out of the pool, showing as state FAILED with large error counter values. (zpool status would also provide some more explanatory details, in the example above including that "applications are unaffected" because sufficient redundancy would still exist; but I'm eliding those here because I don't have them handy and don't feel like creating such a situation just to get example output. The important part is that the disk that dropped off the bus will show as likely UNAVAIL with its internal identifier and a reference to its WWN because of my naming scheme, instead of as completely missing. Solution is to get a replacement disk, plug it in, execute "sudo zpool replace tank $numeric_id $new_device_path", and wait a while, all the while I can still use the system normally.) No matter what kind of storage solution you're using - hardware RAID, software RAID, no redundancy, whichever - or how you're doing backups (assuming that you are, for some value of "you"), you can't just ignore issues with it. That way lies data loss. -- Michael Kjörling 🔗 https://michael.kjorling.se “Remember when, on the Internet, nobody cared that you were a dog?”
[toc] | [prev] | [next] | [standalone]
| From | Felix Miata <mrmazda@earthlink.net> |
|---|---|
| Date | 2024-01-28 21:50 +0100 |
| Subject | Re: Data and hardware protection measures |
| Message-ID | <I1kgh-6dyZ-1@gated-at.bofh.it> |
| In reply to | #266688 |
Michael Kjörling composed on 2024-01-28 19:23 (UTC): > On 28 Jan 2024 19:19 +0100, from hw@adminart.net (hw): >> On Fri, 2024-01-26 at 15:56 +0000, Michael Kjörling wrote: >>> It's also worth talking to your local electrician about installing an >>> incoming-mains overvoltage protection for lightning protection. >> Hm I thought it's expensive. > So did I until I actually asked someone who could give me a quote for > actually installing it. Old construction used meter "boxes" that don't support accessories like those. My utility won't touch mine unless I pay an electrician for an expensive full service upgrade. -- Evolution as taught in public schools is, like religion, based on faith, not based on science. Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata
[toc] | [prev] | [next] | [standalone]
| From | Ralph Aichinger <ra@h5.or.at> |
|---|---|
| Date | 2024-01-27 14:10 +0100 |
| Message-ID | <I0QBA-5VRQ-31@gated-at.bofh.it> |
| In reply to | #266627 |
On Fri, 2024-01-26 at 16:11 +0100, hw wrote: > I've never had issues with any UPS due to self tests. The batteries > need to be replaced when they are worn out. How often that is > required depends on the UPS and the conditions it is working in, > usually every 3--5 years. It was with some small to mid APC model, I think. We had about 1 to 2kW worth of servers on it, so it was not that small, definitely no consumer type. When I took over maintenance somebody had configured some sort of weekly or biweekly self-test, that switched over to battery, was supposed to run the battery down to 25% or similar, and then return to mains power/charging. Except once what the UPS considered 25% charge seemingly was not, and everything shut down instantly. > I rather spend the money on new batteries (EUR 40 last time after 5 > years) every couple years rather than spending thousands on replacing > the hardware when a power surge damages it which could have been > prevented by the UPS, and it's better to have the machines shut down > properly rather taking risks with potential data loss, regardless of > file systems and RAID setups in use. I think having hardware for "thousands" and having a UPS with that cheap batteries is not that common. In above company we certainly had hardware for thousands, but changing batteries cost hundreds of Euros, even with off-brand aftermarket parts. It also was complicated to order the right parts etc. > RAID isn't as complicated as you think. Hardware RAID is most > simple, > followed by btrfs, followed by mdadm. I have to disagree with that too. Some hardware RAIDs might be simple, but others are not. Tracking down the rebrandings of Adaptec, aquisitions and mergers, is a science by itself. As is finding and installing their Firmware and utilites. Are they still calles Avago, or something new again? Or all that BBU stuff: Tracking the state of battery backup units on the controller, and ordering and replacing the correct battery is also not really easy. Clearly enterprise IT type of stuff, keeping even knowledgeable people busy for hours, if you don't do it at scale and regularily. Also often Linux support is problematic. Yes, it will work, but sometimes certain utilities are not available or work as good as with Windows. On the other hand mdadm software RAID is well documented and painless. > > With hardware RAID I can instruct someone who has no idea what > they're > doing to replace a failed disk remotely. Same goes for btrfs and > mdadm, though it better be someone who isn't entirely clueless In fact this was my job for some time: Administering hardware RAID equipped servers, and instructing "remote hands" or customers to swap harddisks. It was not always easy, not always were the correct disks pulled, even though it was correctly labelled. Sometimes clueless people tried swapping by themselves, mixing stuff up. We also had one server with wrong labelling, for whatever reason. That was no fun ;) Now I won't dispute that RAID has its place in data centers and many other applications. I just doubt that it is the correct choice for many home users. > More importantly, the hassle involved in trying to recover from a > failed disk is ridiculously enormous without RAID and can get > expensive when hours of work were lost. With RAID, you don't even > notice unless you keep an eye on it, and when a disk has failed, you > simply order a replacement and plug it in. Yes, that can happen. But more often than not the scenario is like it is with most notebooks today. You send your notebook in for repair, and have to reinstall anyway. Happened to me. I backed up my Debian system, sent the device in for hardware repair, got it back with Windows 10 ;) And no, it was not the disk that was broken, but the touchpad. > > It's not like you could go to a hardware store around the corner and > get a new disk same or next day. Even if you have a store around, > they will need to order the disk, and that can, these days, take > weeks > or months or longer if it's a small store. For consumer hard disks? I just go to my favourite shop if I need a replacement, and they've got maybe 20 or 30 types of hard disk in stock, to be bought right away. Even more with SSDs. And I am in a smallish city, pop. 250.000. > That is simply wrong. RAID doesn't protect you from malware, and > nothing protects you from user error. If you have data losses from > malware and/or user error more often than from failed disks, you're > doing something majorly wrong. In my experience user error is the main source of data loss. By far. > This shows that you have no experience with RAID and is not an > argument. I've got years of experience with RAID, both in my personal use and with employers doing stuff on RAID for customers and internal services. In my experience RAID is a nice solution for data center type setups. RAID often is problematic for home users or even small offices. > Making backups is way more complicated than RAID. You can way more > easily overwrite the wrong backup or misinterpret error messages of > your backup solution than you can pull the wrong disk from a RAID or > misinterpret error messages from your RAID. Yes. Making backups is hard. But my main point is: You need good backups anyway. So RAID does not help you here. If you neglect backups because you've got RAID, you are living dangerously. And some people actually do this. I think it is wrong and asking for problems. > How exactly would you pull the wrong disk from a RAID and thus cause > data loss? Before you pull one, you make a backup. Often people with RAID setups do not do this (making a full backup before a disk swap) for various reasons. > When the disk has > been pulled, its contents remain unchanged and when you put it back > in, your data is still there --- plus you have the backup. Sure it > can sometimes be difficult to tell which disk you need to replace, > and it's not an issue because you can always figure out which one you > need to replace. You can always tell with a good hardware RAID > because it will indicate on the trays which disk has failed and the > controller tells you. I've seen (well not seen, they were on the other end of the phone) people delay disk swaps in RAID until not one but two (of 5 or 6) physical drives were broken. Then they were instructed to pull e.g. drive 2 and 3, pulled the wrong ones (e.g. 1 and 2), they tried to correct their error by pulling another one, and boom. Yes, in theory data probably were still there and could have been restored with forensic techniques. But by that time the array was offline, and it was major repair/data rescue/restore backup time. > No, I generally don't have spares, and I don't leave my backup server > running all the time to make backups every few hours or every day > because electricity is way too expensive, plus it's somewhat loud and > gives off quite a bit of heat. This is the personal compromise that I make. > How often do you verify that you can actually restore everything from > your backups, and how do you do that? Well I don't regularily try to restore "everything", but as some stuff breaks regularily, most often my laptop, I get to try out replacing systems once in a while. The last disk I had fail on me was an SSD in my Proxmox server at home, I did a clean install of Proxmox, restored guest backups, worked like a charm. Finding a replacement disk was the hardest part. Also when migrating to new hardware, it is a good time to test your backups. When I moved my Home Assistant from one hardware to a new type of setup on a different hardware, I implicitly tested the backups. Worked flawlessly. /ralph
[toc] | [prev] | [next] | [standalone]
| From | Stefan Monnier <monnier@iro.umontreal.ca> |
|---|---|
| Date | 2024-01-18 05:20 +0100 |
| Message-ID | <HXs2J-3QiC-1@gated-at.bofh.it> |
| In reply to | #266119 |
> However, I have read that using rsync --delete instead of rsync --
> delete-after is faster and uses less memory, and so is more efficient.
I'd be surprised if it makes a significant difference.
Stefan
[toc] | [prev] | [next] | [standalone]
| From | Andy Smith <andy@strugglers.net> |
|---|---|
| Date | 2024-01-18 15:40 +0100 |
| Message-ID | <HXBIJ-3W5l-7@gated-at.bofh.it> |
| In reply to | #266165 |
Hello, On Wed, Jan 17, 2024 at 11:09:35PM -0500, Stefan Monnier wrote: > > However, I have read that using rsync --delete instead of rsync -- > > delete-after is faster and uses less memory, and so is more efficient. > > I'd be surprised if it makes a significant difference. If you use --delete-after (and some other options) then rsync has to check every file before it can do any work, whereas normally it will find a few files to work on and start work, meanwhile incrementally scanning for more. Each file that it's working on uses 100 bytes of memory, so that's about 100MB for each million files. Not a huge amount of memory either way so probably won't cause an issue. It would be the thing about not starting any work until having fully scanned the file tree. It's all covered in the man page where it talks about those options. Thanks, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting
[toc] | [prev] | [next] | [standalone]
| From | Stefan Monnier <monnier@iro.umontreal.ca> |
|---|---|
| Date | 2024-01-18 16:10 +0100 |
| Message-ID | <HXCbM-3WuI-3@gated-at.bofh.it> |
| In reply to | #266200 |
>> > However, I have read that using rsync --delete instead of rsync --
>> > delete-after is faster and uses less memory, and so is more efficient.
>> I'd be surprised if it makes a significant difference.
> If you use --delete-after (and some other options) then rsync has to
> check every file before it can do any work,
Oh, right, I was thinking of `--delete-delay`.
[ Tho, as you mention, the performance impact of `--delete-after` is
also negligible in many cases. ]
Stefan
[toc] | [prev] | [next] | [standalone]
| From | Michel Verdier <mv524@free.fr> |
|---|---|
| Date | 2024-01-18 22:10 +0100 |
| Message-ID | <HXHO9-3ZQb-1@gated-at.bofh.it> |
| In reply to | #266200 |
On 2024-01-18, Andy Smith wrote: > If you use --delete-after (and some other options) then rsync has to > check every file before it can do any work, whereas normally it will > find a few files to work on and start work, meanwhile incrementally > scanning for more. Not sure of that. rsync always gets a list of files on both sides and compares them before doing anything. But --delete-after forces a second pass on the target for deletion instead of doing it during copying. So a bit more IO.
[toc] | [prev] | [next] | [standalone]
| From | Andy Smith <andy@strugglers.net> |
|---|---|
| Date | 2024-01-18 22:50 +0100 |
| Message-ID | <HXIqR-402L-1@gated-at.bofh.it> |
| In reply to | #266233 |
Hello,
On Thu, Jan 18, 2024 at 10:01:46PM +0100, Michel Verdier wrote:
> On 2024-01-18, Andy Smith wrote:
> > If you use --delete-after (and some other options) then rsync has to
> > check every file before it can do any work, whereas normally it will
> > find a few files to work on and start work, meanwhile incrementally
> > scanning for more.
>
> Not sure of that.
Could check the man page then like I said.
Some options require rsync to know the full file list, so these
options disable the incremental recursion mode. These include:
--delete-before, --delete-after, --prune-empty-dirs, and
--delay-updates.
Thanks,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
[toc] | [prev] | [next] | [standalone]
| From | Default User <hunguponcontent@gmail.com> |
|---|---|
| Date | 2024-01-19 03:40 +0100 |
| Message-ID | <HXMXv-42Kz-11@gated-at.bofh.it> |
| In reply to | #266236 |
On Thu, 2024-01-18 at 21:44 +0000, Andy Smith wrote: > Hello, > > On Thu, Jan 18, 2024 at 10:01:46PM +0100, Michel Verdier wrote: > > On 2024-01-18, Andy Smith wrote: > > > If you use --delete-after (and some other options) then rsync has > > > to > > > check every file before it can do any work, whereas normally it > > > will > > > find a few files to work on and start work, meanwhile > > > incrementally > > > scanning for more. > > > > Not sure of that. > > Could check the man page then like I said. > > Some options require rsync to know the full file list, so these > options disable the incremental recursion mode. These include: > --delete-before, --delete-after, --prune-empty-dirs, and > --delay-updates. > > Thanks, > Andy > Hi, OP here. Just an update. Last night I did the usual copy from the primary backup drive to the secondary backup drive, except I did it using rsync --delete instead of rsync --delete-after. This time, it took 73 minutes. The previous time it took 131 minutes. So it did go significantly quicker. And "seemed" to go okay. Thank you to all who have weighed in on this topic!
[toc] | [prev] | [next] | [standalone]
| From | Michel Verdier <mv524@free.fr> |
|---|---|
| Date | 2024-01-19 08:10 +0100 |
| Message-ID | <HXRaN-45wo-1@gated-at.bofh.it> |
| In reply to | #266236 |
On 2024-01-18, Andy Smith wrote: > Could check the man page then like I said. > > Some options require rsync to know the full file list, so these > options disable the incremental recursion mode. These include: > --delete-before, --delete-after, --prune-empty-dirs, and > --delay-updates. You are right, thanks !
[toc] | [prev] | [standalone]
Page 4 of 4 — ← Prev page 1 2 3 [4]
Back to top | Article view | linux.debian.user
csiph-web