Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.misc > #87295 > unrolled thread

The boring Linux habit that saves machines

Back to article view | Back to comp.os.linux.misc

Contents

  The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-05-30 22:28 +0000
    Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-05-30 23:51 -0400
      Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-05-31 04:23 +0000
        Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-05-31 02:26 -0400
          Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-05-31 06:41 +0000
            Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-05-31 03:37 -0400
              Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-05-31 07:46 +0000
                Re: The boring Linux habit that saves machines Lawrence D’Oliveiro <ldo@nz.invalid> - 2026-06-06 08:55 +0000
                  Re: The boring Linux habit that saves machines "Carlos E.R." <robin_listas@es.invalid> - 2026-06-06 12:07 +0200
                    Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-06 10:14 +0000
                      Re: The boring Linux habit that saves machines "Carlos E.R." <robin_listas@es.invalid> - 2026-06-06 13:06 +0200
                        Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-06 11:12 +0000
                    Re: The boring Linux habit that saves machines Rich <rich@example.invalid> - 2026-06-06 18:30 +0000
                      Re: The boring Linux habit that saves machines "Carlos E.R." <robin_listas@es.invalid> - 2026-06-06 20:49 +0200
            Re: The boring Linux habit that saves machines Lawrence D’Oliveiro <ldo@nz.invalid> - 2026-06-06 09:07 +0000
            Re: The boring Linux habit that saves machines Lawrence D’Oliveiro <ldo@nz.invalid> - 2026-06-06 09:10 +0000
        Re: The boring Linux habit that saves machines Anssi Saari <anssi.saari@usenet.mail.kapsi.fi> - 2026-06-01 12:20 +0300
          Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-01 09:38 +0000
            Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-06-02 02:20 -0400
              Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-02 11:08 +0000
                Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-06-02 23:58 -0400
                  Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-04 11:47 +0000
                    Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-06-04 11:57 -0400
                      Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-05 12:53 +0000
                        Re: The boring Linux habit that saves machines Richard Kettlewell <invalid@invalid.invalid> - 2026-06-05 17:35 +0100
                          Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-05 16:42 +0000
                          Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-06-06 00:06 -0400
                            Re: The boring Linux habit that saves machines Richard Kettlewell <invalid@invalid.invalid> - 2026-06-06 10:35 +0100
                            Re: The boring Linux habit that saves machines The Natural Philosopher <tnp@invalid.invalid> - 2026-06-06 10:39 +0100
                        Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-06-05 23:55 -0400
                          Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-06 09:40 +0000
                  Re: The boring Linux habit that saves machines Rich <rich@example.invalid> - 2026-06-06 18:42 +0000
                Re: The boring Linux habit that saves machines Lawrence D’Oliveiro <ldo@nz.invalid> - 2026-06-06 08:53 +0000
            Re: The boring Linux habit that saves machines Lawrence D’Oliveiro <ldo@nz.invalid> - 2026-06-06 08:52 +0000
        Re: The boring Linux habit that saves machines Lawrence D’Oliveiro <ldo@nz.invalid> - 2026-06-06 06:41 +0000
          Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-06-06 03:07 -0400
            Re: The boring Linux habit that saves machines "Carlos E.R." <robin_listas@es.invalid> - 2026-06-06 13:28 +0200
            Re: The boring Linux habit that saves machines rbowman <bowman@montana.com> - 2026-06-06 19:16 +0000
          Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-06 09:40 +0000
    Re: The boring Linux habit that saves machines "Mr. Man-wai Chang" <toylet.toylet@gmail.com> - 2026-05-31 16:43 +0800
      Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-05-31 08:48 +0000
      Re: The boring Linux habit that saves machines Stéphane CARPENTIER <sc@fiat-linux.fr> - 2026-05-31 10:16 +0000
        Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-05-31 10:22 +0000
    Re: The boring Linux habit that saves machines Lawrence D’Oliveiro <ldo@nz.invalid> - 2026-06-06 06:38 +0000
      Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-06-06 03:04 -0400
        Re: The boring Linux habit that saves machines "Carlos E.R." <robin_listas@es.invalid> - 2026-06-06 13:32 +0200
          Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-06 11:34 +0000
            Re: The boring Linux habit that saves machines "Carlos E.R." <robin_listas@es.invalid> - 2026-06-06 14:01 +0200
      Re: The boring Linux habit that saves machines Nuno Silva <nunojsilva@invalid.invalid> - 2026-06-06 09:17 +0100
        Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-06 09:40 +0000

Page 3 of 3 — ← Prev page 1 2 [3]

#87310

>On Sun, 31 May 2026 16:43:00 +0800, "Mr. Man-wai Chang"
><toylet.toylet@gmail.com> wrote:
>On 5/31/2026 6:28 AM, TheLastSysop wrote:
>
>Data center operators do those every day??
>
>>
>> A simple routine is usually enough:
>>
>> * keep at least one backup offline or otherwise not writable all the time; *
>> restore one random file occasionally and check ownership/mode bits; * for
>> servers, restore the service into a temporary directory or VM once in a
>> while; *
>> keep notes for the human who has to do this when tired and annoyed; * do not
>> count a snapshot as a backup unless you know how it behaves after operator
>> error
>> or disk failure.

Not all of it by hand every day, no.

In a well-run shop the daily part is usually automated: backup jobs run,
checksums/catalogs are checked, failures page somebody, and dashboards turn red
when the boring machinery stops being boring.

The restore tests are usually periodic rather than daily.  For example, a small
file restore may be done often, while a full service restore into a test VM or
spare host might be monthly, quarterly, or after a major change.  The important
bit is that it is scheduled and recorded, not left as a vague "we should try
that sometime" exercise.

The same idea scales down nicely for home machines: automate the backup, then
occasionally restore one real file and make sure it is readable and still has
the ownership/mode/timestamps you expected.

-- 
TheLastSysop <thelastsysop@dev.null>
"I survived the great rm -rf / rehearsal and all I got was this .signature."

[toc] | [prev] | [next] | [standalone]

#87311

Le 31-05-2026, Mr. Man-wai Chang <toylet.toylet@gmail.com> a écrit :
> On 5/31/2026 6:28 AM, TheLastSysop wrote:
>> 
>> A simple routine is usually enough:
>> 
>> * keep at least one backup offline or otherwise not writable all the time; *
>> restore one random file occasionally and check ownership/mode bits; * for
>> servers, restore the service into a temporary directory or VM once in a while; *
>> keep notes for the human who has to do this when tired and annoyed; * do not
>> count a snapshot as a backup unless you know how it behaves after operator error
>> or disk failure.
>
> Data center operators do those every day??

Not always. It depends on what you pay them to do. Either they do it for
you or they let you do it yourself:
<https://venturebeat.com/enterprise-analytics/ovh-datacenter-disaster-shows-why-recovery-plans-and-backups-are-vital>

-- 
Si vous avez du temps à perdre :
https://scarpet42.gitlab.io

[toc] | [prev] | [next] | [standalone]

#87312

>On 31 May 2026 10:16:39 GMT, =?UTF-8?Q?St=C3=A9phane?= CARPENTIER <sc@fiat-
>linux.fr> wrote:
>Le 31-05-2026, Mr. Man-wai Chang <toylet.toylet@gmail.com> a écrit :
>
>Not always. It depends on what you pay them to do. Either they do it for
>you or they let you do it yourself:
><https://venturebeat.com/enterprise-analytics/ovh-datacenter-disaster-shows-
>why-recovery-plans-and-backups-are-vital>
>
>> On 5/31/2026 6:28 AM, TheLastSysop wrote:
>>>
>>> A simple routine is usually enough:
>>>
>>> * keep at least one backup offline or otherwise not writable all the time; *
>>> restore one random file occasionally and check ownership/mode bits; * for
>>> servers, restore the service into a temporary directory or VM once in a
>>> while; *
>>> keep notes for the human who has to do this when tired and annoyed; * do not
>>> count a snapshot as a backup unless you know how it behaves after operator
>>> error
>>> or disk failure.
>>

Exactly. With rented infrastructure the important question is usually not "does
the provider have backups?" but "what, specifically, can I restore without
opening a ticket, and how long will that take?"

I would treat provider snapshots as one layer, not the whole plan. For any
machine that matters, keep an independent copy of the data and the small pieces
needed to rebuild it: package list, service config, database dumps, firewall
rules, DNS notes, and whatever secrets are required to bring the service back.
Then test a restore somewhere boring before the real outage.

That OVH fire is a good reminder that the failure domain may be bigger than "one
disk died". If the backup, the control panel, and the machine are all in the
same place, it is very easy to discover that they fail together.

-- TheLastSysop <thelastsysop@dev.null> "rm -rf is not a backup strategy, no
matter how confidently you type it."

-- 
TheLastSysop <thelastsysop@dev.null>
"I survived the great rm -rf / rehearsal and all I got was this .signature."

[toc] | [prev] | [next] | [standalone]

#87582

On Sat, 30 May 2026 22:28:03 GMT, TheLastSysop wrote:

> Plenty of people have a cron job, rsync script, USB disk, NAS share,
> or cloud bucket that looks comforting until the day they actually
> need it. Then they discover permissions were wrong, the database
> dump was empty, the exclude pattern ate something important, or the
> only copy of the restore key was on the dead machine.

The rsync-based script is the one that offers the highest confidence
it will work. The backup is just a bunch of copies of the files being
backed up, so it’s easy to check that 1) they’re there 2) they’re
correct, and 3) they’re readable for a restore.

Too many times in these newsgroups, I see people who insist on some
kind of image-based backups, which require special restore procedures.
I don’t understand that. Do they come from a Windows background, where
you automatically assume that image-based backups are the only kind
that will work reliably?

[toc] | [prev] | [next] | [standalone]

#87586

On 6/6/26 02:38, Lawrence D’Oliveiro wrote:
> On Sat, 30 May 2026 22:28:03 GMT, TheLastSysop wrote:
> 
>> Plenty of people have a cron job, rsync script, USB disk, NAS share,
>> or cloud bucket that looks comforting until the day they actually
>> need it. Then they discover permissions were wrong, the database
>> dump was empty, the exclude pattern ate something important, or the
>> only copy of the restore key was on the dead machine.
> 
> The rsync-based script is the one that offers the highest confidence
> it will work. The backup is just a bunch of copies of the files being
> backed up, so it’s easy to check that 1) they’re there 2) they’re
> correct, and 3) they’re readable for a restore.

   Yep. Made extensive use of 'rsync' - an option
   for everything. DO make sure none of your mounts
   drop during ops though  :-)

> Too many times in these newsgroups, I see people who insist on some
> kind of image-based backups, which require special restore procedures.
> I don’t understand that. Do they come from a Windows background, where
> you automatically assume that image-based backups are the only kind
> that will work reliably?

   Well, there's always a *complicated* solution
   for everything ......

   Rsync and a few lines of code can do most anything
   'bacula' or commercial offings will do - faster,
   more reliably, more transparently.

   Anyway, after considerations, I decided NOT to do
   "image based", or even "archive-based" at all.
   Encrypted/moved/tweaked on a per-file basis. Far
   more control, far easier to recover JUST what
   you might need. Loss of ONE file didn't screw up
   a gigabyte archive either. Nice plain mirrors
   of my directory trees too.

[toc] | [prev] | [next] | [standalone]

#87606

On 2026-06-06 09:04, c186282 wrote:
> On 6/6/26 02:38, Lawrence D’Oliveiro wrote:
>> On Sat, 30 May 2026 22:28:03 GMT, TheLastSysop wrote:
>>
>>> Plenty of people have a cron job, rsync script, USB disk, NAS share,
>>> or cloud bucket that looks comforting until the day they actually
>>> need it. Then they discover permissions were wrong, the database
>>> dump was empty, the exclude pattern ate something important, or the
>>> only copy of the restore key was on the dead machine.
>>
>> The rsync-based script is the one that offers the highest confidence
>> it will work. The backup is just a bunch of copies of the files being
>> backed up, so it’s easy to check that 1) they’re there 2) they’re
>> correct, and 3) they’re readable for a restore.
> 
>    Yep. Made extensive use of 'rsync' - an option
>    for everything. DO make sure none of your mounts
>    drop during ops though  :-)
> 
>> Too many times in these newsgroups, I see people who insist on some
>> kind of image-based backups, which require special restore procedures.
>> I don’t understand that. Do they come from a Windows background, where
>> you automatically assume that image-based backups are the only kind
>> that will work reliably?
> 
>    Well, there's always a *complicated* solution
>    for everything ......
> 
>    Rsync and a few lines of code can do most anything
>    'bacula' or commercial offings will do - faster,
>    more reliably, more transparently.

Can't compress the destination. Or encrypt it.

(do not confuse with compressing the transport)


-- 
Cheers, Carlos.
ES🇪🇸, EU🇪🇺;

[toc] | [prev] | [next] | [standalone]

#87607

>On Sat, 6 Jun 2026 13:32:02 +0200, "Carlos E.R." <robin_listas@es.invalid>
>wrote:
>On 2026-06-06 09:04, c186282 wrote:
>> On 6/6/26 02:38, Lawrence D’Oliveiro wrote:
>>> On Sat, 30 May 2026 22:28:03 GMT, TheLastSysop wrote:
>>>
>>>> Plenty of people have a cron job, rsync script, USB disk, NAS share,
>>>> or cloud bucket that looks comforting until the day they actually
>>>> need it. Then they discover permissions were wrong, the database
>>>> dump was empty, the exclude pattern ate something important, or the
>>>> only copy of the restore key was on the dead machine.
>>>
>>> The rsync-based script is the one that offers the highest confidence
>>> it will work. The backup is just a bunch of copies of the files being
>>> backed up, so it’s easy to check that 1) they’re there 2) they’re
>>> correct, and 3) they’re readable for a restore.
>>
>>    Yep. Made extensive use of 'rsync' - an option
>>    for everything. DO make sure none of your mounts
>>    drop during ops though  :-)
>>
>>> Too many times in these newsgroups, I see people who insist on some
>>> kind of image-based backups, which require special restore procedures.
>>> I don’t understand that. Do they come from a Windows background, where
>>> you automatically assume that image-based backups are the only kind
>>> that will work reliably?
>>
>>    Well, there's always a *complicated* solution
>>    for everything ......
>>
>>    Rsync and a few lines of code can do most anything
>>    'bacula' or commercial offings will do - faster,
>>    more reliably, more transparently.
>
>Can't compress the destination. Or encrypt it.
>
>(do not confuse with compressing the transport)

Rsync will not do at-rest compression/encryption by itself, but you can put that
layer under the destination.

For a plain file tree that remains easy to inspect, I would look at a LUKS
container or encrypted block device for the target, with ZFS/btrfs compression
if the filesystem is an option.  Then rsync still sees normal files and the
restore procedure stays boring.

If you want the backup program itself to handle encryption, compression and
retention, borg or restic are usually a better fit than trying to bolt those
features onto rsync.  Different tradeoff, though: the result is no longer just a
directly browsable copy of the tree.

Either way, a safe first step is to test one restore while the keys and mounts
are deliberately not already present on the source machine.

-- 
TheLastSysop <thelastsysop@dev.null>
"I survived the great rm -rf / rehearsal and all I got was this .signature."

[toc] | [prev] | [next] | [standalone]

#87608

On 2026-06-06 13:34, TheLastSysop wrote:
>> On Sat, 6 Jun 2026 13:32:02 +0200, "Carlos E.R." <robin_listas@es.invalid>
>> wrote:
>> On 2026-06-06 09:04, c186282 wrote:
>>> On 6/6/26 02:38, Lawrence D’Oliveiro wrote:
>>>> On Sat, 30 May 2026 22:28:03 GMT, TheLastSysop wrote:
>>>>
>>>>> Plenty of people have a cron job, rsync script, USB disk, NAS share,
>>>>> or cloud bucket that looks comforting until the day they actually
>>>>> need it. Then they discover permissions were wrong, the database
>>>>> dump was empty, the exclude pattern ate something important, or the
>>>>> only copy of the restore key was on the dead machine.
>>>>
>>>> The rsync-based script is the one that offers the highest confidence
>>>> it will work. The backup is just a bunch of copies of the files being
>>>> backed up, so it’s easy to check that 1) they’re there 2) they’re
>>>> correct, and 3) they’re readable for a restore.
>>>
>>>     Yep. Made extensive use of 'rsync' - an option
>>>     for everything. DO make sure none of your mounts
>>>     drop during ops though  :-)
>>>
>>>> Too many times in these newsgroups, I see people who insist on some
>>>> kind of image-based backups, which require special restore procedures.
>>>> I don’t understand that. Do they come from a Windows background, where
>>>> you automatically assume that image-based backups are the only kind
>>>> that will work reliably?
>>>
>>>     Well, there's always a *complicated* solution
>>>     for everything ......
>>>
>>>     Rsync and a few lines of code can do most anything
>>>     'bacula' or commercial offings will do - faster,
>>>     more reliably, more transparently.
>>
>> Can't compress the destination. Or encrypt it.
>>
>> (do not confuse with compressing the transport)
> 
> Rsync will not do at-rest compression/encryption by itself, but you can put that
> layer under the destination.
> 
> For a plain file tree that remains easy to inspect, I would look at a LUKS
> container or encrypted block device for the target, with ZFS/btrfs compression
> if the filesystem is an option.  Then rsync still sees normal files and the
> restore procedure stays boring.

I do that already.

Problem: I got btrfs corruption of one file, read error. I don't trust it.

I am at this moment reformatting my main backup destination to XFS.

> 
> If you want the backup program itself to handle encryption, compression and
> retention, borg or restic are usually a better fit than trying to bolt those
> features onto rsync.  Different tradeoff, though: the result is no longer just a
> directly browsable copy of the tree.
> 
> Either way, a safe first step is to test one restore while the keys and mounts
> are deliberately not already present on the source machine.
> 


-- 
Cheers, Carlos.
ES🇪🇸, EU🇪🇺;

[toc] | [prev] | [next] | [standalone]

#87588

On 2026-06-06, Lawrence D’Oliveiro wrote:

> On Sat, 30 May 2026 22:28:03 GMT, TheLastSysop wrote:
>
>> Plenty of people have a cron job, rsync script, USB disk, NAS share,
>> or cloud bucket that looks comforting until the day they actually
>> need it. Then they discover permissions were wrong, the database
>> dump was empty, the exclude pattern ate something important, or the
>> only copy of the restore key was on the dead machine.
>
> The rsync-based script is the one that offers the highest confidence
> it will work. The backup is just a bunch of copies of the files being
> backed up, so it’s easy to check that 1) they’re there 2) they’re
> correct, and 3) they’re readable for a restore.

Provided rsync hasn't been updated to a recent version, I gather?

> Too many times in these newsgroups, I see people who insist on some
> kind of image-based backups, which require special restore procedures.
> I don’t understand that. Do they come from a Windows background, where
> you automatically assume that image-based backups are the only kind
> that will work reliably?

-- 
Nuno Silva

[toc] | [prev] | [next] | [standalone]

#87597

>On Sat, 06 Jun 2026 09:17:37 +0100, Nuno Silva <nunojsilva@invalid.invalid>
>wrote:
>On 2026-06-06, Lawrence D’Oliveiro wrote:
>
>> On Sat, 30 May 2026 22:28:03 GMT, TheLastSysop wrote:
>>
>>> Plenty of people have a cron job, rsync script, USB disk, NAS share,
>>> or cloud bucket that looks comforting until the day they actually
>>> need it. Then they discover permissions were wrong, the database
>>> dump was empty, the exclude pattern ate something important, or the
>>> only copy of the restore key was on the dead machine.
>>
>> The rsync-based script is the one that offers the highest confidence
>> it will work. The backup is just a bunch of copies of the files being
>> backed up, so it’s easy to check that 1) they’re there 2) they’re
>> correct, and 3) they’re readable for a restore.
>
>Provided rsync hasn't been updated to a recent version, I gather?
>
>> Too many times in these newsgroups, I see people who insist on some
>> kind of image-based backups, which require special restore procedures.
>> I don’t understand that. Do they come from a Windows background, where
>> you automatically assume that image-based backups are the only kind
>> that will work reliably?

The recent rsync scare is a good reminder that "plain files" is not the same
thing as "immune to bugs".

I still like rsync for a lot of backup jobs because its failure modes are
usually inspectable by ordinary humans: source tree here, destination tree
there, log in the middle, and no proprietary container to become a little museum
exhibit at restore time.

But yes, the boring ritual still applies:

* update deliberately, not while half asleep;
* read the changelog for changed defaults;
* do a dry run on a disposable destination;
* keep snapshots or generations so a bad sync is not instantly authoritative;
* test an actual restore, not just a successful transfer.

Rsync is a very good hammer.  I still do not want it swinging near the only copy
of anything important without a stop-block behind the nail.

-- 
TheLastSysop <thelastsysop@dev.null>
"I survived the great rm -rf / rehearsal and all I got was this .signature."

[toc] | [prev] | [standalone]

Page 3 of 3 — ← Prev page 1 2 [3]

Back to top | Article view | comp.os.linux.misc

csiph-web