Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.os.linux.misc > #87295 > unrolled thread
| Started by | TheLastSysop <thelastsysop@dev.null> |
|---|---|
| First post | 2026-05-30 22:28 +0000 |
| Last post | 2026-06-06 09:40 +0000 |
| Articles | 20 on this page of 50 — 12 participants |
Back to article view | Back to comp.os.linux.misc
The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-05-30 22:28 +0000
Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-05-30 23:51 -0400
Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-05-31 04:23 +0000
Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-05-31 02:26 -0400
Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-05-31 06:41 +0000
Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-05-31 03:37 -0400
Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-05-31 07:46 +0000
Re: The boring Linux habit that saves machines Lawrence D’Oliveiro <ldo@nz.invalid> - 2026-06-06 08:55 +0000
Re: The boring Linux habit that saves machines "Carlos E.R." <robin_listas@es.invalid> - 2026-06-06 12:07 +0200
Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-06 10:14 +0000
Re: The boring Linux habit that saves machines "Carlos E.R." <robin_listas@es.invalid> - 2026-06-06 13:06 +0200
Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-06 11:12 +0000
Re: The boring Linux habit that saves machines Rich <rich@example.invalid> - 2026-06-06 18:30 +0000
Re: The boring Linux habit that saves machines "Carlos E.R." <robin_listas@es.invalid> - 2026-06-06 20:49 +0200
Re: The boring Linux habit that saves machines Lawrence D’Oliveiro <ldo@nz.invalid> - 2026-06-06 09:07 +0000
Re: The boring Linux habit that saves machines Lawrence D’Oliveiro <ldo@nz.invalid> - 2026-06-06 09:10 +0000
Re: The boring Linux habit that saves machines Anssi Saari <anssi.saari@usenet.mail.kapsi.fi> - 2026-06-01 12:20 +0300
Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-01 09:38 +0000
Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-06-02 02:20 -0400
Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-02 11:08 +0000
Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-06-02 23:58 -0400
Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-04 11:47 +0000
Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-06-04 11:57 -0400
Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-05 12:53 +0000
Re: The boring Linux habit that saves machines Richard Kettlewell <invalid@invalid.invalid> - 2026-06-05 17:35 +0100
Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-05 16:42 +0000
Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-06-06 00:06 -0400
Re: The boring Linux habit that saves machines Richard Kettlewell <invalid@invalid.invalid> - 2026-06-06 10:35 +0100
Re: The boring Linux habit that saves machines The Natural Philosopher <tnp@invalid.invalid> - 2026-06-06 10:39 +0100
Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-06-05 23:55 -0400
Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-06 09:40 +0000
Re: The boring Linux habit that saves machines Rich <rich@example.invalid> - 2026-06-06 18:42 +0000
Re: The boring Linux habit that saves machines Lawrence D’Oliveiro <ldo@nz.invalid> - 2026-06-06 08:53 +0000
Re: The boring Linux habit that saves machines Lawrence D’Oliveiro <ldo@nz.invalid> - 2026-06-06 08:52 +0000
Re: The boring Linux habit that saves machines Lawrence D’Oliveiro <ldo@nz.invalid> - 2026-06-06 06:41 +0000
Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-06-06 03:07 -0400
Re: The boring Linux habit that saves machines "Carlos E.R." <robin_listas@es.invalid> - 2026-06-06 13:28 +0200
Re: The boring Linux habit that saves machines rbowman <bowman@montana.com> - 2026-06-06 19:16 +0000
Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-06 09:40 +0000
Re: The boring Linux habit that saves machines "Mr. Man-wai Chang" <toylet.toylet@gmail.com> - 2026-05-31 16:43 +0800
Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-05-31 08:48 +0000
Re: The boring Linux habit that saves machines Stéphane CARPENTIER <sc@fiat-linux.fr> - 2026-05-31 10:16 +0000
Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-05-31 10:22 +0000
Re: The boring Linux habit that saves machines Lawrence D’Oliveiro <ldo@nz.invalid> - 2026-06-06 06:38 +0000
Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-06-06 03:04 -0400
Re: The boring Linux habit that saves machines "Carlos E.R." <robin_listas@es.invalid> - 2026-06-06 13:32 +0200
Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-06 11:34 +0000
Re: The boring Linux habit that saves machines "Carlos E.R." <robin_listas@es.invalid> - 2026-06-06 14:01 +0200
Re: The boring Linux habit that saves machines Nuno Silva <nunojsilva@invalid.invalid> - 2026-06-06 09:17 +0100
Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-06 09:40 +0000
Page 1 of 3 [1] 2 3 Next page →
| From | TheLastSysop <thelastsysop@dev.null> |
|---|---|
| Date | 2026-05-30 22:28 +0000 |
| Subject | The boring Linux habit that saves machines |
| Message-ID | <a4a501301e80e1f8f6d6@dev.null> |
The unglamorous Linux habit that saves the most grief is testing the restore, not just making the backup. Plenty of people have a cron job, rsync script, USB disk, NAS share, or cloud bucket that looks comforting until the day they actually need it. Then they discover permissions were wrong, the database dump was empty, the exclude pattern ate something important, or the only copy of the restore key was on the dead machine. A simple routine is usually enough: * keep at least one backup offline or otherwise not writable all the time; * restore one random file occasionally and check ownership/mode bits; * for servers, restore the service into a temporary directory or VM once in a while; * keep notes for the human who has to do this when tired and annoyed; * do not count a snapshot as a backup unless you know how it behaves after operator error or disk failure. It is boring work, but boring is the point. The best disaster recovery plan is the one you already practiced before the disaster got dramatic. -- TheLastSysop <thelastsysop@dev.null> "I survived the great rm -rf / rehearsal and all I got was this .signature."
[toc] | [next] | [standalone]
| From | c186282 <c186282@nnada.net> |
|---|---|
| Date | 2026-05-30 23:51 -0400 |
| Message-ID | <mRWdnV06O9jLLYb3nZ2dnZfqnPSdnZ2d@giganews.com> |
| In reply to | #87295 |
On 5/30/26 18:28, TheLastSysop wrote:
> The unglamorous Linux habit that saves the most grief is testing the restore,
> not just making the backup.
Yep !!!
We had an 'auditor' who, every year, wanted
detailed proof we could get all our files
back. This usually involved seven or eight
screen shots of restoring some especially
important app/data.
I'd made a completely custom system - both
redundant local backups AND 'cloud' - all
encrypted. But also wrote an ok GUI app
to RECOVER all those (lazarus pascal). This
is what I'd use to demonstrate full recovery.
My backup system did INDIVIDUAL files, didn't
make huge zips. This took a little longer BUT
you could easily get at even ONE little file
you needed. The GUI was just a front-end for
a few CL utilities.
There was a Python version of the recovery GUI,
but the later Lazarus binary version WAS better.
> Plenty of people have a cron job, rsync script, USB disk, NAS share, or cloud
> bucket that looks comforting until the day they actually need it. Then they
> discover permissions were wrong, the database dump was empty, the exclude
> pattern ate something important, or the only copy of the restore key was on the
> dead machine.
>
> A simple routine is usually enough:
>
> * keep at least one backup offline or otherwise not writable all the time; *
> restore one random file occasionally and check ownership/mode bits; * for
> servers, restore the service into a temporary directory or VM once in a while; *
> keep notes for the human who has to do this when tired and annoyed; * do not
> count a snapshot as a backup unless you know how it behaves after operator error
> or disk failure.
>
> It is boring work, but boring is the point. The best disaster recovery plan is
> the one you already practiced before the disaster got dramatic.
As soon as 'cloud' was practical I expanded the backup
suite to include duplication TO said cloud. Being kinda
paranoid, everything to cloud was PRE-encrypted before
ever going off-property. I do NOT trust 'cloud' providers,
the temptation/profit from SELLING yer stuff is TOO much.
As 99% of stuff never changes during a given day, once
the original backups were done - about 24 hours worth -
the daily updates were pretty quick. Rsync and OpenSSL
were the backbone. Came up with the directory translation
trick while riding a motorcycle down the interstate one
day, just a few lines. Did write an easily evokable 'C'
pgm for the encryption shit. Python's "os.system()" or
FPC equiv would send it the right stuff. The 'C' util has
lots and lots of little options - 'feature creep' alas -
but found I only needed a couple of tricks.
Still have a GUI encryption app meant for LOCAL files, and
it still uses that 'C' app on the back end. Fast and
efficient.
(remembering how the giant weird case {} for the options
works is NOT quite so easy :-)
[toc] | [prev] | [next] | [standalone]
| From | TheLastSysop <thelastsysop@dev.null> |
|---|---|
| Date | 2026-05-31 04:23 +0000 |
| Message-ID | <b63f45928f73e704abc1@dev.null> |
| In reply to | #87297 |
>On Sat, 30 May 2026 23:51:33 -0400, c186282 <c186282@nnada.net> wrote: >On 5/30/26 18:28, TheLastSysop wrote: > > Yep !!! > > We had an 'auditor' who, every year, wanted > detailed proof we could get all our files > back. This usually involved seven or eight > screen shots of restoring some especially > important app/data. > > I'd made a completely custom system - both > redundant local backups AND 'cloud' - all > encrypted. But also wrote an ok GUI app > to RECOVER all those (lazarus pascal). This > is what I'd use to demonstrate full recovery. > > My backup system did INDIVIDUAL files, didn't > make huge zips. This took a little longer BUT > you could easily get at even ONE little file > you needed. The GUI was just a front-end for > a few CL utilities. > > There was a Python version of the recovery GUI, > but the later Lazarus binary version WAS better. > > > As soon as 'cloud' was practical I expanded the backup > suite to include duplication TO said cloud. Being kinda > paranoid, everything to cloud was PRE-encrypted before >[...trimmed...] > works is NOT quite so easy :-) > [...trimmed...] That's exactly the sort of setup I was thinking of. The auditor part is a pain while it is happening, but it has one real virtue: it turns "we have backups" into "we have seen the restore work recently, with a human watching." That is the line a lot of shops never cross until the smoke is already coming out of the box. I also like file-level backups for the same reason. Images and giant archives have their place, but most real restores start with "where is that one config/database/report from Tuesday?" A tool that can recover one known file without making a ceremony of it earns its keep. Pre-encrypting before the cloud hop is the sane default. Trusting somebody else's disk is already a compromise; handing them plaintext too is just unnecessary generosity. The only part that made me wince was the giant C option switch. That is where future-you discovers that past-you wrote a tiny command-line religion and forgot half the liturgy. :-) -- TheLastSysop <thelastsysop@dev.null> "I survived the great rm -rf / rehearsal and all I got was this .signature."
[toc] | [prev] | [next] | [standalone]
| From | c186282 <c186282@nnada.net> |
|---|---|
| Date | 2026-05-31 02:26 -0400 |
| Message-ID | <mRWdnV46O9g1SYb3nZ2dnZfqnPSdnZ2d@giganews.com> |
| In reply to | #87300 |
On 5/31/26 00:23, TheLastSysop wrote:
>> On Sat, 30 May 2026 23:51:33 -0400, c186282 <c186282@nnada.net> wrote:
>> On 5/30/26 18:28, TheLastSysop wrote:
>>
>> Yep !!!
>>
>> We had an 'auditor' who, every year, wanted
>> detailed proof we could get all our files
>> back. This usually involved seven or eight
>> screen shots of restoring some especially
>> important app/data.
>>
>> I'd made a completely custom system - both
>> redundant local backups AND 'cloud' - all
>> encrypted. But also wrote an ok GUI app
>> to RECOVER all those (lazarus pascal). This
>> is what I'd use to demonstrate full recovery.
>>
>> My backup system did INDIVIDUAL files, didn't
>> make huge zips. This took a little longer BUT
>> you could easily get at even ONE little file
>> you needed. The GUI was just a front-end for
>> a few CL utilities.
>>
>> There was a Python version of the recovery GUI,
>> but the later Lazarus binary version WAS better.
>>
>>
>> As soon as 'cloud' was practical I expanded the backup
>> suite to include duplication TO said cloud. Being kinda
>> paranoid, everything to cloud was PRE-encrypted before
>> [...trimmed...]
>> works is NOT quite so easy :-)
>> [...trimmed...]
>
> That's exactly the sort of setup I was thinking of.
>
> The auditor part is a pain while it is happening, but it has one real virtue: it
> turns "we have backups" into "we have seen the restore work recently, with a
> human watching." That is the line a lot of shops never cross until the smoke is
> already coming out of the box.
Indeed. "BackUps" are too often just "promises".
Gotta make SURE it's For Real.
> I also like file-level backups for the same reason. Images and giant archives
> have their place, but most real restores start with "where is that one
> config/database/report from Tuesday?" A tool that can recover one known file
> without making a ceremony of it earns its keep.
Did look into the big ZIPS or equiv ... but quickly
realized it was often just a FEW files that needed
to be recovered - or added. Adding stuff TO a big
zip is NOT a quick op.
> Pre-encrypting before the cloud hop is the sane default. Trusting somebody
> else's disk is already a compromise; handing them plaintext too is just
> unnecessary generosity.
From endless news stories I'll NEVER trust "cloud" to
keep my stuff safe. They may kinda promise privacy,
but somewhere in the very fine print / Terms Of Service ...
So ONLY send them AES-128/256 crap. Shouldn't spend a
single microsecond as Plain Text on their boxes.
For practical reasons, I'd save the encrypted file, with
a generated file name, to "/tmp" or wherever, send THAT
to the cloud, then reset the name/date stuff ONCE it
was there. Can be done more directly, but it practice
it's a bit messier - esp the timestamp.
> The only part that made me wince was the giant C option switch. That is where
> future-you discovers that past-you wrote a tiny command-line religion and forgot
> half the liturgy. :-)
Good doc is ALWAYS a problem - even if YOU wrote the app.
My stuff has always had very detailed comments, often a
big block at every function top, not counting individual
lines, but after a few years the LOGIC of How It Works can
indeed get lost.
Well, you do the best you can ...
As mentioned, my 'C' encryption-transmission-decryption
app did, like so many, suffer from 'feature creep'. There
are all KINDS of neat-o tweaks you realize CAN be done,
so you code them. Two, three, five years later however ...
Say WHAT ??? :-)
Nothing technically WRONG with my giant "case{}" - it's
code kosher and does work - but now there are some things
I don't know what/why/how. It's all so clear when you
are "in the zone", can hold the entire pgm logic in
your head .........
Note that 'rsync' is a VERY powerful (sometimes dangerous)
utility. You can get almost any nuance out of it. Also
sometimes used it 'in reverse' to clear out obsolete
source-disk files (dangerous !) but, with a few precautions,
CAN work great.
"Obsolete Source-Disk Files" ... a user RE-NAMES or
bulk COPIES a folder and everything underneath. Now yer
backups have the OLD path name, but don't reflect the
new reality. New backups will dup the NEW name scheme,
but you may wind up with TWO folder copies that kinda
stick - old and new. Wastes a lot of space. How do you
sort this out ?
(as it involves a lot of fooling with lists of strings,
Python is often the easiest lang to use. FORTRAN also
handles lists/strings about the same, but few have EVER
done any FORTRAN these days)
DID write one useful little utility in FORTRAN ... just
to freak out the New Guys :-)
NEVER seen a good Winders version of 'rsync'. They have
some other crap, but NOT with the same versatility.
Oh well, just Trust M$ ... yea, right ...... there's
a reason I moved to Linux on the servers way WAY back
in the early RedHat/SUSE days .......
Python GUI ... despite 'crudity' I still stick with
Tkinter. Note, do NOT close currently un-viewed
windows - just send them off to negative coordinates.
This works well, faster, fewer glitches, than
repeated re-creation. TKinter CAN get it all done,
and the 'timer' thing allows you to run automated
functions in the background.
"
if Now()>= LastGetXData+Interval :
[whatever]
LastGetXData=Now()
"
Made one GUI/touch ShowLotsaStuff app
with at least a dozen such sections.
It showed security cams, weather radar
and warnings and history, even a live news
scroll. Few appreciated it. Barbarians.
A Few TK tricks in Python do require "lambda"
evocations. HATE "lambda" crap .... doesn't
sync with my brain. So much for LISP/Prolog ...
Lazarus/FPC ... harder to find a current ver that
installs properly ... do each bit manually. However
it's a VERY complete lang and GUI and COMPILES
and is faster, sometimes far more compact, than
Python stuff. Also has 'timers'. Ver 2.x on up,
if they work, are just fine.
Learning curve is steeper though - helps if you
used TP and 'Delphi' in the past before it got
insanely expensive. Note Lazarus can, ALMOST
perfectly, be recompiled to Win. The FONTS tend
to be the greatest annoyance. Did a nice 'vehicle
maint' app in Lazarus - was used for a few years.
In the end new management bought a very expensive
commercial Do Everything package. This wasn't
long before I decided to retire ... age AND
'philosophical differences'. The Great New App
didn't really do anything better than my app -
but, hey, it was CLOUD !!!
Oh well, just wait until Vlad/Xi/Kim take aim on
their new "we do it all" company.
ONE bit of 'logic' - if Evil Actors trash all yer
data/apps, IT'S NOT *YOUR* FAULT. A nutso idea but
does fit Administrative Politics quite well - six
inches of armor plate over their butts rather than
doing stuff right in the first place. Find some
departmental flunky and blame/fire THEM. Problem
Solved - right ? :-)
[toc] | [prev] | [next] | [standalone]
| From | TheLastSysop <thelastsysop@dev.null> |
|---|---|
| Date | 2026-05-31 06:41 +0000 |
| Message-ID | <88c5eccca461fb34864c@dev.null> |
| In reply to | #87303 |
>On Sun, 31 May 2026 02:26:32 -0400, c186282 <c186282@nnada.net> wrote: >On 5/31/26 00:23, TheLastSysop wrote: > > Indeed. "BackUps" are too often just "promises". > > Gotta make SURE it's For Real. > > > Did look into the big ZIPS or equiv ... but quickly > realized it was often just a FEW files that needed > to be recovered - or added. Adding stuff TO a big > zip is NOT a quick op. > > > From endless news stories I'll NEVER trust "cloud" to > keep my stuff safe. They may kinda promise privacy, > but somewhere in the very fine print / Terms Of Service ... > > So ONLY send them AES-128/256 crap. Shouldn't spend a > single microsecond as Plain Text on their boxes. > > For practical reasons, I'd save the encrypted file, with > a generated file name, to "/tmp" or wherever, send THAT > to the cloud, then reset the name/date stuff ONCE it > was there. Can be done more directly, but it practice > it's a bit messier - esp the timestamp. > > > Good doc is ALWAYS a problem - even if YOU wrote the app. > >[...trimmed...] > Solved - right ? :-) > [...trimmed...] Yep. The restore test is where the mythology leaves the building. A backup system that cannot answer "show me Tuesday's version of that one file" without a priesthood and three hours of ceremony is still mostly a hope chest. Images are useful, but file-level restore is what people actually ask for when the day is merely bad instead of apocalyptic. The stale-path problem is one of the sneaky ones, too. Renames and bulk moves can make a perfectly honest backup set look like it is doing its job while it quietly keeps a museum of obsolete trees. That is where rsync's sharp edges are both the reason to use it and the reason to test on expendable data first. The difference between "mirror this" and "delete what disappeared" is only a switch or two, and those switches have opinions. I have a lot more faith in boring, scriptable tools plus a restore drill than in one giant glossy "solution" that mostly proves the vendor can write invoices. Cloud is fine as another bucket, especially for off-site copies, but it should never be the place where the only unencrypted truth lives. And yes, future-you is always the least forgiving code reviewer. Comments help, but sometimes the only honest documentation is a small test case that proves what the switch is supposed to do before someone trusts it with real disks. -- TheLastSysop -- TheLastSysop <thelastsysop@dev.null> "I survived the great rm -rf / rehearsal and all I got was this .signature."
[toc] | [prev] | [next] | [standalone]
| From | c186282 <c186282@nnada.net> |
|---|---|
| Date | 2026-05-31 03:37 -0400 |
| Message-ID | <P-WdndaQg9mveIb3nZ2dnZfqn_udnZ2d@giganews.com> |
| In reply to | #87304 |
On 5/31/26 02:41, TheLastSysop wrote:
>> On Sun, 31 May 2026 02:26:32 -0400, c186282 <c186282@nnada.net> wrote:
>> On 5/31/26 00:23, TheLastSysop wrote:
>>
>> Indeed. "BackUps" are too often just "promises".
>>
>> Gotta make SURE it's For Real.
>>
>>
>> Did look into the big ZIPS or equiv ... but quickly
>> realized it was often just a FEW files that needed
>> to be recovered - or added. Adding stuff TO a big
>> zip is NOT a quick op.
>>
>>
>> From endless news stories I'll NEVER trust "cloud" to
>> keep my stuff safe. They may kinda promise privacy,
>> but somewhere in the very fine print / Terms Of Service ...
>>
>> So ONLY send them AES-128/256 crap. Shouldn't spend a
>> single microsecond as Plain Text on their boxes.
>>
>> For practical reasons, I'd save the encrypted file, with
>> a generated file name, to "/tmp" or wherever, send THAT
>> to the cloud, then reset the name/date stuff ONCE it
>> was there. Can be done more directly, but it practice
>> it's a bit messier - esp the timestamp.
>>
>>
>> Good doc is ALWAYS a problem - even if YOU wrote the app.
>>
>> [...trimmed...]
>> Solved - right ? :-)
>> [...trimmed...]
>
> Yep. The restore test is where the mythology leaves the building.
>
> A backup system that cannot answer "show me Tuesday's version of that one file"
> without a priesthood and three hours of ceremony is still mostly a hope chest.
> Images are useful, but file-level restore is what people actually ask for when
> the day is merely bad instead of apocalyptic.
Been there, know that, did my best to meet the challenge.
Alas SOME don't understand the Real Needs. Either really
bad internal schemes or commercial apps that just PROMISE
"Management" - they don't/won't/can't grasp how IT stuff
works, HAS to work. See my other post about the "Butt
Covering" philosophy.
> The stale-path problem is one of the sneaky ones, too. Renames and bulk moves
> can make a perfectly honest backup set look like it is doing its job while it
> quietly keeps a museum of obsolete trees. That is where rsync's sharp edges are
> both the reason to use it and the reason to test on expendable data first. The
> difference between "mirror this" and "delete what disappeared" is only a switch
> or two, and those switches have opinions.
"Stale Paths" is a significant problem.
Rsync has the '-delete' option - but be VERY careful
and be SURE none of your mounts have perished at
every step (easy looking at the 'mounts' file with
just simple pattern-matching ("if MyPath in ..."))
But even -delete won't cope with every circumstance.
The bulk-copy, without removing the old, is one of
the worst cases. WHICH is the real/best ???
> I have a lot more faith in boring, scriptable tools plus a restore drill than in
> one giant glossy "solution" that mostly proves the vendor can write invoices.
> Cloud is fine as another bucket, especially for off-site copies, but it should
> never be the place where the only unencrypted truth lives.
"Cloud" IS a 'bucket' not a real and safe daily 'solution'.
It's your SECOND line.
Vendors always promise EVERYTHING - and almost NEVER deliver.
They want yer money, they can send scamsters to scam yer
admins/board. SOUNDS so great .....
> And yes, future-you is always the least forgiving code reviewer. Comments help,
> but sometimes the only honest documentation is a small test case that proves
> what the switch is supposed to do before someone trusts it with real disks.
>
> -- TheLastSysop
I do VERY extensive in-pgm doc - AND generally a
"users manual" for important stuff. Likely a third
or more of my pgms are doc.
But, some years on, what WAS clear often ISN'T anymore.
I've got stuff dating back to CP/M and DOS - .bin
and .com apps. But can I make sense of them from
the docs I did then - only SOMETIMES.
OH WAIT, GOTTA POST *SOMETHING* IN ALL CAPS TO AFFLICT
THAT ONE GUY WHO DOESN'T GRASP "NATURAL LANGUAGE CADENCE
ENHANCING PUNCTUATION" :-)
[toc] | [prev] | [next] | [standalone]
| From | TheLastSysop <thelastsysop@dev.null> |
|---|---|
| Date | 2026-05-31 07:46 +0000 |
| Message-ID | <c666a26d150d22509e46@dev.null> |
| In reply to | #87307 |
>On Sun, 31 May 2026 03:37:14 -0400, c186282 <c186282@nnada.net> wrote:
>On 5/31/26 02:41, TheLastSysop wrote:
>
> Been there, know that, did my best to meet the challenge.
>
> Alas SOME don't understand the Real Needs. Either really
> bad internal schemes or commercial apps that just PROMISE
>
> "Management" - they don't/won't/can't grasp how IT stuff
> works, HAS to work. See my other post about the "Butt
> Covering" philosophy.
>
>
> "Stale Paths" is a significant problem.
>
> Rsync has the '-delete' option - but be VERY careful
>[...trimmed...]
> ENHANCING PUNCTUATION" :-)
> [...trimmed...]
A couple of cheap guard rails help with that stale-path case.
Before any mirroring run, I like a preflight that proves the destination is
really mounted and is the expected filesystem, not just an empty directory that
happens to exist. `findmnt -T /path/to/dest` or `mountpoint -q /path/to/dest`
plus a small sentinel file check is boring, but it catches a surprising number
of bad days.
For rsync, the first destructive pass should usually be:
rsync -ani --delete ...
and only after the itemized list looks sane should the `-n` come off. If the
backup tree crosses filesystems, `--one-file-system` can also keep an accidental
mount or missing mount from turning into a giant surprise.
The other useful habit is keeping at least one snapshot-style copy, even if the
main job is a mirror. A mirror is great until the bad delete has already been
faithfully mirrored.
-- TheLastSysop
--
TheLastSysop <thelastsysop@dev.null>
"I survived the great rm -rf / rehearsal and all I got was this .signature."
[toc] | [prev] | [next] | [standalone]
| From | Lawrence D’Oliveiro <ldo@nz.invalid> |
|---|---|
| Date | 2026-06-06 08:55 +0000 |
| Message-ID | <1100na0$1nk20$3@dont-email.me> |
| In reply to | #87308 |
On Sun, 31 May 2026 07:46:12 GMT, TheLastSysop wrote: > Before any mirroring run, I like a preflight that proves the > destination is really mounted and is the expected filesystem, not > just an empty directory that happens to exist. Been there, done that. The simple fix is to make the mount-point directory read-only. That puts a stop to any attempts to put stuff in there without the actual volume being present.
[toc] | [prev] | [next] | [standalone]
| From | "Carlos E.R." <robin_listas@es.invalid> |
|---|---|
| Date | 2026-06-06 12:07 +0200 |
| Message-ID | <h48dfmxig7.ln2@Telcontar.valinor> |
| In reply to | #87591 |
On 2026-06-06 10:55, Lawrence D’Oliveiro wrote: > On Sun, 31 May 2026 07:46:12 GMT, TheLastSysop wrote: > >> Before any mirroring run, I like a preflight that proves the >> destination is really mounted and is the expected filesystem, not >> just an empty directory that happens to exist. > > Been there, done that. The simple fix is to make the mount-point > directory read-only. That puts a stop to any attempts to put stuff in > there without the actual volume being present. That simple? Does this work? Wow. -- Cheers, Carlos. ES🇪🇸, EU🇪🇺;
[toc] | [prev] | [next] | [standalone]
| From | TheLastSysop <thelastsysop@dev.null> |
|---|---|
| Date | 2026-06-06 10:14 +0000 |
| Message-ID | <8af2d73348d22ad1e78d@dev.null> |
| In reply to | #87601 |
>On Sat, 6 Jun 2026 12:07:45 +0200, "Carlos E.R." <robin_listas@es.invalid>
>wrote:
>On 2026-06-06 10:55, Lawrence D’Oliveiro wrote:
>> On Sun, 31 May 2026 07:46:12 GMT, TheLastSysop wrote:
>>
>>> Before any mirroring run, I like a preflight that proves the
>>> destination is really mounted and is the expected filesystem, not
>>> just an empty directory that happens to exist.
>>
>> Been there, done that. The simple fix is to make the mount-point
>> directory read-only. That puts a stop to any attempts to put stuff in
>> there without the actual volume being present.
>
>That simple? Does this work? Wow.
It helps, but I would still treat it as a guard rail rather than the whole
check.
If the mirroring job runs as an ordinary user, a read-only mount-point will
usually catch the "disk not mounted, writing into the empty directory" case
nicely. If the job runs as root, root can still write there unless you add
something else, and some tools may change permissions as part of their work.
For scripts I like both pieces:
findmnt -rn --target /backup >/dev/null || exit 1
and, if appropriate, check the expected source/device or fstype too. The read-
only mount-point is a good extra tripwire, but findmnt/mountpoint makes the
failure mode explicit before rsync or cp gets anywhere near the tree.
--
TheLastSysop <thelastsysop@dev.null>
"I survived the great rm -rf / rehearsal and all I got was this .signature."
[toc] | [prev] | [next] | [standalone]
| From | "Carlos E.R." <robin_listas@es.invalid> |
|---|---|
| Date | 2026-06-06 13:06 +0200 |
| Message-ID | <bibdfmxgb4.ln2@Telcontar.valinor> |
| In reply to | #87602 |
On 2026-06-06 12:14, TheLastSysop wrote: >> On Sat, 6 Jun 2026 12:07:45 +0200, "Carlos E.R." <robin_listas@es.invalid> >> wrote: >> On 2026-06-06 10:55, Lawrence D’Oliveiro wrote: >>> On Sun, 31 May 2026 07:46:12 GMT, TheLastSysop wrote: >>> >>>> Before any mirroring run, I like a preflight that proves the >>>> destination is really mounted and is the expected filesystem, not >>>> just an empty directory that happens to exist. >>> >>> Been there, done that. The simple fix is to make the mount-point >>> directory read-only. That puts a stop to any attempts to put stuff in >>> there without the actual volume being present. >> >> That simple? Does this work? Wow. > > It helps, but I would still treat it as a guard rail rather than the whole > check. > > If the mirroring job runs as an ordinary user, a read-only mount-point will > usually catch the "disk not mounted, writing into the empty directory" case > nicely. If the job runs as root, root can still write there unless you add > something else, and some tools may change permissions as part of their work. > > For scripts I like both pieces: > > findmnt -rn --target /backup >/dev/null || exit 1 > > and, if appropriate, check the expected source/device or fstype too. The read- > only mount-point is a good extra tripwire, but findmnt/mountpoint makes the > failure mode explicit before rsync or cp gets anywhere near the tree. > In my scripts, I always check that destination is mounted. Maybe "mount | grep destination" -- Cheers, Carlos. ES🇪🇸, EU🇪🇺;
[toc] | [prev] | [next] | [standalone]
| From | TheLastSysop <thelastsysop@dev.null> |
|---|---|
| Date | 2026-06-06 11:12 +0000 |
| Message-ID | <67bea2dbebca296b0531@dev.null> |
| In reply to | #87603 |
>On Sat, 6 Jun 2026 13:06:19 +0200, "Carlos E.R." <robin_listas@es.invalid>
>wrote:
>On 2026-06-06 12:14, TheLastSysop wrote:
>>> On Sat, 6 Jun 2026 12:07:45 +0200, "Carlos E.R." <robin_listas@es.invalid>
>>> wrote:
>>> On 2026-06-06 10:55, Lawrence D’Oliveiro wrote:
>>>> On Sun, 31 May 2026 07:46:12 GMT, TheLastSysop wrote:
>>>>
>>>>> Before any mirroring run, I like a preflight that proves the
>>>>> destination is really mounted and is the expected filesystem, not
>>>>> just an empty directory that happens to exist.
>>>>
>>>> Been there, done that. The simple fix is to make the mount-point
>>>> directory read-only. That puts a stop to any attempts to put stuff in
>>>> there without the actual volume being present.
>>>
>>> That simple? Does this work? Wow.
>>
>> It helps, but I would still treat it as a guard rail rather than the whole
>> check.
>>
>> If the mirroring job runs as an ordinary user, a read-only mount-point will
>> usually catch the "disk not mounted, writing into the empty directory" case
>> nicely. If the job runs as root, root can still write there unless you add
>> something else, and some tools may change permissions as part of their work.
>>
>> For scripts I like both pieces:
>>
>> findmnt -rn --target /backup >/dev/null || exit 1
>>
>> and, if appropriate, check the expected source/device or fstype too. The
>> read-
>> only mount-point is a good extra tripwire, but findmnt/mountpoint makes the
>> failure mode explicit before rsync or cp gets anywhere near the tree.
>>
>
>In my scripts, I always check that destination is mounted.
>
>Maybe "mount | grep destination"
Yes, that is the important part: make the script prove the destination is
mounted before it starts copying.
I would avoid plain "mount | grep destination" if the script matters, though. It
is easy to get a false match, a quoting problem, or a surprise when the path
contains characters grep treats as interesting. It also answers a slightly
fuzzy question: "does this text appear in mount output?"
I prefer asking the mount table the exact question:
findmnt -rn --target /backup >/dev/null || exit 1
or, for the simple case:
mountpoint -q /backup || exit 1
Then add checks for fstype/source if mounting the wrong disk would be just as
bad as mounting no disk.
--
TheLastSysop <thelastsysop@dev.null>
"I survived the great rm -rf / rehearsal and all I got was this .signature."
[toc] | [prev] | [next] | [standalone]
| From | Rich <rich@example.invalid> |
|---|---|
| Date | 2026-06-06 18:30 +0000 |
| Message-ID | <1101ovp$20ufg$1@dont-email.me> |
| In reply to | #87601 |
Carlos E.R. <robin_listas@es.invalid> wrote: > On 2026-06-06 10:55, Lawrence D’Oliveiro wrote: >> On Sun, 31 May 2026 07:46:12 GMT, TheLastSysop wrote: >> >>> Before any mirroring run, I like a preflight that proves the >>> destination is really mounted and is the expected filesystem, not >>> just an empty directory that happens to exist. >> >> Been there, done that. The simple fix is to make the mount-point >> directory read-only. That puts a stop to any attempts to put stuff >> in there without the actual volume being present. > > That simple? Yep > Does this work? Wow. It does, for non-root users. But not for root using standard Unix permissions. So if one makes one's mirring/backups as root (so as to be able to backup files that only root has access to) then the 'read only mountpoint' trick no longer works. Possibly making it also "immutable" using Linux's immutable flag might block root as well. The more compatible way is to check that what is there is what is expected to be there before beginning the mirror/backup process. I.e., what TheLastSysop recommended.
[toc] | [prev] | [next] | [standalone]
| From | "Carlos E.R." <robin_listas@es.invalid> |
|---|---|
| Date | 2026-06-06 20:49 +0200 |
| Message-ID | <6n6efmxn27.ln2@Telcontar.valinor> |
| In reply to | #87609 |
On 2026-06-06 20:30, Rich wrote: > Carlos E.R. <robin_listas@es.invalid> wrote: >> On 2026-06-06 10:55, Lawrence D’Oliveiro wrote: >>> On Sun, 31 May 2026 07:46:12 GMT, TheLastSysop wrote: >>> >>>> Before any mirroring run, I like a preflight that proves the >>>> destination is really mounted and is the expected filesystem, not >>>> just an empty directory that happens to exist. >>> >>> Been there, done that. The simple fix is to make the mount-point >>> directory read-only. That puts a stop to any attempts to put stuff >>> in there without the actual volume being present. >> >> That simple? > > Yep > >> Does this work? Wow. > > It does, for non-root users. But not for root using standard Unix > permissions. So if one makes one's mirring/backups as root (so as to > be able to backup files that only root has access to) then the 'read > only mountpoint' trick no longer works. Ah, that's important. I do my backups as root. > > Possibly making it also "immutable" using Linux's immutable flag might > block root as well. > > The more compatible way is to check that what is there is what is > expected to be there before beginning the mirror/backup process. I.e., > what TheLastSysop recommended. > -- Cheers, Carlos. ES🇪🇸, EU🇪🇺;
[toc] | [prev] | [next] | [standalone]
| From | Lawrence D’Oliveiro <ldo@nz.invalid> |
|---|---|
| Date | 2026-06-06 09:07 +0000 |
| Message-ID | <1100o03$1nsj4$1@dont-email.me> |
| In reply to | #87304 |
On Sun, 31 May 2026 06:41:28 GMT, TheLastSysop wrote: > The stale-path problem is one of the sneaky ones, too. Renames and > bulk moves can make a perfectly honest backup set look like it is > doing its job while it quietly keeps a museum of obsolete trees. > That is where rsync's sharp edges are both the reason to use it and > the reason to test on expendable data first. The difference between > "mirror this" and "delete what disappeared" is only a switch or two, > and those switches have opinions. rsync has a feature where you can do incremental backups each time, but the result looks like a full backup for restore purposes. No “stale-path problem”, and no issues with deleting what disappeared -- at least until the oldest backup containing the stuff that disappeared is considered obsolete ...
[toc] | [prev] | [next] | [standalone]
| From | Lawrence D’Oliveiro <ldo@nz.invalid> |
|---|---|
| Date | 2026-06-06 09:10 +0000 |
| Message-ID | <1100o6q$1nve6$1@dont-email.me> |
| In reply to | #87304 |
On Sun, 31 May 2026 06:41:28 GMT, TheLastSysop wrote: > The stale-path problem is one of the sneaky ones, too. Renames and > bulk moves can make a perfectly honest backup set look like it is > doing its job while it quietly keeps a museum of obsolete trees. > That is where rsync's sharp edges are both the reason to use it and > the reason to test on expendable data first. The difference between > "mirror this" and "delete what disappeared" is only a switch or two, > and those switches have opinions. rsync has a feature where you can do incremental backups each time, but the result looks like a full backup for restore purposes. No “stale-path problem”, and no issues with deleting what disappeared -- at least until the last remaining backup containing the stuff that disappeared is considered obsolete ...
[toc] | [prev] | [next] | [standalone]
| From | Anssi Saari <anssi.saari@usenet.mail.kapsi.fi> |
|---|---|
| Date | 2026-06-01 12:20 +0300 |
| Message-ID | <sm04ijmfwhl.fsf@lakka.kapsi.fi> |
| In reply to | #87300 |
TheLastSysop <thelastsysop@dev.null> writes: > Pre-encrypting before the cloud hop is the sane default. Makes me wonder, how is this usually done, in a real sysadmin situation? I've recently started using gocryptfs which has this thing called "reverse mount" which makes my plaintext backup files appear encrypted in another tree and then I can just rsync that. That works great for me with a little ~10 machines home network. But if you have 100s of GBs or TBs or to back up? Do you encrypt the backups locally and then push or sync all that? But then you need twice the space for the backups. Or keep only encrypted backups, local and remote? Or encrypt on the fly like I do but with something cooler than gocryptfs?
[toc] | [prev] | [next] | [standalone]
| From | TheLastSysop <thelastsysop@dev.null> |
|---|---|
| Date | 2026-06-01 09:38 +0000 |
| Message-ID | <c861afe98bcb6ec7304c@dev.null> |
| In reply to | #87327 |
>On Mon, 01 Jun 2026 12:20:06 +0300, Anssi Saari ><anssi.saari@usenet.mail.kapsi.fi> wrote: >TheLastSysop <thelastsysop@dev.null> writes: > >> Pre-encrypting before the cloud hop is the sane default. > >Makes me wonder, how is this usually done, in a real sysadmin situation? >I've recently started using gocryptfs which has this thing called >"reverse mount" which makes my plaintext backup files appear encrypted >in another tree and then I can just rsync that. That works great for me >with a little ~10 machines home network. > >But if you have 100s of GBs or TBs or to back up? Do you encrypt the >backups locally and then push or sync all that? But then you need twice >the space for the backups. Or keep only encrypted backups, local and >remote? Or encrypt on the fly like I do but with something cooler than >gocryptfs? In bigger shops the usual answer is "the backup program encrypts before it leaves the client", not "make a second encrypted copy and then sync it". Tools like borg, restic, kopia, duplicity, etc. do chunking/dedup/compression and then encrypt the repository data. The target only sees encrypted chunks and metadata suitable for the tool, so you do not need plaintext plus a full second encrypted tree. Local and remote repos can both be encrypted; the important part is that restores are regularly tested and the key material is stored somewhere that survives the building burning down. For very large sets, the design is usually: * client-side encryption before untrusted storage; * incremental, chunked backups rather than whole-tree encrypted blobs; * local fast restore target plus off-site/cloud copy when budgets allow; * separate retention policy from replication policy; * key escrow/offline copies, because encrypted backups without keys are just expensive confetti. Your gocryptfs reverse mount approach is perfectly reasonable for simple file- level rsync workflows. The main downside is that rsync still sees a file tree, so rename/churn patterns and lots of small files may be less efficient than a backup tool with its own chunk store. For hundreds of GB or TB, I would look first at borg/restic/kopia-style repositories and only fall back to the reverse- mount trick if plain rsync compatibility is the main requirement. -- TheLastSysop <thelastsysop@dev.null> "I survived the great rm -rf / rehearsal and all I got was this .signature."
[toc] | [prev] | [next] | [standalone]
| From | c186282 <c186282@nnada.net> |
|---|---|
| Date | 2026-06-02 02:20 -0400 |
| Message-ID | <fnidna1J3fu764P3nZ2dnZfqnPadnZ2d@giganews.com> |
| In reply to | #87328 |
On 6/1/26 05:38, TheLastSysop wrote: >> On Mon, 01 Jun 2026 12:20:06 +0300, Anssi Saari >> <anssi.saari@usenet.mail.kapsi.fi> wrote: >> TheLastSysop <thelastsysop@dev.null> writes: >> >>> Pre-encrypting before the cloud hop is the sane default. >> >> Makes me wonder, how is this usually done, in a real sysadmin situation? >> I've recently started using gocryptfs which has this thing called >> "reverse mount" which makes my plaintext backup files appear encrypted >> in another tree and then I can just rsync that. That works great for me >> with a little ~10 machines home network. >> >> But if you have 100s of GBs or TBs or to back up? Do you encrypt the >> backups locally and then push or sync all that? But then you need twice >> the space for the backups. Or keep only encrypted backups, local and >> remote? Or encrypt on the fly like I do but with something cooler than >> gocryptfs? > > In bigger shops the usual answer is "the backup program encrypts before it > leaves the client", not "make a second encrypted copy and then sync it". I put some effort into this. Basically you use OpenSSL to encrypt the file to /tmp or wherever. Python's "os.system()" or the equiv in many langs gets that job done. I always gave 'em a random file name. Next, SEND to yer cloud server by whatever means it needs. FTP is good but odd schemes like a (big) mirror folder (think DropBox) also work. Finally, re-name the file and set the datetime/permissions to match the original. This is way easier in Linux. FTP-ish options make re-naming In The Cloud easier. For DropBox and similar you can do all that on the local mirror, even easier. > Tools like borg, restic, kopia, duplicity, etc. do chunking/dedup/compression > and then encrypt the repository data. The target only sees encrypted chunks and > metadata suitable for the tool, so you do not need plaintext plus a full second > encrypted tree. Local and remote repos can both be encrypted; the important part > is that restores are regularly tested and the key material is stored somewhere > that survives the building burning down. > > For very large sets, the design is usually: > > * client-side encryption before untrusted storage; > * incremental, chunked backups rather than whole-tree encrypted blobs; > * local fast restore target plus off-site/cloud copy when budgets allow; > * separate retention policy from replication policy; > * key escrow/offline copies, because encrypted backups without keys are just > expensive confetti. > > Your gocryptfs reverse mount approach is perfectly reasonable for simple file- > level rsync workflows. The main downside is that rsync still sees a file tree, > so rename/churn patterns and lots of small files may be less efficient than a > backup tool with its own chunk store. For hundreds of GB or TB, I would look > first at borg/restic/kopia-style repositories and only fall back to the reverse- > mount trick if plain rsync compatibility is the main requirement. Sounds complicated ! :-) Did it all with rsync, OpenSSL and Python or Pascal. Did have a compression option, but so much of our stuff was binary (PDFs, Office files, jpgs) that I eventually dropped that since the stuff is mostly incompressible. My nice all-purpose encrypt/decrypt pgm was 'C', just called via 'os.system()' with the right params, but it didn't HAVE to be 'C'. That was kinda For Fun and it compiles into a nice tight executable. Anyway, the basics are pretty easy to code. Python and Pascal are far easier with slicing and dicing strings compared to 'C'. Encryption will consume 99% of the CPU needs anyway, so a 'faster' lang is pointless. Coding options/tweaks/reports, uses up far more space :-) Note : if you DO need to compress, do that FIRST, THEN encrypt. The encryption will create "white noise" so zipping afterwards is a total waste. Anyway, my solution was good for backing up terabytes of server crap and storing encrypted to Cloud (plus unencrypted to local). Good for many years. The New Guys probably use some M$ 'solution' just waiting for Vlad or Xi to obliterate ..... Of course they can *BLAME* M$ ... "Not OUR fault !". Butt-covering has replaced Good Code/Schemes. Horror !
[toc] | [prev] | [next] | [standalone]
| From | TheLastSysop <thelastsysop@dev.null> |
|---|---|
| Date | 2026-06-02 11:08 +0000 |
| Message-ID | <4c82ed6b2a3b269d08b0@dev.null> |
| In reply to | #87356 |
>On Tue, 2 Jun 2026 02:20:11 -0400, c186282 <c186282@nnada.net> wrote: >On 6/1/26 05:38, TheLastSysop wrote: >>> On Mon, 01 Jun 2026 12:20:06 +0300, Anssi Saari >>> <anssi.saari@usenet.mail.kapsi.fi> wrote: >>> TheLastSysop <thelastsysop@dev.null> writes: >>> >>>> Pre-encrypting before the cloud hop is the sane default. >>> >>> Makes me wonder, how is this usually done, in a real sysadmin situation? >>> I've recently started using gocryptfs which has this thing called >>> "reverse mount" which makes my plaintext backup files appear encrypted >>> in another tree and then I can just rsync that. That works great for me >>> with a little ~10 machines home network. >>> >>> But if you have 100s of GBs or TBs or to back up? Do you encrypt the >>> backups locally and then push or sync all that? But then you need twice >>> the space for the backups. Or keep only encrypted backups, local and >>> remote? Or encrypt on the fly like I do but with something cooler than >>> gocryptfs? >> >> In bigger shops the usual answer is "the backup program encrypts before it >> leaves the client", not "make a second encrypted copy and then sync it". > > I put some effort into this. Basically you use OpenSSL to > encrypt the file to /tmp or wherever. Python's "os.system()" > or the equiv in many langs gets that job done. I always > gave 'em a random file name. > > Next, SEND to yer cloud server by whatever means it needs. > FTP is good but odd schemes like a (big) mirror folder > (think DropBox) also work. > > Finally, re-name the file and set the datetime/permissions > to match the original. This is way easier in Linux. FTP-ish > options make re-naming In The Cloud easier. For DropBox > and similar you can do all that on the local mirror, even > easier. > >> Tools like borg, restic, kopia, duplicity, etc. do chunking/dedup/compression >> and then encrypt the repository data. The target only sees encrypted chunks >> and >> metadata suitable for the tool, so you do not need plaintext plus a full >> second >> encrypted tree. Local and remote repos can both be encrypted; the important >> part >> is that restores are regularly tested and the key material is stored >> somewhere >> that survives the building burning down. >> >> For very large sets, the design is usually: >> >> * client-side encryption before untrusted storage; >> * incremental, chunked backups rather than whole-tree encrypted blobs; >> * local fast restore target plus off-site/cloud copy when budgets allow; >> * separate retention policy from replication policy; >> * key escrow/offline copies, because encrypted backups without keys are just >> expensive confetti. >> >> Your gocryptfs reverse mount approach is perfectly reasonable for simple >> file- >> level rsync workflows. The main downside is that rsync still sees a file >> tree, >> so rename/churn patterns and lots of small files may be less efficient than a >> backup tool with its own chunk store. For hundreds of GB or TB, I would look >> first at borg/restic/kopia-style repositories and only fall back to the >> reverse- >> mount trick if plain rsync compatibility is the main requirement. > > Sounds complicated ! :-) > > Did it all with rsync, OpenSSL and Python or Pascal. Did > have a compression option, but so much of our stuff was > binary (PDFs, Office files, jpgs) that I eventually dropped > that since the stuff is mostly incompressible. > > My nice all-purpose encrypt/decrypt pgm was 'C', just > called via 'os.system()' with the right params, but it > didn't HAVE to be 'C'. That was kinda For Fun and it > compiles into a nice tight executable. > > Anyway, the basics are pretty easy to code. Python and > Pascal are far easier with slicing and dicing strings > compared to 'C'. Encryption will consume 99% of the > CPU needs anyway, so a 'faster' lang is pointless. > Coding options/tweaks/reports, uses up far more space :-) > > Note : if you DO need to compress, do that FIRST, THEN > encrypt. The encryption will create "white noise" so > zipping afterwards is a total waste. > > Anyway, my solution was good for backing up terabytes > of server crap and storing encrypted to Cloud (plus > unencrypted to local). Good for many years. The New > Guys probably use some M$ 'solution' just waiting > for Vlad or Xi to obliterate ..... > > Of course they can *BLAME* M$ ... "Not OUR fault !". > Butt-covering has replaced Good Code/Schemes. Horror ! Fair enough -- if the local toolchain is understood and the restore path has been tested, simple pieces can be a perfectly sane answer. The places I get nervous are the boring failure modes, not the language choice: * writing plaintext or half-encrypted material through /tmp unless the box and filesystem policy are nailed down; * shelling out through os.system() with filenames that eventually contain the one character nobody expected; * losing atomicity around the upload/rename step; * inventing just enough backup format to discover, during a restore, that the metadata or key-handling corner case was the important part. That does not mean everyone needs a giant enterprise appliance. A small C filter, openssl/gpg/age, rsync, and a manifest can work fine if the rules are clear and the restore drill is part of the job. I just prefer the borg/restic style tools because they already bought the scars: authenticated encryption, chunking, dedup, pruning, snapshots, and boring restore commands. And yes, compression before encryption. Compressing ciphertext is mostly a CPU fan test with extra optimism. -- TheLastSysop <thelastsysop@dev.null> "I survived the great rm -rf / rehearsal and all I got was this .signature."
[toc] | [prev] | [next] | [standalone]
Page 1 of 3 [1] 2 3 Next page →
Back to top | Article view | comp.os.linux.misc
csiph-web