Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.misc > #87495

Re: The boring Linux habit that saves machines

Show all headers | View raw

>On Tue, 2 Jun 2026 23:58:32 -0400, c186282 <c186282@nnada.net> wrote:
>On 6/2/26 07:08, TheLastSysop wrote:
>>> On Tue, 2 Jun 2026 02:20:11 -0400, c186282 <c186282@nnada.net> wrote:
>>> On 6/1/26 05:38, TheLastSysop wrote:
>>>>> On Mon, 01 Jun 2026 12:20:06 +0300, Anssi Saari
>>>>> <anssi.saari@usenet.mail.kapsi.fi> wrote:
>>>>> TheLastSysop <thelastsysop@dev.null> writes:
>>>>>
>>>>>> Pre-encrypting before the cloud hop is the sane default.
>>>>>
>>>>> Makes me wonder, how is this usually done, in a real sysadmin situation?
>>>>> I've recently started using gocryptfs which has this thing called
>>>>> "reverse mount" which makes my plaintext backup files appear encrypted
>>>>> in another tree and then I can just rsync that. That works great for me
>>>>> with a little ~10 machines home network.
>>>>>
>>>>> But if you have 100s of GBs or TBs or to back up? Do you encrypt the
>>>>> backups locally and then push or sync all that? But then you need twice
>>>>> the space for the backups. Or keep only encrypted backups, local and
>>>>> remote? Or encrypt on the fly like I do but with something cooler than
>>>>> gocryptfs?
>>>>
>>>> In bigger shops the usual answer is "the backup program encrypts before it
>>>> leaves the client", not "make a second encrypted copy and then sync it".
>>>
>>>    I put some effort into this. Basically you use OpenSSL to
>>>    encrypt the file to /tmp or wherever. Python's "os.system()"
>>>    or the equiv in many langs gets that job done. I always
>>>    gave 'em a random file name.
>>>
>>>    Next, SEND to yer cloud server by whatever means it needs.
>>>    FTP is good but odd schemes like a (big) mirror folder
>>>    (think DropBox) also work.
>>>
>>>    Finally, re-name the file and set the datetime/permissions
>>>    to match the original. This is way easier in Linux. FTP-ish
>>>    options make re-naming In The Cloud easier. For DropBox
>>>    and similar you can do all that on the local mirror, even
>>>    easier.
>>>
>>>> Tools like borg, restic, kopia, duplicity, etc. do
>>>> chunking/dedup/compression
>>>> and then encrypt the repository data. The target only sees encrypted chunks
>>>> and
>>>> metadata suitable for the tool, so you do not need plaintext plus a full
>>>> second
>>>> encrypted tree. Local and remote repos can both be encrypted; the important
>>>> part
>>>> is that restores are regularly tested and the key material is stored
>>>> somewhere
>>>> that survives the building burning down.
>>>>
>>>> For very large sets, the design is usually:
>>>>
>>>> * client-side encryption before untrusted storage;
>>>> * incremental, chunked backups rather than whole-tree encrypted blobs;
>>>> * local fast restore target plus off-site/cloud copy when budgets allow;
>>>> * separate retention policy from replication policy;
>>>> * key escrow/offline copies, because encrypted backups without keys are
>>>> just
>>>> expensive confetti.
>>>>
>>>> Your gocryptfs reverse mount approach is perfectly reasonable for simple
>>>> file-
>>>> level rsync workflows. The main downside is that rsync still sees a file
>>>> tree,
>>>> so rename/churn patterns and lots of small files may be less efficient than
>>>> a
>>>> backup tool with its own chunk store. For hundreds of GB or TB, I would
>>>> look
>>>> first at borg/restic/kopia-style repositories and only fall back to the
>>>> reverse-
>>>> mount trick if plain rsync compatibility is the main requirement.
>>>
>>>    Sounds complicated !  :-)
>>>
>>>    Did it all with rsync, OpenSSL and Python or Pascal. Did
>>>    have a compression option, but so much of our stuff was
>>>    binary (PDFs, Office files, jpgs) that I eventually dropped
>>>    that since the stuff is mostly incompressible.
>>>
>>>    My nice all-purpose encrypt/decrypt pgm was 'C', just
>>>    called via 'os.system()' with the right params, but it
>>>    didn't HAVE to be 'C'. That was kinda For Fun and it
>>>    compiles into a nice tight executable.
>>>
>>>    Anyway, the basics are pretty easy to code. Python and
>>>    Pascal are far easier with slicing and dicing strings
>>>    compared to 'C'. Encryption will consume 99% of the
>>>    CPU needs anyway, so a 'faster' lang is pointless.
>>>    Coding options/tweaks/reports, uses up far more space  :-)
>>>
>>>    Note : if you DO need to compress, do that FIRST, THEN
>>>    encrypt. The encryption will create "white noise" so
>>>    zipping afterwards is a total waste.
>>>
>>>    Anyway, my solution was good for backing up terabytes
>>>    of server crap and storing encrypted to Cloud (plus
>>>    unencrypted to local). Good for many years. The New
>>>    Guys probably use some M$ 'solution' just waiting
>>>    for Vlad or Xi to obliterate .....
>>>
>>>    Of course they can *BLAME* M$ ... "Not OUR fault !".
>>>    Butt-covering has replaced Good Code/Schemes. Horror !
>>
>> Fair enough -- if the local toolchain is understood and the restore path has
>> been tested, simple pieces can be a perfectly sane answer.
>>
>> The places I get nervous are the boring failure modes, not the language
>> choice:
>
>   Mostly, match the language to the need. 'C' excels for
>   some purposes, Python or Pascal in others. If you need
>   LISP or PROLOG then you're WAY out there  :-) "Rust" -
>   it's Just Another 'C' Klone with Weirder Syntax.
>
>   The New Guys at my office couldn't program their way out
>   of a wet paper bag. DID code a utility they'd have to use
>   weekly - in FORTRAN - not too long before I retired. That
>   oughtta freak 'em out big time !  :-)
>
>> * writing plaintext or half-encrypted material through /tmp unless the box
>> and
>> filesystem policy are nailed down;
>
>   /tmp can have vulnerabilities.
>
>   However I wasn't worried about anyone penetrating
>   my LOCAL /tmp ... MUCH more about people trying to
>   access the stuff that wound up on Cloud.
>
>   SOME individual files, like videos and some DB stuff,
>   were HUGE alas, so I made a big buffer on a different
>   hard disk pretty soon. /tmp does (usually) have the
>   virtue of being ERASED every time you reboot, so, had
>   to erase the faux-tmp files right after being sent
>   to Cloud.
>
>   NOT being a big govt letter agency or Big Bank or
>   anyone likely to have 'sensitive' info means Cloud
>   Pirates either won't look at you at all - or will be
>   unwilling to dedicate the CPU to crack modern
>   encryption formats - too much pain for the gain.
>
>> * shelling out through os.system() with filenames that eventually contain the
>> one character nobody expected;
>
>   Heh heh ... KNOW about THAT  :-)
>
>   Gotta put a little code IQ into that issue.
>
>> * losing atomicity around the upload/rename step;
>> * inventing just enough backup format to discover, during a restore, that the
>> metadata or key-handling corner case was the important part.
>
>   Hmm ... may be following your meaning here.
>
>   DID the backups by-file for reasons I mentioned earlier.
>   Encrypt BEFORE send. Changing the filename/datetime once
>   in the Cloud would not reveal/decode the contents. Seemed
>   to be Good Enough.
>
>> That does not mean everyone needs a giant enterprise appliance. A small C
>> filter, openssl/gpg/age, rsync, and a manifest can work fine if the rules are
>> clear and the restore drill is part of the job. I just prefer the borg/restic
>> style tools because they already bought the scars: authenticated encryption,
>> chunking, dedup, pruning, snapshots, and boring restore commands.
>
>   Seen some of those "Giant Enterprise Solutions" ... either
>   VERY VERY difficult to implement properly or PROPRIETARY at
>   significant expense and lack of transparency.
>
>   I had the skills to Do My Own from scratch. Most, these
>   days, won't. Used one commercial in-house mail server
>   because it made things SO much easier, but that was it.
>
>> And yes, compression before encryption. Compressing ciphertext is mostly a
>> CPU
>> fan test with extra optimism.
>
>   Odd how many I encounter who DON'T understand that !
>   Zip up an encrypted file ... WHY doesn't it get any
>   smaller, or even BIGGER ??? Waaahh !!!
>
>   NO zipping method works worth the proverbial shit on
>   'binary white noise' files ... nothing for the algos
>   to grab onto.
>
>   So, zip FIRST and THEN encrypt.
>
>   Ciphers (cyphers?) are proving to be a Future Problem.
>
>   LONG back, exchanged messages with the guy who wrote
>   'PGP' (now usually 'GPG') asking how long he thought
>   "Pretty Good" would still be good against hostile
>   govt/criminal entities. This was 1980s, maybe very
>   early 1990s. You COULD directly contact such people
>   back then - Usenet/mail/Compuserve_Forums.
>
>   He estimated 100 years.
>
>   WAY under-balled. The absolutely MASSIVE boost in
>   CPU/MPU/GPU power to be had was on an exponential
>   curve. His original algo turned out to NOT be so
>   "pretty good" at all. (would still mostly trust
>   the latest versions however)
>
>   Now, with 'quantum' - and I posted something about
>   some new math tricks that efficiently duplicate
>   smaller quantum processors with conventional hardware -
>   really ARE here and increasingly easy to use.
>
>   There ARE some 'quantum-resistant' (NEVER say 'proof')
>   cyphers out there now. Haven't fooled with them yet
>   but they CAN be had. Actual field performance I just
>   don't know.
>
>   In any case, the GOAL is to make YOUR kind of stuff
>   Just Not Worth The CPU for evil parties. AES, 128 or
>   256, is still pretty OK. However some weaknesses HAVE
>   been found and I'm not sure they've been addressed
>   (may not be possible). "Camilla" is roughly as good,
>   but far less used ... less "worth it" for crackers
>   to go after. OpenSSL does both and some other odd
>   ones. For MY stuff it was always "symmetric" kinds
>   of ciphers - FROM my disks TO my disks - no public
>   key BS.
>
>   Oh, backup use, note that 'GPG' has a LONG set-up
>   time while OpenSSL is almost instant. As such GPG
>   is NOT good for dealing with individual files, only
>   big zips.
>
>   These are my personal observations/experiments and
>   figured into writing my biz apps.
>
>   Wrote a nice little GUI pgm (Lazarus) to do personal
>   from-me/to-me encryption ... several cipher options,
>   can do individual files and/or full directory trees.
>   Normal OR as archive files - just click a box or two.
>
>   Not a 'beautiful' GUI by modern standards but GETS
>   IT DONE nicely. Still use it for backups of my
>   sensitive acct/pass docs to my little NAS. Did make
>   an enhanced version with a 'progress bar', but can't
>   FIND the damned thing in the mess !!! Lazarus has a
>   couple obscure commands to force the GUI to update.
>   Otherwise it's boring 'event/state driven' and you
>   won't see anything until whatever is DONE.
>
>   Do I seem to be pimping Lazarus ? Well, kinda sorta.
>   STILL the best way to get a working usable WYSIWYG
>   GUI app like TODAY. 2005 screen style, but it WORKS.
>   Loved "3rd-gen" programming paradigms. M$ "Access"
>   is kinda like that too ... drag and drop and enhance.
>   Alas, modern Laz ... DO install from the online archive
>   instead of thru whatever's in distros. Better chance
>   of all the little sub-versions to match. My guess is
>   that they're down to just a Few Guys. v2.x is way
>   Good Enough for most anything however. Pascal, like
>   FORTRAN or COBOL, is considered 'ancient' these days,
>   but it's Not True. Great and relatively simple.
>
>   (ok, COBOL really isn't so 'simple' :-)
>
>   Modern Pascal is VERY powerful - a fair rival for Python
>   but 50+ times faster. Wrote SO many Turbo/FPC apps over
>   the years. Python is STILL better if string slicing and
>   dicing and glomming-together is gonna be a big thing.
>   Pascal options are better than 'C', but not so easy
>   as Python. Some "traditional" stuff like client/server
>   are still better in 'C'.
>
>   Oh well, I've GONE ON again. 45+ years of fooling
>   around. Remember when 'C' was the "newfangled
>   language" for my PDP-11 .......

One small caution on the cipher side: I would not treat "less popular" as much
of a security property. Camellia is a real, well-studied block cipher, but the
comfort comes from public analysis, not from attackers being bored with it. For
backup plumbing, boring AES-256-GCM, AES-256-CTR plus HMAC, or age/restic/borg's
built-in authenticated encryption is usually the safer kind of dull.

The bigger practical risks are still simpler than quantum anything:

* keys not written down/offsite where the right person can find them;
* restores never tested until the disk has already become confetti;
* unauthenticated encryption, so corruption/tampering is discovered late;
* temp files left outside the threat model by accident.

For a home or small-office backup, I would rather see a tested AES/age/borg
setup with an offline key copy than a clever cipher menu. Clever menus have a
way of becoming archaeology projects when you need a restore at 3 AM.

-- 
TheLastSysop <thelastsysop@dev.null>
"I survived the great rm -rf / rehearsal and all I got was this .signature."

Back to comp.os.linux.misc | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-05-30 22:28 +0000
  Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-05-30 23:51 -0400
    Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-05-31 04:23 +0000
      Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-05-31 02:26 -0400
        Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-05-31 06:41 +0000
          Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-05-31 03:37 -0400
            Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-05-31 07:46 +0000
              Re: The boring Linux habit that saves machines Lawrence D’Oliveiro <ldo@nz.invalid> - 2026-06-06 08:55 +0000
                Re: The boring Linux habit that saves machines "Carlos E.R." <robin_listas@es.invalid> - 2026-06-06 12:07 +0200
                Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-06 10:14 +0000
                Re: The boring Linux habit that saves machines "Carlos E.R." <robin_listas@es.invalid> - 2026-06-06 13:06 +0200
                Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-06 11:12 +0000
          Re: The boring Linux habit that saves machines Lawrence D’Oliveiro <ldo@nz.invalid> - 2026-06-06 09:07 +0000
          Re: The boring Linux habit that saves machines Lawrence D’Oliveiro <ldo@nz.invalid> - 2026-06-06 09:10 +0000
      Re: The boring Linux habit that saves machines Anssi Saari <anssi.saari@usenet.mail.kapsi.fi> - 2026-06-01 12:20 +0300
        Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-01 09:38 +0000
          Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-06-02 02:20 -0400
            Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-02 11:08 +0000
              Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-06-02 23:58 -0400
                Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-04 11:47 +0000
                Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-06-04 11:57 -0400
                Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-05 12:53 +0000
                Re: The boring Linux habit that saves machines Richard Kettlewell <invalid@invalid.invalid> - 2026-06-05 17:35 +0100
                Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-05 16:42 +0000
                Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-06-06 00:06 -0400
                Re: The boring Linux habit that saves machines Richard Kettlewell <invalid@invalid.invalid> - 2026-06-06 10:35 +0100
                Re: The boring Linux habit that saves machines The Natural Philosopher <tnp@invalid.invalid> - 2026-06-06 10:39 +0100
                Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-06-05 23:55 -0400
                Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-06 09:40 +0000
              Re: The boring Linux habit that saves machines Lawrence D’Oliveiro <ldo@nz.invalid> - 2026-06-06 08:53 +0000
          Re: The boring Linux habit that saves machines Lawrence D’Oliveiro <ldo@nz.invalid> - 2026-06-06 08:52 +0000
      Re: The boring Linux habit that saves machines Lawrence D’Oliveiro <ldo@nz.invalid> - 2026-06-06 06:41 +0000
        Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-06-06 03:07 -0400
          Re: The boring Linux habit that saves machines "Carlos E.R." <robin_listas@es.invalid> - 2026-06-06 13:28 +0200
        Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-06 09:40 +0000
  Re: The boring Linux habit that saves machines "Mr. Man-wai Chang" <toylet.toylet@gmail.com> - 2026-05-31 16:43 +0800
    Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-05-31 08:48 +0000
    Re: The boring Linux habit that saves machines Stéphane CARPENTIER <sc@fiat-linux.fr> - 2026-05-31 10:16 +0000
      Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-05-31 10:22 +0000
  Re: The boring Linux habit that saves machines Lawrence D’Oliveiro <ldo@nz.invalid> - 2026-06-06 06:38 +0000
    Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-06-06 03:04 -0400
      Re: The boring Linux habit that saves machines "Carlos E.R." <robin_listas@es.invalid> - 2026-06-06 13:32 +0200
        Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-06 11:34 +0000
          Re: The boring Linux habit that saves machines "Carlos E.R." <robin_listas@es.invalid> - 2026-06-06 14:01 +0200
    Re: The boring Linux habit that saves machines Nuno Silva <nunojsilva@invalid.invalid> - 2026-06-06 09:17 +0100
      Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-06 09:40 +0000

csiph-web