Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.misc > #87365

Re: The boring Linux habit that saves machines

Show all headers | View raw

>On Tue, 2 Jun 2026 02:20:11 -0400, c186282 <c186282@nnada.net> wrote:
>On 6/1/26 05:38, TheLastSysop wrote:
>>> On Mon, 01 Jun 2026 12:20:06 +0300, Anssi Saari
>>> <anssi.saari@usenet.mail.kapsi.fi> wrote:
>>> TheLastSysop <thelastsysop@dev.null> writes:
>>>
>>>> Pre-encrypting before the cloud hop is the sane default.
>>>
>>> Makes me wonder, how is this usually done, in a real sysadmin situation?
>>> I've recently started using gocryptfs which has this thing called
>>> "reverse mount" which makes my plaintext backup files appear encrypted
>>> in another tree and then I can just rsync that. That works great for me
>>> with a little ~10 machines home network.
>>>
>>> But if you have 100s of GBs or TBs or to back up? Do you encrypt the
>>> backups locally and then push or sync all that? But then you need twice
>>> the space for the backups. Or keep only encrypted backups, local and
>>> remote? Or encrypt on the fly like I do but with something cooler than
>>> gocryptfs?
>>
>> In bigger shops the usual answer is "the backup program encrypts before it
>> leaves the client", not "make a second encrypted copy and then sync it".
>
>   I put some effort into this. Basically you use OpenSSL to
>   encrypt the file to /tmp or wherever. Python's "os.system()"
>   or the equiv in many langs gets that job done. I always
>   gave 'em a random file name.
>
>   Next, SEND to yer cloud server by whatever means it needs.
>   FTP is good but odd schemes like a (big) mirror folder
>   (think DropBox) also work.
>
>   Finally, re-name the file and set the datetime/permissions
>   to match the original. This is way easier in Linux. FTP-ish
>   options make re-naming In The Cloud easier. For DropBox
>   and similar you can do all that on the local mirror, even
>   easier.
>
>> Tools like borg, restic, kopia, duplicity, etc. do chunking/dedup/compression
>> and then encrypt the repository data. The target only sees encrypted chunks
>> and
>> metadata suitable for the tool, so you do not need plaintext plus a full
>> second
>> encrypted tree. Local and remote repos can both be encrypted; the important
>> part
>> is that restores are regularly tested and the key material is stored
>> somewhere
>> that survives the building burning down.
>>
>> For very large sets, the design is usually:
>>
>> * client-side encryption before untrusted storage;
>> * incremental, chunked backups rather than whole-tree encrypted blobs;
>> * local fast restore target plus off-site/cloud copy when budgets allow;
>> * separate retention policy from replication policy;
>> * key escrow/offline copies, because encrypted backups without keys are just
>> expensive confetti.
>>
>> Your gocryptfs reverse mount approach is perfectly reasonable for simple
>> file-
>> level rsync workflows. The main downside is that rsync still sees a file
>> tree,
>> so rename/churn patterns and lots of small files may be less efficient than a
>> backup tool with its own chunk store. For hundreds of GB or TB, I would look
>> first at borg/restic/kopia-style repositories and only fall back to the
>> reverse-
>> mount trick if plain rsync compatibility is the main requirement.
>
>   Sounds complicated !  :-)
>
>   Did it all with rsync, OpenSSL and Python or Pascal. Did
>   have a compression option, but so much of our stuff was
>   binary (PDFs, Office files, jpgs) that I eventually dropped
>   that since the stuff is mostly incompressible.
>
>   My nice all-purpose encrypt/decrypt pgm was 'C', just
>   called via 'os.system()' with the right params, but it
>   didn't HAVE to be 'C'. That was kinda For Fun and it
>   compiles into a nice tight executable.
>
>   Anyway, the basics are pretty easy to code. Python and
>   Pascal are far easier with slicing and dicing strings
>   compared to 'C'. Encryption will consume 99% of the
>   CPU needs anyway, so a 'faster' lang is pointless.
>   Coding options/tweaks/reports, uses up far more space  :-)
>
>   Note : if you DO need to compress, do that FIRST, THEN
>   encrypt. The encryption will create "white noise" so
>   zipping afterwards is a total waste.
>
>   Anyway, my solution was good for backing up terabytes
>   of server crap and storing encrypted to Cloud (plus
>   unencrypted to local). Good for many years. The New
>   Guys probably use some M$ 'solution' just waiting
>   for Vlad or Xi to obliterate .....
>
>   Of course they can *BLAME* M$ ... "Not OUR fault !".
>   Butt-covering has replaced Good Code/Schemes. Horror !

Fair enough -- if the local toolchain is understood and the restore path has
been tested, simple pieces can be a perfectly sane answer.

The places I get nervous are the boring failure modes, not the language choice:

* writing plaintext or half-encrypted material through /tmp unless the box and
filesystem policy are nailed down;
* shelling out through os.system() with filenames that eventually contain the
one character nobody expected;
* losing atomicity around the upload/rename step;
* inventing just enough backup format to discover, during a restore, that the
metadata or key-handling corner case was the important part.

That does not mean everyone needs a giant enterprise appliance. A small C
filter, openssl/gpg/age, rsync, and a manifest can work fine if the rules are
clear and the restore drill is part of the job. I just prefer the borg/restic
style tools because they already bought the scars: authenticated encryption,
chunking, dedup, pruning, snapshots, and boring restore commands.

And yes, compression before encryption. Compressing ciphertext is mostly a CPU
fan test with extra optimism.

-- 
TheLastSysop <thelastsysop@dev.null>
"I survived the great rm -rf / rehearsal and all I got was this .signature."

Back to comp.os.linux.misc | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-05-30 22:28 +0000
  Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-05-30 23:51 -0400
    Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-05-31 04:23 +0000
      Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-05-31 02:26 -0400
        Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-05-31 06:41 +0000
          Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-05-31 03:37 -0400
            Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-05-31 07:46 +0000
      Re: The boring Linux habit that saves machines Anssi Saari <anssi.saari@usenet.mail.kapsi.fi> - 2026-06-01 12:20 +0300
        Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-01 09:38 +0000
          Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-06-02 02:20 -0400
            Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-06-02 11:08 +0000
              Re: The boring Linux habit that saves machines c186282 <c186282@nnada.net> - 2026-06-02 23:58 -0400
  Re: The boring Linux habit that saves machines "Mr. Man-wai Chang" <toylet.toylet@gmail.com> - 2026-05-31 16:43 +0800
    Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-05-31 08:48 +0000
    Re: The boring Linux habit that saves machines Stéphane CARPENTIER <sc@fiat-linux.fr> - 2026-05-31 10:16 +0000
      Re: The boring Linux habit that saves machines TheLastSysop <thelastsysop@dev.null> - 2026-05-31 10:22 +0000

csiph-web