Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #204237 > unrolled thread

Looking for advice on tools (or libraries) for unsupervised, bulk symmetric encryption/decryption of files

Back to article view | Back to linux.debian.user

Contents

  Looking for advice on tools (or libraries) for unsupervised, bulk  symmetric encryption/decryption of files Kynn Jones <kynnjo@gmail.com> - 2019-01-09 15:10 +0100
    Re: Looking for advice on tools (or libraries) for unsupervised, bulk  symmetric encryption/decryption of files David Christensen <dpchrist@holgerdanske.com> - 2019-01-09 20:00 +0100
    Re: Looking for advice on tools (or libraries) for unsupervised,          bulk symmetric encryption/decryption of files Linux-Fan <Ma_Sys.ma@web.de> - 2019-01-10 00:50 +0100
    Re: Looking for advice on tools (or libraries) for unsupervised,  bulk symmetric encryption/decryption of files Celejar <celejar@gmail.com> - 2019-01-10 04:20 +0100
      Re: Looking for advice on tools (or libraries) for unsupervised,  bulk symmetric encryption/decryption of files Jonathan Dowland <jmtd@debian.org> - 2019-01-11 22:50 +0100
        Re: Looking for advice on tools (or libraries) for unsupervised,  bulk symmetric encryption/decryption of files Celejar <celejar@gmail.com> - 2019-01-13 04:50 +0100
          Re: Looking for advice on tools (or libraries) for unsupervised, bulk  symmetric encryption/decryption of files John Crawley <john@bunsenlabs.org> - 2019-01-13 10:00 +0100
    Re: Looking for advice on tools (or libraries) for unsupervised, bulk  symmetric encryption/decryption of files Ben Caradoc-Davies <ben@transient.nz> - 2019-01-10 06:10 +0100

#204237 — Looking for advice on tools (or libraries) for unsupervised, bulk symmetric encryption/decryption of files

[Multipart message — attachments visible in raw view] — view raw

I am looking for software to symmetric-encrypt large numbers of files on
disk (terabytes' worth of data), and would appreciate some advice.

My basic requirements:

   - It should be open source and no-cost (though, since I'm asking this
   question here, this goes without saying);
   - I should be able to program scripts (shell, Python, Perl, or Ruby) to
   run this software without human intervention; this rules out tools that are
   designed for interactive use.
   - It should be stable; I should be able to decrypt encrypted files that
   were encrypted several years earlier; (how much earlier?  hard to say;
   let's say 10 years, as a rough ballpark)

In addition, the following would be nice:

   - good documentation;
   - good performance;
   - bindings for a high-level language (preferably Python).

The only encryption tool I have used for encrypting files on my hard drive
is gpg2, which I have used for small, interactive encryption tasks
(half-dozen files, at most).

Therefore, my initial attempt was to use gpg2 for this new bulk-encryption
task, but I found myself constantly fighting with it, and finally had to
recognize that I was trying to use gpg2 for something it is not primarily
designed for.  (I am also a bit concerned with gpg2's future stability.
AFAICT, It's design has varied significantly over the years, and as a
result there's a lot of confusion on its use.  That has been my experience,
in any case.)

So I am back to square one.

I stress that I am interested only in symmetric encryption.  The issues
that asymmetric encryption addresses are not at all part of the problem I
am dealing with, and therefore I don't want to have to deal with the
complexities of asymmetric encryption.

What Debian packages would you recommend?

Many thanks in advance!

kj

[toc] | [next] | [standalone]

#204259

On 1/9/19 6:05 AM, Kynn Jones wrote:
> I am looking for software to symmetric-encrypt large numbers of files on
> disk (terabytes' worth of data), and would appreciate some advice.
> 
> My basic requirements:
> 
>     - It should be open source and no-cost (though, since I'm asking this
>     question here, this goes without saying);
>     - I should be able to program scripts (shell, Python, Perl, or Ruby) to
>     run this software without human intervention; this rules out tools that are
>     designed for interactive use.
>     - It should be stable; I should be able to decrypt encrypted files that
>     were encrypted several years earlier; (how much earlier?  hard to say;
>     let's say 10 years, as a rough ballpark)
> 
> In addition, the following would be nice:
> 
>     - good documentation;
>     - good performance;
>     - bindings for a high-level language (preferably Python).
> 
> The only encryption tool I have used for encrypting files on my hard drive
> is gpg2, which I have used for small, interactive encryption tasks
> (half-dozen files, at most).
> 
> Therefore, my initial attempt was to use gpg2 for this new bulk-encryption
> task, but I found myself constantly fighting with it, and finally had to
> recognize that I was trying to use gpg2 for something it is not primarily
> designed for.  (I am also a bit concerned with gpg2's future stability.
> AFAICT, It's design has varied significantly over the years, and as a
> result there's a lot of confusion on its use.  That has been my experience,
> in any case.)
> 
> So I am back to square one.
> 
> I stress that I am interested only in symmetric encryption.  The issues
> that asymmetric encryption addresses are not at all part of the problem I
> am dealing with, and therefore I don't want to have to deal with the
> complexities of asymmetric encryption.
> 
> What Debian packages would you recommend?
> 
> Many thanks in advance!
> 
> kj

I use the ccrypt(1) suite for encrypting files using only a password 
(e.g. not PKI).  I wrap the CLI tools with Bash and/or Perl scripts to 
automate repetitive tasks (Python, Ruby, and other scripting languages 
should also work).  It is available via the Debian package 'ccrypt'.

http://ccrypt.sourceforge.net/


David

[toc] | [prev] | [next] | [standalone]

#204272 — Re: Looking for advice on tools (or libraries) for unsupervised, bulk symmetric encryption/decryption of files

[Multipart message — attachments visible in raw view] — view raw

Kynn Jones writes:

> I am looking for software to symmetric-encrypt large numbers of files on disk  
> (terabytes' worth of data), and would appreciate some advice.
>
> My basic requirements:
>
> • It should be open source and no-cost (though, since I'm asking this  
> question here, this goes without saying);
>
> • I should be able to program scripts (shell, Python, Perl, or Ruby) to run  
> this software without human intervention; this rules out tools that are  
> designed for interactive use.
>
> • It should be stable; I should be able to decrypt encrypted files that were  
> encrypted several years earlier; (how much earlier?  hard to say; let's say  
> 10 years, as a rough ballpark)
> In addition, the following would be nice:
>
> • good documentation;
>
> • good performance;
>
> • bindings for a high-level language (preferably Python).
>
> The only encryption tool I have used for encrypting files on my hard drive is  
> gpg2, which I have used for small, interactive encryption tasks (half-dozen  
> files, at most).

[...]

> What Debian packages would you recommend?

I actually tend to use 7-Zip for symmetric file encryption a lot because it  
ensures cross-platform compatibility and many users have 7-Zip already  
installed (I am always afraid that one day decryption software might not be  
available because then data would be close to being lost). The Debian  
package is `p7zip-full`.

Another program which I like is AESCrypt. Unfortunately it seems it is not  
included in Debian, but one might be able to install it via PIP (I have only  
ever used the Java and C version). I have actually read the Java  
implementation (which is also available as a very minimalistic commandline  
utility) and found the code understandable which is always a bonus when it  
comes to security :) Additionally, it was simple to adapt the library to  
provide a slightly different API and the result is still compatible with the  
AESCrypt commandline. As a result, you can use the API to automatically  
process data and the commandline utility to manually extract the data should  
something go wrong with the automatism. AESCrypt also seems to have Python  
bindings (but I have not used them). See https://www.aescrypt.com.

HTH
Linux-Fan

[toc] | [prev] | [next] | [standalone]

#204279 — Re: Looking for advice on tools (or libraries) for unsupervised, bulk symmetric encryption/decryption of files

On Wed, 9 Jan 2019 09:05:32 -0500
Kynn Jones <kynnjo@gmail.com> wrote:

> I am looking for software to symmetric-encrypt large numbers of files on
> disk (terabytes' worth of data), and would appreciate some advice.
> 
> My basic requirements:
> 
>    - It should be open source and no-cost (though, since I'm asking this
>    question here, this goes without saying);
>    - I should be able to program scripts (shell, Python, Perl, or Ruby) to
>    run this software without human intervention; this rules out tools that are
>    designed for interactive use.
>    - It should be stable; I should be able to decrypt encrypted files that
>    were encrypted several years earlier; (how much earlier?  hard to say;
>    let's say 10 years, as a rough ballpark)
> 
> In addition, the following would be nice:
> 
>    - good documentation;
>    - good performance;
>    - bindings for a high-level language (preferably Python).

The standard encryption technology for linux is LUKS. It works on the
block device level, not the file level. [I may be using the terminology
inaccurately.] You'll find the best compatibility, stability, and
documentation with LUKS, but you don't use it to create encrypted
copies of files - rather, you create a LUKS encrypted device, and
copy / move your files there. The LUKS tools (cryptsetup) are only
necessary for the creation and management of the encrypted volume; once
it's set up, you use the ondinary filesystem tools (cp, mv, ls, etc.)
to access files and move them to and from encrypted storage.

I believe that the most commonly used software for file level
encryption is EncFS. I haven't really used it much, and can't speak to
its long term stablity.



Celejar

[toc] | [prev] | [next] | [standalone]

#204326 — Re: Looking for advice on tools (or libraries) for unsupervised, bulk symmetric encryption/decryption of files

On Wed, Jan 09, 2019 at 10:18:47PM -0500, Celejar wrote:
>The standard encryption technology for linux is LUKS. It works on the
>block device level, not the file level.

LUKS would be no good if the user wants to move/copy/share the encrypted
files, encrypted, elsewhere: they didn't say so explicitly but that's
the impression I got reading their message.

>I believe that the most commonly used software for file level
>encryption is EncFS. I haven't really used it much, and can't speak to
>its long term stablity.

EncFS should not be used for any new file encryption project, IMHO.
There was the following report in 2014:
https://defuse.ca/audits/encfs.htm
This is referenced in the NEWS file in the EncFS package
https://salsa.debian.org/debian/encfs/blob/debian/sid/debian/NEWS

Both the report and the NEWS file are 5 years sold so I am not sure of
its current status but I'd want to seek positive assurance.

-- 

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Jonathan Dowland
⢿⡄⠘⠷⠚⠋⠀ https://jmtd.net
⠈⠳⣄⠀⠀⠀⠀ Please do not CC me, I am subscribed to the list.

[toc] | [prev] | [next] | [standalone]

#204365 — Re: Looking for advice on tools (or libraries) for unsupervised, bulk symmetric encryption/decryption of files

On Fri, 11 Jan 2019 21:45:57 +0000
Jonathan Dowland <jmtd@debian.org> wrote:

> On Wed, Jan 09, 2019 at 10:18:47PM -0500, Celejar wrote:
> >The standard encryption technology for linux is LUKS. It works on the
> >block device level, not the file level.
> 
> LUKS would be no good if the user wants to move/copy/share the encrypted
> files, encrypted, elsewhere: they didn't say so explicitly but that's
> the impression I got reading their message.

You're probably right; I realized after I wrote my reply (and read some
of the other replies) that my solutions likely weren't really what the
OP was looking for.

> >I believe that the most commonly used software for file level
> >encryption is EncFS. I haven't really used it much, and can't speak to
> >its long term stablity.
> 
> EncFS should not be used for any new file encryption project, IMHO.
> There was the following report in 2014:
> https://defuse.ca/audits/encfs.htm
> This is referenced in the NEWS file in the EncFS package
> https://salsa.debian.org/debian/encfs/blob/debian/sid/debian/NEWS
> 
> Both the report and the NEWS file are 5 years sold so I am not sure of
> its current status but I'd want to seek positive assurance.

Huh - good to know. But I was wondering, along similar (but less
informed) lines, how good some of the other suggestions were, e.g.
ccrypt. I know very little about ccrypt, but has it even been
audited at all? Is it sufficiently widely used that any vulnerablities
or misimplementations of the sort discovered by the EncFS audit would
have been looked for or turned up?

Celejar

[toc] | [prev] | [next] | [standalone]

#204368

On 13/01/2019 12.46, Celejar wrote:
> On Fri, 11 Jan 2019 21:45:57 +0000
>>> I believe that the most commonly used software for file level
>>> encryption is EncFS. I haven't really used it much, and can't speak to
>>> its long term stablity.
>>
>> EncFS should not be used for any new file encryption project, IMHO.
>> There was the following report in 2014:
>> https://defuse.ca/audits/encfs.htm
>> This is referenced in the NEWS file in the EncFS package
>> https://salsa.debian.org/debian/encfs/blob/debian/sid/debian/NEWS
>>
>> Both the report and the NEWS file are 5 years sold so I am not sure of
>> its current status but I'd want to seek positive assurance.
> 
> Huh - good to know. But I was wondering, along similar (but less
> informed) lines, how good some of the other suggestions were, e.g.
> ccrypt. I know very little about ccrypt, but has it even been
> audited at all? Is it sufficiently widely used that any vulnerablities
> or misimplementations of the sort discovered by the EncFS audit would
> have been looked for or turned up?

Looking at encfs, gocryptfs showed up, which claims "This project was  
inspired by EncFS and strives to fix its security issues while providing  
good performance":
https://packages.debian.org/stretch/gocryptfs
https://github.com/rfjakob/gocryptfs

No personal experience (yet) of using it though.
-- 
John

[toc] | [prev] | [next] | [standalone]

#204281

On 10/01/2019 03:05, Kynn Jones wrote:
> The only encryption tool I have used for encrypting files on my hard drive
> is gpg2, which I have used for small, interactive encryption tasks
> (half-dozen files, at most).
> Therefore, my initial attempt was to use gpg2 for this new bulk-encryption
> task, but I found myself constantly fighting with it, and finally had to
> recognize that I was trying to use gpg2 for something it is not primarily
> designed for.  (I am also a bit concerned with gpg2's future stability.
> AFAICT, It's design has varied significantly over the years, and as a
> result there's a lot of confusion on its use.  That has been my experience,
> in any case.)

I use a pipe with gpg2 as one component for symmetric encryption:

gpg --batch --symmetric --cipher-algo AES256 --s2k-digest-algo SHA512 
--compress-algo none --passphrase-file $PASSPHRASE_FILE

My pipe input is usually a tar file gzipped with pigz for parallel 
compression, hence the "--compress-algo none". I then add another "pigz 
-0" wrapper to get a cryptographically weak checksum to allow testing 
for media failures without the passphrase. I like tar because it 
preserves file metadata and filesystem structure and is a very stable 
format. Other formats may be better for random access.

Recently I used gpg2 to decrypt files that were encrypted over 15 years 
ago; note that these were much smaller files and a simpler invocation of 
gpg1 (the then default cipher was CAST5 IIRC). The gpg file format seems 
well-documented and stable. Regular decryption tests are prudent to 
catch problems after gpg upgrade. Yes, the new interactive predilections 
of gpg2 were a pain at first when compared to gpg1, but "--batch" and 
"--passphrase-file" seem sufficient for batch symmetric encryption, if 
you do not mind your passphrase being in plain text on your filesystem.

Kind regards,

-- 
Ben Caradoc-Davies <ben@transient.nz>
Director
Transient Software Limited <https://transient.nz/>
New Zealand

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web