Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD

From	piorunz <piorunz@gmx.com>
Newsgroups	linux.debian.user
Subject	Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD
Date	2022-01-03 00:40 +0100
Message-ID	<DBiCe-VP-5@gated-at.bofh.it> (permalink)
References	<Ds67g-4Z6-7@gated-at.bofh.it> <Ds6Ai-58M-3@gated-at.bofh.it> <DsaXf-7O5-1@gated-at.bofh.it> <Dse4P-1bb-9@gated-at.bofh.it> <DBc3M-5qD-19@gated-at.bofh.it>
Organization	linux.* mail to news gateway

Show all headers | View raw

On 02/01/2022 16:33, Jorge P. de Morais Neto wrote:
> Hi Piotr!  Happy 2022!

Hi Jorge!

Likewise!

> Have you revised after kernel Linux 4.15?  The btrfs(5) manpage says:
>
>             Since kernel 4.15, a set of heuristic algorithms have been
>             improved by using frequency sampling, repeated pattern
>             detection and Shannon entropy calculation to avoid that.
>
> Therefore, it looks like after kernel Linux 4.15 the compress option (or
> compress=ALG:LEVEL), instead of compress-force, became more interesting.

Full paragraph says:
"Otherwise some simple heuristics are applied to detect an
incompressible file. If the first blocks written to a file are not
compressible, the whole file is permanently marked to skip compression.
As this is too simple, the compress-force is a workaround that will
compress most of the files at the cost of some wasted CPU cycles on
failed attempts. Since kernel 4.15, a set of heuristic algorithms have
been improved by using frequency sampling, repeated pattern detection
and Shannon entropy calculation to avoid that."

Meaning, compress has been improved to detect first block-uncompressible
rest-compressible files, and so on. But only improved.
Compress-force still remains 100% efficient in compressing everything,
for example 40GB VM disk file, where first 20GB is well used, but
remaining 20GB are empty with zeros, or zero-like patterns, which are
highly compressible. However "improved" compress is, I doubt it will
catch all these edge cases. I entrust compress-force to compress
everything for me without misses, at the cost of some CPU usage, most
likely much smaller CPU footprint than chosen compression level itself.

> I am currently using compress-force=zstd:12 for the SSD and
> compress=zstd:12 for both HDD (internal SATA and external USB3)¹.
> Despite the strong compression level, performance is pretty good.  Yet,
> when the system settles, I intend to reduce compression level to 9 or 6
> (as you earlier recommended).  This should make performance even better,
> while saving a lot of space because most data was compressed at
> level 12.

I run compress-force=zstd:6 on my fast PC, compress-force=zstd:3 on my
server (to give it a bit more breathing space), and also same level 3 on
my laptop. 12 is quite high for SSD, are you sure you not slowing down
peak performance of your SSD by intense CPU usage? Or is it by design,
to reduce number of writes to SSD?

> And I may also change compress-force to compress, even for the SSD,
> because I run kernel 5.10 which is later than 4.15.  I may ask the
> linux-btrfs mailing list first.  I have subscribed to it.  Are you there
> too?

No, I am not on this list, where is it? Please send a link!
>
> Regards
>
> * Footnotes
>
> ¹ Both HDD have compress (rather than compress-force) because most of
> their files are already compressed---pictures, videos, music, compressed
> archives etc.

Makes sense!

--
With kindest regards, Piotr.

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org/
⠈⠳⣄⠀⠀⠀⠀

Thread

Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD "Jorge P. de Morais Neto" <jorge+list@disroot.org> - 2021-12-08 15:30 +0100
  Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD piorunz <piorunz@gmx.com> - 2021-12-08 16:00 +0100
    Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD Jorge P. de Morais Neto <jorge+list@disroot.org> - 2021-12-08 20:40 +0100
      Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD piorunz <piorunz@gmx.com> - 2021-12-09 00:00 +0100
        Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD Jorge P. de Morais Neto <jorge+list@disroot.org> - 2022-01-02 17:40 +0100
          Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD piorunz <piorunz@gmx.com> - 2022-01-03 00:40 +0100
            Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD Jorge P. de Morais Neto <jorge+list@disroot.org> - 2022-01-03 12:50 +0100
              Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD Michael Stone <mstone@debian.org> - 2022-01-03 16:20 +0100
                Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD Dan Ritter <dsr@randomstring.org> - 2022-01-03 17:00 +0100
                Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD Michael Stone <mstone@debian.org> - 2022-01-03 19:30 +0100
                Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD Jorge P. de Morais Neto <jorge+list@disroot.org> - 2022-01-04 01:00 +0100
                Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD Michael Stone <mstone@debian.org> - 2022-01-04 19:20 +0100
      Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD piorunz <piorunz@gmx.com> - 2021-12-09 14:30 +0100
  Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD "Alexander V. Makartsev" <avbetev@gmail.com> - 2021-12-09 01:20 +0100
    Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD Andy Smith <andy@strugglers.net> - 2021-12-09 02:10 +0100
      Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD Jorge P. de Morais Neto <jorge+list@disroot.org> - 2021-12-15 15:50 +0100
        Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD Andy Smith <andy@strugglers.net> - 2021-12-16 04:30 +0100
    Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD piorunz <piorunz@gmx.com> - 2021-12-09 11:30 +0100
      Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD "Alexander V. Makartsev" <avbetev@gmail.com> - 2021-12-09 16:30 +0100
        Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD piorunz <piorunz@gmx.com> - 2021-12-09 17:20 +0100
          Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD piorunz <piorunz@gmx.com> - 2021-12-09 18:30 +0100
    Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD Jorge P. de Morais Neto <jorge+list@disroot.org> - 2021-12-15 15:40 +0100
  Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD David Christensen <dpchrist@holgerdanske.com> - 2021-12-09 07:10 +0100
    Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD Jorge P. de Morais Neto <jorge+list@disroot.org> - 2021-12-14 18:50 +0100
      Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD David Christensen <dpchrist@holgerdanske.com> - 2021-12-15 03:50 +0100
        Re: Why MBR partitioning (was: Reasonably simple setup for 1...) Felix Miata <mrmazda@earthlink.net> - 2021-12-15 04:00 +0100
          Re: Why MBR partitioning (was: Reasonably simple setup for 1...) David Christensen <dpchrist@holgerdanske.com> - 2021-12-15 05:20 +0100
          Re: Why MBR partitioning (was: Reasonably simple setup for 1...) "Thomas Schmitt" <scdbackup@gmx.net> - 2021-12-15 09:10 +0100
  Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD Anssi Saari <as@sci.fi> - 2021-12-09 10:40 +0100
    Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD Dan Ritter <dsr@randomstring.org> - 2021-12-09 13:10 +0100
    Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD Jorge P. de Morais Neto <jorge+list@disroot.org> - 2021-12-14 18:40 +0100
      Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD Andrei POPESCU <andreimpopescu@gmail.com> - 2021-12-15 21:30 +0100
  Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD "hdv@gmail" <hdv.jadev@gmail.com> - 2021-12-09 15:10 +0100
    Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD Dan Ritter <dsr@randomstring.org> - 2021-12-09 16:10 +0100
      Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD "hdv@gmail" <hdv.jadev@gmail.com> - 2021-12-09 16:50 +0100
        Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD Curt <curty@free.fr> - 2021-12-09 18:30 +0100
          Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD David Wright <deblis@lionunicorn.co.uk> - 2021-12-10 06:00 +0100
            Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD Darac Marjal <mailinglist@darac.org.uk> - 2021-12-10 10:30 +0100
      Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD Stefan Monnier <monnier@iro.umontreal.ca> - 2021-12-10 05:10 +0100
      Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD Andrei POPESCU <andreimpopescu@gmail.com> - 2021-12-10 10:40 +0100
        Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD Dan Ritter <dsr@randomstring.org> - 2021-12-10 13:40 +0100
          Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD Andrei POPESCU <andreimpopescu@gmail.com> - 2021-12-10 14:00 +0100
            Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD Dan Ritter <dsr@randomstring.org> - 2021-12-10 15:10 +0100
              Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD Charles Curley <charlescurley@charlescurley.com> - 2021-12-11 02:10 +0100
                Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD "Andrew M.A. Cater" <amacater@einval.com> - 2021-12-11 12:30 +0100
          Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD Darac Marjal <mailinglist@darac.org.uk> - 2021-12-10 14:20 +0100
    Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD Jorge P. de Morais Neto <jorge+list@disroot.org> - 2021-12-15 16:00 +0100
      Re: Reasonably simple setup for 1TB HDD and 250GB M.2 NVMe SSD "hdv@gmail" <hdv.jadev@gmail.com> - 2021-12-15 16:30 +0100

csiph-web