Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #243180 > unrolled thread

Btrfs best practices

Back to article view | Back to linux.debian.user

Contents

  Btrfs best practices "Jorge P. de Morais Neto" <jorge+list@disroot.org> - 2021-12-16 18:20 +0100
    Re: Btrfs best practices Eduardo M KALINOWSKI <eduardo@kalinowski.com.br> - 2021-12-16 19:00 +0100
      Re: Btrfs best practices Jorge P. de Morais Neto <jorge+list@disroot.org> - 2021-12-16 19:20 +0100
    Re: Btrfs best practices Jorge P. de Morais Neto <jorge+list@disroot.org> - 2021-12-16 19:20 +0100
    Re: Btrfs best practices: defrag? piorunz <piorunz@gmx.com> - 2022-01-26 21:40 +0100
      Re: Btrfs best practices: defrag? Andy Smith <andy@strugglers.net> - 2022-01-27 23:20 +0100
    Re: Btrfs best practices Jorge P. de Morais Neto <jorge+list@disroot.org> - 2022-01-29 21:00 +0100

#243180 — Btrfs best practices

Hi!  I own a Dell Inspiron 5570 laptop with 1 TB SATA HDD, a new 250 GB
NVMe SSD¹ and 16 GiB RAM.  I seek reliability, durability, performance
and power efficiency.

I do weekly duplicity backups to external 1.5 TB USB3 HDD.  I'll start
also daily rsyncing some of the SSD data to the SATA HDD.

The SSD will have 50 GB extra over provisioning and a 200 GB partition,
besides the special UEFI partition.  The SATA HDD will start with 16 GiB
swap partition then a big partition.  I'll put system and /home on the
SSD but all XDG user dirs² on the HDD, and tmpfs on /tmp.  All three
drives will have Btrfs with space_cache=v2, noatime, zstd compression
and reasonable free breathing space.

I use Gnome and:
- GNU Emacs
- notmuch and offlineimap (I may switch to mbsync)
- GNU IceCat, Mozilla Firefox and ungoogled-chromium
- Gajim and GNU Jami
- Gnome Boxes or Virtual Machine Manager running a VM with 2 GiB RAM and
  one .qcow2 disk image currently weighting 24 GB.
- mpv
- Nextcloud (always running but rarely syncing changes)

I use Debian stable with only official repositories, including
bullseye-backports.  I manually installed GNU Guix package manager and
have 163 packages on main Guix profile.

* Doubts
** Backported kernel
Should I use a backported kernel as Btrfs [wiki][] recommends?  I worry
that bullseye-backports comes from Debian testing with poor security.

[wiki]: https://btrfs.wiki.kernel.org/index.php/Getting_started#Before_you_start

** Strong compression during install
For lifetime and space saving, I intend to install Debian to the SSD
with compress-force=zstd:12, but then adopt compress-force=zstd.  Thus
the installation will be slow---I'll do something else while the
installer works---but the installed system will be efficient, right?

** HDD Compression
Both HDD have a lot of already-compressed data: videos, audio, photos
and compressed archives and disk images; compress-force would force
Btrfs to recompress it all, only to discard the recompressed data and
store the original.  Therefore compress=zstd:4 would be better, right?

** Fragmentation
Is fragmentation a concern?  Is the [Gotchas][] article accurate?

[Gotchas]: https://btrfs.wiki.kernel.org/index.php/Gotchas#Fragmentation

** Subvolumes
What about
https://fedoraproject.org/wiki/Changes/BtrfsByDefault#Additional_subvolumes ?

** Swappiness
Most performance-critical data will be on the SSD, so there will be much
less need for RAM caches; therefore I should decrease swappiness
(especially if I put swap on the SSD), right?  By how much?

* Footnotes
¹ A 250 GB WD Blue SN550 rated for 150 TBW.
² See the xdg-user-dir manpage.

Kindest regards,
  Jorge

-- 
- Many people hate injustice but few check the facts; this causes more
  injustice.  Ask me about <https://stallmansupport.org>
- I am Brazilian.  I hope my English is correct and I welcome feedback.
- https://www.defectivebydesign.org
- https://www.gnu.org

[toc] | [next] | [standalone]

#243181

On 16/12/2021 14:13, Jorge P. de Morais Neto wrote:
> I'll put system and /home on the
> SSD but all XDG user dirs² on the HDD [snip]

I don't have that manpage installed, but if you're refering to 
~/.config, ~/.local, etc, these are exactly the kinds of things that 
should be on the SSD - it'll help with application startup times as 
files in those directories are read when applications start.

Put large files (photos, music, video, etc) on the HD. For everything 
else, prefer the SSD.

As for the other questions, I cannot help specifically, but as a general 
advice, don't overthink it and don't bother trying to optimize 
everything up to the smallest details. Unless you have some very 
specific use, default settings are good enough, you should'nt notice any 
different in day to day use.

-- 
The best way to get rid of worries is to let them die of neglect.

Eduardo M KALINOWSKI
eduardo@kalinowski.com.br

[toc] | [prev] | [next] | [standalone]

#243182

Hi.

Em [2021-12-16 qui 14:55:23-0300], Eduardo M KALINOWSKI escreveu:

> On 16/12/2021 14:13, Jorge P. de Morais Neto wrote:
>> I'll put system and /home on the SSD but all XDG user dirs² on the
>> HDD [snip]
>
> I don't have that manpage installed, but if you're refering to
> ~/.config, ~/.local, etc, these are exactly the kinds of things that
> should be on the SSD - it'll help with application startup times as
> files in those directories are read when applications start.

No, I was referring to:

    DESKTOP
    DOWNLOAD
    TEMPLATES
    PUBLICSHARE
    DOCUMENTS
    MUSIC
    PICTURES
    VIDEOS

> As for the other questions, I cannot help specifically, but as a
> general advice, don't overthink it and don't bother trying to optimize
> everything up to the smallest details.  Unless you have some very
> specific use, default settings are good enough, you should'nt notice
> any different in day to day use.

Thank you for the advice!  I will think about it.  I must not return to
the days when I riced Gentoo for hours every week trying to gain a tiny
bit of extra performance.  There must be moderation.

Regards

-- 
- Many people hate injustice but few check the facts; this causes more
  injustice.  Ask me about <https://stallmansupport.org>
- Please adopt free/libre formats like PDF, Org, LaTeX, ODF, Opus, WebM and 7z.
- Libre apps for AOSP (Replicant, LineageOS, etc.) and Android: F-Droid
- https://www.gnu.org/philosophy/free-sw.html "What is free software?"

[toc] | [prev] | [next] | [standalone]

#243183

Hi.  I must add the information that I use zswap:

GRUB_CMDLINE_LINUX_DEFAULT="quiet zswap.enabled=1 zswap.zpool=z3fold zswap.compressor=lzo-rle"

Em [2021-12-16 qui 14:13:05-0300], Jorge P. de Morais Neto escreveu:

> Hi!  I own a Dell Inspiron 5570 laptop with 1 TB SATA HDD, a new 250 GB
> NVMe SSD¹ and 16 GiB RAM.  I seek reliability, durability, performance
> and power efficiency.
>
> I do weekly duplicity backups to external 1.5 TB USB3 HDD.  I'll start
> also daily rsyncing some of the SSD data to the SATA HDD.
>
> The SSD will have 50 GB extra over provisioning and a 200 GB partition,
> besides the special UEFI partition.  The SATA HDD will start with 16 GiB
> swap partition then a big partition.  I'll put system and /home on the
> SSD but all XDG user dirs² on the HDD, and tmpfs on /tmp.  All three
> drives will have Btrfs with space_cache=v2, noatime, zstd compression
> and reasonable free breathing space.
>
> I use Gnome and:
> - GNU Emacs
> - notmuch and offlineimap (I may switch to mbsync)
> - GNU IceCat, Mozilla Firefox and ungoogled-chromium
> - Gajim and GNU Jami
> - Gnome Boxes or Virtual Machine Manager running a VM with 2 GiB RAM and
>   one .qcow2 disk image currently weighting 24 GB.
> - mpv
> - Nextcloud (always running but rarely syncing changes)
>
> I use Debian stable with only official repositories, including
> bullseye-backports.  I manually installed GNU Guix package manager and
> have 163 packages on main Guix profile.
>
> * Doubts
> ** Backported kernel
> Should I use a backported kernel as Btrfs [wiki][] recommends?  I worry
> that bullseye-backports comes from Debian testing with poor security.
>
> [wiki]: https://btrfs.wiki.kernel.org/index.php/Getting_started#Before_you_start
>
> ** Strong compression during install
> For lifetime and space saving, I intend to install Debian to the SSD
> with compress-force=zstd:12, but then adopt compress-force=zstd.  Thus
> the installation will be slow---I'll do something else while the
> installer works---but the installed system will be efficient, right?
>
> ** HDD Compression
> Both HDD have a lot of already-compressed data: videos, audio, photos
> and compressed archives and disk images; compress-force would force
> Btrfs to recompress it all, only to discard the recompressed data and
> store the original.  Therefore compress=zstd:4 would be better, right?
>
> ** Fragmentation
> Is fragmentation a concern?  Is the [Gotchas][] article accurate?
>
> [Gotchas]: https://btrfs.wiki.kernel.org/index.php/Gotchas#Fragmentation
>
> ** Subvolumes
> What about
> https://fedoraproject.org/wiki/Changes/BtrfsByDefault#Additional_subvolumes ?
>
> ** Swappiness
> Most performance-critical data will be on the SSD, so there will be much
> less need for RAM caches; therefore I should decrease swappiness
> (especially if I put swap on the SSD), right?  By how much?
>
> * Footnotes
> ¹ A 250 GB WD Blue SN550 rated for 150 TBW.
> ² See the xdg-user-dir manpage.
>
> Kindest regards,
>   Jorge
>
> -- 
> - Many people hate injustice but few check the facts; this causes more
>   injustice.  Ask me about <https://stallmansupport.org>
> - I am Brazilian.  I hope my English is correct and I welcome feedback.
> - https://www.defectivebydesign.org
> - https://www.gnu.org

-- 
- Many people hate injustice but few check the facts; this causes more
  injustice.  Ask me about <https://stallmansupport.org>
- I am Brazilian.  I hope my English is correct and I welcome feedback.
- Free Software Supporter: https://www.fsf.org/free-software-supporter
- If an email of mine arrives at your spam box, please notify me.

[toc] | [prev] | [next] | [standalone]

#244641 — Re: Btrfs best practices: defrag?

I'd like to add question to this topic.

Is it safe & recommended to run autodefrag mount option in fstab?
I am considering two machines here, normal desktop which has Btrfs as
/home, and server with VM and other databases also btrfs /home. Both
Btrfs RAID10 types. Both are heavily fragmented. I never defragmented
them, in fact.

Running manual defrag on server machine, like:
sudo btrfs filesystem defrag -v -t4G -r /home
takes ages and can cause 120 second timeout kernel error in dmesg due to
service timeouts. I prefer to autodefrag gradually, overtime, mount
option seems to be good for that.

My current fstab mounting:

noatime,space_cache=v2,compress-force=zstd:3 0 2

Will autodefrag break COW files? Like I copy paste a file and I save
space, but defrag with destroy this space saving?
Also, will autodefrag compress files automatically, as mount option
enforces (compress-force=zstd:3)?

Any suggestions welcome.

--
With kindest regards, Piotr.

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org/
⠈⠳⣄⠀⠀⠀⠀

[toc] | [prev] | [next] | [standalone]

#244697 — Re: Btrfs best practices: defrag?

Hello,

On Wed, Jan 26, 2022 at 08:38:41PM +0000, piorunz wrote:
> My current fstab mounting:
> 
> noatime,space_cache=v2,compress-force=zstd:3 0 2
> 
> Will autodefrag break COW files? Like I copy paste a file and I save
> space, but defrag with destroy this space saving?

Yes, I believe so. That is, if you have files sharing extents (like
if you de-duplicated them or did a "cp --reflink=always") and defrag
touches those files, it will undo the extent sharing.

> Also, will autodefrag compress files automatically, as mount option
> enforces (compress-force=zstd:3)?

Yes.

> Any suggestions welcome.

If you're not relying on extent sharing then I'd think that
automated defrag is okay.

Cheers,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

[toc] | [prev] | [next] | [standalone]

#244776

Hello!  I think I should inform this list about my choices so far:

Em [2021-12-16 qui 14:13:05-0300], Jorge P. de Morais Neto escreveu:

> Should I use a backported kernel as Btrfs [wiki][] recommends?  I worry
> that bullseye-backports comes from Debian testing with poor security.

I'm just using bullseye's kernel (5.10 LTS).

> For lifetime and space saving, I intend to install Debian to the SSD
> with compress-force=zstd:12, but then adopt compress-force=zstd.  Thus
> the installation will be slow---I'll do something else while the
> installer works---but the installed system will be efficient, right?

I'm still using compress=zstd:12 and it's performing well.  Notice I
went from "compress-force=zstd:12" to just "compress=zstd:12".  That is
because of:

    Using the forcing compression is not recommended, the heuristics are
    supposed to decide that and compression algorithms internally detect
    incompressible data too.[1]

    Btrfs contains an internal heuristics that determines if some data
    is compressible so that it doesn't try to compress data that isn't
    compressible as this wastes CPU time.  The compress-force mount
    option bypasses this heuristics in order to gain better compression
    ratios.  A downside is that this increases fragmentation with
    non-compressible files.[2]

1: https://btrfs.readthedocs.io/en/latest/Compression.html "Compression —
   BTRFS documentation"
2: https://wiki.tnonline.net/w/Btrfs/Compression#The_compress-force_mount_option
   "Btrfs/Compression - Forza's ramblings"

In the near future I intend to reduce this strong compression level (12)
to something more usual, in order to reduce power usage.

> Is fragmentation a concern?  Is the [Gotchas][] article accurate?

I now have little to worry about fragmentation, because:

1. I dedicated a raw partition to my qemu-KVM virtual machine, bypassing
   Btrfs.
2. I moved the caches of ungoogled-chromium, GNU IceCat, Firefox,
   Evolution and GNU Guix to the HDD, because they (especially the web
   browser caches) were writing too much temporary data to the SSD.
   Thus, if they ever become too fragmented, I can now just defrag them,
   without the danger of wearing the SSD.
3. I made a script to find heavily fragmented files (using compsize's
   output) and so far I have nothing to worry about.

> ** Subvolumes

I read https://en.opensuse.org/SDB:BTRFS and laid out subvolumes
according to this fstab excerpt:

LABEL=SSD /		     btrfs noatime,space_cache=v2,compress=zstd:12,subvol=@rootfs      0       0
LABEL=SSD /var/cache         btrfs noatime,space_cache=v2,compress=zstd:12,subvol=@var-cache   0       0
LABEL=SSD /var/backups       btrfs noatime,space_cache=v2,compress=zstd:12,subvol=@var-backups 0       0
LABEL=SSD /var/mail          btrfs noatime,space_cache=v2,compress=zstd:12,subvol=@var-mail    0       0
LABEL=SSD /var/spool         btrfs noatime,space_cache=v2,compress=zstd:12,subvol=@var-spool   0       0
LABEL=SSD /var/tmp           btrfs noatime,space_cache=v2,compress=zstd:12,subvol=@var-tmp     0       0
LABEL=HDD /var/log	     btrfs noatime,space_cache=v2,compress=zstd:12,subvol=@var-log     0       0
LABEL=SSD /var/lib/libvirt/images btrfs noatime,space_cache=v2,compress=zstd:12,subvol=@libvirt-images 0 0
LABEL=HDD /var/cache/apt/archives btrfs noatime,space_cache=v2,compress=zstd:12,subvol=@apt-archives 0 0
LABEL=HDD /root	             btrfs noatime,space_cache=v2,compress=zstd:12,subvol=@~root       0       0
LABEL=SSD /home		     btrfs noatime,space_cache=v2,compress=zstd:12,subvol=@home        0       0
LABEL=SSD /home/cache        btrfs noatime,space_cache=v2,compress=zstd:12,subvol=@home-cache  0       0
LABEL=HDD /home-HDD          btrfs noatime,space_cache=v2,compress=zstd:12,subvol=@home        0       0
LABEL=HDD /home-HDD/cache    btrfs noatime,space_cache=v2,compress=zstd:12,subvol=@home-cache  0       0
LABEL=SSD /gnu		     btrfs noatime,space_cache=v2,compress=zstd:12,subvol=@guix-store  0       0
LABEL=SSD /var/guix	     btrfs noatime,space_cache=v2,compress=zstd:12,subvol=@guix-var    0       0
LABEL=SSD /usr/local	     btrfs noatime,space_cache=v2,compress=zstd:12,subvol=@usr-local   0       0

Rationale:
1. In the future I could snapshot @rootfs before certain system
   operations (say, large upgrades).  If I then rollback the system to a
   snapshot, I'll still want the latest logs, user data, cache,
   libvirt-images etc., so these should be outside the @rootfs
   subvolume.  Also, including them in snapshots would be very expensive
   because some of these directories have too much variable data.
2. If I snapshot @home (probably for backup) I don't want to snapshot
   user cache (see above).
3. Some user data should be on the HDD, such as videos, music, pictures,
   downloads etc.  They are large files that would fill the SSD; and
   their usage characteristics don't require SSD performance.
4. Also some of the user cache should be on the HDD (such as web browser
   caches), to avoid wearing the SSD.

> ** Swappiness

I have not messed with swappiness.

Speaking of swap, I dedicated a 16 GiB swap partition at the start of
the HDD.

All is working well, except for some error messages I get when shutting
down.  See my email from Thu, 20 Jan 2022 08:57:35 -0300 with subject
'"Failed unmounting "/{root,var/cache} error messages when shutting down' 
(Message-ID <87zgnq62mo.fsf@disroot.org>).

Kind regards,
  Jorge

-- 
- Many people hate injustice but few check the facts; this causes more
  injustice.  Ask me about <https://stallmansupport.org>
- I am Brazilian.  I hope my English is correct and I welcome feedback.
- https://www.defectivebydesign.org
- https://www.gnu.org

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web