Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #200631

Re: btrfs and deduplication

Show all headers | View raw

On 2018-09-26 00:40, Anders Andersson wrote:
> On Tue, Sep 25, 2018 at 9:56 AM, Markus Raps <m.raps@rapsplace.de> 
> wrote:
>> Hi there,
>> 
>> currently iam trying to get deduplication working in debian/btrfs
>> 
>> #so i created a btrfs filesystem
>> 
>> mkfs.btrfs /dev/vdb1
>> mkdir /mnt/btrfs
>> mount /dev/vdb1 /mnt/btrfs
>> 
>> # create some random file
>> dd if=/dev/urandom of=/mnt/btrfs/img bs=1M count=1024
>> for i in {1..30}; do cp /mnt/btrfs/img /mnt/btrfs/img$i; done
>> 
>> # dedup this stuff
>> jdupes -S -B -r /mnt/btrfs/
>> Examining 31 files, 1 dirs (in 1 specified)
>> Deduplication done (30 files processed)
>> 
>> raps-debian btrfs # btrfs fi df /mnt/btrfs/
>> Data, single: total=31.50GiB, used=30.96GiB
>> System, DUP: total=32.00MiB, used=16.00KiB
>> Metadata, DUP: total=222.94MiB, used=33.69MiB
>> GlobalReserve, single: total=33.56MiB, used=0.00B
>> 
>> raps-debian btrfs # btrfs fi usage /mnt/btrfs/
>> Overall:
>>     Device size:                  32.00GiB
>>     Device allocated:             32.00GiB
>>     Device unallocated:            1.04MiB
>>     Device missing:                  0.00B
>>     Used:                         31.03GiB
>>     Free (estimated):            549.41MiB      (min: 549.41MiB)
>>     Data ratio:                       1.00
>>     Metadata ratio:                   2.00
>>     Global reserve:               33.56MiB      (used: 0.00B)
>> 
>> Data,single: Size:31.50GiB, Used:30.96GiB
>>    /dev/vdb1      31.50GiB
>> 
>> Metadata,DUP: Size:222.94MiB, Used:33.69MiB
>>    /dev/vdb1     445.88MiB
>> 
>> System,DUP: Size:32.00MiB, Used:16.00KiB
>>    /dev/vdb1      64.00MiB
>> 
>> Unallocated:
>>    /dev/vdb1       1.04MiB
>> raps-debian btrfs #
>> 
>> 
>> 
>> hm ... hasn't worked.
>> wrong tool ? did i missed something
>> or have i completely misunderstood deduplication?
> 
> 
> I can not answer this, but as a long time user of btrfs who has never
> tried deduplication I got curious, so I tried to recreate your result
> on my debian testing workstation, and for me it worked. Here is the
> log of exactly how I created the filesystem, and the output from btrfs
> fi usage before and after running jdupes. As you can see from the
> usage after deduplication, both the allocated and used space has
> decreased to what would be expected.
> 
> Perhaps your kernel is too old? It is possible that you are being hit
> by the issue described here:
> https://btrfs.wiki.kernel.org/index.php/Changelog#By_feature
> 
> "The range for out-of-band deduplication implemented by the
> EXTENT_SAME ioctl will split the range into 16MiB chunks. Up to now
> this was the overall limit and effectively only the first 16MiB was
> deduplicated."
> 
> I can't say I understand much about it, but it was fixed in kernel
> 4.18 which happens to be what I use.
> 
> 
> 
> # btrfs --version
> btrfs-progs v4.17
> # uname -a
> Linux spacelab 4.18.0-1-amd64 #1 SMP Debian 4.18.6-1 (2018-09-06)
> x86_64 GNU/Linux
> # lvcreate spacelab -n dedup -L 100G
> # mkfs.btrfs /dev/spacelab/dedup
> # mount /dev/spacelab/dedup /mnt/
> # dd if=/dev/urandom of=/mnt/img bs=1M count=1024
> # for i in {1..30}; do cp /mnt/img /mnt/img$i;done
> # btrfs fi usage /mnt
> Overall:
>     Device size:         100.00GiB
>     Device allocated:          33.02GiB
>     Device unallocated:          66.98GiB
>     Device missing:             0.00B
>     Used:              31.04GiB
>     Free (estimated):          67.98GiB    (min: 67.98GiB)
>     Data ratio:                  1.00
>     Metadata ratio:              1.00
>     Global reserve:          33.23MiB    (used: 0.00B)
> 
> Data,single: Size:32.01GiB, Used:31.01GiB
>    /dev/mapper/spacelab-dedup      32.01GiB
> 
> Metadata,single: Size:1.01GiB, Used:33.36MiB
>    /dev/mapper/spacelab-dedup       1.01GiB
> 
> System,single: Size:4.00MiB, Used:16.00KiB
>    /dev/mapper/spacelab-dedup       4.00MiB
> 
> Unallocated:
>    /dev/mapper/spacelab-dedup      66.98GiB
> # jdupes -S -B -r /mnt
> Scanning: 31 files, 1 items (in 1 specified)
> Deduplication done (30 files processed)
> # btrfs fi usage /mnt
> Overall:
>     Device size:         100.00GiB
>     Device allocated:           2.02GiB
>     Device unallocated:          97.98GiB
>     Device missing:             0.00B
>     Used:               1.00GiB
>     Free (estimated):          97.99GiB    (min: 97.99GiB)
>     Data ratio:                  1.00
>     Metadata ratio:              1.00
>     Global reserve:          16.00MiB    (used: 0.00B)
> 
> Data,single: Size:1.01GiB, Used:1.00GiB
>    /dev/mapper/spacelab-dedup       1.01GiB
> 
> Metadata,single: Size:1.01GiB, Used:1.38MiB
>    /dev/mapper/spacelab-dedup       1.01GiB
> 
> System,single: Size:4.00MiB, Used:16.00KiB
>    /dev/mapper/spacelab-dedup       4.00MiB
> 
> Unallocated:
>    /dev/mapper/spacelab-dedup      97.98GiB

just updated the kernel to 4.18
now it works

thank you



-- 
Mit freundlichen Grüßen / best regards
Markus Raps

Back to linux.debian.user | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

btrfs and deduplication Markus Raps <m.raps@rapsplace.de> - 2018-09-25 10:20 +0200
  Re: btrfs and deduplication Anders Andersson <pipatron@gmail.com> - 2018-09-26 00:50 +0200
    Re: btrfs and deduplication Markus Raps <m.raps@rapsplace.de> - 2018-09-26 11:40 +0200

csiph-web