Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.debian.user > #200575 > unrolled thread
| Started by | Markus Raps <m.raps@rapsplace.de> |
|---|---|
| First post | 2018-09-25 10:20 +0200 |
| Last post | 2018-09-26 11:40 +0200 |
| Articles | 3 — 2 participants |
Back to article view | Back to linux.debian.user
btrfs and deduplication Markus Raps <m.raps@rapsplace.de> - 2018-09-25 10:20 +0200
Re: btrfs and deduplication Anders Andersson <pipatron@gmail.com> - 2018-09-26 00:50 +0200
Re: btrfs and deduplication Markus Raps <m.raps@rapsplace.de> - 2018-09-26 11:40 +0200
| From | Markus Raps <m.raps@rapsplace.de> |
|---|---|
| Date | 2018-09-25 10:20 +0200 |
| Subject | btrfs and deduplication |
| Message-ID | <wBQJz-8lp-3@gated-at.bofh.it> |
Hi there,
currently iam trying to get deduplication working in debian/btrfs
#so i created a btrfs filesystem
mkfs.btrfs /dev/vdb1
mkdir /mnt/btrfs
mount /dev/vdb1 /mnt/btrfs
# create some random file
dd if=/dev/urandom of=/mnt/btrfs/img bs=1M count=1024
for i in {1..30}; do cp /mnt/btrfs/img /mnt/btrfs/img$i; done
# dedup this stuff
jdupes -S -B -r /mnt/btrfs/
Examining 31 files, 1 dirs (in 1 specified)
Deduplication done (30 files processed)
raps-debian btrfs # btrfs fi df /mnt/btrfs/
Data, single: total=31.50GiB, used=30.96GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=222.94MiB, used=33.69MiB
GlobalReserve, single: total=33.56MiB, used=0.00B
raps-debian btrfs # btrfs fi usage /mnt/btrfs/
Overall:
Device size: 32.00GiB
Device allocated: 32.00GiB
Device unallocated: 1.04MiB
Device missing: 0.00B
Used: 31.03GiB
Free (estimated): 549.41MiB (min: 549.41MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 33.56MiB (used: 0.00B)
Data,single: Size:31.50GiB, Used:30.96GiB
/dev/vdb1 31.50GiB
Metadata,DUP: Size:222.94MiB, Used:33.69MiB
/dev/vdb1 445.88MiB
System,DUP: Size:32.00MiB, Used:16.00KiB
/dev/vdb1 64.00MiB
Unallocated:
/dev/vdb1 1.04MiB
raps-debian btrfs #
hm ... hasn't worked.
wrong tool ? did i missed something
or have i completely misunderstood deduplication?
Greetings
Markus
[toc] | [next] | [standalone]
| From | Anders Andersson <pipatron@gmail.com> |
|---|---|
| Date | 2018-09-26 00:50 +0200 |
| Message-ID | <wC4jv-7Jt-5@gated-at.bofh.it> |
| In reply to | #200575 |
On Tue, Sep 25, 2018 at 9:56 AM, Markus Raps <m.raps@rapsplace.de> wrote:
> Hi there,
>
> currently iam trying to get deduplication working in debian/btrfs
>
> #so i created a btrfs filesystem
>
> mkfs.btrfs /dev/vdb1
> mkdir /mnt/btrfs
> mount /dev/vdb1 /mnt/btrfs
>
> # create some random file
> dd if=/dev/urandom of=/mnt/btrfs/img bs=1M count=1024
> for i in {1..30}; do cp /mnt/btrfs/img /mnt/btrfs/img$i; done
>
> # dedup this stuff
> jdupes -S -B -r /mnt/btrfs/
> Examining 31 files, 1 dirs (in 1 specified)
> Deduplication done (30 files processed)
>
> raps-debian btrfs # btrfs fi df /mnt/btrfs/
> Data, single: total=31.50GiB, used=30.96GiB
> System, DUP: total=32.00MiB, used=16.00KiB
> Metadata, DUP: total=222.94MiB, used=33.69MiB
> GlobalReserve, single: total=33.56MiB, used=0.00B
>
> raps-debian btrfs # btrfs fi usage /mnt/btrfs/
> Overall:
> Device size: 32.00GiB
> Device allocated: 32.00GiB
> Device unallocated: 1.04MiB
> Device missing: 0.00B
> Used: 31.03GiB
> Free (estimated): 549.41MiB (min: 549.41MiB)
> Data ratio: 1.00
> Metadata ratio: 2.00
> Global reserve: 33.56MiB (used: 0.00B)
>
> Data,single: Size:31.50GiB, Used:30.96GiB
> /dev/vdb1 31.50GiB
>
> Metadata,DUP: Size:222.94MiB, Used:33.69MiB
> /dev/vdb1 445.88MiB
>
> System,DUP: Size:32.00MiB, Used:16.00KiB
> /dev/vdb1 64.00MiB
>
> Unallocated:
> /dev/vdb1 1.04MiB
> raps-debian btrfs #
>
>
>
> hm ... hasn't worked.
> wrong tool ? did i missed something
> or have i completely misunderstood deduplication?
I can not answer this, but as a long time user of btrfs who has never
tried deduplication I got curious, so I tried to recreate your result
on my debian testing workstation, and for me it worked. Here is the
log of exactly how I created the filesystem, and the output from btrfs
fi usage before and after running jdupes. As you can see from the
usage after deduplication, both the allocated and used space has
decreased to what would be expected.
Perhaps your kernel is too old? It is possible that you are being hit
by the issue described here:
https://btrfs.wiki.kernel.org/index.php/Changelog#By_feature
"The range for out-of-band deduplication implemented by the
EXTENT_SAME ioctl will split the range into 16MiB chunks. Up to now
this was the overall limit and effectively only the first 16MiB was
deduplicated."
I can't say I understand much about it, but it was fixed in kernel
4.18 which happens to be what I use.
# btrfs --version
btrfs-progs v4.17
# uname -a
Linux spacelab 4.18.0-1-amd64 #1 SMP Debian 4.18.6-1 (2018-09-06)
x86_64 GNU/Linux
# lvcreate spacelab -n dedup -L 100G
# mkfs.btrfs /dev/spacelab/dedup
# mount /dev/spacelab/dedup /mnt/
# dd if=/dev/urandom of=/mnt/img bs=1M count=1024
# for i in {1..30}; do cp /mnt/img /mnt/img$i;done
# btrfs fi usage /mnt
Overall:
Device size: 100.00GiB
Device allocated: 33.02GiB
Device unallocated: 66.98GiB
Device missing: 0.00B
Used: 31.04GiB
Free (estimated): 67.98GiB (min: 67.98GiB)
Data ratio: 1.00
Metadata ratio: 1.00
Global reserve: 33.23MiB (used: 0.00B)
Data,single: Size:32.01GiB, Used:31.01GiB
/dev/mapper/spacelab-dedup 32.01GiB
Metadata,single: Size:1.01GiB, Used:33.36MiB
/dev/mapper/spacelab-dedup 1.01GiB
System,single: Size:4.00MiB, Used:16.00KiB
/dev/mapper/spacelab-dedup 4.00MiB
Unallocated:
/dev/mapper/spacelab-dedup 66.98GiB
# jdupes -S -B -r /mnt
Scanning: 31 files, 1 items (in 1 specified)
Deduplication done (30 files processed)
# btrfs fi usage /mnt
Overall:
Device size: 100.00GiB
Device allocated: 2.02GiB
Device unallocated: 97.98GiB
Device missing: 0.00B
Used: 1.00GiB
Free (estimated): 97.99GiB (min: 97.99GiB)
Data ratio: 1.00
Metadata ratio: 1.00
Global reserve: 16.00MiB (used: 0.00B)
Data,single: Size:1.01GiB, Used:1.00GiB
/dev/mapper/spacelab-dedup 1.01GiB
Metadata,single: Size:1.01GiB, Used:1.38MiB
/dev/mapper/spacelab-dedup 1.01GiB
System,single: Size:4.00MiB, Used:16.00KiB
/dev/mapper/spacelab-dedup 4.00MiB
Unallocated:
/dev/mapper/spacelab-dedup 97.98GiB
[toc] | [prev] | [next] | [standalone]
| From | Markus Raps <m.raps@rapsplace.de> |
|---|---|
| Date | 2018-09-26 11:40 +0200 |
| Message-ID | <wCesy-5kf-9@gated-at.bofh.it> |
| In reply to | #200618 |
On 2018-09-26 00:40, Anders Andersson wrote:
> On Tue, Sep 25, 2018 at 9:56 AM, Markus Raps <m.raps@rapsplace.de>
> wrote:
>> Hi there,
>>
>> currently iam trying to get deduplication working in debian/btrfs
>>
>> #so i created a btrfs filesystem
>>
>> mkfs.btrfs /dev/vdb1
>> mkdir /mnt/btrfs
>> mount /dev/vdb1 /mnt/btrfs
>>
>> # create some random file
>> dd if=/dev/urandom of=/mnt/btrfs/img bs=1M count=1024
>> for i in {1..30}; do cp /mnt/btrfs/img /mnt/btrfs/img$i; done
>>
>> # dedup this stuff
>> jdupes -S -B -r /mnt/btrfs/
>> Examining 31 files, 1 dirs (in 1 specified)
>> Deduplication done (30 files processed)
>>
>> raps-debian btrfs # btrfs fi df /mnt/btrfs/
>> Data, single: total=31.50GiB, used=30.96GiB
>> System, DUP: total=32.00MiB, used=16.00KiB
>> Metadata, DUP: total=222.94MiB, used=33.69MiB
>> GlobalReserve, single: total=33.56MiB, used=0.00B
>>
>> raps-debian btrfs # btrfs fi usage /mnt/btrfs/
>> Overall:
>> Device size: 32.00GiB
>> Device allocated: 32.00GiB
>> Device unallocated: 1.04MiB
>> Device missing: 0.00B
>> Used: 31.03GiB
>> Free (estimated): 549.41MiB (min: 549.41MiB)
>> Data ratio: 1.00
>> Metadata ratio: 2.00
>> Global reserve: 33.56MiB (used: 0.00B)
>>
>> Data,single: Size:31.50GiB, Used:30.96GiB
>> /dev/vdb1 31.50GiB
>>
>> Metadata,DUP: Size:222.94MiB, Used:33.69MiB
>> /dev/vdb1 445.88MiB
>>
>> System,DUP: Size:32.00MiB, Used:16.00KiB
>> /dev/vdb1 64.00MiB
>>
>> Unallocated:
>> /dev/vdb1 1.04MiB
>> raps-debian btrfs #
>>
>>
>>
>> hm ... hasn't worked.
>> wrong tool ? did i missed something
>> or have i completely misunderstood deduplication?
>
>
> I can not answer this, but as a long time user of btrfs who has never
> tried deduplication I got curious, so I tried to recreate your result
> on my debian testing workstation, and for me it worked. Here is the
> log of exactly how I created the filesystem, and the output from btrfs
> fi usage before and after running jdupes. As you can see from the
> usage after deduplication, both the allocated and used space has
> decreased to what would be expected.
>
> Perhaps your kernel is too old? It is possible that you are being hit
> by the issue described here:
> https://btrfs.wiki.kernel.org/index.php/Changelog#By_feature
>
> "The range for out-of-band deduplication implemented by the
> EXTENT_SAME ioctl will split the range into 16MiB chunks. Up to now
> this was the overall limit and effectively only the first 16MiB was
> deduplicated."
>
> I can't say I understand much about it, but it was fixed in kernel
> 4.18 which happens to be what I use.
>
>
>
> # btrfs --version
> btrfs-progs v4.17
> # uname -a
> Linux spacelab 4.18.0-1-amd64 #1 SMP Debian 4.18.6-1 (2018-09-06)
> x86_64 GNU/Linux
> # lvcreate spacelab -n dedup -L 100G
> # mkfs.btrfs /dev/spacelab/dedup
> # mount /dev/spacelab/dedup /mnt/
> # dd if=/dev/urandom of=/mnt/img bs=1M count=1024
> # for i in {1..30}; do cp /mnt/img /mnt/img$i;done
> # btrfs fi usage /mnt
> Overall:
> Device size: 100.00GiB
> Device allocated: 33.02GiB
> Device unallocated: 66.98GiB
> Device missing: 0.00B
> Used: 31.04GiB
> Free (estimated): 67.98GiB (min: 67.98GiB)
> Data ratio: 1.00
> Metadata ratio: 1.00
> Global reserve: 33.23MiB (used: 0.00B)
>
> Data,single: Size:32.01GiB, Used:31.01GiB
> /dev/mapper/spacelab-dedup 32.01GiB
>
> Metadata,single: Size:1.01GiB, Used:33.36MiB
> /dev/mapper/spacelab-dedup 1.01GiB
>
> System,single: Size:4.00MiB, Used:16.00KiB
> /dev/mapper/spacelab-dedup 4.00MiB
>
> Unallocated:
> /dev/mapper/spacelab-dedup 66.98GiB
> # jdupes -S -B -r /mnt
> Scanning: 31 files, 1 items (in 1 specified)
> Deduplication done (30 files processed)
> # btrfs fi usage /mnt
> Overall:
> Device size: 100.00GiB
> Device allocated: 2.02GiB
> Device unallocated: 97.98GiB
> Device missing: 0.00B
> Used: 1.00GiB
> Free (estimated): 97.99GiB (min: 97.99GiB)
> Data ratio: 1.00
> Metadata ratio: 1.00
> Global reserve: 16.00MiB (used: 0.00B)
>
> Data,single: Size:1.01GiB, Used:1.00GiB
> /dev/mapper/spacelab-dedup 1.01GiB
>
> Metadata,single: Size:1.01GiB, Used:1.38MiB
> /dev/mapper/spacelab-dedup 1.01GiB
>
> System,single: Size:4.00MiB, Used:16.00KiB
> /dev/mapper/spacelab-dedup 4.00MiB
>
> Unallocated:
> /dev/mapper/spacelab-dedup 97.98GiB
just updated the kernel to 4.18
now it works
thank you
--
Mit freundlichen Grüßen / best regards
Markus Raps
[toc] | [prev] | [standalone]
Back to top | Article view | linux.debian.user
csiph-web