Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1281751 > unrolled thread
| Started by | Baolin Wang <baolin.wang@linaro.org> |
|---|---|
| First post | 2015-12-02 13:50 +0100 |
| Last post | 2015-12-04 06:00 +0100 |
| Articles | 10 — 4 participants |
Back to article view | Back to linux.kernel
This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by
below is the oldest one visible, not the original post.
Re: [PATCH 0/2] Introduce the request handling for dm-crypt Baolin Wang <baolin.wang@linaro.org> - 2015-12-02 13:50 +0100
Re: [dm-devel] [PATCH 0/2] Introduce the request handling for dm-crypt Alasdair G Kergon <agk@redhat.com> - 2015-12-02 21:00 +0100
Re: [dm-devel] [PATCH 0/2] Introduce the request handling for dm-crypt Baolin Wang <baolin.wang@linaro.org> - 2015-12-03 04:00 +0100
Re: [dm-devel] [PATCH 0/2] Introduce the request handling for dm-crypt Baolin Wang <baolin.wang@linaro.org> - 2015-12-03 11:40 +0100
Re: [dm-devel] [PATCH 0/2] Introduce the request handling for dm-crypt Zdenek Kabelac <zkabelac@redhat.com> - 2015-12-03 12:10 +0100
Re: [dm-devel] [PATCH 0/2] Introduce the request handling for dm-crypt Baolin Wang <baolin.wang@linaro.org> - 2015-12-03 12:30 +0100
Re: [dm-devel] [PATCH 0/2] Introduce the request handling for dm-crypt Mikulas Patocka <mpatocka@redhat.com> - 2015-12-03 16:50 +0100
Re: [dm-devel] [PATCH 0/2] Introduce the request handling for dm-crypt Baolin Wang <baolin.wang@linaro.org> - 2015-12-04 06:00 +0100
Re: [dm-devel] [PATCH 0/2] Introduce the request handling for dm-crypt Mikulas Patocka <mpatocka@redhat.com> - 2015-12-03 16:50 +0100
Re: [dm-devel] [PATCH 0/2] Introduce the request handling for dm-crypt Baolin Wang <baolin.wang@linaro.org> - 2015-12-04 06:00 +0100
| From | Baolin Wang <baolin.wang@linaro.org> |
|---|---|
| Date | 2015-12-02 13:50 +0100 |
| Subject | Re: [PATCH 0/2] Introduce the request handling for dm-crypt |
| Message-ID | <qBf17-7eb-9@gated-at.bofh.it> |
Hi All, These are the benchmarks for request based dm-crypt. Please check it. 一、Environment 1. Hardware configuration Board: Beaglebone black Processor: Am335x 1GHz ARM Cortex-A8 RAM:512M SD card:8G Kernel version:4.4-rc1 2. Encryption method (1) Use cbc(aes) cipher to encrypt the block device with dmsetup tool dmsetup create dm-0 --table “0 `blockdev --getsize /dev/mmcblk0p1` crypt aes-cbc-plain:sha256 babebabebabebabebabebabebabebabe 0 /dev/mmcblk0p1 0” (2) Enable the AES engine by config CONFIG_CRYPTO_HW=y CONFIG_CRYPTO_DEV_OMAP_AES=y (3) Limitation We want to test it on ramdisk devices rather than slow media devices (SD card) firstly, but here we can't use ramdisk to be mapped with dmsetup tool. Cause ramdisk device is non-request-stackable device, it can not be used for request-based dm. 二. Result summary 1. Results table ----------------------------------------------------------------------------------------------- | Test | size | bio | request | % change | ----------------------------------------------------------------------------------------------- | dd sequential read | 1G | 5.6Mb/s | 11.3Mb/s | +101.8% | ----------------------------------------------------------------------------------------------- | dd sequential write | 1G | 4.2Mb/s | 6.8Mb/s | +61.9% | ----------------------------------------------------------------------------------------------- | fio sequential read | 1G | 5336KB/s | 10928KB/s | +104.8% | ----------------------------------------------------------------------------------------------- | fio sequential write | 1G | 4049KB/s | 6574KB/s | +62.4% | ----------------------------------------------------------------------------------------------- 2. Summary From all the test data with the dd/fio tools, the test results basically are coincident though the different tools, so it can basically reflect the IO performance effection by request based opimization. It has a larger IO performance effection for reading speed, and it'll have a larger improvement (at least a double improvement) when enabling the request based opimization. Also it will have a big improvement for writing speed, and it is increased about 50% when enabling the request based opimization. But for random writing, it has a litle difference limited by slow hardware random accessing. 三. DD test procedure dd can be used for simplified copying of data at the low level with operating the raw devices. dd can provide good basic coverage but isn't very realistic, and only provide sequential IO accessing. But we can use dd to read/write the raw devices without the filesystem's caches effection, test result as below: 1. Sequential read: (1) Sequential read 1G with bio based: time dd if=/dev/dm-0 of=/dev/null bs=64K count=16384 iflag=direct 1073741824 bytes (1.1 GB) copied, 192.091 s, 5.6 MB/s real 3m12.112s user 0m0.070s sys 0m3.820s (2) Sequential read 1G with requset based: time dd if=/dev/dm-0 of=/dev/null bs=64K count=16384 iflag=direct 1073741824 bytes (1.1 GB) copied, 94.8922 s, 11.3 MB/s real 1m34.908s user 0m0.030s sys 0m4.000s (3) Sequential read 1G without encryption: time dd if=/dev/mmcblk0p1 of=/dev/null bs=64K count=16384 iflag=direct 1073741824 bytes (1.1 GB) copied, 58.49 s, 18.4 MB/s real 0m58.505s user 0m0.040s sys 0m3.050s 2. Sequential write: (1) Sequential write 1G with bio based: time dd if=/dev/zero of=/dev/dm-0 bs=64K count=16384 oflag=direct 1073741824 bytes (1.1 GB) copied, 253.477 s, 4.2 MB/s real 4m13.497s user 0m0.130s sys 0m3.990s (2) Sequential write 1G with requset based: time dd if=/dev/zero of=/dev/dm-0 bs=64K count=16384 oflag=direct 1073741824 bytes (1.1 GB) copied, 157.396 s, 6.8 MB/s real 2m37.414s user 0m0.130s sys 0m4.190s (3) Sequential write 1G without encryption: time dd if=/dev/zero of=/dev/mmcblk0p1 bs=64K count=16384 oflag=direct 1073741824 bytes (1.1 GB) copied, 120.452 s, 8.9 MB/s real 2m0.471s user 0m0.050s sys 0m3.820s 3. Summary: we can see the sequential read/write speed with bio based is: 5.6MB/s and 4.2MB/s, but when encrypting the block device with request based things, the sequential read/write speed can be increased to: 11.3MB/s and 6.8MBs. So sequential reading and writing speed have a big different with request based, speed are increased by 101.8% and 61.9%. Meanwhile we also can see the difference in 'sys' time with request based optimizations. 三、Fio test procedure We specify the block size is 64K, and command like: fio --filename=/dev/dm-0 --direct=1 --iodepth=1 --rw=read --bs=64K --size=1G --group_reporting --numjobs=1 --name=test_read 1. Sequential read 1G with bio based: READ: io=1024.0MB, aggrb=5336KB/s, minb=5336KB/s, maxb=5336KB/s, mint=196494msec, maxt=196494msec 2. Sequential write 1G with bio based: WRITE: io=1024.0MB, aggrb=4049KB/s, minb=4049KB/s, maxb=4049KB/s, mint=258954msec, maxt=258954msec 3. Sequential read 1G with request based: READ: io=1024.0MB, aggrb=10928KB/s, minb=10928KB/s, maxb=10928KB/s, mint=95947msec, maxt=95947msec 4. Sequential write 1G with request based: WRITE: io=1024.0MB, aggrb=6574KB/s, minb=6574KB/s, maxb=6574KB/s, mint=159493msec, maxt=159493msec 5. Summary: (1) read: The sequential read speed has a big improvment with reuest based things, which is increased by 104.8% when the reuest based things are enabled for dm-crypt. It can not be a really random read if we specify the block size, so the data doesn't list the random read speed improvements, though it show big improvements. (2) write: The sequential write speed has some improvements with request based, which is increased about by 62.4%. But for random write, this part is very hard to measure on an SD card though, because any random write smaller than the underlying block size will cause long I/O latencies at some point, which is can not show the improvements. 四、IO block size test We also change the block size from 4K to 1M (most IO block size in practice are much smaller than 1M) to see the block size influences with reuest based for dm-crypt. 1. Sequential read 1G (1) block size = 4k time dd if=/dev/dm-0 of=/dev/null bs=4k count=262144 iflag=direct 1073741824 bytes (1.1 GB) copied, 310.598 s, 3.5 MB/s real 5m10.614s user 0m0.610s sys 0m36.040s (2) block size = 64k 1073741824 bytes (1.1 GB) copied, 95.0489 s, 11.3 MB/s real 1m35.071s user 0m0.040s sys 0m4.030s (3) block size = 256k 1073741824 bytes (1.1 GB) copied, 84.3311 s, 12.7 MB/s real 1m24.347s user 0m0.050s sys 0m1.950s (4) block size = 1M 1073741824 bytes (1.1 GB) copied, 80.8778 s, 13.3 MB/s real 1m20.893s user 0m0.010s sys 0m1.390s 2. Sequential write 1G (1) block size = 4k time dd if=/dev/zero of=/dev/dm-0 bs=4K count=262144 oflag=direct 1073741824 bytes (1.1 GB) copied, 629.656 s, 1.7 MB/s real 10m29.671s user 0m0.790s sys 0m33.550s (2) block size = 64k 1073741824 bytes (1.1 GB) copied, 155.697 s, 6.9 MB/s real 2m35.713s user 0m0.040s sys 0m4.110s (3) block size = 256k 1073741824 bytes (1.1 GB) copied, 143.682 s, 7.5 MB/s real 2m23.698s user 0m0.040s sys 0m2.500s (4) block size = 1M 1073741824 bytes (1.1 GB) copied, 140.654 s, 7.6 MB/s real 2m20.670s user 0m0.040s sys 0m2.090s 3. Summary For request based things, some sequential bios/requests can merged into one request to expand the IO size to be a big block handled by hardware engine at one time. With the hardware acceleration, it can improve the encryption/decryption speed, so the hardware engine can play the best performance with big block size. From the data, we also can see the reading/writing speed can be increased by expanding the block size, which means that it doesn't help much for small sizes in. But when the block size is above 64K, the speed dose not get the corresponding performance benefits, I think the speed limitation is also in cryto that lets the bigger bios can't get the similar performance benefits, which is need more investigations. On 13 November 2015 at 19:51, Mark Brown <broonie@kernel.org> wrote: > On Thu, Nov 12, 2015 at 08:26:26AM -0700, Jens Axboe wrote: >> On 11/12/2015 03:04 AM, Mark Brown wrote: > >> >Android now wants to encrypt phones and tablets by default and have been >> >seeing substantial performance hits as a result, we can try to get >> >people to share performance data from productionish systems but it might >> >be difficult. > >> Well, shame on them for developing out-of-tree, looks like they are reaping >> all the benefits of that. > >> Guys, we need some numbers, enough with the hand waving. There's no point >> discussing this further until we know how much of a difference it makes to >> handle X MB chunks instead of Y MB chunks. As was previously stated, unless >> there's a _substantial_ performance benefit, this patchset isn't going >> anywhere. > > Yeah, what I'm saying here is that there will issues getting the numbers > from relevant production systems - we are most likely to be looking at > proxies which are hopefully reasonably representative but there's likely > to be more divergence than you'd see just running benchmark workloads on > similar systems to those used in production. -- Baolin.wang Best Regards -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [next] | [standalone]
| From | Alasdair G Kergon <agk@redhat.com> |
|---|---|
| Date | 2015-12-02 21:00 +0100 |
| Subject | Re: [dm-devel] [PATCH 0/2] Introduce the request handling for dm-crypt |
| Message-ID | <qBlJh-3cy-15@gated-at.bofh.it> |
| In reply to | #1281751 |
On Wed, Dec 02, 2015 at 08:46:54PM +0800, Baolin Wang wrote: > These are the benchmarks for request based dm-crypt. Please check it. Now please put request-based dm-crypt completely to one side and focus just on the existing bio-based code. Why is it slower and what can be adjusted to improve this? People aren't going to take a request-based solution seriously until you can explain in full detail *why* bio-based is slower AND why it's impossible to improve its performance. > For request based things, some sequential bios/requests can merged > into one request to expand the IO size to be a big block handled by > hardware engine at one time. Bio-based also merges I/O so that does not provide justification. Investigate in much more detail the actual merging and scheduling involved in the cases you need to optimise. See if blktrace gives you any clues, or add your own instrumentation. You could even look at some of the patches we've had in the list archives for optimising bio-based crypt in different situations. Alasdair -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Baolin Wang <baolin.wang@linaro.org> |
|---|---|
| Date | 2015-12-03 04:00 +0100 |
| Subject | Re: [dm-devel] [PATCH 0/2] Introduce the request handling for dm-crypt |
| Message-ID | <qBshI-7mN-5@gated-at.bofh.it> |
| In reply to | #1282318 |
On 3 December 2015 at 03:56, Alasdair G Kergon <agk@redhat.com> wrote: > On Wed, Dec 02, 2015 at 08:46:54PM +0800, Baolin Wang wrote: >> These are the benchmarks for request based dm-crypt. Please check it. > > Now please put request-based dm-crypt completely to one side and focus > just on the existing bio-based code. Why is it slower and what can be > adjusted to improve this? > OK. I think I find something need to be point out. 1. From the IO block size test in the performance report, for the request based, we can find it can not get the corresponding performance if we just expand the IO size. Because In dm crypt, it will map the data buffer of one request with scatterlists, and send all scatterlists of one request to the encryption engine to encrypt or decrypt. I found if the scatterlist list number is small and each scatterlist length is bigger, it will improve the encryption speed, that helps the engine palys best performance. But a big IO size does not mean bigger scatterlists (maybe many scatterlists with small length), that's why we can not get the corresponding performance if we just expand the IO size I think. 2. Why bio based is slower? If you understand 1, you can obviously understand the crypto engine likes bigger scatterlists to improve the performance. But for bio based, it only send one scatterlist (the scatterlist's length is always '1 << SECTOR_SHIFT' = 512) to the crypto engine at one time. It means if the bio size is 1M, the bio based will send 2048 times (evey time the only one scatterlist length is 512 bytes) to crypto engine to handle, which is more time-consuming and ineffective for the crypto engine. But for request based, it can map the whole request with many scatterlists (not just one scatterlist), and send all the scatterlists to the crypto engine which can improve the performance, is it right? Another optimization solution I think is we can expand the scatterlist entry number for bio based. > People aren't going to take a request-based solution seriously until > you can explain in full detail *why* bio-based is slower AND why it's > impossible to improve its performance. > >> For request based things, some sequential bios/requests can merged >> into one request to expand the IO size to be a big block handled by >> hardware engine at one time. > > Bio-based also merges I/O so that does not provide justification. > Investigate in much more detail the actual merging and scheduling > involved in the cases you need to optimise. See if blktrace gives you > any clues, or add your own instrumentation. You could even look at some > of the patches we've had in the list archives for optimising bio-based > crypt in different situations. > > Alasdair > -- Baolin.wang Best Regards -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Baolin Wang <baolin.wang@linaro.org> |
|---|---|
| Date | 2015-12-03 11:40 +0100 |
| Subject | Re: [dm-devel] [PATCH 0/2] Introduce the request handling for dm-crypt |
| Message-ID | <qBzsT-3Jm-43@gated-at.bofh.it> |
| In reply to | #1282665 |
On 3 December 2015 at 10:56, Baolin Wang <baolin.wang@linaro.org> wrote: > On 3 December 2015 at 03:56, Alasdair G Kergon <agk@redhat.com> wrote: >> On Wed, Dec 02, 2015 at 08:46:54PM +0800, Baolin Wang wrote: >>> These are the benchmarks for request based dm-crypt. Please check it. >> >> Now please put request-based dm-crypt completely to one side and focus >> just on the existing bio-based code. Why is it slower and what can be >> adjusted to improve this? >> > > OK. I think I find something need to be point out. > 1. From the IO block size test in the performance report, for the > request based, we can find it can not get the corresponding > performance if we just expand the IO size. Because In dm crypt, it > will map the data buffer of one request with scatterlists, and send > all scatterlists of one request to the encryption engine to encrypt or > decrypt. I found if the scatterlist list number is small and each > scatterlist length is bigger, it will improve the encryption speed, > that helps the engine palys best performance. But a big IO size does > not mean bigger scatterlists (maybe many scatterlists with small > length), that's why we can not get the corresponding performance if we > just expand the IO size I think. > > 2. Why bio based is slower? > If you understand 1, you can obviously understand the crypto engine > likes bigger scatterlists to improve the performance. But for bio > based, it only send one scatterlist (the scatterlist's length is > always '1 << SECTOR_SHIFT' = 512) to the crypto engine at one time. It > means if the bio size is 1M, the bio based will send 2048 times (evey > time the only one scatterlist length is 512 bytes) to crypto engine to > handle, which is more time-consuming and ineffective for the crypto > engine. But for request based, it can map the whole request with many > scatterlists (not just one scatterlist), and send all the scatterlists > to the crypto engine which can improve the performance, is it right? > > Another optimization solution I think is we can expand the scatterlist > entry number for bio based. > I did some testing about my assumption of expanding the scatterlist entry number for bio based. I did some modification for the bio based to support multiple scatterlists, then it will get the same performance as the request based things. 1. bio based with expanding the scatterlist entry time dd if=/dev/dm-0 of=/dev/null bs=64K count=16384 iflag=direct 1073741824 bytes (1.1 GB) copied, 94.5458 s, 11.4 MB/s real 1m34.562s user 0m0.030s sys 0m3.850s 2. Sequential read 1G with requset based: time dd if=/dev/dm-0 of=/dev/null bs=64K count=16384 iflag=direct 1073741824 bytes (1.1 GB) copied, 94.8922 s, 11.3 MB/s real 1m34.908s user 0m0.030s sys 0m4.000s From the data, we can find the bio based also can get the same performance as the request based. So if someone still don't like the request based things, I think we can optimize the bio based by expanding the scatterlists number. Thanks. >> >> Alasdair >> > > > > -- > Baolin.wang > Best Regards -- Baolin.wang Best Regards -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Zdenek Kabelac <zkabelac@redhat.com> |
|---|---|
| Date | 2015-12-03 12:10 +0100 |
| Subject | Re: [dm-devel] [PATCH 0/2] Introduce the request handling for dm-crypt |
| Message-ID | <qBzVT-4ak-11@gated-at.bofh.it> |
| In reply to | #1282923 |
Dne 3.12.2015 v 11:36 Baolin Wang napsal(a): > On 3 December 2015 at 10:56, Baolin Wang <baolin.wang@linaro.org> wrote: >> On 3 December 2015 at 03:56, Alasdair G Kergon <agk@redhat.com> wrote: >>> On Wed, Dec 02, 2015 at 08:46:54PM +0800, Baolin Wang wrote: >>>> These are the benchmarks for request based dm-crypt. Please check it. >>> >>> Now please put request-based dm-crypt completely to one side and focus >>> just on the existing bio-based code. Why is it slower and what can be >>> adjusted to improve this? >>> >> >> OK. I think I find something need to be point out. >> 1. From the IO block size test in the performance report, for the >> request based, we can find it can not get the corresponding >> performance if we just expand the IO size. Because In dm crypt, it >> will map the data buffer of one request with scatterlists, and send >> all scatterlists of one request to the encryption engine to encrypt or >> decrypt. I found if the scatterlist list number is small and each >> scatterlist length is bigger, it will improve the encryption speed, >> that helps the engine palys best performance. But a big IO size does >> not mean bigger scatterlists (maybe many scatterlists with small >> length), that's why we can not get the corresponding performance if we >> just expand the IO size I think. >> >> 2. Why bio based is slower? >> If you understand 1, you can obviously understand the crypto engine >> likes bigger scatterlists to improve the performance. But for bio >> based, it only send one scatterlist (the scatterlist's length is >> always '1 << SECTOR_SHIFT' = 512) to the crypto engine at one time. It >> means if the bio size is 1M, the bio based will send 2048 times (evey >> time the only one scatterlist length is 512 bytes) to crypto engine to >> handle, which is more time-consuming and ineffective for the crypto >> engine. But for request based, it can map the whole request with many >> scatterlists (not just one scatterlist), and send all the scatterlists >> to the crypto engine which can improve the performance, is it right? >> >> Another optimization solution I think is we can expand the scatterlist >> entry number for bio based. >> > > I did some testing about my assumption of expanding the scatterlist > entry number for bio based. I did some modification for the bio based > to support multiple scatterlists, then it will get the same > performance as the request based things. > > 1. bio based with expanding the scatterlist entry > time dd if=/dev/dm-0 of=/dev/null bs=64K count=16384 iflag=direct > 1073741824 bytes (1.1 GB) copied, 94.5458 s, 11.4 MB/s > real 1m34.562s > user 0m0.030s > sys 0m3.850s > > 2. Sequential read 1G with requset based: > time dd if=/dev/dm-0 of=/dev/null bs=64K count=16384 iflag=direct > 1073741824 bytes (1.1 GB) copied, 94.8922 s, 11.3 MB/s > real 1m34.908s > user 0m0.030s > sys 0m4.000s > > From the data, we can find the bio based also can get the same > performance as the request based. So if someone still don't like the > request based things, I think we can optimize the bio based by > expanding the scatterlists number. Thanks. > Hi Do you see any performance impact if you use with cryptsetup options: --perf-same_cpu_crypt --perf-submit_from_crypt_cpus with your regular unpatched kernel. Zdenek -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Baolin Wang <baolin.wang@linaro.org> |
|---|---|
| Date | 2015-12-03 12:30 +0100 |
| Subject | Re: [dm-devel] [PATCH 0/2] Introduce the request handling for dm-crypt |
| Message-ID | <qBAfg-4hh-9@gated-at.bofh.it> |
| In reply to | #1282943 |
On 3 December 2015 at 19:07, Zdenek Kabelac <zkabelac@redhat.com> wrote: > Dne 3.12.2015 v 11:36 Baolin Wang napsal(a): > >> On 3 December 2015 at 10:56, Baolin Wang <baolin.wang@linaro.org> wrote: >>> >>> On 3 December 2015 at 03:56, Alasdair G Kergon <agk@redhat.com> wrote: >>>> >>>> On Wed, Dec 02, 2015 at 08:46:54PM +0800, Baolin Wang wrote: >>>>> >>>>> These are the benchmarks for request based dm-crypt. Please check it. >>>> >>>> >>>> Now please put request-based dm-crypt completely to one side and focus >>>> just on the existing bio-based code. Why is it slower and what can be >>>> adjusted to improve this? >>>> >>> >>> OK. I think I find something need to be point out. >>> 1. From the IO block size test in the performance report, for the >>> request based, we can find it can not get the corresponding >>> performance if we just expand the IO size. Because In dm crypt, it >>> will map the data buffer of one request with scatterlists, and send >>> all scatterlists of one request to the encryption engine to encrypt or >>> decrypt. I found if the scatterlist list number is small and each >>> scatterlist length is bigger, it will improve the encryption speed, >>> that helps the engine palys best performance. But a big IO size does >>> not mean bigger scatterlists (maybe many scatterlists with small >>> length), that's why we can not get the corresponding performance if we >>> just expand the IO size I think. >>> >>> 2. Why bio based is slower? >>> If you understand 1, you can obviously understand the crypto engine >>> likes bigger scatterlists to improve the performance. But for bio >>> based, it only send one scatterlist (the scatterlist's length is >>> always '1 << SECTOR_SHIFT' = 512) to the crypto engine at one time. It >>> means if the bio size is 1M, the bio based will send 2048 times (evey >>> time the only one scatterlist length is 512 bytes) to crypto engine to >>> handle, which is more time-consuming and ineffective for the crypto >>> engine. But for request based, it can map the whole request with many >>> scatterlists (not just one scatterlist), and send all the scatterlists >>> to the crypto engine which can improve the performance, is it right? >>> >>> Another optimization solution I think is we can expand the scatterlist >>> entry number for bio based. >>> >> >> I did some testing about my assumption of expanding the scatterlist >> entry number for bio based. I did some modification for the bio based >> to support multiple scatterlists, then it will get the same >> performance as the request based things. >> >> 1. bio based with expanding the scatterlist entry >> time dd if=/dev/dm-0 of=/dev/null bs=64K count=16384 iflag=direct >> 1073741824 bytes (1.1 GB) copied, 94.5458 s, 11.4 MB/s >> real 1m34.562s >> user 0m0.030s >> sys 0m3.850s >> >> 2. Sequential read 1G with requset based: >> time dd if=/dev/dm-0 of=/dev/null bs=64K count=16384 iflag=direct >> 1073741824 bytes (1.1 GB) copied, 94.8922 s, 11.3 MB/s >> real 1m34.908s >> user 0m0.030s >> sys 0m4.000s >> >> From the data, we can find the bio based also can get the same >> performance as the request based. So if someone still don't like the >> request based things, I think we can optimize the bio based by >> expanding the scatterlists number. Thanks. >> > > > Hi > > Do you see any performance impact if you use with cryptsetup options: > > --perf-same_cpu_crypt > --perf-submit_from_crypt_cpus > > with your regular unpatched kernel. I did not see the performance impact with these options you said, cause my board only has one cpu. > > Zdenek > -- Baolin.wang Best Regards -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Mikulas Patocka <mpatocka@redhat.com> |
|---|---|
| Date | 2015-12-03 16:50 +0100 |
| Subject | Re: [dm-devel] [PATCH 0/2] Introduce the request handling for dm-crypt |
| Message-ID | <qBEiS-71h-3@gated-at.bofh.it> |
| In reply to | #1282923 |
On Thu, 3 Dec 2015, Baolin Wang wrote: > On 3 December 2015 at 10:56, Baolin Wang <baolin.wang@linaro.org> wrote: > > On 3 December 2015 at 03:56, Alasdair G Kergon <agk@redhat.com> wrote: > >> On Wed, Dec 02, 2015 at 08:46:54PM +0800, Baolin Wang wrote: > >>> These are the benchmarks for request based dm-crypt. Please check it. > >> > >> Now please put request-based dm-crypt completely to one side and focus > >> just on the existing bio-based code. Why is it slower and what can be > >> adjusted to improve this? > >> > > > > OK. I think I find something need to be point out. > > 1. From the IO block size test in the performance report, for the > > request based, we can find it can not get the corresponding > > performance if we just expand the IO size. Because In dm crypt, it > > will map the data buffer of one request with scatterlists, and send > > all scatterlists of one request to the encryption engine to encrypt or > > decrypt. I found if the scatterlist list number is small and each > > scatterlist length is bigger, it will improve the encryption speed, > > that helps the engine palys best performance. But a big IO size does > > not mean bigger scatterlists (maybe many scatterlists with small > > length), that's why we can not get the corresponding performance if we > > just expand the IO size I think. > > > > 2. Why bio based is slower? > > If you understand 1, you can obviously understand the crypto engine > > likes bigger scatterlists to improve the performance. But for bio > > based, it only send one scatterlist (the scatterlist's length is > > always '1 << SECTOR_SHIFT' = 512) to the crypto engine at one time. It > > means if the bio size is 1M, the bio based will send 2048 times (evey > > time the only one scatterlist length is 512 bytes) to crypto engine to > > handle, which is more time-consuming and ineffective for the crypto > > engine. But for request based, it can map the whole request with many > > scatterlists (not just one scatterlist), and send all the scatterlists > > to the crypto engine which can improve the performance, is it right? > > > > Another optimization solution I think is we can expand the scatterlist > > entry number for bio based. > > > > I did some testing about my assumption of expanding the scatterlist > entry number for bio based. I did some modification for the bio based > to support multiple scatterlists, then it will get the same > performance as the request based things. > > 1. bio based with expanding the scatterlist entry > time dd if=/dev/dm-0 of=/dev/null bs=64K count=16384 iflag=direct > 1073741824 bytes (1.1 GB) copied, 94.5458 s, 11.4 MB/s > real 1m34.562s > user 0m0.030s > sys 0m3.850s > > 2. Sequential read 1G with requset based: > time dd if=/dev/dm-0 of=/dev/null bs=64K count=16384 iflag=direct > 1073741824 bytes (1.1 GB) copied, 94.8922 s, 11.3 MB/s > real 1m34.908s > user 0m0.030s > sys 0m4.000s Measuring the system time this way is completely wrong because it doesn't account for the time spent in kernel threads. Mikulas -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Baolin Wang <baolin.wang@linaro.org> |
|---|---|
| Date | 2015-12-04 06:00 +0100 |
| Subject | Re: [dm-devel] [PATCH 0/2] Introduce the request handling for dm-crypt |
| Message-ID | <qBQDn-6ue-1@gated-at.bofh.it> |
| In reply to | #1283140 |
On 3 December 2015 at 23:49, Mikulas Patocka <mpatocka@redhat.com> wrote: > > > On Thu, 3 Dec 2015, Baolin Wang wrote: > >> On 3 December 2015 at 10:56, Baolin Wang <baolin.wang@linaro.org> wrote: >> > On 3 December 2015 at 03:56, Alasdair G Kergon <agk@redhat.com> wrote: >> >> On Wed, Dec 02, 2015 at 08:46:54PM +0800, Baolin Wang wrote: >> >>> These are the benchmarks for request based dm-crypt. Please check it. >> >> >> >> Now please put request-based dm-crypt completely to one side and focus >> >> just on the existing bio-based code. Why is it slower and what can be >> >> adjusted to improve this? >> >> >> > >> > OK. I think I find something need to be point out. >> > 1. From the IO block size test in the performance report, for the >> > request based, we can find it can not get the corresponding >> > performance if we just expand the IO size. Because In dm crypt, it >> > will map the data buffer of one request with scatterlists, and send >> > all scatterlists of one request to the encryption engine to encrypt or >> > decrypt. I found if the scatterlist list number is small and each >> > scatterlist length is bigger, it will improve the encryption speed, >> > that helps the engine palys best performance. But a big IO size does >> > not mean bigger scatterlists (maybe many scatterlists with small >> > length), that's why we can not get the corresponding performance if we >> > just expand the IO size I think. >> > >> > 2. Why bio based is slower? >> > If you understand 1, you can obviously understand the crypto engine >> > likes bigger scatterlists to improve the performance. But for bio >> > based, it only send one scatterlist (the scatterlist's length is >> > always '1 << SECTOR_SHIFT' = 512) to the crypto engine at one time. It >> > means if the bio size is 1M, the bio based will send 2048 times (evey >> > time the only one scatterlist length is 512 bytes) to crypto engine to >> > handle, which is more time-consuming and ineffective for the crypto >> > engine. But for request based, it can map the whole request with many >> > scatterlists (not just one scatterlist), and send all the scatterlists >> > to the crypto engine which can improve the performance, is it right? >> > >> > Another optimization solution I think is we can expand the scatterlist >> > entry number for bio based. >> > >> >> I did some testing about my assumption of expanding the scatterlist >> entry number for bio based. I did some modification for the bio based >> to support multiple scatterlists, then it will get the same >> performance as the request based things. >> >> 1. bio based with expanding the scatterlist entry >> time dd if=/dev/dm-0 of=/dev/null bs=64K count=16384 iflag=direct >> 1073741824 bytes (1.1 GB) copied, 94.5458 s, 11.4 MB/s >> real 1m34.562s >> user 0m0.030s >> sys 0m3.850s >> >> 2. Sequential read 1G with requset based: >> time dd if=/dev/dm-0 of=/dev/null bs=64K count=16384 iflag=direct >> 1073741824 bytes (1.1 GB) copied, 94.8922 s, 11.3 MB/s >> real 1m34.908s >> user 0m0.030s >> sys 0m4.000s > > Measuring the system time this way is completely wrong because it doesn't > account for the time spent in kernel threads. > OK. Thanks for your suggestions. > Mikulas -- Baolin.wang Best Regards -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Mikulas Patocka <mpatocka@redhat.com> |
|---|---|
| Date | 2015-12-03 16:50 +0100 |
| Subject | Re: [dm-devel] [PATCH 0/2] Introduce the request handling for dm-crypt |
| Message-ID | <qBEiR-71h-1@gated-at.bofh.it> |
| In reply to | #1282665 |
On Thu, 3 Dec 2015, Baolin Wang wrote: > On 3 December 2015 at 03:56, Alasdair G Kergon <agk@redhat.com> wrote: > > On Wed, Dec 02, 2015 at 08:46:54PM +0800, Baolin Wang wrote: > >> These are the benchmarks for request based dm-crypt. Please check it. > > > > Now please put request-based dm-crypt completely to one side and focus > > just on the existing bio-based code. Why is it slower and what can be > > adjusted to improve this? > > > > OK. I think I find something need to be point out. > 1. From the IO block size test in the performance report, for the > request based, we can find it can not get the corresponding > performance if we just expand the IO size. Because In dm crypt, it > will map the data buffer of one request with scatterlists, and send > all scatterlists of one request to the encryption engine to encrypt or > decrypt. I found if the scatterlist list number is small and each > scatterlist length is bigger, it will improve the encryption speed, This optimization is only applicable to XTS mode. XTS has its weaknesses and it is not recommended for encryption of more than 1TB of data ( http://grouper.ieee.org/groups/1619/email/msg02357.html ) You can optimize bio-based dm-crypt as well (use larger encryption chunk than 512 bytes when the mode is XTS). The most commonly used mode aes-cbc-essiv:sha256 can't be optimized that way. You have to do encryption and decryption sector by sector because every sector has different IV. Mikulas > that helps the engine palys best performance. But a big IO size does > not mean bigger scatterlists (maybe many scatterlists with small > length), that's why we can not get the corresponding performance if we > just expand the IO size I think. > > 2. Why bio based is slower? > If you understand 1, you can obviously understand the crypto engine > likes bigger scatterlists to improve the performance. But for bio > based, it only send one scatterlist (the scatterlist's length is > always '1 << SECTOR_SHIFT' = 512) to the crypto engine at one time. It > means if the bio size is 1M, the bio based will send 2048 times (evey > time the only one scatterlist length is 512 bytes) to crypto engine to > handle, which is more time-consuming and ineffective for the crypto > engine. But for request based, it can map the whole request with many > scatterlists (not just one scatterlist), and send all the scatterlists > to the crypto engine which can improve the performance, is it right? > > Another optimization solution I think is we can expand the scatterlist > entry number for bio based. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Baolin Wang <baolin.wang@linaro.org> |
|---|---|
| Date | 2015-12-04 06:00 +0100 |
| Subject | Re: [dm-devel] [PATCH 0/2] Introduce the request handling for dm-crypt |
| Message-ID | <qBQDn-6ue-3@gated-at.bofh.it> |
| In reply to | #1283139 |
On 3 December 2015 at 23:47, Mikulas Patocka <mpatocka@redhat.com> wrote: > > > On Thu, 3 Dec 2015, Baolin Wang wrote: > >> On 3 December 2015 at 03:56, Alasdair G Kergon <agk@redhat.com> wrote: >> > On Wed, Dec 02, 2015 at 08:46:54PM +0800, Baolin Wang wrote: >> >> These are the benchmarks for request based dm-crypt. Please check it. >> > >> > Now please put request-based dm-crypt completely to one side and focus >> > just on the existing bio-based code. Why is it slower and what can be >> > adjusted to improve this? >> > >> >> OK. I think I find something need to be point out. >> 1. From the IO block size test in the performance report, for the >> request based, we can find it can not get the corresponding >> performance if we just expand the IO size. Because In dm crypt, it >> will map the data buffer of one request with scatterlists, and send >> all scatterlists of one request to the encryption engine to encrypt or >> decrypt. I found if the scatterlist list number is small and each >> scatterlist length is bigger, it will improve the encryption speed, > > This optimization is only applicable to XTS mode. XTS has its weaknesses > and it is not recommended for encryption of more than 1TB of data > ( http://grouper.ieee.org/groups/1619/email/msg02357.html ) > > You can optimize bio-based dm-crypt as well (use larger encryption chunk > than 512 bytes when the mode is XTS). > > The most commonly used mode aes-cbc-essiv:sha256 can't be optimized that > way. You have to do encryption and decryption sector by sector because > every sector has different IV. Make sense. We'll optimize bio-based dm-crypt for XTS mode, and do some investigations for none XTS mode. > > Mikulas > -- Baolin.wang Best Regards -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web