Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1267058 > unrolled thread

[PATCH 0/2] Introduce the request handling for dm-crypt

Back to article view | Back to linux.kernel

Contents

  [PATCH 0/2] Introduce the request handling for dm-crypt Baolin Wang <baolin.wang@linaro.org> - 2015-11-11 10:40 +0100
    [PATCH 1/2] block: Introduce BIO_ENDIO_FREE for bio flags Baolin Wang <baolin.wang@linaro.org> - 2015-11-11 10:40 +0100
      Re: [PATCH 1/2] block: Introduce BIO_ENDIO_FREE for bio flags Mike Snitzer <snitzer@redhat.com> - 2015-11-11 19:00 +0100
        Re: [PATCH 1/2] block: Introduce BIO_ENDIO_FREE for bio flags Baolin Wang <baolin.wang@linaro.org> - 2015-11-12 05:10 +0100
    Re: [PATCH 0/2] Introduce the request handling for dm-crypt Christoph Hellwig <hch@infradead.org> - 2015-11-11 10:50 +0100
      Re: [PATCH 0/2] Introduce the request handling for dm-crypt Baolin Wang <baolin.wang@linaro.org> - 2015-11-12 03:20 +0100
        Re: [PATCH 0/2] Introduce the request handling for dm-crypt Jan Kara <jack@suse.cz> - 2015-11-12 10:20 +0100
          Re: [PATCH 0/2] Introduce the request handling for dm-crypt Baolin Wang <baolin.wang@linaro.org> - 2015-11-12 10:50 +0100
            Re: [PATCH 0/2] Introduce the request handling for dm-crypt Jan Kara <jack@suse.cz> - 2015-11-12 12:10 +0100
              Re: [PATCH 0/2] Introduce the request handling for dm-crypt Baolin Wang <baolin.wang@linaro.org> - 2015-11-12 12:50 +0100
                Re: [PATCH 0/2] Introduce the request handling for dm-crypt Jan Kara <jack@suse.cz> - 2015-11-12 13:30 +0100
                  Re: [PATCH 0/2] Introduce the request handling for dm-crypt Arnd Bergmann <arnd@arndb.de> - 2015-11-12 14:00 +0100
                    Re: [PATCH 0/2] Introduce the request handling for dm-crypt Mark Brown <broonie@kernel.org> - 2015-11-12 16:10 +0100
                      Re: [PATCH 0/2] Introduce the request handling for dm-crypt Baolin Wang <baolin.wang@linaro.org> - 2015-11-13 04:30 +0100
                    Re: [PATCH 0/2] Introduce the request handling for dm-crypt Baolin Wang <baolin.wang@linaro.org> - 2015-11-13 04:30 +0100
                  Re: [PATCH 0/2] Introduce the request handling for dm-crypt Baolin Wang <baolin.wang@linaro.org> - 2015-11-12 14:00 +0100
                    Re: [PATCH 0/2] Introduce the request handling for dm-crypt Jan Kara <jack@suse.cz> - 2015-11-12 14:00 +0100
                      Re: [PATCH 0/2] Introduce the request handling for dm-crypt Baolin Wang <baolin.wang@linaro.org> - 2015-11-13 03:10 +0100
                        Re: [PATCH 0/2] Introduce the request handling for dm-crypt Arnd Bergmann <arnd@arndb.de> - 2015-11-13 10:10 +0100
                          Re: [PATCH 0/2] Introduce the request handling for dm-crypt Baolin Wang <baolin.wang@linaro.org> - 2015-11-13 12:40 +0100
    Re: [PATCH 0/2] Introduce the request handling for dm-crypt Mike Snitzer <snitzer@redhat.com> - 2015-11-11 19:20 +0100
      Re: [PATCH 0/2] Introduce the request handling for dm-crypt Baolin Wang <baolin.wang@linaro.org> - 2015-11-12 03:40 +0100
        Re: [dm-devel] [PATCH 0/2] Introduce the request handling for  dm-crypt Christoph Hellwig <hch@infradead.org> - 2015-11-12 10:10 +0100
      Re: [PATCH 0/2] Introduce the request handling for dm-crypt Baolin Wang <baolin.wang@linaro.org> - 2015-11-12 09:30 +0100
        Re: [PATCH 0/2] Introduce the request handling for dm-crypt Mark Brown <broonie@kernel.org> - 2015-11-12 11:10 +0100
          Re: [PATCH 0/2] Introduce the request handling for dm-crypt Jens Axboe <axboe@kernel.dk> - 2015-11-12 16:30 +0100
            Re: [PATCH 0/2] Introduce the request handling for dm-crypt Baolin Wang <baolin.wang@linaro.org> - 2015-11-13 03:10 +0100
            Re: [PATCH 0/2] Introduce the request handling for dm-crypt Mark Brown <broonie@kernel.org> - 2015-11-13 13:00 +0100
      Re: [dm-devel] [PATCH 0/2] Introduce the request handling for  dm-crypt Christoph Hellwig <hch@infradead.org> - 2015-11-12 10:10 +0100

Page 1 of 2  [1] 2  Next page →

#1267058 — [PATCH 0/2] Introduce the request handling for dm-crypt

Now the dm-crypt code only implemented the 'based-bio' method to encrypt/
decrypt block data, which can only hanle one bio at one time. As we know,
one bio must use the sequential physical address and it also has a limitation
of length. Thus it may limit the big block encyrtion/decryption when some
hardware support the big block data encryption.

This patch series introduc the 'based-request' method to handle the data
encryption/decryption. One request can contain multiple bios, so it can
handle big block data to improve the efficiency.

Also this patch refers to the Dinesh K Garg <dineshg@codeaurora.org> and
franciscofranco <franciscofranco.1990@gmail.com> submission of 'Request
based dm-crypt', and the original link is:
https://github.com/major91/Zeta-Chromium-N5/commit/621d4821da04cfde383329ebdf4ce8711ec3329c

Now it can encrypt/decrypt successfully with read/write testing, and it shows
some efficiency improvements with hardware acceleration. But I really hope to
need more comments about the structure and conception. Thanks for your review.

Baolin Wang (2):
  block: Introduce BIO_ENDIO_FREE for bio flags
  md: dm-crypt: Introduce the request handling for dm-crypt

 block/blk-core.c              |    6 +-
 drivers/md/Kconfig            |    6 +
 drivers/md/dm-crypt.c         |  831 ++++++++++++++++++++++++++++++++++++++++-
 drivers/md/dm.c               |   13 +-
 include/linux/blk_types.h     |    6 +
 include/linux/device-mapper.h |    5 +
 6 files changed, 861 insertions(+), 6 deletions(-)

-- 
1.7.9.5

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1267059 — [PATCH 1/2] block: Introduce BIO_ENDIO_FREE for bio flags

When we use dm-crypt to decrypt block data, it will decrypt the block data
in endio() when one IO is completed. In this situation we don't want the
cloned bios is freed before calling the endio().

Thus introduce 'BIO_ENDIO_FREE' flag to support the request handling for dm-crypt,
this flag will ensure that blk layer does not complete the cloned bios before
completing the request. When the crypt endio is called, post-processsing is
done and then the dm layer will complete the bios (clones) and free them.

Signed-off-by: Baolin Wang <baolin.wang@linaro.org>
---
 block/blk-core.c              |    6 +++++-
 drivers/md/dm.c               |   13 ++++++++++---
 include/linux/blk_types.h     |    6 ++++++
 include/linux/device-mapper.h |    5 +++++
 4 files changed, 26 insertions(+), 4 deletions(-)

diff --git a/block/blk-core.c b/block/blk-core.c
index 60912e9..6838936 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -1411,7 +1411,7 @@ void __blk_put_request(struct request_queue *q, struct request *req)
 	elv_completed_request(q, req);
 
 	/* this is a bio leak */
-	WARN_ON(req->bio != NULL);
+	WARN_ON(req->bio != NULL && !bio_flagged(req->bio, BIO_ENDIO_FREE));
 
 	/*
 	 * Request may not have originated from ll_rw_blk. if not,
@@ -2521,6 +2521,10 @@ bool blk_update_request(struct request *req, int error, unsigned int nr_bytes)
 	blk_account_io_completion(req, nr_bytes);
 
 	total_bytes = 0;
+
+	if (bio_flagged(req->bio, BIO_ENDIO_FREE))
+		return false;
+
 	while (req->bio) {
 		struct bio *bio = req->bio;
 		unsigned bio_bytes = min(bio->bi_iter.bi_size, nr_bytes);
diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index 6264781..2c18a34 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -1047,6 +1047,13 @@ static struct dm_rq_target_io *tio_from_request(struct request *rq)
 	return (rq->q->mq_ops ? blk_mq_rq_to_pdu(rq) : rq->special);
 }
 
+struct request *dm_get_orig_rq(struct request *clone)
+{
+	struct dm_rq_target_io *tio = clone->end_io_data;
+
+	return tio->orig;
+}
+
 static void rq_end_stats(struct mapped_device *md, struct request *orig)
 {
 	if (unlikely(dm_stats_used(&md->stats))) {
@@ -1118,7 +1125,7 @@ static void free_rq_clone(struct request *clone)
  * Must be called without clone's queue lock held,
  * see end_clone_request() for more details.
  */
-static void dm_end_request(struct request *clone, int error)
+void dm_end_request(struct request *clone, int error)
 {
 	int rw = rq_data_dir(clone);
 	struct dm_rq_target_io *tio = clone->end_io_data;
@@ -1311,7 +1318,7 @@ static void dm_complete_request(struct request *rq, int error)
  * Target's rq_end_io() function isn't called.
  * This may be used when the target's map_rq() or clone_and_map_rq() functions fail.
  */
-static void dm_kill_unmapped_request(struct request *rq, int error)
+void dm_kill_unmapped_request(struct request *rq, int error)
 {
 	rq->cmd_flags |= REQ_FAILED;
 	dm_complete_request(rq, error);
@@ -1758,7 +1765,7 @@ int dm_request_based(struct mapped_device *md)
 	return blk_queue_stackable(md->queue);
 }
 
-static void dm_dispatch_clone_request(struct request *clone, struct request *rq)
+void dm_dispatch_clone_request(struct request *clone, struct request *rq)
 {
 	int r;
 
diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h
index e813013..30aee54 100644
--- a/include/linux/blk_types.h
+++ b/include/linux/blk_types.h
@@ -122,6 +122,12 @@ struct bio {
 #define BIO_REFFED	8	/* bio has elevated ->bi_cnt */
 
 /*
+ * Added for Req based dm which need to perform post processing. This flag
+ * ensures blk_update_request does not free the bios or request, this is done
+ * at the dm level.
+ */
+#define BIO_ENDIO_FREE 12
+/*
  * Flags starting here get preserved by bio_reset() - this includes
  * BIO_POOL_IDX()
  */
diff --git a/include/linux/device-mapper.h b/include/linux/device-mapper.h
index 76d23fa..f636c50 100644
--- a/include/linux/device-mapper.h
+++ b/include/linux/device-mapper.h
@@ -407,6 +407,11 @@ union map_info *dm_get_rq_mapinfo(struct request *rq);
 
 struct queue_limits *dm_get_queue_limits(struct mapped_device *md);
 
+void dm_end_request(struct request *clone, int error);
+void dm_kill_unmapped_request(struct request *rq, int error);
+void dm_dispatch_clone_request(struct request *clone, struct request *rq);
+struct request *dm_get_orig_rq(struct request *clone);
+
 /*
  * Geometry functions.
  */
-- 
1.7.9.5

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1267352 — Re: [PATCH 1/2] block: Introduce BIO_ENDIO_FREE for bio flags

On Wed, Nov 11 2015 at  4:31am -0500,
Baolin Wang <baolin.wang@linaro.org> wrote:

> When we use dm-crypt to decrypt block data, it will decrypt the block data
> in endio() when one IO is completed. In this situation we don't want the
> cloned bios is freed before calling the endio().
> 
> Thus introduce 'BIO_ENDIO_FREE' flag to support the request handling for dm-crypt,
> this flag will ensure that blk layer does not complete the cloned bios before
> completing the request. When the crypt endio is called, post-processsing is
> done and then the dm layer will complete the bios (clones) and free them.

Not following why request-based DM's partial completion handling
(drivers/md/dm.c:end_clone_bio) isn't a sufficient hook -- no need to
add block complexity.

But that aside, I'm not liking the idea of a request-based dm-crypt.

> diff --git a/include/linux/device-mapper.h b/include/linux/device-mapper.h
> index 76d23fa..f636c50 100644
> --- a/include/linux/device-mapper.h
> +++ b/include/linux/device-mapper.h
> @@ -407,6 +407,11 @@ union map_info *dm_get_rq_mapinfo(struct request *rq);
>  
>  struct queue_limits *dm_get_queue_limits(struct mapped_device *md);
>  
> +void dm_end_request(struct request *clone, int error);
> +void dm_kill_unmapped_request(struct request *rq, int error);
> +void dm_dispatch_clone_request(struct request *clone, struct request *rq);
> +struct request *dm_get_orig_rq(struct request *clone);
> +
>  /*
>   * Geometry functions.
>   */

I have no interest in seeing any request-based DM interfaces exported.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1267602 — Re: [PATCH 1/2] block: Introduce BIO_ENDIO_FREE for bio flags

On 12 November 2015 at 01:54, Mike Snitzer <snitzer@redhat.com> wrote:
> On Wed, Nov 11 2015 at  4:31am -0500,
> Baolin Wang <baolin.wang@linaro.org> wrote:
>
>> When we use dm-crypt to decrypt block data, it will decrypt the block data
>> in endio() when one IO is completed. In this situation we don't want the
>> cloned bios is freed before calling the endio().
>>
>> Thus introduce 'BIO_ENDIO_FREE' flag to support the request handling for dm-crypt,
>> this flag will ensure that blk layer does not complete the cloned bios before
>> completing the request. When the crypt endio is called, post-processsing is
>> done and then the dm layer will complete the bios (clones) and free them.
>
> Not following why request-based DM's partial completion handling
> (drivers/md/dm.c:end_clone_bio) isn't a sufficient hook -- no need to
> add block complexity.
>

Sorry for lacking of more explanation for that. The dm-crypt will
decrypt block data in the end_io() callback function when one request
is completed, so we don't want the bios of this request is freed when
calling the end_io() callback. Thus we introduce a flag to indicate
these type bios of this request will be freed at dm layer not in block
layer.

> But that aside, I'm not liking the idea of a request-based dm-crypt.
>
>> diff --git a/include/linux/device-mapper.h b/include/linux/device-mapper.h
>> index 76d23fa..f636c50 100644
>> --- a/include/linux/device-mapper.h
>> +++ b/include/linux/device-mapper.h
>> @@ -407,6 +407,11 @@ union map_info *dm_get_rq_mapinfo(struct request *rq);
>>
>>  struct queue_limits *dm_get_queue_limits(struct mapped_device *md);
>>
>> +void dm_end_request(struct request *clone, int error);
>> +void dm_kill_unmapped_request(struct request *rq, int error);
>> +void dm_dispatch_clone_request(struct request *clone, struct request *rq);
>> +struct request *dm_get_orig_rq(struct request *clone);
>> +
>>  /*
>>   * Geometry functions.
>>   */
>
> I have no interest in seeing any request-based DM interfaces exported.

OK.


-- 
Baolin.wang
Best Regards
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1267062

On Wed, Nov 11, 2015 at 05:31:43PM +0800, Baolin Wang wrote:
> Now the dm-crypt code only implemented the 'based-bio' method to encrypt/
> decrypt block data, which can only hanle one bio at one time. As we know,
> one bio must use the sequential physical address and it also has a limitation
> of length. Thus it may limit the big block encyrtion/decryption when some
> hardware support the big block data encryption.
> 
> This patch series introduc the 'based-request' method to handle the data
> encryption/decryption. One request can contain multiple bios, so it can
> handle big block data to improve the efficiency.

NAK for more request based stacking or DM drivers.  They are a major
pain to deal with, and adding more with different requirements then
dm-multipath is not helping in actually making that one work properly.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1267587

On 11 November 2015 at 17:48, Christoph Hellwig <hch@infradead.org> wrote:
> On Wed, Nov 11, 2015 at 05:31:43PM +0800, Baolin Wang wrote:
>> Now the dm-crypt code only implemented the 'based-bio' method to encrypt/
>> decrypt block data, which can only hanle one bio at one time. As we know,
>> one bio must use the sequential physical address and it also has a limitation
>> of length. Thus it may limit the big block encyrtion/decryption when some
>> hardware support the big block data encryption.
>>
>> This patch series introduc the 'based-request' method to handle the data
>> encryption/decryption. One request can contain multiple bios, so it can
>> handle big block data to improve the efficiency.
>
> NAK for more request based stacking or DM drivers.  They are a major
> pain to deal with, and adding more with different requirements then
> dm-multipath is not helping in actually making that one work properly.

But now many vendors supply the hardware engine to handle the
encyrtion/decryption. The hardware really need a big block to indicate
its performance with request based things. Another thing is now the
request based things is used by many vendors (Qualcomm, Spreadtrum and
so on) to improve their performance and there's a real performance
requirement here (I can show the performance result later).

I don't think you will worry if any one can work properly, we will
remove the bio based things in future if the request things are
accepted and proved effectively. Thanks for your comment.

-- 
Baolin.wang
Best Regards
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1267745

On Thu 12-11-15 10:15:32, Baolin Wang wrote:
> On 11 November 2015 at 17:48, Christoph Hellwig <hch@infradead.org> wrote:
> > On Wed, Nov 11, 2015 at 05:31:43PM +0800, Baolin Wang wrote:
> >> Now the dm-crypt code only implemented the 'based-bio' method to encrypt/
> >> decrypt block data, which can only hanle one bio at one time. As we know,
> >> one bio must use the sequential physical address and it also has a limitation
> >> of length. Thus it may limit the big block encyrtion/decryption when some
> >> hardware support the big block data encryption.
> >>
> >> This patch series introduc the 'based-request' method to handle the data
> >> encryption/decryption. One request can contain multiple bios, so it can
> >> handle big block data to improve the efficiency.
> >
> > NAK for more request based stacking or DM drivers.  They are a major
> > pain to deal with, and adding more with different requirements then
> > dm-multipath is not helping in actually making that one work properly.
> 
> But now many vendors supply the hardware engine to handle the
> encyrtion/decryption. The hardware really need a big block to indicate
> its performance with request based things. Another thing is now the
> request based things is used by many vendors (Qualcomm, Spreadtrum and
> so on) to improve their performance and there's a real performance
> requirement here (I can show the performance result later).

So you've mentioned several times that hardware needs big blocks. How big
those blocks need to be? Ideally, can you give some numbers on how the
throughput of the encryption hw grows with the block size?

Because as Mike had said there are downsides to having request based
dm-crypt as well. E.g. if you want to have encrypted raid5 volume then
you'd rather want to put encryption on top of raid5 (easier management,
larger sequential blocks to encrypt, ...) but you cannot do that when
dm-crypt would be request based. So modifying bio-based dm-crypt to form
larger chunks for encryption HW would be superior in this regard.

You mentioned that you use requests because of size limitations on bios - I
had a look and current struct bio can easily describe 1MB requests (that's
assuming 64-bit architecture, 4KB pages) when we have 1 page worth of
struct bio_vec. Is that not enough?

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1267758

On 12 November 2015 at 17:17, Jan Kara <jack@suse.cz> wrote:
> On Thu 12-11-15 10:15:32, Baolin Wang wrote:
>> On 11 November 2015 at 17:48, Christoph Hellwig <hch@infradead.org> wrote:
>> > On Wed, Nov 11, 2015 at 05:31:43PM +0800, Baolin Wang wrote:
>> >> Now the dm-crypt code only implemented the 'based-bio' method to encrypt/
>> >> decrypt block data, which can only hanle one bio at one time. As we know,
>> >> one bio must use the sequential physical address and it also has a limitation
>> >> of length. Thus it may limit the big block encyrtion/decryption when some
>> >> hardware support the big block data encryption.
>> >>
>> >> This patch series introduc the 'based-request' method to handle the data
>> >> encryption/decryption. One request can contain multiple bios, so it can
>> >> handle big block data to improve the efficiency.
>> >
>> > NAK for more request based stacking or DM drivers.  They are a major
>> > pain to deal with, and adding more with different requirements then
>> > dm-multipath is not helping in actually making that one work properly.
>>
>> But now many vendors supply the hardware engine to handle the
>> encyrtion/decryption. The hardware really need a big block to indicate
>> its performance with request based things. Another thing is now the
>> request based things is used by many vendors (Qualcomm, Spreadtrum and
>> so on) to improve their performance and there's a real performance
>> requirement here (I can show the performance result later).
>
> So you've mentioned several times that hardware needs big blocks. How big
> those blocks need to be? Ideally, can you give some numbers on how the
> throughput of the encryption hw grows with the block size?

It depends on the hardware design. My beaglebone black board's AES
engine can handle 1M at one time which is not big. As I know some
other AES engine can handle 16M data at one time or more.

>
> Because as Mike had said there are downsides to having request based
> dm-crypt as well. E.g. if you want to have encrypted raid5 volume then
> you'd rather want to put encryption on top of raid5 (easier management,
> larger sequential blocks to encrypt, ...) but you cannot do that when
> dm-crypt would be request based. So modifying bio-based dm-crypt to form
> larger chunks for encryption HW would be superior in this regard.
>

Make sense.

> You mentioned that you use requests because of size limitations on bios - I
> had a look and current struct bio can easily describe 1MB requests (that's
> assuming 64-bit architecture, 4KB pages) when we have 1 page worth of
> struct bio_vec. Is that not enough?

Usually one bio does not always use the full 1M, maybe some 1k/2k/8k
or some other small chunks. But request can combine some sequential
small bios to be a big block and it is better than bio at least.

>
>                                                                 Honza
> --
> Jan Kara <jack@suse.com>
> SUSE Labs, CR



-- 
Baolin.wang
Best Regards
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1267823

On Thu 12-11-15 17:40:59, Baolin Wang wrote:
> On 12 November 2015 at 17:17, Jan Kara <jack@suse.cz> wrote:
> > On Thu 12-11-15 10:15:32, Baolin Wang wrote:
> >> On 11 November 2015 at 17:48, Christoph Hellwig <hch@infradead.org> wrote:
> >> > On Wed, Nov 11, 2015 at 05:31:43PM +0800, Baolin Wang wrote:
> >> >> Now the dm-crypt code only implemented the 'based-bio' method to encrypt/
> >> >> decrypt block data, which can only hanle one bio at one time. As we know,
> >> >> one bio must use the sequential physical address and it also has a limitation
> >> >> of length. Thus it may limit the big block encyrtion/decryption when some
> >> >> hardware support the big block data encryption.
> >> >>
> >> >> This patch series introduc the 'based-request' method to handle the data
> >> >> encryption/decryption. One request can contain multiple bios, so it can
> >> >> handle big block data to improve the efficiency.
> >> >
> >> > NAK for more request based stacking or DM drivers.  They are a major
> >> > pain to deal with, and adding more with different requirements then
> >> > dm-multipath is not helping in actually making that one work properly.
> >>
> >> But now many vendors supply the hardware engine to handle the
> >> encyrtion/decryption. The hardware really need a big block to indicate
> >> its performance with request based things. Another thing is now the
> >> request based things is used by many vendors (Qualcomm, Spreadtrum and
> >> so on) to improve their performance and there's a real performance
> >> requirement here (I can show the performance result later).
> >
> > So you've mentioned several times that hardware needs big blocks. How big
> > those blocks need to be? Ideally, can you give some numbers on how the
> > throughput of the encryption hw grows with the block size?
> 
> It depends on the hardware design. My beaglebone black board's AES
> engine can handle 1M at one time which is not big. As I know some
> other AES engine can handle 16M data at one time or more.

Well, one question is "can handle" and other question is how big gain in
throughput it will bring compared to say 1M chunks. I suppose there's some
constant overhead to issue a request to the crypto hw and by the time it is
encrypting 1M it may be that this overhead is well amortized by the cost of
the encryption itself which is in principle linear in the size of the
block. That's why I'd like to get idea of the real numbers...

> > You mentioned that you use requests because of size limitations on bios - I
> > had a look and current struct bio can easily describe 1MB requests (that's
> > assuming 64-bit architecture, 4KB pages) when we have 1 page worth of
> > struct bio_vec. Is that not enough?
> 
> Usually one bio does not always use the full 1M, maybe some 1k/2k/8k
> or some other small chunks. But request can combine some sequential
> small bios to be a big block and it is better than bio at least.

As Christoph mentions 4.3 should be better in submitting larger bios. Did
you check it?

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1267856

On 12 November 2015 at 19:06, Jan Kara <jack@suse.cz> wrote:
> On Thu 12-11-15 17:40:59, Baolin Wang wrote:
>> On 12 November 2015 at 17:17, Jan Kara <jack@suse.cz> wrote:
>> > On Thu 12-11-15 10:15:32, Baolin Wang wrote:
>> >> On 11 November 2015 at 17:48, Christoph Hellwig <hch@infradead.org> wrote:
>> >> > On Wed, Nov 11, 2015 at 05:31:43PM +0800, Baolin Wang wrote:
>> >> >> Now the dm-crypt code only implemented the 'based-bio' method to encrypt/
>> >> >> decrypt block data, which can only hanle one bio at one time. As we know,
>> >> >> one bio must use the sequential physical address and it also has a limitation
>> >> >> of length. Thus it may limit the big block encyrtion/decryption when some
>> >> >> hardware support the big block data encryption.
>> >> >>
>> >> >> This patch series introduc the 'based-request' method to handle the data
>> >> >> encryption/decryption. One request can contain multiple bios, so it can
>> >> >> handle big block data to improve the efficiency.
>> >> >
>> >> > NAK for more request based stacking or DM drivers.  They are a major
>> >> > pain to deal with, and adding more with different requirements then
>> >> > dm-multipath is not helping in actually making that one work properly.
>> >>
>> >> But now many vendors supply the hardware engine to handle the
>> >> encyrtion/decryption. The hardware really need a big block to indicate
>> >> its performance with request based things. Another thing is now the
>> >> request based things is used by many vendors (Qualcomm, Spreadtrum and
>> >> so on) to improve their performance and there's a real performance
>> >> requirement here (I can show the performance result later).
>> >
>> > So you've mentioned several times that hardware needs big blocks. How big
>> > those blocks need to be? Ideally, can you give some numbers on how the
>> > throughput of the encryption hw grows with the block size?
>>
>> It depends on the hardware design. My beaglebone black board's AES
>> engine can handle 1M at one time which is not big. As I know some
>> other AES engine can handle 16M data at one time or more.
>
> Well, one question is "can handle" and other question is how big gain in
> throughput it will bring compared to say 1M chunks. I suppose there's some
> constant overhead to issue a request to the crypto hw and by the time it is
> encrypting 1M it may be that this overhead is well amortized by the cost of
> the encryption itself which is in principle linear in the size of the
> block. That's why I'd like to get idea of the real numbers...

Please correct me if I misunderstood your point. Let's suppose the AES
engine can handle 16M at one time. If we give the size of data is less
than 16M, the engine can handle it at one time. But if the data size
is 20M (more than 16M), the engine driver will split the data with 16M
and 4M to deal with. I can not say how many numbers, but I think the
engine is like to big chunks than small chunks which is the hardware
engine's advantage.

>
>> > You mentioned that you use requests because of size limitations on bios - I
>> > had a look and current struct bio can easily describe 1MB requests (that's
>> > assuming 64-bit architecture, 4KB pages) when we have 1 page worth of
>> > struct bio_vec. Is that not enough?
>>
>> Usually one bio does not always use the full 1M, maybe some 1k/2k/8k
>> or some other small chunks. But request can combine some sequential
>> small bios to be a big block and it is better than bio at least.
>
> As Christoph mentions 4.3 should be better in submitting larger bios. Did
> you check it?

I'm sorry I didn't check it. What's the limitation of one bio on 4.3?
But I think it is better to choose a bigger one ( one request ) for
crypto which is suitable for hardware engine.

>
>                                                                 Honza
> --
> Jan Kara <jack@suse.com>
> SUSE Labs, CR



-- 
Baolin.wang
Best Regards
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1267878

On Thu 12-11-15 19:46:26, Baolin Wang wrote:
> On 12 November 2015 at 19:06, Jan Kara <jack@suse.cz> wrote:
> > On Thu 12-11-15 17:40:59, Baolin Wang wrote:
> >> On 12 November 2015 at 17:17, Jan Kara <jack@suse.cz> wrote:
> >> > On Thu 12-11-15 10:15:32, Baolin Wang wrote:
> >> >> On 11 November 2015 at 17:48, Christoph Hellwig <hch@infradead.org> wrote:
> >> >> > On Wed, Nov 11, 2015 at 05:31:43PM +0800, Baolin Wang wrote:
> >> >> >> Now the dm-crypt code only implemented the 'based-bio' method to encrypt/
> >> >> >> decrypt block data, which can only hanle one bio at one time. As we know,
> >> >> >> one bio must use the sequential physical address and it also has a limitation
> >> >> >> of length. Thus it may limit the big block encyrtion/decryption when some
> >> >> >> hardware support the big block data encryption.
> >> >> >>
> >> >> >> This patch series introduc the 'based-request' method to handle the data
> >> >> >> encryption/decryption. One request can contain multiple bios, so it can
> >> >> >> handle big block data to improve the efficiency.
> >> >> >
> >> >> > NAK for more request based stacking or DM drivers.  They are a major
> >> >> > pain to deal with, and adding more with different requirements then
> >> >> > dm-multipath is not helping in actually making that one work properly.
> >> >>
> >> >> But now many vendors supply the hardware engine to handle the
> >> >> encyrtion/decryption. The hardware really need a big block to indicate
> >> >> its performance with request based things. Another thing is now the
> >> >> request based things is used by many vendors (Qualcomm, Spreadtrum and
> >> >> so on) to improve their performance and there's a real performance
> >> >> requirement here (I can show the performance result later).
> >> >
> >> > So you've mentioned several times that hardware needs big blocks. How big
> >> > those blocks need to be? Ideally, can you give some numbers on how the
> >> > throughput of the encryption hw grows with the block size?
> >>
> >> It depends on the hardware design. My beaglebone black board's AES
> >> engine can handle 1M at one time which is not big. As I know some
> >> other AES engine can handle 16M data at one time or more.
> >
> > Well, one question is "can handle" and other question is how big gain in
> > throughput it will bring compared to say 1M chunks. I suppose there's some
> > constant overhead to issue a request to the crypto hw and by the time it is
> > encrypting 1M it may be that this overhead is well amortized by the cost of
> > the encryption itself which is in principle linear in the size of the
> > block. That's why I'd like to get idea of the real numbers...
> 
> Please correct me if I misunderstood your point. Let's suppose the AES
> engine can handle 16M at one time. If we give the size of data is less
> than 16M, the engine can handle it at one time. But if the data size
> is 20M (more than 16M), the engine driver will split the data with 16M
> and 4M to deal with. I can not say how many numbers, but I think the
> engine is like to big chunks than small chunks which is the hardware
> engine's advantage.

No, I meant something different. I meant that if HW can encrypt 1M in say
1.05 ms and it can encrypt 16M in 16.05 ms, then although using 16 M blocks
gives you some advantage it becomes diminishingly small.

> >> > You mentioned that you use requests because of size limitations on bios - I
> >> > had a look and current struct bio can easily describe 1MB requests (that's
> >> > assuming 64-bit architecture, 4KB pages) when we have 1 page worth of
> >> > struct bio_vec. Is that not enough?
> >>
> >> Usually one bio does not always use the full 1M, maybe some 1k/2k/8k
> >> or some other small chunks. But request can combine some sequential
> >> small bios to be a big block and it is better than bio at least.
> >
> > As Christoph mentions 4.3 should be better in submitting larger bios. Did
> > you check it?
> 
> I'm sorry I didn't check it. What's the limitation of one bio on 4.3?

On 4.3 it is 1 MB (which should be enough because requests are limited to
512 KB by default anyway). Previously the maximum bio size depended on the
queue parameters such as max number of segments etc.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1267895

On Thursday 12 November 2015 20:51:10 Baolin Wang wrote:
> On 12 November 2015 at 20:24, Jan Kara <jack@suse.cz> wrote:
> > On Thu 12-11-15 19:46:26, Baolin Wang wrote:
> >> On 12 November 2015 at 19:06, Jan Kara <jack@suse.cz> wrote:
> >> > Well, one question is "can handle" and other question is how big gain in
> >> > throughput it will bring compared to say 1M chunks. I suppose there's some
> >> > constant overhead to issue a request to the crypto hw and by the time it is
> >> > encrypting 1M it may be that this overhead is well amortized by the cost of
> >> > the encryption itself which is in principle linear in the size of the
> >> > block. That's why I'd like to get idea of the real numbers...
> >>
> >> Please correct me if I misunderstood your point. Let's suppose the AES
> >> engine can handle 16M at one time. If we give the size of data is less
> >> than 16M, the engine can handle it at one time. But if the data size
> >> is 20M (more than 16M), the engine driver will split the data with 16M
> >> and 4M to deal with. I can not say how many numbers, but I think the
> >> engine is like to big chunks than small chunks which is the hardware
> >> engine's advantage.
> >
> > No, I meant something different. I meant that if HW can encrypt 1M in say
> > 1.05 ms and it can encrypt 16M in 16.05 ms, then although using 16 M blocks
> > gives you some advantage it becomes diminishingly small.
> >
> 
> But if it encrypts 16M with 1M one by one, it will be much more than
> 16.05ms (should be consider the SW submits bio one by one).

The example that Jan gave was meant to illustrate the case where it's not
much more than 16.05ms, just slightly more.

The point is that we need real numbers to show at what size we stop
getting significant returns from increased block sizes.

> >> >> > You mentioned that you use requests because of size limitations on bios - I
> >> >> > had a look and current struct bio can easily describe 1MB requests (that's
> >> >> > assuming 64-bit architecture, 4KB pages) when we have 1 page worth of
> >> >> > struct bio_vec. Is that not enough?
> >> >>
> >> >> Usually one bio does not always use the full 1M, maybe some 1k/2k/8k
> >> >> or some other small chunks. But request can combine some sequential
> >> >> small bios to be a big block and it is better than bio at least.
> >> >
> >> > As Christoph mentions 4.3 should be better in submitting larger bios. Did
> >> > you check it?
> >>
> >> I'm sorry I didn't check it. What's the limitation of one bio on 4.3?
> >
> > On 4.3 it is 1 MB (which should be enough because requests are limited to
> > 512 KB by default anyway). Previously the maximum bio size depended on the
> > queue parameters such as max number of segments etc.
> 
> But it maybe not enough for HW engine which can handle maybe 10M/20M
> at one time.

Given that you have already done measurements, can you find out how much
you lose in overall performance with your existing patch if you artificially
limit the maximum size to sizes like 256kb, 1MB, 4MB, ...?

	Arnd
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1268010

[Multipart message — attachments visible in raw view] — view raw

On Thu, Nov 12, 2015 at 01:57:27PM +0100, Arnd Bergmann wrote:
> On Thursday 12 November 2015 20:51:10 Baolin Wang wrote:

> > But it maybe not enough for HW engine which can handle maybe 10M/20M
> > at one time.

> Given that you have already done measurements, can you find out how much
> you lose in overall performance with your existing patch if you artificially
> limit the maximum size to sizes like 256kb, 1MB, 4MB, ...?

It's probably also worth looking at the impact on CPU utilisation as
well as throughput in your benchmarking since the system will often not
be idle when it's doing a lot of I/O - I know you've done some
measurements in that area before, including them when looking at block
sizes might be interesting.

[toc] | [prev] | [next] | [standalone]

#1268561

On 12 November 2015 at 23:02, Mark Brown <broonie@kernel.org> wrote:
> On Thu, Nov 12, 2015 at 01:57:27PM +0100, Arnd Bergmann wrote:
>> On Thursday 12 November 2015 20:51:10 Baolin Wang wrote:
>
>> > But it maybe not enough for HW engine which can handle maybe 10M/20M
>> > at one time.
>
>> Given that you have already done measurements, can you find out how much
>> you lose in overall performance with your existing patch if you artificially
>> limit the maximum size to sizes like 256kb, 1MB, 4MB, ...?
>
> It's probably also worth looking at the impact on CPU utilisation as
> well as throughput in your benchmarking since the system will often not
> be idle when it's doing a lot of I/O - I know you've done some
> measurements in that area before, including them when looking at block
> sizes might be interesting.

Make sense.



-- 
Baolin.wang
Best Regards
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1268563

On 12 November 2015 at 20:57, Arnd Bergmann <arnd@arndb.de> wrote:
> On Thursday 12 November 2015 20:51:10 Baolin Wang wrote:
>> On 12 November 2015 at 20:24, Jan Kara <jack@suse.cz> wrote:
>> > On Thu 12-11-15 19:46:26, Baolin Wang wrote:
>> >> On 12 November 2015 at 19:06, Jan Kara <jack@suse.cz> wrote:
>> >> > Well, one question is "can handle" and other question is how big gain in
>> >> > throughput it will bring compared to say 1M chunks. I suppose there's some
>> >> > constant overhead to issue a request to the crypto hw and by the time it is
>> >> > encrypting 1M it may be that this overhead is well amortized by the cost of
>> >> > the encryption itself which is in principle linear in the size of the
>> >> > block. That's why I'd like to get idea of the real numbers...
>> >>
>> >> Please correct me if I misunderstood your point. Let's suppose the AES
>> >> engine can handle 16M at one time. If we give the size of data is less
>> >> than 16M, the engine can handle it at one time. But if the data size
>> >> is 20M (more than 16M), the engine driver will split the data with 16M
>> >> and 4M to deal with. I can not say how many numbers, but I think the
>> >> engine is like to big chunks than small chunks which is the hardware
>> >> engine's advantage.
>> >
>> > No, I meant something different. I meant that if HW can encrypt 1M in say
>> > 1.05 ms and it can encrypt 16M in 16.05 ms, then although using 16 M blocks
>> > gives you some advantage it becomes diminishingly small.
>> >
>>
>> But if it encrypts 16M with 1M one by one, it will be much more than
>> 16.05ms (should be consider the SW submits bio one by one).
>
> The example that Jan gave was meant to illustrate the case where it's not
> much more than 16.05ms, just slightly more.
>
> The point is that we need real numbers to show at what size we stop
> getting significant returns from increased block sizes.
>

Got it. Thanks.

>> >> >> > You mentioned that you use requests because of size limitations on bios - I
>> >> >> > had a look and current struct bio can easily describe 1MB requests (that's
>> >> >> > assuming 64-bit architecture, 4KB pages) when we have 1 page worth of
>> >> >> > struct bio_vec. Is that not enough?
>> >> >>
>> >> >> Usually one bio does not always use the full 1M, maybe some 1k/2k/8k
>> >> >> or some other small chunks. But request can combine some sequential
>> >> >> small bios to be a big block and it is better than bio at least.
>> >> >
>> >> > As Christoph mentions 4.3 should be better in submitting larger bios. Did
>> >> > you check it?
>> >>
>> >> I'm sorry I didn't check it. What's the limitation of one bio on 4.3?
>> >
>> > On 4.3 it is 1 MB (which should be enough because requests are limited to
>> > 512 KB by default anyway). Previously the maximum bio size depended on the
>> > queue parameters such as max number of segments etc.
>>
>> But it maybe not enough for HW engine which can handle maybe 10M/20M
>> at one time.
>
> Given that you have already done measurements, can you find out how much
> you lose in overall performance with your existing patch if you artificially
> limit the maximum size to sizes like 256kb, 1MB, 4MB, ...?
>

Cause my board AES engine throughput is 1M, I just did a simple dd
test with small chunks. Results are in last email.

>         Arnd



-- 
Baolin.wang
Best Regards
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1267898

On 12 November 2015 at 20:24, Jan Kara <jack@suse.cz> wrote:
> On Thu 12-11-15 19:46:26, Baolin Wang wrote:
>> On 12 November 2015 at 19:06, Jan Kara <jack@suse.cz> wrote:
>> > On Thu 12-11-15 17:40:59, Baolin Wang wrote:
>> >> On 12 November 2015 at 17:17, Jan Kara <jack@suse.cz> wrote:
>> >> > On Thu 12-11-15 10:15:32, Baolin Wang wrote:
>> >> >> On 11 November 2015 at 17:48, Christoph Hellwig <hch@infradead.org> wrote:
>> >> >> > On Wed, Nov 11, 2015 at 05:31:43PM +0800, Baolin Wang wrote:
>> >> >> >> Now the dm-crypt code only implemented the 'based-bio' method to encrypt/
>> >> >> >> decrypt block data, which can only hanle one bio at one time. As we know,
>> >> >> >> one bio must use the sequential physical address and it also has a limitation
>> >> >> >> of length. Thus it may limit the big block encyrtion/decryption when some
>> >> >> >> hardware support the big block data encryption.
>> >> >> >>
>> >> >> >> This patch series introduc the 'based-request' method to handle the data
>> >> >> >> encryption/decryption. One request can contain multiple bios, so it can
>> >> >> >> handle big block data to improve the efficiency.
>> >> >> >
>> >> >> > NAK for more request based stacking or DM drivers.  They are a major
>> >> >> > pain to deal with, and adding more with different requirements then
>> >> >> > dm-multipath is not helping in actually making that one work properly.
>> >> >>
>> >> >> But now many vendors supply the hardware engine to handle the
>> >> >> encyrtion/decryption. The hardware really need a big block to indicate
>> >> >> its performance with request based things. Another thing is now the
>> >> >> request based things is used by many vendors (Qualcomm, Spreadtrum and
>> >> >> so on) to improve their performance and there's a real performance
>> >> >> requirement here (I can show the performance result later).
>> >> >
>> >> > So you've mentioned several times that hardware needs big blocks. How big
>> >> > those blocks need to be? Ideally, can you give some numbers on how the
>> >> > throughput of the encryption hw grows with the block size?
>> >>
>> >> It depends on the hardware design. My beaglebone black board's AES
>> >> engine can handle 1M at one time which is not big. As I know some
>> >> other AES engine can handle 16M data at one time or more.
>> >
>> > Well, one question is "can handle" and other question is how big gain in
>> > throughput it will bring compared to say 1M chunks. I suppose there's some
>> > constant overhead to issue a request to the crypto hw and by the time it is
>> > encrypting 1M it may be that this overhead is well amortized by the cost of
>> > the encryption itself which is in principle linear in the size of the
>> > block. That's why I'd like to get idea of the real numbers...
>>
>> Please correct me if I misunderstood your point. Let's suppose the AES
>> engine can handle 16M at one time. If we give the size of data is less
>> than 16M, the engine can handle it at one time. But if the data size
>> is 20M (more than 16M), the engine driver will split the data with 16M
>> and 4M to deal with. I can not say how many numbers, but I think the
>> engine is like to big chunks than small chunks which is the hardware
>> engine's advantage.
>
> No, I meant something different. I meant that if HW can encrypt 1M in say
> 1.05 ms and it can encrypt 16M in 16.05 ms, then although using 16 M blocks
> gives you some advantage it becomes diminishingly small.
>

But if it encrypts 16M with 1M one by one, it will be much more than
16.05ms (should be consider the SW submits bio one by one).

>> >> > You mentioned that you use requests because of size limitations on bios - I
>> >> > had a look and current struct bio can easily describe 1MB requests (that's
>> >> > assuming 64-bit architecture, 4KB pages) when we have 1 page worth of
>> >> > struct bio_vec. Is that not enough?
>> >>
>> >> Usually one bio does not always use the full 1M, maybe some 1k/2k/8k
>> >> or some other small chunks. But request can combine some sequential
>> >> small bios to be a big block and it is better than bio at least.
>> >
>> > As Christoph mentions 4.3 should be better in submitting larger bios. Did
>> > you check it?
>>
>> I'm sorry I didn't check it. What's the limitation of one bio on 4.3?
>
> On 4.3 it is 1 MB (which should be enough because requests are limited to
> 512 KB by default anyway). Previously the maximum bio size depended on the
> queue parameters such as max number of segments etc.
>

But it maybe not enough for HW engine which can handle maybe 10M/20M
at one time.


>                                                                 Honza
> --
> Jan Kara <jack@suse.com>
> SUSE Labs, CR



-- 
Baolin.wang
Best Regards
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1267899

On Thu 12-11-15 20:51:10, Baolin Wang wrote:
> On 12 November 2015 at 20:24, Jan Kara <jack@suse.cz> wrote:
> > On Thu 12-11-15 19:46:26, Baolin Wang wrote:
> >> On 12 November 2015 at 19:06, Jan Kara <jack@suse.cz> wrote:
> >> > On Thu 12-11-15 17:40:59, Baolin Wang wrote:
> >> >> On 12 November 2015 at 17:17, Jan Kara <jack@suse.cz> wrote:
> >> >> > On Thu 12-11-15 10:15:32, Baolin Wang wrote:
> >> >> >> On 11 November 2015 at 17:48, Christoph Hellwig <hch@infradead.org> wrote:
> >> >> >> > On Wed, Nov 11, 2015 at 05:31:43PM +0800, Baolin Wang wrote:
> >> >> >> >> Now the dm-crypt code only implemented the 'based-bio' method to encrypt/
> >> >> >> >> decrypt block data, which can only hanle one bio at one time. As we know,
> >> >> >> >> one bio must use the sequential physical address and it also has a limitation
> >> >> >> >> of length. Thus it may limit the big block encyrtion/decryption when some
> >> >> >> >> hardware support the big block data encryption.
> >> >> >> >>
> >> >> >> >> This patch series introduc the 'based-request' method to handle the data
> >> >> >> >> encryption/decryption. One request can contain multiple bios, so it can
> >> >> >> >> handle big block data to improve the efficiency.
> >> >> >> >
> >> >> >> > NAK for more request based stacking or DM drivers.  They are a major
> >> >> >> > pain to deal with, and adding more with different requirements then
> >> >> >> > dm-multipath is not helping in actually making that one work properly.
> >> >> >>
> >> >> >> But now many vendors supply the hardware engine to handle the
> >> >> >> encyrtion/decryption. The hardware really need a big block to indicate
> >> >> >> its performance with request based things. Another thing is now the
> >> >> >> request based things is used by many vendors (Qualcomm, Spreadtrum and
> >> >> >> so on) to improve their performance and there's a real performance
> >> >> >> requirement here (I can show the performance result later).
> >> >> >
> >> >> > So you've mentioned several times that hardware needs big blocks. How big
> >> >> > those blocks need to be? Ideally, can you give some numbers on how the
> >> >> > throughput of the encryption hw grows with the block size?
> >> >>
> >> >> It depends on the hardware design. My beaglebone black board's AES
> >> >> engine can handle 1M at one time which is not big. As I know some
> >> >> other AES engine can handle 16M data at one time or more.
> >> >
> >> > Well, one question is "can handle" and other question is how big gain in
> >> > throughput it will bring compared to say 1M chunks. I suppose there's some
> >> > constant overhead to issue a request to the crypto hw and by the time it is
> >> > encrypting 1M it may be that this overhead is well amortized by the cost of
> >> > the encryption itself which is in principle linear in the size of the
> >> > block. That's why I'd like to get idea of the real numbers...
> >>
> >> Please correct me if I misunderstood your point. Let's suppose the AES
> >> engine can handle 16M at one time. If we give the size of data is less
> >> than 16M, the engine can handle it at one time. But if the data size
> >> is 20M (more than 16M), the engine driver will split the data with 16M
> >> and 4M to deal with. I can not say how many numbers, but I think the
> >> engine is like to big chunks than small chunks which is the hardware
> >> engine's advantage.
> >
> > No, I meant something different. I meant that if HW can encrypt 1M in say
> > 1.05 ms and it can encrypt 16M in 16.05 ms, then although using 16 M blocks
> > gives you some advantage it becomes diminishingly small.
> >
> 
> But if it encrypts 16M with 1M one by one, it will be much more than
> 16.05ms (should be consider the SW submits bio one by one).

Really? In my example, it would take 16.8 ms if we encrypted 16M in 1M
chunks and 16.05 ms if done in one chunk. That is a difference for which I
would not be willing to bend over backwards. Now these numbers are
completely made up and that's why I wanted to see the real numbers...

> >> >> > You mentioned that you use requests because of size limitations on bios - I
> >> >> > had a look and current struct bio can easily describe 1MB requests (that's
> >> >> > assuming 64-bit architecture, 4KB pages) when we have 1 page worth of
> >> >> > struct bio_vec. Is that not enough?
> >> >>
> >> >> Usually one bio does not always use the full 1M, maybe some 1k/2k/8k
> >> >> or some other small chunks. But request can combine some sequential
> >> >> small bios to be a big block and it is better than bio at least.
> >> >
> >> > As Christoph mentions 4.3 should be better in submitting larger bios. Did
> >> > you check it?
> >>
> >> I'm sorry I didn't check it. What's the limitation of one bio on 4.3?
> >
> > On 4.3 it is 1 MB (which should be enough because requests are limited to
> > 512 KB by default anyway). Previously the maximum bio size depended on the
> > queue parameters such as max number of segments etc.
> 
> But it maybe not enough for HW engine which can handle maybe 10M/20M
> at one time.

Currently, you would not be able to create larger than 512K / 1M chunks
even with request based dm-crypt since requests have limits on number of
data they can carry as well... So this is kind of abstract discussion.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1268544

On 12 November 2015 at 20:59, Jan Kara <jack@suse.cz> wrote:
> On Thu 12-11-15 20:51:10, Baolin Wang wrote:
>> On 12 November 2015 at 20:24, Jan Kara <jack@suse.cz> wrote:
>> > On Thu 12-11-15 19:46:26, Baolin Wang wrote:
>> >> On 12 November 2015 at 19:06, Jan Kara <jack@suse.cz> wrote:
>> >> > On Thu 12-11-15 17:40:59, Baolin Wang wrote:
>> >> >> On 12 November 2015 at 17:17, Jan Kara <jack@suse.cz> wrote:
>> >> >> > On Thu 12-11-15 10:15:32, Baolin Wang wrote:
>> >> >> >> On 11 November 2015 at 17:48, Christoph Hellwig <hch@infradead.org> wrote:
>> >> >> >> > On Wed, Nov 11, 2015 at 05:31:43PM +0800, Baolin Wang wrote:
>> >> >> >> >> Now the dm-crypt code only implemented the 'based-bio' method to encrypt/
>> >> >> >> >> decrypt block data, which can only hanle one bio at one time. As we know,
>> >> >> >> >> one bio must use the sequential physical address and it also has a limitation
>> >> >> >> >> of length. Thus it may limit the big block encyrtion/decryption when some
>> >> >> >> >> hardware support the big block data encryption.
>> >> >> >> >>
>> >> >> >> >> This patch series introduc the 'based-request' method to handle the data
>> >> >> >> >> encryption/decryption. One request can contain multiple bios, so it can
>> >> >> >> >> handle big block data to improve the efficiency.
>> >> >> >> >
>> >> >> >> > NAK for more request based stacking or DM drivers.  They are a major
>> >> >> >> > pain to deal with, and adding more with different requirements then
>> >> >> >> > dm-multipath is not helping in actually making that one work properly.
>> >> >> >>
>> >> >> >> But now many vendors supply the hardware engine to handle the
>> >> >> >> encyrtion/decryption. The hardware really need a big block to indicate
>> >> >> >> its performance with request based things. Another thing is now the
>> >> >> >> request based things is used by many vendors (Qualcomm, Spreadtrum and
>> >> >> >> so on) to improve their performance and there's a real performance
>> >> >> >> requirement here (I can show the performance result later).
>> >> >> >
>> >> >> > So you've mentioned several times that hardware needs big blocks. How big
>> >> >> > those blocks need to be? Ideally, can you give some numbers on how the
>> >> >> > throughput of the encryption hw grows with the block size?
>> >> >>
>> >> >> It depends on the hardware design. My beaglebone black board's AES
>> >> >> engine can handle 1M at one time which is not big. As I know some
>> >> >> other AES engine can handle 16M data at one time or more.
>> >> >
>> >> > Well, one question is "can handle" and other question is how big gain in
>> >> > throughput it will bring compared to say 1M chunks. I suppose there's some
>> >> > constant overhead to issue a request to the crypto hw and by the time it is
>> >> > encrypting 1M it may be that this overhead is well amortized by the cost of
>> >> > the encryption itself which is in principle linear in the size of the
>> >> > block. That's why I'd like to get idea of the real numbers...
>> >>
>> >> Please correct me if I misunderstood your point. Let's suppose the AES
>> >> engine can handle 16M at one time. If we give the size of data is less
>> >> than 16M, the engine can handle it at one time. But if the data size
>> >> is 20M (more than 16M), the engine driver will split the data with 16M
>> >> and 4M to deal with. I can not say how many numbers, but I think the
>> >> engine is like to big chunks than small chunks which is the hardware
>> >> engine's advantage.
>> >
>> > No, I meant something different. I meant that if HW can encrypt 1M in say
>> > 1.05 ms and it can encrypt 16M in 16.05 ms, then although using 16 M blocks
>> > gives you some advantage it becomes diminishingly small.
>> >
>>
>> But if it encrypts 16M with 1M one by one, it will be much more than
>> 16.05ms (should be consider the SW submits bio one by one).
>
> Really? In my example, it would take 16.8 ms if we encrypted 16M in 1M
> chunks and 16.05 ms if done in one chunk. That is a difference for which I
> would not be willing to bend over backwards. Now these numbers are
> completely made up and that's why I wanted to see the real numbers...
>

Well, I did a simple test with dd reading, cause my engine limitation is 1M,
(1) so the time like below when handle 1M at one time.
1048576 bytes (1.0 MB) copied, 0.0841235 s, 12.5 MB/s
1048576 bytes (1.0 MB) copied, 0.0836294 s, 12.5 MB/s
1048576 bytes (1.0 MB) copied, 0.0836526 s, 12.5 MB/s

(2) These handle 64K at one time * 16 times
1048576 bytes (1.0 MB) copied, 0.0937223 s, 11.2 MB/s
1048576 bytes (1.0 MB) copied, 0.097205 s, 10.8 MB/s
1048576 bytes (1.0 MB) copied, 0.0935884 s, 11.2 MB/s

Here is a 10ms level difference, try to image if the hardware engine's
throughput is bigger than that. But like Jens said, we can measure it
by the performance data. Thanks.

>> >> >> > You mentioned that you use requests because of size limitations on bios - I
>> >> >> > had a look and current struct bio can easily describe 1MB requests (that's
>> >> >> > assuming 64-bit architecture, 4KB pages) when we have 1 page worth of
>> >> >> > struct bio_vec. Is that not enough?
>> >> >>
>> >> >> Usually one bio does not always use the full 1M, maybe some 1k/2k/8k
>> >> >> or some other small chunks. But request can combine some sequential
>> >> >> small bios to be a big block and it is better than bio at least.
>> >> >
>> >> > As Christoph mentions 4.3 should be better in submitting larger bios. Did
>> >> > you check it?
>> >>
>> >> I'm sorry I didn't check it. What's the limitation of one bio on 4.3?
>> >
>> > On 4.3 it is 1 MB (which should be enough because requests are limited to
>> > 512 KB by default anyway). Previously the maximum bio size depended on the
>> > queue parameters such as max number of segments etc.
>>
>> But it maybe not enough for HW engine which can handle maybe 10M/20M
>> at one time.
>
> Currently, you would not be able to create larger than 512K / 1M chunks
> even with request based dm-crypt since requests have limits on number of
> data they can carry as well... So this is kind of abstract discussion.
>

OK.  But I think if it is that it should change the default limitation
for the DM device.

>                                                                 Honza
> --
> Jan Kara <jack@suse.com>
> SUSE Labs, CR



-- 
Baolin.wang
Best Regards
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1268704

On Friday 13 November 2015 10:05:28 Baolin Wang wrote:
> 
> Well, I did a simple test with dd reading, cause my engine limitation is 1M,
> (1) so the time like below when handle 1M at one time.
> 1048576 bytes (1.0 MB) copied, 0.0841235 s, 12.5 MB/s
> 1048576 bytes (1.0 MB) copied, 0.0836294 s, 12.5 MB/s
> 1048576 bytes (1.0 MB) copied, 0.0836526 s, 12.5 MB/s
> 
> (2) These handle 64K at one time * 16 times
> 1048576 bytes (1.0 MB) copied, 0.0937223 s, 11.2 MB/s
> 1048576 bytes (1.0 MB) copied, 0.097205 s, 10.8 MB/s
> 1048576 bytes (1.0 MB) copied, 0.0935884 s, 11.2 MB/s
> 
> Here is a 10ms level difference, try to image if the hardware engine's
> throughput is bigger than that. But like Jens said, we can measure it
> by the performance data.

The absolute numbers look really low. Does this include writing to
a hard drive? That would certainly make the difference appear
less significant.

Could you try backing this with a ram disk backing for comparison,
and also use 'time dd' to show the CPU utilization for all cases?
For completeness, including cpu-only performance might also help
put this into perspective.

	Arnd
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1268881

On 13 November 2015 at 17:05, Arnd Bergmann <arnd@arndb.de> wrote:
> On Friday 13 November 2015 10:05:28 Baolin Wang wrote:
>>
>> Well, I did a simple test with dd reading, cause my engine limitation is 1M,
>> (1) so the time like below when handle 1M at one time.
>> 1048576 bytes (1.0 MB) copied, 0.0841235 s, 12.5 MB/s
>> 1048576 bytes (1.0 MB) copied, 0.0836294 s, 12.5 MB/s
>> 1048576 bytes (1.0 MB) copied, 0.0836526 s, 12.5 MB/s
>>
>> (2) These handle 64K at one time * 16 times
>> 1048576 bytes (1.0 MB) copied, 0.0937223 s, 11.2 MB/s
>> 1048576 bytes (1.0 MB) copied, 0.097205 s, 10.8 MB/s
>> 1048576 bytes (1.0 MB) copied, 0.0935884 s, 11.2 MB/s
>>
>> Here is a 10ms level difference, try to image if the hardware engine's
>> throughput is bigger than that. But like Jens said, we can measure it
>> by the performance data.
>
> The absolute numbers look really low. Does this include writing to
> a hard drive? That would certainly make the difference appear
> less significant.
>

OK, I'll supply the complete performance data to measure it.

> Could you try backing this with a ram disk backing for comparison,
> and also use 'time dd' to show the CPU utilization for all cases?
> For completeness, including cpu-only performance might also help
> put this into perspective.
>

OK, I'll try. Thanks.

>         Arnd



-- 
Baolin.wang
Best Regards
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.kernel

csiph-web