Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1564555 > unrolled thread

[PATCH v4 0/4] Update LZ4 compressor module

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  [PATCH v4 0/4] Update LZ4 compressor module Sven Schmidt <4sschmid@informatik.uni-hamburg.de> - 2017-01-22 20:40 +0100
    [PATCH v4 2/4] lib/decompress_unlz4: Change module to work with new LZ4 module version Sven Schmidt <4sschmid@informatik.uni-hamburg.de> - 2017-01-22 20:40 +0100
    [PATCH v4 4/4] fs/pstore: fs/squashfs: Change usage of LZ4 to work with new LZ4 version Sven Schmidt <4sschmid@informatik.uni-hamburg.de> - 2017-01-22 20:40 +0100
    Re: [PATCH v4 1/4] lib: Update LZ4 compressor module Andrew Morton <akpm@linux-foundation.org> - 2017-01-24 01:30 +0100
      Re: [PATCH v4 1/4] lib: Update LZ4 compressor module Sven Schmidt <4sschmid@informatik.uni-hamburg.de> - 2017-01-24 18:00 +0100

#1564555 — [PATCH v4 0/4] Update LZ4 compressor module

This patchset is for updating the LZ4 compression module to a version based
on LZ4 v1.7.3 allowing to use the fast compression algorithm aka LZ4 fast
which provides an "acceleration" parameter as a tradeoff between
high compression ratio and high compression speed.

We want to use LZ4 fast in order to support compression in lustre
and (mostly, based on that) investigate data reduction techniques in behalf of
storage systems.

Also, it will be useful for other users of LZ4 compression, as with LZ4 fast
it is possible to enable applications to use fast and/or high compression
depending on the usecase.
For instance, ZRAM is offering a LZ4 backend and could benefit from an updated
LZ4 in the kernel.

LZ4 homepage: http://www.lz4.org/
LZ4 source repository: https://github.com/lz4/lz4
Source version: 1.7.3

Benchmark (taken from [1], Core i5-4300U @1.9GHz):
----------------|--------------|----------------|----------
Compressor      | Compression  | Decompression  | Ratio
----------------|--------------|----------------|----------
memcpy          |  4200 MB/s   |  4200 MB/s     | 1.000
LZ4 fast 50     |  1080 MB/s   |  2650 MB/s     | 1.375
LZ4 fast 17     |   680 MB/s   |  2220 MB/s     | 1.607
LZ4 fast 5      |   475 MB/s   |  1920 MB/s     | 1.886
LZ4 default     |   385 MB/s   |  1850 MB/s     | 2.101

[1] http://fastcompression.blogspot.de/2015/04/sampling-or-faster-lz4.html
fs/pstore: fs/squashfs: Change usage of LZ4 to work with new LZ4 version

[PATCH 1/4] lib: Update LZ4 compressor module
[PATCH 2/4] lib/decompress_unlz4: Change module to work with new LZ4 module version
[PATCH 3/4] crypto: Change LZ4 modules to work with new LZ4 module version
[PATCH 4/4] fs/pstore: fs/squashfs: Change usage of LZ4 to work with new LZ4 version

v2:
- Changed order of the patches since in the initial patchset the lz4.h was in the
  last patch but was referenced by the other ones
- Split lib/decompress_unlz4.c in an own patch
- Fixed errors reported by the buildbot
- Further refactorings
- Added more appropriate copyright note to include/linux/lz4.h

v3:
- Adjusted the code to satisfy kernel coding style (checkpatch.pl)
- Made sure the changes to LZ4 in Kernel (overflow checks etc.)
  are included in the new module (they are)
- Removed the second LZ4_compressBound function with related name but
  different return type
- Corrected version number (was LZ4 1.7.3)
- Added missing LZ4 streaming functions

v4:
- Fixed kbuild errors
- Re-added lz4_compressbound as alias for LZ4_compressBound
  to ensure backwards compatibility
- Wrapped LZ4_hash5 with check for LZ4_ARCH64 since it is only used there
  and triggers an unused function warning when false

[toc] | [next] | [standalone]

#1564556 — [PATCH v4 2/4] lib/decompress_unlz4: Change module to work with new LZ4 module version

This patch updates the unlz4 wrapper to work with the
updated LZ4 kernel module version.

Signed-off-by: Sven Schmidt <4sschmid@informatik.uni-hamburg.de>
---
 lib/decompress_unlz4.c | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/lib/decompress_unlz4.c b/lib/decompress_unlz4.c
index 036fc88..1b0baf3 100644
--- a/lib/decompress_unlz4.c
+++ b/lib/decompress_unlz4.c
@@ -72,7 +72,7 @@ STATIC inline int INIT unlz4(u8 *input, long in_len,
 		error("NULL input pointer and missing fill function");
 		goto exit_1;
 	} else {
-		inp = large_malloc(lz4_compressbound(uncomp_chunksize));
+		inp = large_malloc(LZ4_compressBound(uncomp_chunksize));
 		if (!inp) {
 			error("Could not allocate input buffer");
 			goto exit_1;
@@ -136,7 +136,7 @@ STATIC inline int INIT unlz4(u8 *input, long in_len,
 			inp += 4;
 			size -= 4;
 		} else {
-			if (chunksize > lz4_compressbound(uncomp_chunksize)) {
+			if (chunksize > LZ4_compressBound(uncomp_chunksize)) {
 				error("chunk length is longer than allocated");
 				goto exit_2;
 			}
@@ -152,11 +152,14 @@ STATIC inline int INIT unlz4(u8 *input, long in_len,
 			out_len -= dest_len;
 		} else
 			dest_len = out_len;
-		ret = lz4_decompress(inp, &chunksize, outp, dest_len);
+
+		ret = LZ4_decompress_fast(inp, outp, dest_len);
+		chunksize = ret;
 #else
 		dest_len = uncomp_chunksize;
-		ret = lz4_decompress_unknownoutputsize(inp, chunksize, outp,
-				&dest_len);
+
+		ret = LZ4_decompress_safe(inp, outp, chunksize, dest_len);
+		dest_len = ret;
 #endif
 		if (ret < 0) {
 			error("Decoding failed");
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1564557 — [PATCH v4 4/4] fs/pstore: fs/squashfs: Change usage of LZ4 to work with new LZ4 version

This patch updates fs/pstore and fs/squashfs to use the updated
functions from the new LZ4 module.

Signed-off-by: Sven Schmidt <4sschmid@informatik.uni-hamburg.de>
---
 fs/pstore/platform.c      | 14 +++++++-------
 fs/squashfs/lz4_wrapper.c | 12 ++++++------
 2 files changed, 13 insertions(+), 13 deletions(-)

diff --git a/fs/pstore/platform.c b/fs/pstore/platform.c
index 729677e..85c4c58 100644
--- a/fs/pstore/platform.c
+++ b/fs/pstore/platform.c
@@ -342,31 +342,31 @@ static int compress_lz4(const void *in, void *out, size_t inlen, size_t outlen)
 {
 	int ret;
 
-	ret = lz4_compress(in, inlen, out, &outlen, workspace);
-	if (ret) {
+	ret = LZ4_compress_default(in, out, inlen, outlen, workspace);
+	if (!ret) {
 		pr_err("lz4_compress error, ret = %d!\n", ret);
 		return -EIO;
 	}
 
-	return outlen;
+	return ret;
 }
 
 static int decompress_lz4(void *in, void *out, size_t inlen, size_t outlen)
 {
 	int ret;
 
-	ret = lz4_decompress_unknownoutputsize(in, inlen, out, &outlen);
-	if (ret) {
+	ret = LZ4_decompress_safe(in, out, inlen, outlen);
+	if (ret < 0) {
 		pr_err("lz4_decompress error, ret = %d!\n", ret);
 		return -EIO;
 	}
 
-	return outlen;
+	return ret;
 }
 
 static void allocate_lz4(void)
 {
-	big_oops_buf_sz = lz4_compressbound(psinfo->bufsize);
+	big_oops_buf_sz = LZ4_compressBound(psinfo->bufsize);
 	big_oops_buf = kmalloc(big_oops_buf_sz, GFP_KERNEL);
 	if (big_oops_buf) {
 		workspace = kmalloc(LZ4_MEM_COMPRESS, GFP_KERNEL);
diff --git a/fs/squashfs/lz4_wrapper.c b/fs/squashfs/lz4_wrapper.c
index ff4468b..95da653 100644
--- a/fs/squashfs/lz4_wrapper.c
+++ b/fs/squashfs/lz4_wrapper.c
@@ -97,7 +97,6 @@ static int lz4_uncompress(struct squashfs_sb_info *msblk, void *strm,
 	struct squashfs_lz4 *stream = strm;
 	void *buff = stream->input, *data;
 	int avail, i, bytes = length, res;
-	size_t dest_len = output->length;
 
 	for (i = 0; i < b; i++) {
 		avail = min(bytes, msblk->devblksize - offset);
@@ -108,12 +107,13 @@ static int lz4_uncompress(struct squashfs_sb_info *msblk, void *strm,
 		put_bh(bh[i]);
 	}
 
-	res = lz4_decompress_unknownoutputsize(stream->input, length,
-					stream->output, &dest_len);
-	if (res)
+	res = LZ4_decompress_safe(stream->input, stream->output,
+		length, output->length);
+
+	if (res < 0)
 		return -EIO;
 
-	bytes = dest_len;
+	bytes = res;
 	data = squashfs_first_page(output);
 	buff = stream->output;
 	while (data) {
@@ -128,7 +128,7 @@ static int lz4_uncompress(struct squashfs_sb_info *msblk, void *strm,
 	}
 	squashfs_finish_page(output);
 
-	return dest_len;
+	return res;
 }
 
 const struct squashfs_decompressor squashfs_lz4_comp_ops = {
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1565386 — Re: [PATCH v4 1/4] lib: Update LZ4 compressor module

On Sun, 22 Jan 2017 20:35:14 +0100 Sven Schmidt <4sschmid@informatik.uni-hamburg.de> wrote:

> This patch updates LZ4 kernel module to LZ4 v1.7.3 by Yann Collet.
> The kernel module is inspired by the previous work by Chanho Min.
> The updated LZ4 module will not break existing code since there were alias
> methods added to ensure backwards compatibility.
> 
> API changes:
> 
> New method LZ4_compress_fast which differs from the variant available
> in kernel by the new acceleration parameter,
> allowing to trade compression ratio for more compression speed
> and vice versa.
> 
> LZ4_decompress_fast is the respective decompression method, featuring a very
> fast decoder (multiple GB/s per core), able to reach RAM speed in multi-core
> systems. The decompressor allows to decompress data compressed with
> LZ4 fast as well as the LZ4 HC (high compression) algorithm.
> 
> Also the useful functions LZ4_decompress_safe_partial
> LZ4_compress_destsize were added. The latter reverses the logic by trying to
> compress as much data as possible from source to dest while the former aims
> to decompress partial blocks of data.
> 
> A bunch of streaming functions were also added
> which allow compressig/decompressing data in multiple steps
> (so called "streaming mode").
> 
> The methods lz4_compress and lz4_decompress_unknownoutputsize
> are now known as LZ4_compress_default respectivley LZ4_decompress_safe.
> The old methods are still available for providing backwards compatibility.
> 
> ...
>
> +/*
> + * For backward compatibility
> + */
> +static inline int lz4_compressbound(size_t isize)
>  {

Do we actually need the back-compat wrappers?  After your other three
patches we have no callers, correct?  If so, we should go ahead and
eliminate such back-compatibility interfaces.


> -	return isize + (isize / 255) + 16;
> +	return LZ4_COMPRESSBOUND(isize);
>  }

[toc] | [prev] | [next] | [standalone]

#1566006 — Re: [PATCH v4 1/4] lib: Update LZ4 compressor module

On Mon, Jan 23, 2017 at 04:23:57PM -0800, Andrew Morton wrote:
> On Sun, 22 Jan 2017 20:35:14 +0100 Sven Schmidt <4sschmid@informatik.uni-hamburg.de> wrote:
> 
> > This patch updates LZ4 kernel module to LZ4 v1.7.3 by Yann Collet.
> > The kernel module is inspired by the previous work by Chanho Min.
> > The updated LZ4 module will not break existing code since there were alias
> > methods added to ensure backwards compatibility.
> > 
> > API changes:
> > 
> > New method LZ4_compress_fast which differs from the variant available
> > in kernel by the new acceleration parameter,
> > allowing to trade compression ratio for more compression speed
> > and vice versa.
> > 
> > LZ4_decompress_fast is the respective decompression method, featuring a very
> > fast decoder (multiple GB/s per core), able to reach RAM speed in multi-core
> > systems. The decompressor allows to decompress data compressed with
> > LZ4 fast as well as the LZ4 HC (high compression) algorithm.
> > 
> > Also the useful functions LZ4_decompress_safe_partial
> > LZ4_compress_destsize were added. The latter reverses the logic by trying to
> > compress as much data as possible from source to dest while the former aims
> > to decompress partial blocks of data.
> > 
> > A bunch of streaming functions were also added
> > which allow compressig/decompressing data in multiple steps
> > (so called "streaming mode").
> > 
> > The methods lz4_compress and lz4_decompress_unknownoutputsize
> > are now known as LZ4_compress_default respectivley LZ4_decompress_safe.
> > The old methods are still available for providing backwards compatibility.
> > 
> > ...
> >
> > +/*
> > + * For backward compatibility
> > + */
> > +static inline int lz4_compressbound(size_t isize)
> >  {
> 
> Do we actually need the back-compat wrappers?  After your other three
> patches we have no callers, correct?  If so, we should go ahead and
> eliminate such back-compatibility interfaces.
> 
> 
> > -	return isize + (isize / 255) + 16;
> > +	return LZ4_COMPRESSBOUND(isize);
> >  }
> 

Hey Andrew,

you're absolutely right. Since I changed the callers to use the new functions, we do not need these wrappers anymore.

I will add a patch removing them and send an update.

Thanks,

Sven

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web