Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.compression > #3255 > unrolled thread

TurboBench - Back to the future (incl. LzTurbo)

Started bypowturbo <powturbo@gmail.com>
First post2015-10-21 15:09 -0700
Last post2017-05-20 04:45 -0700
Articles 19 — 2 participants

Back to article view | Back to comp.compression

Contents

  TurboBench - Back to the future (incl. LzTurbo) powturbo <powturbo@gmail.com> - 2015-10-21 15:09 -0700
    Re: TurboBench - Back to the future (incl. LzTurbo) powturbo <powturbo@gmail.com> - 2015-11-16 10:59 -0800
      Re: TurboBench - Back to the future (incl. LzTurbo) powturbo <powturbo@gmail.com> - 2015-11-25 04:27 -0800
        Re: TurboBench - Back to the future (incl. LzTurbo) powturbo <powturbo@gmail.com> - 2015-12-02 04:22 -0800
        TurboBench - Back to the future (incl. LzTurbo) powturbo <powturbo@gmail.com> - 2015-12-02 04:24 -0800
          TurboBench - Back to the future (incl. LzTurbo) powturbo <powturbo@gmail.com> - 2015-12-25 14:31 -0800
            Re: TurboBench - Back to the future (incl. LzTurbo) powturbo <powturbo@gmail.com> - 2016-01-09 06:42 -0800
              Re: TurboBench - Back to the future (incl. LzTurbo) powturbo <powturbo@gmail.com> - 2016-04-17 12:19 -0700
    Re: TurboBench - Back to the future (incl. LzTurbo) powturbo <powturbo@gmail.com> - 2016-06-18 10:16 -0700
      Re: TurboBench - Back to the future (incl. LzTurbo) powturbo <powturbo@gmail.com> - 2016-08-17 02:25 -0700
        Re: TurboBench - Back to the future (incl. LzTurbo) powturbo <powturbo@gmail.com> - 2016-08-19 13:38 -0700
          Re: TurboBench - Back to the future (incl. LzTurbo) powturbo <powturbo@gmail.com> - 2017-03-03 01:49 -0800
            Re: TurboBench - Back to the future (incl. LzTurbo) powturbo <powturbo@gmail.com> - 2017-03-27 03:36 -0700
              Re: TurboBench - Back to the future (incl. LzTurbo) powturbo <powturbo@gmail.com> - 2017-04-15 01:20 -0700
                Re: TurboBench - Back to the future (incl. LzTurbo) powturbo <powturbo@gmail.com> - 2017-04-21 11:23 -0700
                  Re: TurboBench - Back to the future (incl. LzTurbo) powturbo <powturbo@gmail.com> - 2017-05-10 04:21 -0700
                    Re: TurboBench - Back to the future (incl. LzTurbo) xbouzidi@googlemail.com - 2017-05-15 05:57 -0700
                    Re: TurboBench - Back to the future (incl. LzTurbo) powturbo <powturbo@gmail.com> - 2017-05-16 10:28 -0700
                      Re: TurboBench - Back to the future (incl. LzTurbo) powturbo <powturbo@gmail.com> - 2017-05-20 04:45 -0700

#3255 — TurboBench - Back to the future (incl. LzTurbo)

Frompowturbo <powturbo@gmail.com>
Date2015-10-21 15:09 -0700
SubjectTurboBench - Back to the future (incl. LzTurbo)
Message-ID<2084d0c1-4f5c-4e92-abf2-84d585c350cf@googlegroups.com>
Compressor Benchmark : Back to the future version incl. LzTurbo


- 100% in memory benchmark, no I/O overhead
- Include most popular, latest or fastest compressors in one compiled package 
- Minimum plugins call overhead
- Set one, a group or several compressors to benchmark at the command line
- Multiple input files with recursive directories
- Concatenate multiple small files into one multiblock file
- Benchmark multiblock file as one large block, but each block processed separatelly
- Avoid cache szenario found in other benchmarks with small files
- Set block size, file size limit,...
- Set number of iterations, number of runs, set max. time per run
- Automatic sort by compressed length
- 64 bits Linux and Windows versions (gcc 5.2/mingw64 gcc5.2 w. -O3, only for Sandy bridge and later CPUs)
- The only benchmark program including LzTurbo
- Benchmarking entropy coders, lz77 and bwt compressors

benchmark program source code will be released in the near future.


Download TurboBench at: https://sites.google.com/site/powturbo/downloads

Including: LzTurbo, lz4, zlib, brotli, lzham, lzma, ...

[toc] | [next] | [standalone]

#3277

Frompowturbo <powturbo@gmail.com>
Date2015-11-16 10:59 -0800
Message-ID<2ab8f369-0a40-403d-b4c9-2cbf8f21231a@googlegroups.com>
In reply to#3255
Update incl. Benchmark program source code
- Include (>50) allmost all popular, latest or fastest compressors in one compiled package 
- Set number of iterations, number of runs, benchamrks, set max. time per run
- Benchmarking Entropy Coders, Lz77 and BWT compressors
- Text, html, csv, markdown and other output formats
- show pareto frontier
- Transfer speed sheet for different connections or devices: GPRS,3G,4G,DSL,Network,HDD,SSD,RAM


Benchmark program source code: https://github.com/powturbo/TurboBench

Compressor Benchmark w/ transfer speed sheet: http://encode.ru/threads/2333-TurboBench-Back-to-the-future-(incl-LzTurbo)

[toc] | [prev] | [next] | [standalone]

#3292

Frompowturbo <powturbo@gmail.com>
Date2015-11-25 04:27 -0800
Message-ID<99d992a1-ce72-4b1a-8251-6bf83ffdb01e@googlegroups.com>
In reply to#3277
Turbobench update incl. Transfer Speed Plot:
https://github.com/powturbo/TurboBench

- new - automatic update & merge of result files
- new - Text, html, csv, markdown and other output formats without retesting
- new - html output with sortable tables
- new - Transfer speed sheet for different connections or devices: GPRS,2G,3G,4G,DSL,Network,HDD,SSD,RAM
- Speedup html plot
- Linux and Windows binaries
- All in one executable, no hassless installing of additional package

[toc] | [prev] | [next] | [standalone]

#3304

Frompowturbo <powturbo@gmail.com>
Date2015-12-02 04:22 -0800
Message-ID<ce6addb7-1938-4dfd-80e5-c7bb181b0e7b@googlegroups.com>
In reply to#3292
Turbobench update: https://github.com/powturbo/TurboBench

- New plugin: libzpaq
- New plugin: shoco
- New version: zstd 
- New update brotli and all submodules
- New option "-s" to benchmark small files accurately
- New option "-g" disable update of result file
- stdin / stdout file I/O
- Source code: 32 bits ready for linux and windows
- Gipfeli updated
- New: Clang support
- New: Ratio/Speed plot

[toc] | [prev] | [next] | [standalone]

#3305

Frompowturbo <powturbo@gmail.com>
Date2015-12-02 04:24 -0800
Message-ID<822fbd1f-9686-4bad-a7ec-deeff2cb6b37@googlegroups.com>
In reply to#3292
On Wednesday, November 25, 2015 at 1:27:19 PM UTC+1, powturbo wrote:
Turbobench update: https://github.com/powturbo/TurboBench 

- New plugin: libzpaq 
- New plugin: shoco 
- New version: zstd 
- New update brotli and all submodules 
- New option "-s" to benchmark small files accurately 
- New option "-g" disable update of result file 
- stdin / stdout file I/O 
- Source code: 32 bits ready for linux and windows 
- Gipfeli updated 
- New: Clang support 
- New: Ratio/Speed plot

[toc] | [prev] | [next] | [standalone]

#3333

Frompowturbo <powturbo@gmail.com>
Date2015-12-25 14:31 -0800
Message-ID<81a27359-85a1-48ba-8d02-78c48d4f4ca0@googlegroups.com>
In reply to#3305
Turbobench github update all submodules.

- New: build in peak memory usage and reporting on html output.
Only for linux and tracking only malloc/free memory calls.

- New: Sample codec "mycodec" to show how to include and build your own functions.

[toc] | [prev] | [next] | [standalone]

#3360

Frompowturbo <powturbo@gmail.com>
Date2016-01-09 06:42 -0800
Message-ID<38b17715-36ee-4107-bad3-64ad3d9e1fc9@googlegroups.com>
In reply to#3333
TurboBench update

Turbobench github update all submodules.

- blosc
- zstd
- lz5
- CSC
- brotli
- libzling

- New: build in peak memory usage and reporting on html output.
Only for linux and tracking only malloc/free memory calls.

- New: Sample codec "mycodec" to show how to include and build your own functions.

- New Entropy Coders: fpaq0p_sh, fqz0, ppmdec, nania rANS, fpaqc
- New: miniz
- New: Visual Studio c++ 2015 support

- New: You can now disable the brotli dictionary in TurboBench with the parameter 'D'.
Ex.
./turbobench -ebrotli,11,11D canterbury/cp.html

[toc] | [prev] | [next] | [standalone]

#3417

Frompowturbo <powturbo@gmail.com>
Date2016-04-17 12:19 -0700
Message-ID<1f25e38e-761a-46ca-a15b-8cd2a4aef0d4@googlegroups.com>
In reply to#3360
Turbobench github update all submodules.
 - blosc
 - zstd
 - brotli
 - bcm
 - zpaq
 - new: Libdeflate
 - new: lzoma
 - new: LZSSE

[toc] | [prev] | [next] | [standalone]

#3445

Frompowturbo <powturbo@gmail.com>
Date2016-06-18 10:16 -0700
Message-ID<a723deec-4902-4625-bd9e-46f4aea55e87@googlegroups.com>
In reply to#3255
Turbobench update:
- new LZFSE apple's compression algorithm


TurboBench : New efficiency benchmark including LZFSE and the fastest compressors with similar compression ratio. 

CPU i7-2600k at 4,2 GHz (all compressors with latest version 18 JUN 2016)
app3-Portable App Suite(binary) from http://compressionratings.com/app3.html

      C Size  ratio%     C MB/s     D MB/s   Name            File              (bold = pareto) MB=1.000.0000
    36775168    36.7      66.85    1460.28   lzturbo 32      app3.tar
    46562400    46.5     198.73    1220.31   lzturbo 31      app3.tar
    46804845    46.8      50.60     964.08   zstd 9          app3.tar
    47754454    47.7      41.19     603.81   xpack 2         app3.tar
    48788658    48.7      83.87     350.78   brotli 4        app3.tar
    49159780    49.1     133.69     880.02   zstd 4          app3.tar
    49521949    49.5      69.80     622.48   lzfse           app3.tar
    49618923    49.6      78.59     520.27   libdeflate 7    app3.tar
    49784097    49.7      51.23    2011.81   lzturbo 22      app3.tar
    49962678    49.9      35.23     308.44   zlib 6          app3.tar
    50152281    50.1      43.90     306.01   zlib 5          app3.tar
    51578720    51.5      37.90    1446.18   lz5 8           app3.tar

      C Size  ratio%     C MB/s     D MB/s   Name            File              (bold = pareto) MB=1.000.0000
    59827047    28.2      53.30    1002.34   lzturbo 32      silesia.tar
    60416836    28.5      48.19     839.14   zstd 9          silesia.tar
    65312617    30.8      60.39     633.68   xpack 2         silesia.tar
    66035804    31.2     183.94    1020.04   lzturbo 31      silesia.tar
    66404949    31.3     130.11     736.24   zstd 4          silesia.tar
    66858972    31.5      80.24     452.16   brotli 4        silesia.tar
    67611164    31.9      70.55     721.22   libdeflate 7    silesia.tar
    67624724    31.9      69.51     734.42   lzfse           silesia.tar
    68225985    32.2      32.61     362.39   zlib 6          silesia.tar
    69159650    32.6      45.93     355.86   zlib 5          silesia.tar
    73815607    34.8      54.92    1821.74   lzturbo 22      silesia.tar
    77999252    36.8      48.15    1286.94   lz5 8           silesia.tar

[toc] | [prev] | [next] | [standalone]

#3468

Frompowturbo <powturbo@gmail.com>
Date2016-08-17 02:25 -0700
Message-ID<165c3c49-cd30-464c-bdc3-8368bbcd35ea@googlegroups.com>
In reply to#3445
New TurboBench benchmark : http://encode.ru/threads/2333-TurboBench-Back-to-the-future-(incl-LzTurbo)?p=49454&viewfull=1#post49454

TurboBench - Fastest Lz77 - ARM 64 bits - Cortex-A53 2Ghz

[toc] | [prev] | [next] | [standalone]

#3471

Frompowturbo <powturbo@gmail.com>
Date2016-08-19 13:38 -0700
Message-ID<297d5a35-3ab3-444b-909b-dc71123c78f1@googlegroups.com>
In reply to#3468
New TurboBench benchmark : http://encode.ru/threads/2575-SLZ-stateless-zip-fast-zlib-compatible-compressor?p=49537&viewfull=1#post49537 

TurboBench - Dynamic Web Content Compression Benchmark 

Intel & ARM CPU

[toc] | [prev] | [next] | [standalone]

#3547

Frompowturbo <powturbo@gmail.com>
Date2017-03-03 01:49 -0800
Message-ID<fc614c7b-d556-4897-91a1-2127c346532d@googlegroups.com>
In reply to#3471
New update TurboBench Compressor Benchmark: https://github.com/powturbo/TurboBench

Latest compressor versions at 2017.03.03
- brotli
- zlib + zlib-ng
- lz4
- rans_static
- snappy
- zstd
- and others

[toc] | [prev] | [next] | [standalone]

#3560

Frompowturbo <powturbo@gmail.com>
Date2017-03-27 03:36 -0700
Message-ID<726893bb-8908-4503-b939-c4ef8c6f884a@googlegroups.com>
In reply to#3547
TurboBench new update
TurboBench binaries for windows and linux updated with the latest compressor versions:
https://sites.google.com/site/powturbo/downloads

You can find the latest source code on Github: 
https://github.com/powturbo/TurboBench

[toc] | [prev] | [next] | [standalone]

#3576

Frompowturbo <powturbo@gmail.com>
Date2017-04-15 01:20 -0700
Message-ID<080f8c1b-2d6f-41f3-89d5-da9b920597c0@googlegroups.com>
In reply to#3560
New Text Data Compression Benchmark:
- Ryzen 1700 3.9 GHz
- Haswell i5-4670K 4.6 GHz
- i5-7200u 3.1GHz (notebook)

https://drive.google.com/file/d/0BzKgu_YpO6uZd0RkcGRGTk50WTA/view

[toc] | [prev] | [next] | [standalone]

#3580

Frompowturbo <powturbo@gmail.com>
Date2017-04-21 11:23 -0700
Message-ID<4704587b-b039-45cf-ba60-c2bdef1e094e@googlegroups.com>
In reply to#3576
Update: Static/Dynamic web content compression benchmark with TurboBench

https://sites.google.com/site/powturbo/home/web-compression

- Latest version for all compressor
- New: zstd included for comparison only (content-encoding not supported)

[toc] | [prev] | [next] | [standalone]

#3593

Frompowturbo <powturbo@gmail.com>
Date2017-05-10 04:21 -0700
Message-ID<628acb80-3a11-4baf-9cd7-fa485aaa4cf9@googlegroups.com>
In reply to#3580
TurboBench Compressor benchmark github update:
https://github.com/powturbo/TurboBench

- Thanks to Turbobench submodule architecture, all codecs with latest version as usual
- Benchmark updated with latest version of all compressors.


Included the new pareto "lzturbo 29" without any entropy coding.
For this binary file it beats the max. lizard,49 (huffman) and even max. brotli 11 in compression ratio.

 28,788,842 for enwik8
230,792,563 for enwik9 (better than cabarc)

[toc] | [prev] | [next] | [standalone]

#3596

Fromxbouzidi@googlemail.com
Date2017-05-15 05:57 -0700
Message-ID<fd40ef14-4f80-4a88-9093-a6356be87375@googlegroups.com>
In reply to#3593
In Memory Compression Benchmark 
https://sites.google.com/site/powturbo/compression-benchmark

- new update: binary app, text enwik8, game data, silesia corpus

[toc] | [prev] | [next] | [standalone]

#3597

Frompowturbo <powturbo@gmail.com>
Date2017-05-16 10:28 -0700
Message-ID<7772f17f-f73f-4270-92c9-7ad9a342ce36@googlegroups.com>
In reply to#3593
In Memory Compression Benchmark 
https://sites.google.com/site/powturbo/compression-benchmark 

- new update: binary app, text enwik8, game data, silesia corpus, http log file

[toc] | [prev] | [next] | [standalone]

#3598

Frompowturbo <powturbo@gmail.com>
Date2017-05-20 04:45 -0700
Message-ID<100bfa90-cd85-4ebf-aca5-707cb7dfd388@googlegroups.com>
In reply to#3597
TurboBench Compressor benchmark github update:
https://github.com/powturbo/TurboBench

- new igzip: Intel(R) Intelligent Storage Acceleration Library
https://github.com/01org/isa-l

Paper: ISA-L Igzip: Improvements to a Fast Deflate (DCC2017)
http://ieeexplore.ieee.org/document/7923748/

see also: 
Static/Dynamic web content compression benchmark
https://sites.google.com/site/powturbo/home/web-compression

[toc] | [prev] | [standalone]

Back to top | Article view | comp.compression

csiph-web