Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.compression > #2841 > unrolled thread
| Started by | "Dex" <admin@127.0.0.1> |
|---|---|
| First post | 2015-02-05 15:59 +0800 |
| Last post | 2015-02-08 11:13 +0800 |
| Articles | 3 — 2 participants |
Back to article view | Back to comp.compression
Pre-process before packing a file thats 2/3 nullchars? "Dex" <admin@127.0.0.1> - 2015-02-05 15:59 +0800
Re: Pre-process before packing a file thats 2/3 nullchars? jacko <jackokring@gmail.com> - 2015-02-07 05:18 -0800
Re: Pre-process before packing a file thats 2/3 nullchars? "Dex" <admin@127.0.0.1> - 2015-02-08 11:13 +0800
| From | "Dex" <admin@127.0.0.1> |
|---|---|
| Date | 2015-02-05 15:59 +0800 |
| Subject | Pre-process before packing a file thats 2/3 nullchars? |
| Message-ID | <mav7sd$73d$1@adenine.netfront.net> |
Im working with a custom file format that is often maybe two-thirds full of nullchars ... not really too many in a row (its not an "RLE thing"), but just lots of them scattered throughout as most of the contents are 32bit integer array with usually one or two bytes null per int. Should I just leave it up to the main compression algorithm (lzma) to deal with those, or would it be worthwhile using a pre-processor first (and post-processor afterwards as it needs to be lossless), and if so what type? Thankyou! --- news://freenews.netfront.net/ - complaints: news@netfront.net ---
[toc] | [next] | [standalone]
| From | jacko <jackokring@gmail.com> |
|---|---|
| Date | 2015-02-07 05:18 -0800 |
| Message-ID | <d5c95d67-ddd4-4b2b-9183-a79ace0f7568@googlegroups.com> |
| In reply to | #2841 |
You might get a shrink with a pre/post processing, but would the speed impact be worth it? For example by building a bit map of nul or not nul, and then having just the not nuls to compress would suggest a reduction in size to 1/3 rd plus an entropy stream 1/8 th size with some bit bias to compress out in the map, so about 11/24 th in size by pre-compression. After the lzma (although this may not be the best), you might get a little extra shrink, but then again you might not. You'd have to test.
[toc] | [prev] | [next] | [standalone]
| From | "Dex" <admin@127.0.0.1> |
|---|---|
| Date | 2015-02-08 11:13 +0800 |
| Message-ID | <mb6k88$2158$1@adenine.netfront.net> |
| In reply to | #2858 |
Thankyou very much jacko Yes I'll have to do some tests to determine overall if the ratio gain is worth the extra processing, but that's the sort of information i was after --- news://freenews.netfront.net/ - complaints: news@netfront.net ---
[toc] | [prev] | [standalone]
Back to top | Article view | comp.compression
csiph-web