Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.compression > #1992

Re: Generating data that can be compressed with a known ratio

Show all headers | View raw

Ian Collins <ian-news@hotmail.com> writes:
> I'm looking for an algorithm I can use to generate data that will have
> a fixed compress ratio when compressed with an ideal compression
> algorithm (or gzip as a practical compressor).
> 
> I want to produce files that will compress by a given percentage in
> order to test performance of a filesystem with compression enabled.
> 
> Any ideas?

On top of all the other answers given (reduce the alphabet, and 
bias the bit distribution), assuming your files are long, the 
easiest *dumb* solution is to just generate the appropriate ratio 
of known incompressible data from a decent PRNG, and intersperse 
chunks of that with chunks of trivially compressible data (runs of
0, say). If might take a short while for the model to learn its in
a run of zeros, but the overhead shouldn't be too much (also, the
incompressible data may expand by a bit, as that's what compression
functions do, and that would need to be taken into consideration if
you want accuracy.)

However, it's probably better to just use a corpus of real world
files, as that's what people using the algorithm in the real world
will be doing.

Phil
-- 
If "law-abiding citizens have nothing to fear" from privacy-invading 
technologies and policies, then law-abiding governments should have
nothing to fear from whistleblowers.

Back to comp.compression | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

Generating data that can be compressed with a known ratio Ian Collins <ian-news@hotmail.com> - 2013-06-23 10:59 +1200
  Re: Generating data that can be compressed with a known ratio glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2013-06-22 23:24 +0000
  Re: Generating data that can be compressed with a known ratio SG <sgesemann@gmail.invalid> - 2013-06-23 01:45 +0200
  Re: Generating data that can be compressed with a known ratio hitchmanr@gmail.com - 2013-06-24 11:53 -0700
    Re: Generating data that can be compressed with a known ratio glen herrmannsfeldt <gah@ugcs.caltech.edu> - 2013-06-24 19:15 +0000
      Re: Generating data that can be compressed with a known ratio hitchmanr@gmail.com - 2013-06-24 13:44 -0700
        Re: Generating data that can be compressed with a known ratio Ian Collins <ian-news@hotmail.com> - 2013-06-25 10:18 +1200
          Re: Generating data that can be compressed with a known ratio hitchmanr@gmail.com - 2013-06-24 16:57 -0700
            Re: Generating data that can be compressed with a known ratio Ian Collins <ian-news@hotmail.com> - 2013-06-25 13:58 +1200
  Re: Generating data that can be compressed with a known ratio Phil Carmody <thefatphil_demunged@yahoo.co.uk> - 2013-06-27 10:08 +0300

csiph-web