Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.compression > #1406 > unrolled thread

multi-member gzip usage

Back to article view | Back to comp.compression

Contents

  multi-member gzip usage jefflad <jladouceur@gmail.com> - 2012-07-09 16:17 -0700
    Re: multi-member gzip usage Eli the Bearded <*@eli.users.panix.com> - 2012-07-10 19:25 +0000
      Re: multi-member gzip usage Sven Köhler <remove-sven.koehler@gmail.com> - 2012-07-11 21:13 +0200
    Re: multi-member gzip usage Thomas Pornin <pornin@bolet.org> - 2012-07-11 11:06 +0000

#1406 — multi-member gzip usage

What are the use cases for multi-member gzip ? Are there any applications that uses it ?

For instance would http 1.1 chunk encoding use this ?

thanks

[toc] | [next] | [standalone]

#1410

In comp.compression, jefflad  <jladouceur@gmail.com> wrote:
> What are the use cases for multi-member gzip ? Are there any
> applications that uses it ?

I've used it in the past for log archives. For cases where you
want to keep more logs around than you can fit uncompressed on
disk, it can fill a niche: Daily log files compressed daily
into monthly archives, that can then be fed to a log parser
able process from a compressed stream.

> For instance would http 1.1 chunk encoding use this ?

I don't know if it does, but that seems a reasonable use case.

Elijah
------
now just uses daily files

[toc] | [prev] | [next] | [standalone]

#1418

Am 10.07.2012 21:25, schrieb Eli the Bearded:
> In comp.compression, jefflad  <jladouceur@gmail.com> wrote:
>> For instance would http 1.1 chunk encoding use this ?
> 
> I don't know if it does, but that seems a reasonable use case.

No. I think chunked transfer-encoding was invented to solve one
essential problem: if you don't know the length of the content in
advance (content-length HTTP response header), you cannot know when the
HTTP response ends. Prior to to chunked encoding, you had to read till
EOF and then throw the HTTP connection away. In order to allow the
re-use of the connection, you send the data in chunks, terminating the
stream of chunks with a zero length chunk. It's simply another way of
encoding an EOF. No content-length header is needed.

Also, wikipedia states, that gzip and chunk encoding are at different
layers: content-encoding (gzip) and transfer-encoding (chunks):
http://en.wikipedia.org/wiki/Chunked_transfer_encoding
So first, the data is gzipped, then the gzip data is split into chunks.

A HTTP-oriented use-case of multi-member gzip could be the following:
If the server wants to perform a flush (i.e. emptying any buffers,
making sure any data sent so far successfully arrives at the client),
the server could terminate the current member and start a new one.
But aren't there other means of flushing? Also, I don't know whether
some webservers actually implement flushing of gzip-compressed HTTP
connections.


Regards,
  Sven

[toc] | [prev] | [next] | [standalone]

#1413

According to jefflad  <jladouceur@gmail.com>:
> What are the use cases for multi-member gzip ? Are there any
> applications that uses it ?

I have seen "mutt" using it. This is a Unix application for reading and
sending emails; it can read mailboxes in the traditional format (i.e.
all mails concatenated in a single file), and it can also use compressed
mailboxes (the same file, compressed with gzip). When adding new emails
to a mailbox (a common case of an "archive mailbox"), it may do it by
appending the gzip of the new emails to the existing file (i.e. a
multi-member gzip), thus avoiding the cost of decompressing and
recompressing the existing mailbox (CPU cost, and also the temporary
storage cost).

(This was more than 15 years ago, and it was quite handy when the system
was a 25 MHz Sparc system, and the mailbox was in the 10MB+ range. Since
then, machines have grown, and mailboxes have grown, too, but not as
fast.)


	--Thomas Pornin

[toc] | [prev] | [standalone]

Back to top | Article view | comp.compression

csiph-web