Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #6354

Re: filters

Show all headers | View raw

On 06/18/2011 06:15 AM, lee wrote:
> Hi,
> 
> it´s probably a FAQ, though I haven´t found any good info on it: How do
> you go about writing a filter, i. e. a program that reads data from
> stdin and processes it?
> 
> Since indefinte amounts of data could be read from stdin, they can´t
> just be put into some (ever increasing) buffer. Using a buffer of
> limited size can make it difficult to process the data because the
> buffer could be too small.
> 
> What´s the solution for this?

The term "filter" is generally reserved for the kind of application that
only needs to keep a small portion of the input in memory at any given
time. A typical Unix filter is "cut". It parses each line of input up
into fields, which can be either fixed width or delimited by a
user-specifiable character which defaults to '\t'. It writes out a
specified subset of the fields, with the delimiter optionally replaced
with an arbitrary string. I've never attempted implementing it, but it
seems to me that it should be implementable in a way that never keeps
more than one character of input in memory at any given time.

However, if for some reason your program does need to store the entire
input, then you need expandable storage, and the C standard library
provides some. Start by allocating a buffer with malloc(). Whenever the
buffer gets full, call realloc() to expand it; I recommend increasing
the size by a fixed factor; 2 would be a good value. Note, there are a
couple of tricky points in connection with calling realloc():
* if realloc() fails, it returns a null pointer, and pointers into the
old buffer are still valid. Therefore, if you make the mistake of
storing the value returned by realloc() directly into the same pointer
you were using to keep track of your buffer, you'll lose your ability to
access that buffer if realloc() fails.
* if realloc() succeeds, it may have moved your data to a new location
in memory, invalidating any pointers you may have been keeping that
pointed into your old buffer. You can't safely do anything with any of
the old pointer values, not even comparing them for equality with new
ones to determine whether or not the buffer was moved. For each such
pointer, determine its offset from the beginning of the buffer before
calling realloc(). If realloc() succeeds, calculate the new value for
the corresponding pointer by adding that offset to the start of the new
buffer.

If realloc() does fail, you'll have to switch to a different approach.
One option is to create a temporary file using tmpfile(). Read from
standard input, then write to the temporary file. Once the entire file
is read in, you can move around in the temporary file using fseek(),
something you cannot do with stdin.
-- 
James Kuyper

Back to comp.lang.c | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

filters lee <lee@yun.yagibdah.de> - 2011-06-18 12:15 +0200
  Re: filters Ben Bacarisse <ben.usenet@bsb.me.uk> - 2011-06-18 11:37 +0100
    Re: filters cri@tiac.net (Richard Harter) - 2011-06-18 18:42 +0000
      Re: filters Ben Bacarisse <ben.usenet@bsb.me.uk> - 2011-06-18 23:13 +0100
        Re: filters cri@tiac.net (Richard Harter) - 2011-06-19 19:02 +0000
          Re: filters Ben Bacarisse <ben.usenet@bsb.me.uk> - 2011-06-19 21:16 +0100
            Re: filters lee <lee@yun.yagibdah.de> - 2011-06-20 16:06 +0200
              Re: filters Ben Bacarisse <ben.usenet@bsb.me.uk> - 2011-06-20 15:53 +0100
                Re: filters lee <lee@yun.yagibdah.de> - 2011-06-27 22:20 +0200
  Re: filters Francois Grieu <fgrieu@gmail.com> - 2011-06-18 13:56 +0200
    Re: filters "Bill Cunningham" <nospam@nspam.invalid> - 2011-06-18 14:24 -0400
      Re: filters Angel <angel+news@spamcop.net> - 2011-06-18 18:51 +0000
        Re: filters Keith Thompson <kst-u@mib.org> - 2011-06-18 11:55 -0700
          Re: filters Angel <angel+news@spamcop.net> - 2011-06-18 19:46 +0000
            Re: filters Keith Thompson <kst-u@mib.org> - 2011-06-18 13:18 -0700
      Re: filters Lew Pitcher <lpitcher@teksavvy.com> - 2011-06-18 14:55 -0400
  Re: filters James Kuyper <jameskuyper@verizon.net> - 2011-06-18 10:14 -0400
    Re: filters cri@tiac.net (Richard Harter) - 2011-06-18 19:58 +0000
    Re: filters Tim Rentsch <txr@alumni.caltech.edu> - 2011-06-19 16:09 -0700
      Re: filters Jorgen Grahn <grahn+nntp@snipabacken.se> - 2011-06-20 07:51 +0000
        Re: filters Tim Rentsch <txr@alumni.caltech.edu> - 2011-06-20 12:33 -0700
  Re: filters Nobody <nobody@nowhere.com> - 2011-06-18 15:25 +0100
  Re: filters gordonb.f4hvo@burditt.org (Gordon Burditt) - 2011-06-18 15:17 -0500

csiph-web