Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #6375

Re: filters

Show all headers | View raw

On Sat, 18 Jun 2011 10:14:55 -0400, James Kuyper
<jameskuyper@verizon.net> wrote:

>On 06/18/2011 06:15 AM, lee wrote:
>> Hi,
>> 
>> it´s probably a FAQ, though I haven´t found any good info on it: How do
>> you go about writing a filter, i. e. a program that reads data from
>> stdin and processes it?
>> 
>> Since indefinte amounts of data could be read from stdin, they can´t
>> just be put into some (ever increasing) buffer. Using a buffer of
>> limited size can make it difficult to process the data because the
>> buffer could be too small.
>> 
>> What´s the solution for this?
>
>The term "filter" is generally reserved for the kind of application that
>only needs to keep a small portion of the input in memory at any given
>time. A typical Unix filter is "cut". It parses each line of input up
>into fields, which can be either fixed width or delimited by a
>user-specifiable character which defaults to '\t'. It writes out a
>specified subset of the fields, with the delimiter optionally replaced
>with an arbitrary string. I've never attempted implementing it, but it
>seems to me that it should be implementable in a way that never keeps
>more than one character of input in memory at any given time.
>
>However, if for some reason your program does need to store the entire
>input, then you need expandable storage, and the C standard library
>provides some. Start by allocating a buffer with malloc(). Whenever the
>buffer gets full, call realloc() to expand it; I recommend increasing
>the size by a fixed factor; 2 would be a good value. Note, there are a
>couple of tricky points in connection with calling realloc():
>* if realloc() fails, it returns a null pointer, and pointers into the
>old buffer are still valid. Therefore, if you make the mistake of
>storing the value returned by realloc() directly into the same pointer
>you were using to keep track of your buffer, you'll lose your ability to
>access that buffer if realloc() fails.
>* if realloc() succeeds, it may have moved your data to a new location
>in memory, invalidating any pointers you may have been keeping that
>pointed into your old buffer. You can't safely do anything with any of
>the old pointer values, not even comparing them for equality with new
>ones to determine whether or not the buffer was moved. For each such
>pointer, determine its offset from the beginning of the buffer before
>calling realloc(). If realloc() succeeds, calculate the new value for
>the corresponding pointer by adding that offset to the start of the new
>buffer.
>
>If realloc() does fail, you'll have to switch to a different approach.
>One option is to create a temporary file using tmpfile(). Read from
>standard input, then write to the temporary file. Once the entire file
>is read in, you can move around in the temporary file using fseek(),
>something you cannot do with stdin.

Excellent post.  Just as an addendum, if you do need a variable size
buffer, it can be a good idea to use indices into the buffer rather
than pointers.  Thus, if your buffer is actually holding a data
structure, e.g., a tree or a linked list, then the node links hold
array indices rather than pointers.  If you do this, the node links
are not affected by relocation of the buffer, whereas pointers are.

Back to comp.lang.c | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

filters lee <lee@yun.yagibdah.de> - 2011-06-18 12:15 +0200
  Re: filters Ben Bacarisse <ben.usenet@bsb.me.uk> - 2011-06-18 11:37 +0100
    Re: filters cri@tiac.net (Richard Harter) - 2011-06-18 18:42 +0000
      Re: filters Ben Bacarisse <ben.usenet@bsb.me.uk> - 2011-06-18 23:13 +0100
        Re: filters cri@tiac.net (Richard Harter) - 2011-06-19 19:02 +0000
          Re: filters Ben Bacarisse <ben.usenet@bsb.me.uk> - 2011-06-19 21:16 +0100
            Re: filters lee <lee@yun.yagibdah.de> - 2011-06-20 16:06 +0200
              Re: filters Ben Bacarisse <ben.usenet@bsb.me.uk> - 2011-06-20 15:53 +0100
                Re: filters lee <lee@yun.yagibdah.de> - 2011-06-27 22:20 +0200
  Re: filters Francois Grieu <fgrieu@gmail.com> - 2011-06-18 13:56 +0200
    Re: filters "Bill Cunningham" <nospam@nspam.invalid> - 2011-06-18 14:24 -0400
      Re: filters Angel <angel+news@spamcop.net> - 2011-06-18 18:51 +0000
        Re: filters Keith Thompson <kst-u@mib.org> - 2011-06-18 11:55 -0700
          Re: filters Angel <angel+news@spamcop.net> - 2011-06-18 19:46 +0000
            Re: filters Keith Thompson <kst-u@mib.org> - 2011-06-18 13:18 -0700
      Re: filters Lew Pitcher <lpitcher@teksavvy.com> - 2011-06-18 14:55 -0400
  Re: filters James Kuyper <jameskuyper@verizon.net> - 2011-06-18 10:14 -0400
    Re: filters cri@tiac.net (Richard Harter) - 2011-06-18 19:58 +0000
    Re: filters Tim Rentsch <txr@alumni.caltech.edu> - 2011-06-19 16:09 -0700
      Re: filters Jorgen Grahn <grahn+nntp@snipabacken.se> - 2011-06-20 07:51 +0000
        Re: filters Tim Rentsch <txr@alumni.caltech.edu> - 2011-06-20 12:33 -0700
  Re: filters Nobody <nobody@nowhere.com> - 2011-06-18 15:25 +0100
  Re: filters gordonb.f4hvo@burditt.org (Gordon Burditt) - 2011-06-18 15:17 -0500

csiph-web