Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #101894

Re: strtok() and the mistery of the same address strings with different lengths...

Show all headers | View raw

Les Cargill <lcargill99@comcast.com> writes:
> Keith Thompson wrote:
>> James Kuyper <jameskuyper@verizon.net> writes:
>> [...]
>>> Keep in mind that the same features that make strtok() problematic in
>>> multi-threaded contexts also make it problematic in a far more ordinary
>>> cases, one of which I actually ran into. The calling function was doing
>>> a strtok() loop, and during each pass through the loop, it called a
>>> subroutine. That subroutine was in a library I inherited responsibility
>>> for, which also called strtok() for it's own purposes, rather thoroughly
>>> messing up the calling routine.
>>> I re-wrote the library function to do the same thing that strtok does,
>>> but with explicit code rather than by calling strtok().
>>
>> Perhaps a more straightforward example: Suppose you want to parse
>> a string with strtok(), where each substring needs to be parsed
>> with strtok().  For example, you might have several space-delimited
>> fields, where each field contains several comma-delimited subfields.
>>
>> You can't use strtok() while another sequence of strtok() calls
>> is pending.  It saves a static pointer into the string passed on the
>> first call -- and that pointer is shared by *all* calls to strtok(),
>> not just calls in a particular parsed string.
>
> You can use strtok(). You just can't use it with a NULL first
> parameter. Whether you should use it at all or not is up to you.

OK, more precisely, if you call strtok() while another sequence of
strtok() calls is pending, you can't resume the original sequence.

The usual way to use strtok() is to call it once with the string to be
parsed, then to call it repeatedly with a null pointer as the first
argument until it runs out of tokens.  Calling it with a non-null
pointer each time, advancing that pointer through the string being
parsed, is ... interesting.  (I hardly ever use strtok(), and I don't
know how commmon that usage pattern is.)

> ( in the example, we don't care what happens to "buffer". In real
> life, you'd probably need to make a copy. )
>
>
> #include <stdio.h>
> #include <string.h>
>
> int main(void)
> {
> 	static char buffer[] = "this,is,separated,by,,,commas,";
>
> 	char *p = strtok(buffer,",");
>
> 	while (p)
> 	{
> 		const int pos = p-buffer;
> 		printf("p=:%s:%d:%d:\n",p,pos,strlen(p));
> 		char *q = p + (strlen(p)+1);
>          	p  = strtok(q,",");
> 	}
> 	return 0;
> }

Note that you have to rescan each token to determine its length, which
is a bit inefficient.  It does clobber the static data that strtok()
uses internally, but since that static data is never used unless you
call it with a null pointer, that doesn't matter.

> Output:
>
> p=:this:0:4:
> p=:is:5:2:
> p=:separated:8:9:
> p=:by:18:2:
> p=:commas:23:6:

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

Back to comp.lang.c | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-13 16:16 -0800
  Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-13 16:30 -0800
  Re: strtok() and the mistery of the same address strings with different lengths... Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-14 00:44 +0000
    Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-13 17:43 -0800
    Re: strtok() and the mistery of the same address strings with different lengths... Manfred <noname@invalid.add> - 2017-01-14 16:52 +0100
      Re: strtok() and the mistery of the same address strings with different lengths... Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-14 19:38 +0000
  Re: strtok() and the mistery of the same address strings with different lengths... Keith Thompson <kst-u@mib.org> - 2017-01-13 16:49 -0800
    Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-13 17:39 -0800
      Re: strtok() and the mistery of the same address strings with different lengths... James Kuyper <jameskuyper@verizon.net> - 2017-01-13 22:00 -0500
        Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 03:11 -0800
          Re: strtok() and the mistery of the same address strings with different lengths... James Kuyper <jameskuyper@verizon.net> - 2017-01-14 10:50 -0500
            Re: strtok() and the mistery of the same address strings with different lengths... Keith Thompson <kst-u@mib.org> - 2017-01-14 14:27 -0800
              Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 15:21 -0800
              Re: strtok() and the mistery of the same address strings with different lengths... Les Cargill <lcargill99@comcast.com> - 2017-01-15 11:48 -0600
                Re: strtok() and the mistery of the same address strings with different lengths... Keith Thompson <kst-u@mib.org> - 2017-01-15 15:24 -0800
                Re: strtok() and the mistery of the same address strings with different lengths... Les Cargill <lcargill99@comcast.com> - 2017-01-15 22:11 -0600
      Re: strtok() and the mistery of the same address strings with different lengths... Manfred <noname@invalid.add> - 2017-01-14 16:48 +0100
  Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 03:18 -0800
    Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 04:16 -0800
      Re: strtok() and the mistery of the same address strings with different lengths... Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-14 15:14 +0000
        Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 09:58 -0800
          Re: strtok() and the mistery of the same address strings with different lengths... Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-14 19:51 +0000
            Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 15:18 -0800
        Re: strtok() and the mistery of the same address strings with different lengths... Keith Thompson <kst-u@mib.org> - 2017-01-14 14:17 -0800
    Re: strtok() and the mistery of the same address strings with different lengths... Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-14 15:09 +0000
      Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 11:09 -0800
        Re: strtok() and the mistery of the same address strings with different lengths... luser droog <luser.droog@gmail.com> - 2017-01-14 11:41 -0800
        Re: strtok() and the mistery of the same address strings with different lengths... Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-14 20:04 +0000
        Re: strtok() and the mistery of the same address strings with different lengths... James Kuyper <jameskuyper@verizon.net> - 2017-01-14 15:09 -0500
          Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 15:22 -0800
      Re: strtok() and the mistery of the same address strings with different lengths... Keith Thompson <kst-u@mib.org> - 2017-01-14 14:21 -0800

csiph-web