Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #101909

Re: strtok() and the mistery of the same address strings with different lengths...

Show all headers | View raw

Keith Thompson wrote:
> Les Cargill <lcargill99@comcast.com> writes:
>> Keith Thompson wrote:
>>> James Kuyper <jameskuyper@verizon.net> writes:
>>> [...]
>>>> Keep in mind that the same features that make strtok() problematic in
>>>> multi-threaded contexts also make it problematic in a far more ordinary
>>>> cases, one of which I actually ran into. The calling function was doing
>>>> a strtok() loop, and during each pass through the loop, it called a
>>>> subroutine. That subroutine was in a library I inherited responsibility
>>>> for, which also called strtok() for it's own purposes, rather thoroughly
>>>> messing up the calling routine.
>>>> I re-wrote the library function to do the same thing that strtok does,
>>>> but with explicit code rather than by calling strtok().
>>>
>>> Perhaps a more straightforward example: Suppose you want to parse
>>> a string with strtok(), where each substring needs to be parsed
>>> with strtok().  For example, you might have several space-delimited
>>> fields, where each field contains several comma-delimited subfields.
>>>
>>> You can't use strtok() while another sequence of strtok() calls
>>> is pending.  It saves a static pointer into the string passed on the
>>> first call -- and that pointer is shared by *all* calls to strtok(),
>>> not just calls in a particular parsed string.
>>
>> You can use strtok(). You just can't use it with a NULL first
>> parameter. Whether you should use it at all or not is up to you.
>
> OK, more precisely, if you call strtok() while another sequence of
> strtok() calls is pending, you can't resume the original sequence.
>
> The usual way to use strtok() is to call it once with the string to be
> parsed, then to call it repeatedly with a null pointer as the first
> argument until it runs out of tokens.  Calling it with a non-null
> pointer each time, advancing that pointer through the string being
> parsed, is ... interesting.  (I hardly ever use strtok(), and I don't
> know how commmon that usage pattern is.)
>


I don't know either  it's probably a good thing to not use, but I just
felt like the counterexample was warranted. The fact that it replaces
characters in he buffer with NUL character in the buffer is kind of
creepy.

>> ( in the example, we don't care what happens to "buffer". In real
>> life, you'd probably need to make a copy. )
>>
>>
>> #include <stdio.h>
>> #include <string.h>
>>
>> int main(void)
>> {
>> 	static char buffer[] = "this,is,separated,by,,,commas,";
>>
>> 	char *p = strtok(buffer,",");
>>
>> 	while (p)
>> 	{
>> 		const int pos = p-buffer;
>> 		printf("p=:%s:%d:%d:\n",p,pos,strlen(p));
>> 		char *q = p + (strlen(p)+1);
>>          	p  = strtok(q,",");
>> 	}
>> 	return 0;
>> }
>
> Note that you have to rescan each token to determine its length, which
> is a bit inefficient.

I don't think it's possible to improve on that, though.

Generally, when parsing, I lean towards state machines.

> It does clobber the static data that strtok()
> uses internally,

... which is pretty ugly, and may be bad in multithreaded
situations...

>  but since that static data is never used unless you
> call it with a null pointer, that doesn't matter.
>
>> Output:
>>
>> p=:this:0:4:
>> p=:is:5:2:
>> p=:separated:8:9:
>> p=:by:18:2:
>> p=:commas:23:6:
>

-- 
Les Cargill

Back to comp.lang.c | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-13 16:16 -0800
  Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-13 16:30 -0800
  Re: strtok() and the mistery of the same address strings with different lengths... Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-14 00:44 +0000
    Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-13 17:43 -0800
    Re: strtok() and the mistery of the same address strings with different lengths... Manfred <noname@invalid.add> - 2017-01-14 16:52 +0100
      Re: strtok() and the mistery of the same address strings with different lengths... Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-14 19:38 +0000
  Re: strtok() and the mistery of the same address strings with different lengths... Keith Thompson <kst-u@mib.org> - 2017-01-13 16:49 -0800
    Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-13 17:39 -0800
      Re: strtok() and the mistery of the same address strings with different lengths... James Kuyper <jameskuyper@verizon.net> - 2017-01-13 22:00 -0500
        Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 03:11 -0800
          Re: strtok() and the mistery of the same address strings with different lengths... James Kuyper <jameskuyper@verizon.net> - 2017-01-14 10:50 -0500
            Re: strtok() and the mistery of the same address strings with different lengths... Keith Thompson <kst-u@mib.org> - 2017-01-14 14:27 -0800
              Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 15:21 -0800
              Re: strtok() and the mistery of the same address strings with different lengths... Les Cargill <lcargill99@comcast.com> - 2017-01-15 11:48 -0600
                Re: strtok() and the mistery of the same address strings with different lengths... Keith Thompson <kst-u@mib.org> - 2017-01-15 15:24 -0800
                Re: strtok() and the mistery of the same address strings with different lengths... Les Cargill <lcargill99@comcast.com> - 2017-01-15 22:11 -0600
      Re: strtok() and the mistery of the same address strings with different lengths... Manfred <noname@invalid.add> - 2017-01-14 16:48 +0100
  Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 03:18 -0800
    Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 04:16 -0800
      Re: strtok() and the mistery of the same address strings with different lengths... Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-14 15:14 +0000
        Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 09:58 -0800
          Re: strtok() and the mistery of the same address strings with different lengths... Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-14 19:51 +0000
            Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 15:18 -0800
        Re: strtok() and the mistery of the same address strings with different lengths... Keith Thompson <kst-u@mib.org> - 2017-01-14 14:17 -0800
    Re: strtok() and the mistery of the same address strings with different lengths... Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-14 15:09 +0000
      Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 11:09 -0800
        Re: strtok() and the mistery of the same address strings with different lengths... luser droog <luser.droog@gmail.com> - 2017-01-14 11:41 -0800
        Re: strtok() and the mistery of the same address strings with different lengths... Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-14 20:04 +0000
        Re: strtok() and the mistery of the same address strings with different lengths... James Kuyper <jameskuyper@verizon.net> - 2017-01-14 15:09 -0500
          Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 15:22 -0800
      Re: strtok() and the mistery of the same address strings with different lengths... Keith Thompson <kst-u@mib.org> - 2017-01-14 14:21 -0800

csiph-web