Path: csiph.com!eternal-september.org!feeder.eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail From: Keith Thompson Newsgroups: comp.lang.c Subject: Re: strtok() and the mistery of the same address strings with different lengths... Date: Sun, 15 Jan 2017 15:24:19 -0800 Organization: None to speak of Lines: 78 Message-ID: References: <94292e94-bcb9-40ea-8b8a-2a8563eeb33e@googlegroups.com> <77fd1c6c-16e6-418f-859e-a56547a59389@googlegroups.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Injection-Info: mx02.eternal-september.org; posting-host="0a2d73b4260c7d84c3fe58a86676df00"; logging-data="11598"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1/s44lygMYG0hiPewizqw1t" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.1 (gnu/linux) Cancel-Lock: sha1:oixTORexrF8SepLuxWsTm2f8R2E= sha1:mxpV328k8BW3HpSl6rtdUVBBqtQ= Xref: csiph.com comp.lang.c:101894 Les Cargill writes: > Keith Thompson wrote: >> James Kuyper writes: >> [...] >>> Keep in mind that the same features that make strtok() problematic in >>> multi-threaded contexts also make it problematic in a far more ordinary >>> cases, one of which I actually ran into. The calling function was doing >>> a strtok() loop, and during each pass through the loop, it called a >>> subroutine. That subroutine was in a library I inherited responsibility >>> for, which also called strtok() for it's own purposes, rather thoroughly >>> messing up the calling routine. >>> I re-wrote the library function to do the same thing that strtok does, >>> but with explicit code rather than by calling strtok(). >> >> Perhaps a more straightforward example: Suppose you want to parse >> a string with strtok(), where each substring needs to be parsed >> with strtok(). For example, you might have several space-delimited >> fields, where each field contains several comma-delimited subfields. >> >> You can't use strtok() while another sequence of strtok() calls >> is pending. It saves a static pointer into the string passed on the >> first call -- and that pointer is shared by *all* calls to strtok(), >> not just calls in a particular parsed string. > > You can use strtok(). You just can't use it with a NULL first > parameter. Whether you should use it at all or not is up to you. OK, more precisely, if you call strtok() while another sequence of strtok() calls is pending, you can't resume the original sequence. The usual way to use strtok() is to call it once with the string to be parsed, then to call it repeatedly with a null pointer as the first argument until it runs out of tokens. Calling it with a non-null pointer each time, advancing that pointer through the string being parsed, is ... interesting. (I hardly ever use strtok(), and I don't know how commmon that usage pattern is.) > ( in the example, we don't care what happens to "buffer". In real > life, you'd probably need to make a copy. ) > > > #include > #include > > int main(void) > { > static char buffer[] = "this,is,separated,by,,,commas,"; > > char *p = strtok(buffer,","); > > while (p) > { > const int pos = p-buffer; > printf("p=:%s:%d:%d:\n",p,pos,strlen(p)); > char *q = p + (strlen(p)+1); > p = strtok(q,","); > } > return 0; > } Note that you have to rescan each token to determine its length, which is a bit inefficient. It does clobber the static data that strtok() uses internally, but since that static data is never used unless you call it with a null pointer, that doesn't matter. > Output: > > p=:this:0:4: > p=:is:5:2: > p=:separated:8:9: > p=:by:18:2: > p=:commas:23:6: -- Keith Thompson (The_Other_Keith) kst-u@mib.org Working, but not speaking, for JetHead Development, Inc. "We must do something. This is something. Therefore, we must do this." -- Antony Jay and Jonathan Lynn, "Yes Minister"