Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #101695 > unrolled thread

strtok() and the mistery of the same address strings with different lengths...

Back to article view | Back to comp.lang.c

Contents

  strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-13 16:16 -0800
    Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-13 16:30 -0800
    Re: strtok() and the mistery of the same address strings with different lengths... Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-14 00:44 +0000
      Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-13 17:43 -0800
      Re: strtok() and the mistery of the same address strings with different lengths... Manfred <noname@invalid.add> - 2017-01-14 16:52 +0100
        Re: strtok() and the mistery of the same address strings with different lengths... Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-14 19:38 +0000
    Re: strtok() and the mistery of the same address strings with different lengths... Keith Thompson <kst-u@mib.org> - 2017-01-13 16:49 -0800
      Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-13 17:39 -0800
        Re: strtok() and the mistery of the same address strings with different lengths... James Kuyper <jameskuyper@verizon.net> - 2017-01-13 22:00 -0500
          Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 03:11 -0800
            Re: strtok() and the mistery of the same address strings with different lengths... James Kuyper <jameskuyper@verizon.net> - 2017-01-14 10:50 -0500
              Re: strtok() and the mistery of the same address strings with different lengths... Keith Thompson <kst-u@mib.org> - 2017-01-14 14:27 -0800
                Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 15:21 -0800
                Re: strtok() and the mistery of the same address strings with different lengths... Les Cargill <lcargill99@comcast.com> - 2017-01-15 11:48 -0600
                  Re: strtok() and the mistery of the same address strings with different lengths... Keith Thompson <kst-u@mib.org> - 2017-01-15 15:24 -0800
                    Re: strtok() and the mistery of the same address strings with different lengths... Les Cargill <lcargill99@comcast.com> - 2017-01-15 22:11 -0600
        Re: strtok() and the mistery of the same address strings with different lengths... Manfred <noname@invalid.add> - 2017-01-14 16:48 +0100
    Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 03:18 -0800
      Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 04:16 -0800
        Re: strtok() and the mistery of the same address strings with different lengths... Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-14 15:14 +0000
          Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 09:58 -0800
            Re: strtok() and the mistery of the same address strings with different lengths... Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-14 19:51 +0000
              Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 15:18 -0800
          Re: strtok() and the mistery of the same address strings with different lengths... Keith Thompson <kst-u@mib.org> - 2017-01-14 14:17 -0800
      Re: strtok() and the mistery of the same address strings with different lengths... Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-14 15:09 +0000
        Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 11:09 -0800
          Re: strtok() and the mistery of the same address strings with different lengths... luser droog <luser.droog@gmail.com> - 2017-01-14 11:41 -0800
          Re: strtok() and the mistery of the same address strings with different lengths... Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-14 20:04 +0000
          Re: strtok() and the mistery of the same address strings with different lengths... James Kuyper <jameskuyper@verizon.net> - 2017-01-14 15:09 -0500
            Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 15:22 -0800
        Re: strtok() and the mistery of the same address strings with different lengths... Keith Thompson <kst-u@mib.org> - 2017-01-14 14:21 -0800

Page 1 of 2  [1] 2  Next page →

#101695 — strtok() and the mistery of the same address strings with different lengths...

Ok, this is driving me crazy since 2 hours now.

Given this program that parses the string stored in "line[]":


/**
 * parse-0.c
 */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>


int main(int argc, char *argv[])
{
    char line[] = "GET /this%20is%20a/page.html?q=cats HTTP/1.1";
    char *abs_path = NULL;
    char *query = NULL;
    //
    printf("line = [%s], address = %p\n", line, line);
    char *cmd = strtok(line, " ");
    printf("cmd = [%s], address = %p\n", cmd, cmd);
    char *PathAndQuery = strtok(NULL, " ");
    printf("PathAndQuery = [%s], address = %p\n", PathAndQuery, PathAndQuery);
    //
    char *Http = strtok(NULL, " ");
    printf("Http = [%s], address = %p\n", Http, Http);
    abs_path = strtok(PathAndQuery, "?");
    printf("abs_path = [%s], address = %p\n", abs_path, abs_path);
    query = strtok(NULL, "?");
    printf("query = [%s], address = %p\n", query, query);
    //
    printf("cmd = [%s], address = %p\n", cmd, cmd);
    
    return 0;
}


...How is possible that the function strtok() could return different strings --each with it's own \0 termination-- in different string variables without dinamic memory?

This is the generated output:

fernando@fer-desktop:~/DESA/C-Lang/Tests$ ./parse-0 
line = [GET /this%20is%20a/page.html?q=cats HTTP/1.1], address = 0x7ffc120c8240
cmd = [GET], address = 0x7ffc120c8240
PathAndQuery = [/this%20is%20a/page.html?q=cats], address = 0x7ffc120c8244
Http = [HTTP/1.1], address = 0x7ffc120c8264
abs_path = [/this%20is%20a/page.html], address = 0x7ffc120c8244
query = [q=cats], address = 0x7ffc120c825d
cmd = [GET], address = 0x7ffc120c8240


I have printed out "cmd" twice to make sure that the value is really there.

Look at the addresses of "PathAndQuery" and "abs_path" (0x7ffc120c8240), they are the same! How is possible that 2 vars point at the same address and have different length at the same time?

I have no explanation for this...  :-/

[toc] | [next] | [standalone]

#101697

El sábado, 14 de enero de 2017, 1:17:11 (UTC+1), Fernando D. Bozzo  escribió:
<snip> 
> Look at the addresses of "PathAndQuery" and "abs_path" (0x7ffc120c8240), they are the same! How is possible that 2 vars point at the same address and have different length at the same time?
> 

I made a mistake, the address I'm talking about is 0x7ffc120c8244, not 40.

[toc] | [prev] | [next] | [standalone]

#101698

"Fernando D. Bozzo" <fdbozzo@gmail.com> writes:
<snip>
> /**
>  * parse-0.c
>  */
> #include <stdio.h>
> #include <stdlib.h>
> #include <string.h>
>
>
> int main(int argc, char *argv[])
> {
>     char line[] = "GET /this%20is%20a/page.html?q=cats HTTP/1.1";
>     char *abs_path = NULL;
>     char *query = NULL;
>     //
>     printf("line = [%s], address = %p\n", line, line);
>     char *cmd = strtok(line, " ");
>     printf("cmd = [%s], address = %p\n", cmd, cmd);
>     char *PathAndQuery = strtok(NULL, " ");
>     printf("PathAndQuery = [%s], address = %p\n", PathAndQuery, PathAndQuery);
>     //
>     char *Http = strtok(NULL, " ");
>     printf("Http = [%s], address = %p\n", Http, Http);
>     abs_path = strtok(PathAndQuery, "?");
>     printf("abs_path = [%s], address = %p\n", abs_path, abs_path);
>     query = strtok(NULL, "?");
>     printf("query = [%s], address = %p\n", query, query);
>     //
>     printf("cmd = [%s], address = %p\n", cmd, cmd);
>     
>     return 0;
> }
>
> ...How is possible that the function strtok() could return different
> strings --each with it's own \0 termination-- in different string
> variables without dinamic memory?

It overwrites the storage -- the first token separator that is found
gets replaced by a null and a pointer to the start is returned.  On the
second call, a pointer to the first non-separator character after the
last one found is returned but not before the next separator is replaced
by a null.

Functions like this are seriously bad news, but practically and
theoretically.  They are hard to understand and hard to reason about
because of the internal state they hold that alters the result they
return.  And there are practical concerns to do with multi-threading and
the possibility of calling strtok for some other string before all the
tokens of the first have been processed.

<snip>
-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#101708

El sábado, 14 de enero de 2017, 1:44:12 (UTC+1), Ben Bacarisse  escribió:
<snip>
> It overwrites the storage -- the first token separator that is found
> gets replaced by a null and a pointer to the start is returned.  On the
> second call, a pointer to the first non-separator character after the
> last one found is returned but not before the next separator is replaced
> by a null.
> 
> Functions like this are seriously bad news, but practically and
> theoretically.  They are hard to understand and hard to reason about
> because of the internal state they hold that alters the result they
> return.  And there are practical concerns to do with multi-threading and
> the possibility of calling strtok for some other string before all the
> tokens of the first have been processed.


Thanks Ben! Got it now! :)

Now I think about it, I remember that in what I've read on this function mention another similar but thread-safe, strtok_r()

[toc] | [prev] | [next] | [standalone]

#101775

On 1/14/2017 1:44 AM, Ben Bacarisse wrote:
> And there are practical concerns to do with multi-threading and
> the possibility of calling strtok for some other string before all the
> tokens of the first have been processed.
In fact, besides multithreading, also reentrant functions can yield 
unexpected results with strtok.

[toc] | [prev] | [next] | [standalone]

#101793

Manfred <noname@invalid.add> writes:

> On 1/14/2017 1:44 AM, Ben Bacarisse wrote:
>> And there are practical concerns to do with multi-threading and
>> the possibility of calling strtok for some other string before all the
>> tokens of the first have been processed.
> In fact, besides multithreading, also reentrant functions can yield
> unexpected results with strtok.

Yes, that's what I was trying to say with the last remark above.  I
wanted to avoid the jargon since it's not always helpful and in this
case is often used to cover both kinds of problem.

-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#101699

"Fernando D. Bozzo" <fdbozzo@gmail.com> writes:
> Ok, this is driving me crazy since 2 hours now.
[...]
> ...How is possible that the function strtok() could return different
> strings --each with it's own \0 termination-- in different string
> variables without dinamic memory?
[...]

strtok() modifies its input string in place.

The first time you call it with a non-NULL second argument, it remembers
the location of the string you're tokenizing.  (This is a real problem
if you want to tokenize two different strings concurrently.)

If you start with a string (showing the terminating '\0' explicitly)

    "abc def ghi\0"

and use strtok() to split it on ' ' delimiters, then after you're done
looping over it, the same memory will contain:

    "abc\0def\0ghi\0"

A pointer to the 'a', to the 'd', or to the 'g' will point to a valid
null-terminated string with a length of 3 characters.  The original
11-character string "abc def ghi" will have been clobbered.

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#101705

El sábado, 14 de enero de 2017, 1:49:26 (UTC+1), Keith Thompson  escribió:
> 
> strtok() modifies its input string in place.
> 
> The first time you call it with a non-NULL second argument, it remembers
> the location of the string you're tokenizing.  (This is a real problem
> if you want to tokenize two different strings concurrently.)
> 
> If you start with a string (showing the terminating '\0' explicitly)
> 
>     "abc def ghi\0"
> 
> and use strtok() to split it on ' ' delimiters, then after you're done
> looping over it, the same memory will contain:
> 
>     "abc\0def\0ghi\0"
> 
> A pointer to the 'a', to the 'd', or to the 'g' will point to a valid
> null-terminated string with a length of 3 characters.  The original
> 11-character string "abc def ghi" will have been clobbered.


Amazing! It's just incredible the creativity used to reutilize the same memory segment, subsegmenting it in chunks to make it look as different segments.

It deserves a WOW from my part, really nice.

I understand the consequences when multi-threading, like you and Ben point out, but for monolithic apps it's a great idea.


Now that I understand how it works, I have another question about it:

Why they save the original string as static (which can cause problems when multi-threading) and not used a third parameter with a pointer to next segment to scan, given by the user? Couldn't this resolve the problem with multi-threading? (just speculation, I have the theory about how MT works and used some COM+/STA components with MS IIS server to do the threading, but no deep knowledge on it)

Thanks! Really clarifying answer.-

[toc] | [prev] | [next] | [standalone]

#101721

On 01/13/2017 08:39 PM, Fernando D. Bozzo wrote:
> El sábado, 14 de enero de 2017, 1:49:26 (UTC+1), Keith Thompson  escribió:
>>
>> strtok() modifies its input string in place.
>>
>> The first time you call it with a non-NULL second argument, it remembers
>> the location of the string you're tokenizing.  (This is a real problem
>> if you want to tokenize two different strings concurrently.)
>>
>> If you start with a string (showing the terminating '\0' explicitly)
>>
>>     "abc def ghi\0"
>>
>> and use strtok() to split it on ' ' delimiters, then after you're done
>> looping over it, the same memory will contain:
>>
>>     "abc\0def\0ghi\0"
>>
>> A pointer to the 'a', to the 'd', or to the 'g' will point to a valid
>> null-terminated string with a length of 3 characters.  The original
>> 11-character string "abc def ghi" will have been clobbered.
> 
> 
> Amazing! It's just incredible the creativity used to reutilize the same memory segment, subsegmenting it in chunks to make it look as different segments.
> 
> It deserves a WOW from my part, really nice.
> 
> I understand the consequences when multi-threading, like you and Ben point out, but for monolithic apps it's a great idea.
> 
> 
> Now that I understand how it works, I have another question about it:
> 
> Why they save the original string as static (which can cause problems when multi-threading) and not used a third parameter with a pointer to next segment to scan, given by the user? Couldn't this resolve the problem with multi-threading? (just speculation, I have the theory about how MT works and used some COM+/STA components with MS IIS server to do the threading, but no deep knowledge on it)

strtok() doesn't save the original string, it saves a pointer to the
start of the next token in the string. If a third argument were added,
it would have to be a pointer to a pointer, because the pointer value
needs to be updated each time strtok() is called. As for why it works
that way, I think it was just a bad decision, possibly influenced in
part by a lack of experience with multi-threading.

[toc] | [prev] | [next] | [standalone]

#101730

El sábado, 14 de enero de 2017, 4:00:52 (UTC+1), James Kuyper  escribió:
> <snip>
> As for why it works that way, I think it was just a bad decision, possibly influenced in part by a lack of experience with multi-threading.


I think that when this function was created, the Multi-threading didn't exist.

[toc] | [prev] | [next] | [standalone]

#101774

On 01/14/2017 06:11 AM, Fernando D. Bozzo wrote:
> El sábado, 14 de enero de 2017, 4:00:52 (UTC+1), James Kuyper  escribió:
>> <snip>
>> As for why it works that way, I think it was just a bad decision, possibly influenced in part by a lack of experience with multi-threading.
> 
> 
> I think that when this function was created, the Multi-threading didn't exist.

Possibly - I don't know how long ago strtok was created. However, it
would have to predate C itself by a couple of years (which is entirely
possible - C was not invented out of whole cloth), in order to pre-date
threading. According to
<https://en.wikipedia.org/wiki/Thread_(computing)> 'Threads made an
early appearance in OS/360 Multiprogramming with a Variable Number of
Tasks (MVT) in 1967, in which context they were called "tasks".'

That's why I was cautious, and merely cited "lack of experience with
multi-threading" as a possible explanation, rather than "non-existence
of multi-threading".

Keep in mind that the same features that make strtok() problematic in
multi-threaded contexts also make it problematic in a far more ordinary
cases, one of which I actually ran into. The calling function was doing
a strtok() loop, and during each pass through the loop, it called a
subroutine. That subroutine was in a library I inherited responsibility
for, which also called strtok() for it's own purposes, rather thoroughly
messing up the calling routine.
I re-wrote the library function to do the same thing that strtok does,
but with explicit code rather than by calling strtok().

[toc] | [prev] | [next] | [standalone]

#101815

James Kuyper <jameskuyper@verizon.net> writes:
[...]
> Keep in mind that the same features that make strtok() problematic in
> multi-threaded contexts also make it problematic in a far more ordinary
> cases, one of which I actually ran into. The calling function was doing
> a strtok() loop, and during each pass through the loop, it called a
> subroutine. That subroutine was in a library I inherited responsibility
> for, which also called strtok() for it's own purposes, rather thoroughly
> messing up the calling routine.
> I re-wrote the library function to do the same thing that strtok does,
> but with explicit code rather than by calling strtok().

Perhaps a more straightforward example: Suppose you want to parse
a string with strtok(), where each substring needs to be parsed
with strtok().  For example, you might have several space-delimited
fields, where each field contains several comma-delimited subfields.

You can't use strtok() while another sequence of strtok() calls
is pending.  It saves a static pointer into the string passed on the
first call -- and that pointer is shared by *all* calls to strtok(),
not just calls in a particular parsed string.

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#101825

El sábado, 14 de enero de 2017, 23:27:10 (UTC+1), Keith Thompson  escribió:
> James Kuyper writes:
> [...]
> > Keep in mind that the same features that make strtok() problematic in
> > multi-threaded contexts also make it problematic in a far more ordinary
> > cases, one of which I actually ran into. The calling function was doing
> > a strtok() loop, and during each pass through the loop, it called a
> > subroutine. That subroutine was in a library I inherited responsibility
> > for, which also called strtok() for it's own purposes, rather thoroughly
> > messing up the calling routine.
> > I re-wrote the library function to do the same thing that strtok does,
> > but with explicit code rather than by calling strtok().
> 
> Perhaps a more straightforward example: Suppose you want to parse
> a string with strtok(), where each substring needs to be parsed
> with strtok().  For example, you might have several space-delimited
> fields, where each field contains several comma-delimited subfields.
> 
> You can't use strtok() while another sequence of strtok() calls
> is pending.  It saves a static pointer into the string passed on the
> first call -- and that pointer is shared by *all* calls to strtok(),
> not just calls in a particular parsed string.
> 


Thanks Keith, crystal example.-

[toc] | [prev] | [next] | [standalone]

#101854

Keith Thompson wrote:
> James Kuyper <jameskuyper@verizon.net> writes:
> [...]
>> Keep in mind that the same features that make strtok() problematic in
>> multi-threaded contexts also make it problematic in a far more ordinary
>> cases, one of which I actually ran into. The calling function was doing
>> a strtok() loop, and during each pass through the loop, it called a
>> subroutine. That subroutine was in a library I inherited responsibility
>> for, which also called strtok() for it's own purposes, rather thoroughly
>> messing up the calling routine.
>> I re-wrote the library function to do the same thing that strtok does,
>> but with explicit code rather than by calling strtok().
>
> Perhaps a more straightforward example: Suppose you want to parse
> a string with strtok(), where each substring needs to be parsed
> with strtok().  For example, you might have several space-delimited
> fields, where each field contains several comma-delimited subfields.
>
> You can't use strtok() while another sequence of strtok() calls
> is pending.  It saves a static pointer into the string passed on the
> first call -- and that pointer is shared by *all* calls to strtok(),
> not just calls in a particular parsed string.
>

You can use strtok(). You just can't use it with a NULL first
parameter. Whether you should use it at all or not is up to you.

( in the example, we don't care what happens to "buffer". In real
life, you'd probably need to make a copy. )


#include <stdio.h>
#include <string.h>

int main(void)
{
	static char buffer[] = "this,is,separated,by,,,commas,";

	char *p = strtok(buffer,",");

	while (p)
	{
		const int pos = p-buffer;
		printf("p=:%s:%d:%d:\n",p,pos,strlen(p));
		char *q = p + (strlen(p)+1);
         	p  = strtok(q,",");
	}
	return 0;
}

Output:

p=:this:0:4:
p=:is:5:2:
p=:separated:8:9:
p=:by:18:2:
p=:commas:23:6:

-- 
Les Cargill

[toc] | [prev] | [next] | [standalone]

#101894

Les Cargill <lcargill99@comcast.com> writes:
> Keith Thompson wrote:
>> James Kuyper <jameskuyper@verizon.net> writes:
>> [...]
>>> Keep in mind that the same features that make strtok() problematic in
>>> multi-threaded contexts also make it problematic in a far more ordinary
>>> cases, one of which I actually ran into. The calling function was doing
>>> a strtok() loop, and during each pass through the loop, it called a
>>> subroutine. That subroutine was in a library I inherited responsibility
>>> for, which also called strtok() for it's own purposes, rather thoroughly
>>> messing up the calling routine.
>>> I re-wrote the library function to do the same thing that strtok does,
>>> but with explicit code rather than by calling strtok().
>>
>> Perhaps a more straightforward example: Suppose you want to parse
>> a string with strtok(), where each substring needs to be parsed
>> with strtok().  For example, you might have several space-delimited
>> fields, where each field contains several comma-delimited subfields.
>>
>> You can't use strtok() while another sequence of strtok() calls
>> is pending.  It saves a static pointer into the string passed on the
>> first call -- and that pointer is shared by *all* calls to strtok(),
>> not just calls in a particular parsed string.
>
> You can use strtok(). You just can't use it with a NULL first
> parameter. Whether you should use it at all or not is up to you.

OK, more precisely, if you call strtok() while another sequence of
strtok() calls is pending, you can't resume the original sequence.

The usual way to use strtok() is to call it once with the string to be
parsed, then to call it repeatedly with a null pointer as the first
argument until it runs out of tokens.  Calling it with a non-null
pointer each time, advancing that pointer through the string being
parsed, is ... interesting.  (I hardly ever use strtok(), and I don't
know how commmon that usage pattern is.)

> ( in the example, we don't care what happens to "buffer". In real
> life, you'd probably need to make a copy. )
>
>
> #include <stdio.h>
> #include <string.h>
>
> int main(void)
> {
> 	static char buffer[] = "this,is,separated,by,,,commas,";
>
> 	char *p = strtok(buffer,",");
>
> 	while (p)
> 	{
> 		const int pos = p-buffer;
> 		printf("p=:%s:%d:%d:\n",p,pos,strlen(p));
> 		char *q = p + (strlen(p)+1);
>          	p  = strtok(q,",");
> 	}
> 	return 0;
> }

Note that you have to rescan each token to determine its length, which
is a bit inefficient.  It does clobber the static data that strtok()
uses internally, but since that static data is never used unless you
call it with a null pointer, that doesn't matter.

> Output:
>
> p=:this:0:4:
> p=:is:5:2:
> p=:separated:8:9:
> p=:by:18:2:
> p=:commas:23:6:

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#101909

Keith Thompson wrote:
> Les Cargill <lcargill99@comcast.com> writes:
>> Keith Thompson wrote:
>>> James Kuyper <jameskuyper@verizon.net> writes:
>>> [...]
>>>> Keep in mind that the same features that make strtok() problematic in
>>>> multi-threaded contexts also make it problematic in a far more ordinary
>>>> cases, one of which I actually ran into. The calling function was doing
>>>> a strtok() loop, and during each pass through the loop, it called a
>>>> subroutine. That subroutine was in a library I inherited responsibility
>>>> for, which also called strtok() for it's own purposes, rather thoroughly
>>>> messing up the calling routine.
>>>> I re-wrote the library function to do the same thing that strtok does,
>>>> but with explicit code rather than by calling strtok().
>>>
>>> Perhaps a more straightforward example: Suppose you want to parse
>>> a string with strtok(), where each substring needs to be parsed
>>> with strtok().  For example, you might have several space-delimited
>>> fields, where each field contains several comma-delimited subfields.
>>>
>>> You can't use strtok() while another sequence of strtok() calls
>>> is pending.  It saves a static pointer into the string passed on the
>>> first call -- and that pointer is shared by *all* calls to strtok(),
>>> not just calls in a particular parsed string.
>>
>> You can use strtok(). You just can't use it with a NULL first
>> parameter. Whether you should use it at all or not is up to you.
>
> OK, more precisely, if you call strtok() while another sequence of
> strtok() calls is pending, you can't resume the original sequence.
>
> The usual way to use strtok() is to call it once with the string to be
> parsed, then to call it repeatedly with a null pointer as the first
> argument until it runs out of tokens.  Calling it with a non-null
> pointer each time, advancing that pointer through the string being
> parsed, is ... interesting.  (I hardly ever use strtok(), and I don't
> know how commmon that usage pattern is.)
>


I don't know either  it's probably a good thing to not use, but I just
felt like the counterexample was warranted. The fact that it replaces
characters in he buffer with NUL character in the buffer is kind of
creepy.

>> ( in the example, we don't care what happens to "buffer". In real
>> life, you'd probably need to make a copy. )
>>
>>
>> #include <stdio.h>
>> #include <string.h>
>>
>> int main(void)
>> {
>> 	static char buffer[] = "this,is,separated,by,,,commas,";
>>
>> 	char *p = strtok(buffer,",");
>>
>> 	while (p)
>> 	{
>> 		const int pos = p-buffer;
>> 		printf("p=:%s:%d:%d:\n",p,pos,strlen(p));
>> 		char *q = p + (strlen(p)+1);
>>          	p  = strtok(q,",");
>> 	}
>> 	return 0;
>> }
>
> Note that you have to rescan each token to determine its length, which
> is a bit inefficient.

I don't think it's possible to improve on that, though.

Generally, when parsing, I lean towards state machines.

> It does clobber the static data that strtok()
> uses internally,

... which is pretty ugly, and may be bad in multithreaded
situations...

>  but since that static data is never used unless you
> call it with a null pointer, that doesn't matter.
>
>> Output:
>>
>> p=:this:0:4:
>> p=:is:5:2:
>> p=:separated:8:9:
>> p=:by:18:2:
>> p=:commas:23:6:
>

-- 
Les Cargill

[toc] | [prev] | [next] | [standalone]

#101773

On 1/14/2017 2:39 AM, Fernando D. Bozzo wrote:
> Why they save the original string as static
(which can cause problems when multi-threading)
and not used a third parameter with a pointer to next
segment to scan, given by the user? Couldn't this
resolve the problem with multi-threading?
(just speculation, I have the theory about how MT
works and used some COM+/STA components with MS IIS
server to do the threading, but no deep knowledge on it)
This is what the third parameter of strtok_r is for.

[toc] | [prev] | [next] | [standalone]

#101731

Let's say I create a malloc'ed copy of the original string with strdup(), then use strtok() for parsing it:

1) Is this thread safe, given that I give it a new memory segment?

2) Once I finish using it and the duplicated string is sub-segmented, which end pointing to the first segment: does free() know how to free the entire memory segment with all their sub-segments? I don't want to create a memory leak.

Thanks!

[toc] | [prev] | [next] | [standalone]

#101738

El sábado, 14 de enero de 2017, 12:19:04 (UTC+1), Fernando D. Bozzo  escribió:
> Let's say I create a malloc'ed copy of the original string with strdup(), then use strtok() for parsing it:
> 
> 1) Is this thread safe, given that I give it a new memory segment?
> 
> 2) Once I finish using it and the duplicated string is sub-segmented, which end pointing to the first segment: does free() know how to free the entire memory segment with all their sub-segments? I don't want to create a memory leak.
> 
> Thanks!


I answer myself:

2) NO, doing this totally corrupts the memory segment :(

This is what happen when I try it:

fernando@fer-desktop:~/DESA/C-Lang/Tests$ valgrind ./parse-0 
==25869== Memcheck, a memory error detector
==25869== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==25869== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==25869== Command: ./parse-0
==25869== 
==25869== Invalid write of size 1
==25869==    at 0x4008F1: strduplicate (in /home/fernando/DESA/C-Lang/Tests/parse-0)
==25869==    by 0x4006F6: main (in /home/fernando/DESA/C-Lang/Tests/parse-0)
==25869==  Address 0x5203098 is 24 bytes before an unallocated block of size 4,194,096 in arena "client"
==25869== 

valgrind: m_mallocfree.c:303 (get_bszB_as_is): Assertion 'bszB_lo == bszB_hi' failed.
valgrind: Heap block lo/hi size mismatch: lo = 4194272, hi = 0.
This is probably caused by your program erroneously writing past the
end of a heap block and corrupting heap metadata.  If you fix any
invalid writes reported by Memcheck, this assertion failure will
probably go away.  Please try that before reporting this as a bug.


host stacktrace:
==25869==    at 0x38083F48: show_sched_status_wrk (m_libcassert.c:343)
==25869==    by 0x38084064: report_and_quit (m_libcassert.c:415)
==25869==    by 0x380841F1: vgPlain_assert_fail (m_libcassert.c:481)
==25869==    by 0x380903A1: get_bszB_as_is (m_mallocfree.c:301)
==25869==    by 0x380903A1: get_bszB (m_mallocfree.c:311)
==25869==    by 0x380903A1: get_next_b (m_mallocfree.c:429)
==25869==    by 0x380903A1: unlinkBlock (m_mallocfree.c:1629)
==25869==    by 0x380920F5: vgPlain_arena_malloc (m_mallocfree.c:1810)
==25869==    by 0x38050E84: vgMemCheck_new_block (mc_malloc_wrappers.c:350)
==25869==    by 0x38051056: vgMemCheck_malloc (mc_malloc_wrappers.c:385)
==25869==    by 0x380D4F7B: do_client_request (scheduler.c:1865)
==25869==    by 0x380D4F7B: vgPlain_scheduler (scheduler.c:1425)
==25869==    by 0x380E3946: thread_wrapper (syswrap-linux.c:102)
==25869==    by 0x380E3946: run_a_thread_NORETURN (syswrap-linux.c:155)

sched status:
  running_tid=1

Thread 1: status = VgTs_Runnable (lwpid 25869)
==25869==    at 0x4C2DB8F: malloc (vg_replace_malloc.c:299)
==25869==    by 0x4EA71D4: _IO_file_doallocate (filedoalloc.c:127)
==25869==    by 0x4EB5593: _IO_doallocbuf (genops.c:398)
==25869==    by 0x4EB48F7: _IO_file_overflow@@GLIBC_2.2.5 (fileops.c:820)
==25869==    by 0x4EB328C: _IO_file_xsputn@@GLIBC_2.2.5 (fileops.c:1331)
==25869==    by 0x4E87240: vfprintf (vfprintf.c:1320)
==25869==    by 0x4E8F898: printf (printf.c:33)
==25869==    by 0x40073A: main (in /home/fernando/DESA/C-Lang/Tests/parse-0)


Note: see also the FAQ in the source distribution.
It contains workarounds to several common problems.
In particular, if Valgrind aborted or crashed after
identifying problems in your program, there's a good chance
that fixing those problems will prevent Valgrind aborting or
crashing, especially if it happened in m_mallocfree.c.

If that doesn't help, please report this bug to: www.valgrind.org

In the bug report, send all the above text, the valgrind
version, and what OS and version you are using.  Thanks.

[toc] | [prev] | [next] | [standalone]

#101768

"Fernando D. Bozzo" <fdbozzo@gmail.com> writes:

> El sábado, 14 de enero de 2017, 12:19:04 (UTC+1), Fernando D. Bozzo  escribió:
>> Let's say I create a malloc'ed copy of the original string with
>> strdup(), then use strtok() for parsing it:
>> 
>> 1) Is this thread safe, given that I give it a new memory segment?
>> 
>> 2) Once I finish using it and the duplicated string is
>> sub-segmented, which end pointing to the first segment: does free()
>> know how to free the entire memory segment with all their
>> sub-segments? I don't want to create a memory leak.
>> 
>> Thanks!
>
>
> I answer myself:
>
> 2) NO, doing this totally corrupts the memory segment :(

Maybe I did not understand what you were asking in 2).  The key thing is
that free must be passed a pointer that was returned from malloc (or
friends).  So this:

  char *const p = malloc(len);
  // lots of calls to strtok and strdup
  free(p);

is always OK because p will not have changed.  But you can't do this:

  char *p = malloc(len);
  while (isspace(*p)) p++;
  // anything else
  free(p);

because p might not now hold the same pointer that came back from
malloc.

<snip>
-- 
Ben.

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.c

csiph-web