Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #101695 > unrolled thread

strtok() and the mistery of the same address strings with different lengths...

Back to article view | Back to comp.lang.c

Contents

  strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-13 16:16 -0800
    Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-13 16:30 -0800
    Re: strtok() and the mistery of the same address strings with different lengths... Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-14 00:44 +0000
      Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-13 17:43 -0800
      Re: strtok() and the mistery of the same address strings with different lengths... Manfred <noname@invalid.add> - 2017-01-14 16:52 +0100
        Re: strtok() and the mistery of the same address strings with different lengths... Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-14 19:38 +0000
    Re: strtok() and the mistery of the same address strings with different lengths... Keith Thompson <kst-u@mib.org> - 2017-01-13 16:49 -0800
      Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-13 17:39 -0800
        Re: strtok() and the mistery of the same address strings with different lengths... James Kuyper <jameskuyper@verizon.net> - 2017-01-13 22:00 -0500
          Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 03:11 -0800
            Re: strtok() and the mistery of the same address strings with different lengths... James Kuyper <jameskuyper@verizon.net> - 2017-01-14 10:50 -0500
              Re: strtok() and the mistery of the same address strings with different lengths... Keith Thompson <kst-u@mib.org> - 2017-01-14 14:27 -0800
                Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 15:21 -0800
                Re: strtok() and the mistery of the same address strings with different lengths... Les Cargill <lcargill99@comcast.com> - 2017-01-15 11:48 -0600
                  Re: strtok() and the mistery of the same address strings with different lengths... Keith Thompson <kst-u@mib.org> - 2017-01-15 15:24 -0800
                    Re: strtok() and the mistery of the same address strings with different lengths... Les Cargill <lcargill99@comcast.com> - 2017-01-15 22:11 -0600
        Re: strtok() and the mistery of the same address strings with different lengths... Manfred <noname@invalid.add> - 2017-01-14 16:48 +0100
    Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 03:18 -0800
      Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 04:16 -0800
        Re: strtok() and the mistery of the same address strings with different lengths... Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-14 15:14 +0000
          Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 09:58 -0800
            Re: strtok() and the mistery of the same address strings with different lengths... Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-14 19:51 +0000
              Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 15:18 -0800
          Re: strtok() and the mistery of the same address strings with different lengths... Keith Thompson <kst-u@mib.org> - 2017-01-14 14:17 -0800
      Re: strtok() and the mistery of the same address strings with different lengths... Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-14 15:09 +0000
        Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 11:09 -0800
          Re: strtok() and the mistery of the same address strings with different lengths... luser droog <luser.droog@gmail.com> - 2017-01-14 11:41 -0800
          Re: strtok() and the mistery of the same address strings with different lengths... Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-01-14 20:04 +0000
          Re: strtok() and the mistery of the same address strings with different lengths... James Kuyper <jameskuyper@verizon.net> - 2017-01-14 15:09 -0500
            Re: strtok() and the mistery of the same address strings with different lengths... "Fernando D. Bozzo" <fdbozzo@gmail.com> - 2017-01-14 15:22 -0800
        Re: strtok() and the mistery of the same address strings with different lengths... Keith Thompson <kst-u@mib.org> - 2017-01-14 14:21 -0800

Page 2 of 2 — ← Prev page 1 [2]

#101784

El sábado, 14 de enero de 2017, 16:14:38 (UTC+1), Ben Bacarisse  escribió:
> "Fernando D. Bozzo" writes:
> 
> > El sábado, 14 de enero de 2017, 12:19:04 (UTC+1), Fernando D. Bozzo  escribió:
> >> Let's say I create a malloc'ed copy of the original string with
> >> strdup(), then use strtok() for parsing it:
> >> 
> >> 1) Is this thread safe, given that I give it a new memory segment?
> >> 
> >> 2) Once I finish using it and the duplicated string is
> >> sub-segmented, which end pointing to the first segment: does free()
> >> know how to free the entire memory segment with all their
> >> sub-segments? I don't want to create a memory leak.
> >> 
> >> Thanks!
> >
> >
> > I answer myself:
> >
> > 2) NO, doing this totally corrupts the memory segment :(
> 
> Maybe I did not understand what you were asking in 2).  The key thing is
> that free must be passed a pointer that was returned from malloc (or
> friends).  So this:
> 
>   char *const p = malloc(len);
>   // lots of calls to strtok and strdup
>   free(p);
> 
> is always OK because p will not have changed.  But you can't do this:
> 
>   char *p = malloc(len);
>   while (isspace(*p)) p++;
>   // anything else
>   free(p);
> 
> because p might not now hold the same pointer that came back from
> malloc.
> 


My implementation corrupts the malloc'ed segment, this it what I've did:

/**
 * parse-0.c
 */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

// Prototypes
char *strduplicate(const char *string);

int main(int argc, char *argv[])
{
    char line_orig[] = "GET /this%20is%20a/page.html?q=cats HTTP/1.1";
    char *line = strduplicate(line_orig);
    if (line == NULL)
    {
        return 1;
    }
    
    char *abs_path = NULL;
    char *query = NULL;
    //
    printf("line = [%s], address = %p\n", line_orig, line_orig);
    printf("line = [%s], address = %p\n", line, line);
    char *cmd = strtok(line, " ");
    printf("cmd = [%s], address = %p\n", cmd, cmd);
    char *PathAndQuery = strtok(NULL, " ");
    printf("PathAndQuery = [%s], address = %p\n", PathAndQuery, PathAndQuery);
    //
    char *Http = strtok(NULL, " ");
    printf("Http = [%s], address = %p\n", Http, Http);
    abs_path = strtok(PathAndQuery, "?");
    printf("abs_path = [%s], address = %p\n", abs_path, abs_path);
    query = strtok(NULL, "?");
    printf("query = [%s], address = %p\n", query, query);
    //
    free(line);
    
    return 0;
}


/**
 * 
 */
char *strduplicate(const char *string)
{
    int len = strlen(string);
    printf("len = %d\n", len);
    char *newstr = malloc((len + 1) * sizeof(char));
    printf("newstr len = %lu, addr = %p\n", sizeof(newstr), newstr);
    char *buff = &newstr[0];
    printf("buff len = %lu, addr = %p\n", sizeof(buff), buff);
    while (*string != '\0')
    {
        printf("string = %c, addr = %p   ", *string, string);
        *buff++ = *string++;
        printf("buff = %c, addr = %p\n", *buff, buff);
    }
    printf("string = %c, addr = %p   ", *string, string);
    buff[len] = '\0';
    printf("buff = %c, addr = %p\n", *buff, buff);
    return newstr;
}


I use "buff" (copy of newstr) for adding the characters to the string, but I return newstr.

I can't free it, It's corrupted

[toc] | [prev] | [next] | [standalone]

#101796

"Fernando D. Bozzo" <fdbozzo@gmail.com> writes:
<snip>
> My implementation corrupts the malloc'ed segment, this it what I've
> did:

Yes, it does.

<snip>
> char *strduplicate(const char *string)
> {
>     int len = strlen(string);

I'd use size_t len here.

>     printf("len = %d\n", len);

and then I'd need %zu as the format.

>     char *newstr = malloc((len + 1) * sizeof(char));

sizeof (char) is guaranteed to be 1.  It looks a little odd to put it
there but it's not an error.

>     printf("newstr len = %lu, addr = %p\n", sizeof(newstr), newstr);

sizeof newstr is not the length of newstr so the message you print is
potentially confusing.

>     char *buff = &newstr[0];

Or just char *buff = newstr;

>     printf("buff len = %lu, addr = %p\n", sizeof(buff), buff);

See above.

>     while (*string != '\0')
>     {
>         printf("string = %c, addr = %p   ", *string, string);
>         *buff++ = *string++;
>         printf("buff = %c, addr = %p\n", *buff, buff);

Here you print the value in a location you have not set.  It won't tell
you anything interesting (and valgrind will complain about it).

>     }
>     printf("string = %c, addr = %p   ", *string, string);
>     buff[len] = '\0';
>     printf("buff = %c, addr = %p\n", *buff, buff);
>     return newstr;
> }

Here's the body simplified.  Can you see the problem now?

     size_t len = strlen(string);
     char *newstr = malloc(len + 1);
     char *buff = &newstr[0];
     while (*string != '\0')
         *buff++ = *string++;
     buff[len] = '\0';
     return newstr;

Hint...
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
You've moved buff as you copy, so buf[len] is way too far away for the
null byte to go!

<snip>
-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#101823

El sábado, 14 de enero de 2017, 20:51:49 (UTC+1), Ben Bacarisse  escribió:
> "Fernando D. Bozzo" writes:
> <snip>
> > My implementation corrupts the malloc'ed segment, this it what I've
> > did:
> 
> Yes, it does.
> 
> <snip>
> > char *strduplicate(const char *string)
> > {
> >     int len = strlen(string);
> 
> I'd use size_t len here.
> 
> >     printf("len = %d\n", len);
> 
> and then I'd need %zu as the format.
> 
> >     char *newstr = malloc((len + 1) * sizeof(char));
> 
> sizeof (char) is guaranteed to be 1.  It looks a little odd to put it
> there but it's not an error.
> 
> >     printf("newstr len = %lu, addr = %p\n", sizeof(newstr), newstr);
> 
> sizeof newstr is not the length of newstr so the message you print is
> potentially confusing.
> 
> >     char *buff = &newstr[0];
> 
> Or just char *buff = newstr;
> 
> >     printf("buff len = %lu, addr = %p\n", sizeof(buff), buff);
> 
> See above.
> 
> >     while (*string != '\0')
> >     {
> >         printf("string = %c, addr = %p   ", *string, string);
> >         *buff++ = *string++;
> >         printf("buff = %c, addr = %p\n", *buff, buff);
> 
> Here you print the value in a location you have not set.  It won't tell
> you anything interesting (and valgrind will complain about it).
> 
> >     }
> >     printf("string = %c, addr = %p   ", *string, string);
> >     buff[len] = '\0';
> >     printf("buff = %c, addr = %p\n", *buff, buff);
> >     return newstr;
> > }
> 
> Here's the body simplified.  Can you see the problem now?
> 
>      size_t len = strlen(string);
>      char *newstr = malloc(len + 1);
>      char *buff = &newstr[0];
>      while (*string != '\0')
>          *buff++ = *string++;
>      buff[len] = '\0';
>      return newstr;
> 
> Hint...
> .
> .
> .
> .
> .
> .
> .
> .
> .
> .
> .
> .
> .
> .
> .
> .
> .
> .
> .
> .
> .
> .
> .
> .
> You've moved buff as you copy, so buf[len] is way too far away for the
> null byte to go!
> 
> <snip>
> -- 
> Ben.


Excellent analysis Ben! I understand now my errors: taking out the printf() ones, my last line should be:

buff = '\0';

because the buff pointer was incremented on the last assign in the loop.

I've repeated the test with the same errors just to understand some valgrind messages, like "Conditional jump or move depends on uninitialised value(s)"

        *buff++ = *string++;
        printf("buff = %c, addr = %p\n", *buff, buff); ==> THIS ONE, as you told


Great! It works now, very thanks for all the details!

[toc] | [prev] | [next] | [standalone]

#101812

Ben Bacarisse <ben.usenet@bsb.me.uk> writes:
[...]
> Maybe I did not understand what you were asking in 2).  The key thing is
> that free must be passed a pointer that was returned from malloc (or
> friends).  So this:
>
>   char *const p = malloc(len);
>   // lots of calls to strtok and strdup
>   free(p);
>
> is always OK because p will not have changed.

And also because memory outside the allocated space will not
have changed.  The free(p) call can fail (with undefined behavior)
either if the value of p has changed, or if something has tried to
modify memory before the address p points to, or at or after p+len.
That has undefined behavior anyway, but in this case it can clobber
the metadata used by the memory allocation system.

But changing the data in the allocated chunk will have no effect
on the free(p) call.  The malloc/free system doesn't care what you
do with the contents of the allocated memory.

[...]

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#101767

"Fernando D. Bozzo" <fdbozzo@gmail.com> writes:

> Let's say I create a malloc'ed copy of the original string with
> strdup(), then use strtok() for parsing it:

strdup is not a standard C function.  I mention this only because if you
are using non-standard functions you can use strtok_r too.

> 1) Is this thread safe, given that I give it a new memory segment?

Just don't use it.  Use the _r version instead.  If you are not using
threads, the don't worry about it.

> 2) Once I finish using it and the duplicated string is sub-segmented,
> which end pointing to the first segment: does free() know how to free
> the entire memory segment with all their sub-segments? I don't want to
> create a memory leak.

That's fine.  Free will free the space no matter how many null bytes
there are in it.

I don't think I've ever wanted what strtok does enough to be bothered
with its peculiarities.  What it is doing is relatively simple and I've
always found a better way to get the result I need.

-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#101791

El sábado, 14 de enero de 2017, 16:09:23 (UTC+1), Ben Bacarisse  escribió:
> "Fernando D. Bozzo" writes:
> 
> > Let's say I create a malloc'ed copy of the original string with
> > strdup(), then use strtok() for parsing it:
> 
> strdup is not a standard C function.  I mention this only because if you
> are using non-standard functions you can use strtok_r too.
> 
> > 1) Is this thread safe, given that I give it a new memory segment?
> 
> Just don't use it.  Use the _r version instead.  If you are not using
> threads, the don't worry about it.
> 
> > 2) Once I finish using it and the duplicated string is sub-segmented,
> > which end pointing to the first segment: does free() know how to free
> > the entire memory segment with all their sub-segments? I don't want to
> > create a memory leak.
> 
> That's fine.  Free will free the space no matter how many null bytes
> there are in it.
> 
> I don't think I've ever wanted what strtok does enough to be bothered
> with its peculiarities.  What it is doing is relatively simple and I've
> always found a better way to get the result I need.
> 
> -- 
> Ben.

I've discovered by the hard way that you can't pass a malloc'ed string to strok(), because then you can't free that var.

What a deception  :(

[toc] | [prev] | [next] | [standalone]

#101794

On Saturday, January 14, 2017 at 1:09:35 PM UTC-6, Fernando D. Bozzo wrote:
> El sábado, 14 de enero de 2017, 16:09:23 (UTC+1), Ben Bacarisse  escribió:
> > "Fernando D. Bozzo" writes:
> > 
> > > Let's say I create a malloc'ed copy of the original string with
> > > strdup(), then use strtok() for parsing it:
> > 
> > strdup is not a standard C function.  I mention this only because if you
> > are using non-standard functions you can use strtok_r too.
> > 
> > > 1) Is this thread safe, given that I give it a new memory segment?
> > 
> > Just don't use it.  Use the _r version instead.  If you are not using
> > threads, the don't worry about it.
> > 
> > > 2) Once I finish using it and the duplicated string is sub-segmented,
> > > which end pointing to the first segment: does free() know how to free
> > > the entire memory segment with all their sub-segments? I don't want to
> > > create a memory leak.
> > 
> > That's fine.  Free will free the space no matter how many null bytes
> > there are in it.
> > 
> > I don't think I've ever wanted what strtok does enough to be bothered
> > with its peculiarities.  What it is doing is relatively simple and I've
> > always found a better way to get the result I need.
> > 
> > -- 
> > Ben.
> 
> I've discovered by the hard way that you can't pass a malloc'ed string to strok(), because then you can't free that var.
> 
> What a deception  :(

Sure you can. You just can't free any of the internal
substrings that it parcels off (except the first (but
you still have the original, so just use that)).

[toc] | [prev] | [next] | [standalone]

#101797

"Fernando D. Bozzo" <fdbozzo@gmail.com> writes:
<snip>
> I've discovered by the hard way that you can't pass a malloc'ed string
> to strok(), because then you can't free that var.

No, whatever you have discovered, it is not that.  malloc and strtok are
fine together.

-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#101798

On 01/14/2017 02:09 PM, Fernando D. Bozzo wrote:
...
> I've discovered by the hard way that you can't pass a malloc'ed string to strok(), because then you can't free that var.

No, you haven't. As Ben has pointed out, what you've discovered is that
writing past the end of a dynamically allocated block of memory had
undefined behavior. Among the likely consequences of such code is
corruption of the heap, and one possible symptom of such corruption is
the failure of a subsequent call to free().

[toc] | [prev] | [next] | [standalone]

#101826

El sábado, 14 de enero de 2017, 21:09:35 (UTC+1), James Kuyper  escribió:
> On 01/14/2017 02:09 PM, Fernando D. Bozzo wrote:
> ...
> > I've discovered by the hard way that you can't pass a malloc'ed string to strok(), because then you can't free that var.
> 
> No, you haven't. As Ben has pointed out, what you've discovered is that
> writing past the end of a dynamically allocated block of memory had
> undefined behavior. Among the likely consequences of such code is
> corruption of the heap, and one possible symptom of such corruption is
> the failure of a subsequent call to free().


You are right, I see it now, Thanks!

[toc] | [prev] | [next] | [standalone]

#101814

Ben Bacarisse <ben.usenet@bsb.me.uk> writes:
> "Fernando D. Bozzo" <fdbozzo@gmail.com> writes:
>> Let's say I create a malloc'ed copy of the original string with
>> strdup(), then use strtok() for parsing it:
>
> strdup is not a standard C function.  I mention this only because if you
> are using non-standard functions you can use strtok_r too.
>
>> 1) Is this thread safe, given that I give it a new memory segment?
>
> Just don't use it.  Use the _r version instead.  If you are not using
> threads, the don't worry about it.
[...]

strtok_s is standard, but it's specified in Annex K, which is new in C11
and optional.

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.lang.c

csiph-web