Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #59134 > unrolled thread

what the fork is wrong with strtok_r

Back to article view | Back to comp.lang.c

Contents

  what the fork is wrong with strtok_r what <ceo@iran.gov> - 2015-03-28 14:41 +0000
    Re: what the fork is wrong with strtok_r James Kuyper <jameskuyper@verizon.net> - 2015-03-28 10:57 -0400
      Re: what the fork is wrong with strtok_r Keith Thompson <kst-u@mib.org> - 2015-03-28 17:39 -0700
        Re: what the fork is wrong with strtok_r James Kuyper <jameskuyper@verizon.net> - 2015-03-28 21:33 -0400
        Re: what the fork is wrong with strtok_r ruben safir <ruben@mrbrklyn.com> - 2015-03-29 03:36 -0400
        Re: what the fork is wrong with strtok_r ruben safir <ruben@mrbrklyn.com> - 2015-03-29 03:39 -0400
    Re: what the fork is wrong with strtok_r Lőrinczy Zsigmond <nospam@for.me> - 2015-03-29 08:28 +0200

#59134 — what the fork is wrong with strtok_r

The docs are terrible for strtok_r and I can not predict how it will 
behave.  I have a simple text string from stdin to tokenize on " \n" and  
it seems that this is very difficult task for the C programming language. 

[toc] | [next] | [standalone]

#59136

On 03/28/2015 10:41 AM, what wrote:
> The docs are terrible for strtok_r and I can not predict how it will 
> behave.  I have a simple text string from stdin to tokenize on " \n" and  
> it seems that this is very difficult task for the C programming language. 

That's not a useful question. To get a useful answer, you should provide
the following:
1. The source code that you're using to read the string from stdin and
pass it to strtok_r.
2. The complete text that you want to tokenize.
3. An explanation of what you expected to happen.
4. The evidence provided by the output from your program that makes you
think that something else happened.

Note: if you're using gets() to read the text string, '\n' characters
will be replaced with '\0', which would make tokenizing on '\n' a little
difficult. If that fact is completely irrelevant to your problem, that
demonstrates the need for item 1.

For your own personal investigation, I'd recommend breaking this up into
two separate investigations:
A. Reading the string from input. Check to make sure that what you've
read from input is precisely what you expected to read. This includes
terminating null characters and newline characters.
B. Create the string inside your program without reading it from stdin.
Pass it to strtok_r(), and see what the behavior is.
-- 
James Kuyper

[toc] | [prev] | [next] | [standalone]

#59184

James Kuyper <jameskuyper@verizon.net> writes:
[...]
> Note: if you're using gets() to read the text string, '\n' characters
> will be replaced with '\0', which would make tokenizing on '\n' a little
> difficult. If that fact is completely irrelevant to your problem, that
> demonstrates the need for item 1.

If you're using gets(), that's your first problem.  gets() is inherently
unsafe, and has been removed from the language as of the 2011 standard.

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
Working, but not speaking, for JetHead Development, Inc.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#59185

On 03/28/2015 08:39 PM, Keith Thompson wrote:
> James Kuyper <jameskuyper@verizon.net> writes:
> [...]
>> Note: if you're using gets() to read the text string, '\n' characters
>> will be replaced with '\0', which would make tokenizing on '\n' a little
>> difficult. If that fact is completely irrelevant to your problem, that
>> demonstrates the need for item 1.
> 
> If you're using gets(), that's your first problem.  gets() is inherently
> unsafe, and has been removed from the language as of the 2011 standard.

I was going to get into that later, if he confirmed that gets() was
involved. It's just the simplest way I could think of to make the '\n'
characters he was planning to tokenize on be unintentionally absent.
It's not a particularly plausible way - which is why I didn't want to go
into it in any detail unless and until he confirmed that it was relevant.
-- 
James Kuyper

[toc] | [prev] | [next] | [standalone]

#59191

On 03/28/2015 08:39 PM, Keith Thompson wrote:
> James Kuyper <jameskuyper@verizon.net> writes:
> [...]
>> Note: if you're using gets() to read the text string, '\n' characters
>> will be replaced with '\0', which would make tokenizing on '\n' a little
>> difficult. If that fact is completely irrelevant to your problem, that
>> demonstrates the need for item 1.
> 
> If you're using gets(), that's your first problem.  gets() is inherently
> unsafe, and has been removed from the language as of the 2011 standard.
> 

YOu can just right that down as answer 48, which is wrong in this case,
but answer 48 should always be available when needed.

Just to be clear, I was using getline, just to see how hard it was.

BTW - it is also seemly insecure .

Then again Perl has

while(<>){
}


Now this is really begging for a rewrite:  They should try to get
12 year olds to read and understand these things.  strtok should
not be rocket science

       char *strtok_r(char *str, const char *delim, char **saveptr);

   Feature Test Macro Requirements for glibc (see feature_test_macros(7)):

       strtok_r(): _SVID_SOURCE || _BSD_SOURCE || _POSIX_C_SOURCE >= 1 ||
       _XOPEN_SOURCE || _POSIX_SOURCE

DESCRIPTION
       The strtok() function breaks a string into a sequence of zero  or
 more
       nonempty tokens.  On the first call to strtok() the string to be
parsed
       should be specified in str.  In each subsequent call that should
 parse
       the same string, str must be NULL.

       The  delim argument specifies a set of bytes that delimit the
tokens in
       the parsed string.  The caller may specify different strings  in
 delim
       in successive calls that parse the same string.

       Each  call  to  strtok()  returns a pointer to a null-terminated
string
       containing the next token.  This string does not include the
delimiting
       byte.  If no more tokens are found, strtok() returns NULL.
   A  sequence  of calls to strtok() that operate on the same string main‐
       tains a pointer that determines the point from which to start
searching
       for  the  next  token.  The first call to strtok() sets this
pointer to
       point to the first byte of the string.  The start of the next
token  is
       determined  by  scanning forward for the next nondelimiter byte
in str.
       If such a byte is found, it is taken as the start of  the  next
token.
       If  no  such byte is found, then there are no more tokens, and
strtok()
       returns NULL.  (A string that is empty or that contains only
delimiters
       will thus cause strtok() to return NULL on the first call.)

       The  end  of  each  token is found by scanning forward until
either the
       next delimiter byte is found or until the terminating null byte
('\0')
       is encountered.  If a delimiter byte is found, it is overwritten
with a
       null byte to terminate the current token, and strtok() saves a
pointer
       to  the following byte; that pointer will be used as the starting
point
       when searching for the next token.  In this case,  strtok()
returns  a
       pointer to the start of the found token.

       From  the  above description, it follows that a sequence of two
or more
       contiguous delimiter bytes in the parsed string is considered to
 be  a
       single  delimiter,  and that delimiter bytes at the start or end
of the
       string are ignored.  Put another way: the tokens returned  by
strtok()
       are  always  nonempty  strings.   Thus,  for  example, given the
string
       "aaa;;bbb,", successive calls to strtok() that  specify  the
delimiter
       string  ";,"  would return the strings "aaa" and "bbb", and then
a null
       pointer.

       The strtok_r() function is a reentrant version strtok().   The
saveptr
       argument  is  a pointer to a char * variable that is used
internally by
       strtok_r() in order to maintain context between successive  calls
 that
       parse the same string.

       On  the  first call to strtok_r(), str should point to the string
to be
       parsed, and the value of saveptr is ignored.  In subsequent
calls,  str
       should  be  NULL,  and  saveptr  should be unchanged since the
previous
       call.

       Different strings may be parsed concurrently using sequences  of
 calls
       to strtok_r() that specify different saveptr arguments.



NOW check out this insane example:
 Program source

       #include <stdio.h>
       #include <stdlib.h>
       #include <string.h>

       int
       main(int argc, char *argv[])
       {
           char *str1, *str2, *token, *subtoken;
           char *saveptr1, *saveptr2;
           int j;

           if (argc != 4) {
               fprintf(stderr, "Usage: %s string delim subdelim\n",
                       argv[0]);
               exit(EXIT_FAILURE);
           }

           for (j = 1, str1 = argv[1]; ; j++, str1 = NULL) {
               token = strtok_r(str1, argv[2], &saveptr1);
               if (token == NULL)
                   break;
               printf("%d: %s\n", j, token);

               for (str2 = token; ; str2 = NULL) {
                   subtoken = strtok_r(str2, argv[3], &saveptr2);
                   if (subtoken == NULL)
                       break;
                   printf(" --> %s\n", subtoken);
               }
           }

           exit(EXIT_SUCCESS);
       }



~~~~~~~~~~~~~~~~~~~~~~~~~~

What is NEEDED is a simple example...

Ruben






Ruben

[toc] | [prev] | [next] | [standalone]

#59192

On 03/28/2015 08:39 PM, Keith Thompson wrote:
> James Kuyper <jameskuyper@verizon.net> writes:
> [...]
>> Note: if you're using gets() to read the text string, '\n' characters
>> will be replaced with '\0', which would make tokenizing on '\n' a little
>> difficult. If that fact is completely irrelevant to your problem, that
>> demonstrates the need for item 1.
> 
> If you're using gets(), that's your first problem.  gets() is inherently
> unsafe, and has been removed from the language as of the 2011 standard.
> 


you have to LOVE this line
 Another example program using strtok() can be found in getaddrinfo_a(3).

[toc] | [prev] | [next] | [standalone]

#59190

On 2015.03.28. 15:41, what wrote:
> The docs are terrible for strtok_r and I can not predict how it will
> behave.  I have a simple text string from stdin to tokenize on " \n" and
> it seems that this is very difficult task for the C programming language.

It's sad.
Still, you could try to read the manual and/or
search for example programs.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.c

csiph-web