Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c++ > #10007 > unrolled thread

Best way to tokenize in String

Back to article view | Back to comp.lang.c++

Contents

  Best way to tokenize in String sravanreddy001 <sravanreddy001@gmail.com> - 2011-09-09 19:25 -0700
    Re: Best way to tokenize in String Ian Collins <ian-news@hotmail.com> - 2011-09-10 15:14 +1200
      Re: Best way to tokenize in String Paul <pchristor@yahoo.co.uk> - 2011-09-09 22:30 -0700
        Re: Best way to tokenize in String red floyd <no.spam.here@its.invalid> - 2011-09-09 23:11 -0700
          Re: Best way to tokenize in String "Paul" <pchristor@yahoo.co.uk> - 2011-09-10 11:32 +0100
        Re: Best way to tokenize in String Ben Cottrell <email@address.invalid> - 2011-09-10 13:05 +0100
          Re: Best way to tokenize in String Paul <pchristor@yahoo.co.uk> - 2011-09-10 08:25 -0700
    Re: Best way to tokenize in String Leigh Johnston <leigh@i42.co.uk> - 2011-09-10 13:15 +0100
    Re: Best way to tokenize in String Asger-P <junk@asger-p.dk> - 2011-09-10 14:54 +0200
    Re: Best way to tokenize in String Moshbear dot Net <moshbear.net@gmail.com> - 2011-09-10 22:46 -0700

#10007 — Best way to tokenize in String

Hi,
what is the efficient way to tokenize the string and splitting into
words based on the delimiter?

i've looked at the strtok() in string.h
is there any other better way to do this?
A more efficient one?

[toc] | [next] | [standalone]

#10009

On 09/10/11 02:25 PM, sravanreddy001 wrote:
> Hi,
> what is the efficient way to tokenize the string and splitting into
> words based on the delimiter?

The simplest way is to create an istringstream from the string and 
stream out the words.

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#10011

On Sep 10, 4:14 am, Ian Collins <ian-n...@hotmail.com> wrote:
> On 09/10/11 02:25 PM, sravanreddy001 wrote:
>
> > Hi,
> > what is the efficient way to tokenize the string and splitting into
> > words based on the delimiter?
>
> The simplest way is to create an istringstream from the string and
> stream out the words.
>
But is that the most efficient way?

However simplicty is sometime preferred over efficiency and the stream
method is quite handy, it is described here, bottom of page:
http://www.oopweb.com/CPP/Documents/CPPHOWTO/Volume/C++Programming-HOWTO-7.html

Is there a way to use this stream method when the delimiter is not
whitespace?

[toc] | [prev] | [next] | [standalone]

#10012

On 9/9/2011 10:30 PM, Paul wrote:
> On Sep 10, 4:14 am, Ian Collins<ian-n...@hotmail.com>  wrote:
>> On 09/10/11 02:25 PM, sravanreddy001 wrote:
>>
>>> Hi,
>>> what is the efficient way to tokenize the string and splitting into
>>> words based on the delimiter?
>>
>> The simplest way is to create an istringstream from the string and
>> stream out the words.
>>
> But is that the most efficient way?

Screw "efficiency".  The amount of time parsing should be minimal
compared to either your I/O or processing time.  Until you benchmark
and show that the tokenizing is a bottleneck, go for the simplicity.

[toc] | [prev] | [next] | [standalone]

#10021

"red floyd" <no.spam.here@its.invalid> wrote in message 
news:j4ev39$o1e$1@dont-email.me...
> On 9/9/2011 10:30 PM, Paul wrote:
>> On Sep 10, 4:14 am, Ian Collins<ian-n...@hotmail.com>  wrote:
>>> On 09/10/11 02:25 PM, sravanreddy001 wrote:
>>>
>>>> Hi,
>>>> what is the efficient way to tokenize the string and splitting into
>>>> words based on the delimiter?
>>>
>>> The simplest way is to create an istringstream from the string and
>>> stream out the words.
>>>
>> But is that the most efficient way?
>
> Screw "efficiency".  The amount of time parsing should be minimal
> compared to either your I/O or processing time.  Until you benchmark
> and show that the tokenizing is a bottleneck, go for the simplicity.
>
Yeah I know efficiency wouldn't be mega important in many cases but the OP 
did specifically ask about something more efficient than strtok.

Anyway you can't really do anything other than loop through the string 
testing each char against a delimiter. Any gains in efficiency are probably 
in the way the data is input/output where:
input(file read lines?) ---> tokenizer function ---> output(further 
processing?)

When you find a token what do you do with it; store it or send it for 
further processing?
The design of the function needs to fit the programs needs i.e: return 
value, argument types, does it throw exceptions, is it a template, is it a 
class member etc etc. 


--- Posted via news://freenews.netfront.net/ - Complaints to news@netfront.net ---

[toc] | [prev] | [next] | [standalone]

#10023

Paul wrote:
> On Sep 10, 4:14 am, Ian Collins <ian-n...@hotmail.com> wrote:
> 
>>On 09/10/11 02:25 PM, sravanreddy001 wrote:
>>
>>
>>>Hi,
>>>what is the efficient way to tokenize the string and splitting into
>>>words based on the delimiter?
>>
>>The simplest way is to create an istringstream from the string and
>>stream out the words.
>>
> 
> But is that the most efficient way?
> 
> However simplicty is sometime preferred over efficiency and the stream
> method is quite handy, it is described here, bottom of page:
> http://www.oopweb.com/CPP/Documents/CPPHOWTO/Volume/C++Programming-HOWTO-7.html
> 
> Is there a way to use this stream method when the delimiter is not
> whitespace?
> 
> 
Yes, there's a technique called the "whitespace redefinition approach" 
which will let you do that:

https://groups.google.com/group/comp.lang.c++/msg/4de0be2e4eb8e0ba?output=gplain&hl=de&pli=1


Personally I like the ability to use 'cin >> foo >> bar' style to read 
delimited data from a stream, although most C++ programmers I know don't 
really know/care much about locales, ctypes and facets, and 
unfortunately I would expect they'd probably consider it to be "a bit 
too weird" to use in their code.




Another one which I like, using TR1/Boost RegEx:

#include <string>
#include <regex>
#include <iostream>

int main()
{
     std::string str = "the\t    quick  brown\n-\n- fox"
                       " jumped..over,the,lazy,.dog";
     std::tr1::regex re("[\\s-,.]+");

     std::tr1::sregex_token_iterator
         iter(str.begin(), str.end(), re, -1),
         end;

     while(iter != end)
     {
         std::cout << *iter++ << std::endl;
     }
}

[toc] | [prev] | [next] | [standalone]

#10032

On Sep 10, 1:05 pm, Ben Cottrell <em...@address.invalid> wrote:
> Paul wrote:
> > On Sep 10, 4:14 am, Ian Collins <ian-n...@hotmail.com> wrote:
>
> >>On 09/10/11 02:25 PM, sravanreddy001 wrote:
>
> >>>Hi,
> >>>what is the efficient way to tokenize the string and splitting into
> >>>words based on the delimiter?
>
> >>The simplest way is to create an istringstream from the string and
> >>stream out the words.
>
> > But is that the most efficient way?
>
> > However simplicty is sometime preferred over efficiency and the stream
> > method is quite handy, it is described here, bottom of page:
> >http://www.oopweb.com/CPP/Documents/CPPHOWTO/Volume/C++Programming-HO...
>
> > Is there a way to use this stream method when the delimiter is not
> > whitespace?
>
> Yes, there's a technique called the "whitespace redefinition approach"
> which will let you do that:
>
> https://groups.google.com/group/comp.lang.c++/msg/4de0be2e4eb8e0ba?ou...
>
> Personally I like the ability to use 'cin >> foo >> bar' style to read
> delimited data from a stream, although most C++ programmers I know don't
> really know/care much about locales, ctypes and facets, and
> unfortunately I would expect they'd probably consider it to be "a bit
> too weird" to use in their code.
This looks preety usefull especially if your source comes from a file.
I have to admit I don't know much about locales and facets as I've
never really delved into that part of the language. It looks quite
advanced and think it might take a couple of days studying to get a
good understanding of it all. I'll add it my list of things to do. :)

>
> Another one which I like, using TR1/Boost RegEx:
>
> #include <string>
> #include <regex>
> #include <iostream>
>
> int main()
> {
>      std::string str = "the\t    quick  brown\n-\n- fox"
>                        " jumped..over,the,lazy,.dog";
>      std::tr1::regex re("[\\s-,.]+");
>
>      std::tr1::sregex_token_iterator
>          iter(str.begin(), str.end(), re, -1),
>          end;
>
>      while(iter != end)
>      {
>          std::cout << *iter++ << std::endl;
>      }
>
>
>
This looks very tidy too. More usefull if input is in the form of a
string. Again never really delved into boost RegEx, another one for
the list :)

[toc] | [prev] | [next] | [standalone]

#10024

On 10/09/2011 03:25, sravanreddy001 wrote:
> Hi,
> what is the efficient way to tokenize the string and splitting into
> words based on the delimiter?
>
> i've looked at the strtok() in string.h
> is there any other better way to do this?
> A more efficient one?

Here is my tokenizer; feel free to use for whatever purpose:

namespace neolib
{
	template <typename FwdIter1, typename FwdIter2, typename ResultContainer>
	inline FwdIter1 tokens(FwdIter1 aFirst, FwdIter1 aLast, FwdIter2 
aDelimeterFirst, FwdIter2 aDelimiterLast, ResultContainer& aTokens, 
std::size_t aMaxTokens = 0, bool aSkipEmptyTokens = true, bool 
aDelimeterIsSubsequence = false)
	{
		if (aFirst == aLast)
			return aFirst;
		typedef typename ResultContainer::value_type value_type;
		if (aDelimeterFirst == aDelimiterLast)
		{
			aTokens.push_back(value_type(aFirst, aLast));
			return aLast;
		}
		FwdIter1 b = aFirst;
		FwdIter1 e = aDelimeterIsSubsequence ? std::search(b, aLast, 
aDelimeterFirst, aDelimiterLast) : std::find_first_of(b, aLast, 
aDelimeterFirst, aDelimiterLast);
		std::size_t tokens = 0;
		while(e != aLast && (aMaxTokens == 0 || tokens < aMaxTokens))
		{
			if (b == e && !aSkipEmptyTokens)
			{
				aTokens.push_back(value_type(b, b));
				++tokens;
			}
			else if (b != e)
			{
				aTokens.push_back(value_type(b, e));
				++tokens;
			}
			b = e;
			for (std::size_t i = aDelimeterIsSubsequence ? 
std::distance(aDelimeterFirst, aDelimiterLast) : 1; i > 0; --i)
				++b;
			e = aDelimeterIsSubsequence ? std::search(b, aLast, aDelimeterFirst, 
aDelimiterLast) : std::find_first_of(b, aLast, aDelimeterFirst, 
aDelimiterLast);
		}
		if (b != e && (aMaxTokens == 0 || tokens < aMaxTokens))
		{
			aTokens.push_back(value_type(b, e));
			b = e;
		}
		return b;
	}
	
	template <typename CharT, typename Traits, typename Alloc, typename 
ResultContainer>
	inline void tokens(const std::basic_string<CharT, Traits, Alloc>& 
aLine, const std::basic_string<CharT, Traits, Alloc>& aDelimeter, 
ResultContainer& aTokens, std::size_t aMaxTokens = 0, bool 
aSkipEmptyTokens = true, bool aDelimeterIsSubsequence = false)
	{
		tokens(aLine.begin(), aLine.end(), aDelimeter.begin(), 
aDelimeter.end(), aTokens, aMaxTokens, aSkipEmptyTokens, 
aDelimeterIsSubsequence);
	}
}

Example of use:

int main(int argc, char* argv[])
{
	std::string string = "The quick brown fox jumped over the lazy dog.";
	typedef std::vector<std::string> tokens_t;
	tokens_t tokens;
	neolib::tokens(string, std::string(" "), tokens);
	for (tokens_t::const_iterator i = tokens.begin(); i != tokens.end(); ++i)
		std::cout << "[" << *i << "]" << std::endl;
}

/Leigh

[toc] | [prev] | [next] | [standalone]

#10025

Hi sravanreddy001

On the: 10. of september-2011 At: 04:25 sravanreddy001 wrote:

> Hi,
> what is the efficient way to tokenize the string and splitting into
> words based on the delimiter?

You write delimiter and not delimiters, is that because You
have only one delimiter to consider ?

If thats the case of if You have only a few known delimiters
then You can do a it a lot faster then strtok, if You write
Your own tokenizer.


Best regards
Asger-P

[toc] | [prev] | [next] | [standalone]

#10043

On Sep 9, 10:25 pm, sravanreddy001 <sravanreddy...@gmail.com> wrote:
> Hi,
> what is the efficient way to tokenize the string and splitting into
> words based on the delimiter?
>
> i've looked at the strtok() in string.h
> is there any other better way to do this?
> A more efficient one?

I've had good results with boost.tokenizer.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.c++

csiph-web