Sane C++ Libraries
C++ Platform Abstraction Libraries
SC::StringViewTokenizer Struct Reference

Splits a StringView in tokens according to separators. More...

#include <StringView.h>

Public Types

enum  Options {
  IncludeEmpty ,
  SkipEmpty
}
 

Public Member Functions

 StringViewTokenizer (StringView text)
 Build a tokenizer operating on the given text string view. More...
 
bool tokenizeNext (Span< const StringCodePoint > separators, Options options=Options::SkipEmpty)
 Splits the string along a list of separators. More...
 
bool tokenizeNextLine ()
 Tokenizes from current position to first newline. More...
 
StringViewTokenizercountTokens (Span< const StringCodePoint > separators)
 Count the number of tokens that exist in the string view passed in constructor, when splitted along the given separators. More...
 
bool isFinished () const
 Check if the tokenizer has processed the entire the string view passed in the constructor. More...
 

Public Attributes

StringCodePoint splittingCharacter = 0
 The last splitting character matched in current tokenization. More...
 
size_t numSplitsNonEmpty = 0
 How many non-empty splits have occurred in current tokenization. More...
 
size_t numSplitsTotal = 0
 How many total splits have occurred in current tokenization. More...
 
StringView component
 Current component that has been tokenized by tokenizeNext. More...
 
StringView processed
 Substring of original string passed in constructor processed so far. More...
 
StringView remaining
 Substring from current position until the end of original text. More...
 

Detailed Description

Splits a StringView in tokens according to separators.

Member Enumeration Documentation

◆ Options

Enumerator
IncludeEmpty 

If to tokenizeNext should return also empty tokens.

SkipEmpty 

If to tokenizeNext should NOT return also empty tokens.

Constructor & Destructor Documentation

◆ StringViewTokenizer()

SC::StringViewTokenizer::StringViewTokenizer ( StringView  text)
inline

Build a tokenizer operating on the given text string view.

Member Function Documentation

◆ countTokens()

StringViewTokenizer & SC::StringViewTokenizer::countTokens ( Span< const StringCodePoint separators)

Count the number of tokens that exist in the string view passed in constructor, when splitted along the given separators.

Parameters
separatorsSeparators to split the original string with
Returns
Current StringViewTokenizer to inspect SC::StringViewTokenizer::numSplitsNonEmpty or SC::StringViewTokenizer::numSplitsTotal.
Example:
#define SC_TEST_EXPECT(e)
Records a test expectation (eventually aborting or breaking o n failed test)
Definition: Testing.h:113
StringViewTokenizer(StringView text)
Build a tokenizer operating on the given text string view.
Definition: StringView.h:589
StringViewTokenizer & countTokens(Span< const StringCodePoint > separators)
Count the number of tokens that exist in the string view passed in constructor, when splitted along t...
size_t numSplitsNonEmpty
How many non-empty splits have occurred in current tokenization.
Definition: StringView.h:575
size_t numSplitsTotal
How many total splits have occurred in current tokenization.
Definition: StringView.h:576

◆ isFinished()

bool SC::StringViewTokenizer::isFinished ( ) const

Check if the tokenizer has processed the entire the string view passed in the constructor.

◆ tokenizeNext()

bool SC::StringViewTokenizer::tokenizeNext ( Span< const StringCodePoint separators,
Options  options = Options::SkipEmpty 
)

Splits the string along a list of separators.

Parameters
separatorsList of separators
optionsIf to skip empty tokens or not
Returns
true if there are additional tokens to parse
Example:
StringViewTokenizer tokenizer("bring,me,the,horizon");
while (tokenizer.tokenizeNext(',', StringViewTokenizer::SkipEmpty))
{
console.printLine(tokenizer.component);
}
@ SkipEmpty
If to tokenizeNext should NOT return also empty tokens.
Definition: StringView.h:585

◆ tokenizeNextLine()

bool SC::StringViewTokenizer::tokenizeNextLine ( )
inline

Tokenizes from current position to first newline.

Returns
true if a new line has been found
Example:
StringViewTokenizer lines("Line1\nLine2\nLine3\n");
SC_TEST_EXPECT(lines.tokenizeNextLine());
SC_TEST_EXPECT(lines.component == "Line1");
SC_TEST_EXPECT(lines.tokenizeNextLine());
SC_TEST_EXPECT(lines.component == "Line2");
SC_TEST_EXPECT(lines.tokenizeNextLine());
SC_TEST_EXPECT(lines.component == "Line3");
SC_TEST_EXPECT(not lines.tokenizeNextLine());

Member Data Documentation

◆ component

StringView SC::StringViewTokenizer::component

Current component that has been tokenized by tokenizeNext.

◆ numSplitsNonEmpty

size_t SC::StringViewTokenizer::numSplitsNonEmpty = 0

How many non-empty splits have occurred in current tokenization.

◆ numSplitsTotal

size_t SC::StringViewTokenizer::numSplitsTotal = 0

How many total splits have occurred in current tokenization.

◆ processed

StringView SC::StringViewTokenizer::processed

Substring of original string passed in constructor processed so far.

◆ remaining

StringView SC::StringViewTokenizer::remaining

Substring from current position until the end of original text.

◆ splittingCharacter

StringCodePoint SC::StringViewTokenizer::splittingCharacter = 0

The last splitting character matched in current tokenization.


The documentation for this struct was generated from the following file: