SC::StringViewTokenizer Struct Reference

Splits a StringView in tokens according to separators. More...

#include <StringView.h>

Public Types
enum	Options {   IncludeEmpty ,   SkipEmpty }

Public Member Functions
	StringViewTokenizer (StringView text)
	Build a tokenizer operating on the given text string view.
bool	tokenizeNext (Span< const StringCodePoint > separators, Options options=Options::SkipEmpty)
	Splits the string along a list of separators.
bool	tokenizeNextLine ()
	Tokenizes from current position to first newline.
StringViewTokenizer &	countTokens (Span< const StringCodePoint > separators)
	Count the number of tokens that exist in the string view passed in constructor, when splitted along the given separators.
bool	isFinished () const
	Check if the tokenizer has processed the entire the string view passed in the constructor.

Public Attributes
StringCodePoint	splittingCharacter = 0
	The last splitting character matched in current tokenization.
size_t	numSplitsNonEmpty = 0
	How many non-empty splits have occurred in current tokenization.
size_t	numSplitsTotal = 0
	How many total splits have occurred in current tokenization.
StringView	component
	Current component that has been tokenized by tokenizeNext.
StringView	processed
	Substring of original string passed in constructor processed so far.
StringView	remaining
	Substring from current position until the end of original text.

Detailed Description

Splits a StringView in tokens according to separators.

Member Enumeration Documentation

Options

enum SC::StringViewTokenizer::Options

Enumerator
IncludeEmpty	If to tokenizeNext should return also empty tokens.
SkipEmpty	If to tokenizeNext should NOT return also empty tokens.

Constructor & Destructor Documentation

StringViewTokenizer()

SC::StringViewTokenizer::StringViewTokenizer ( StringView text )

inline

Build a tokenizer operating on the given text string view.

Member Function Documentation

countTokens()

StringViewTokenizer & SC::StringViewTokenizer::countTokens

(

Span< const StringCodePoint >

separators

)

Count the number of tokens that exist in the string view passed in constructor, when splitted along the given separators.

Parameters

separators

Separators to split the original string with

Returns

Current StringViewTokenizer to inspect SC::StringViewTokenizer::numSplitsNonEmpty or SC::StringViewTokenizer::numSplitsTotal.
Example:

SC_TEST_EXPECT(StringViewTokenizer("___").countTokens('_').numSplitsNonEmpty == 0);
SC_TEST_EXPECT(StringViewTokenizer("___").countTokens('_').numSplitsTotal == 3);

isFinished()

bool SC::StringViewTokenizer::isFinished ( ) const

nodiscard

Check if the tokenizer has processed the entire the string view passed in the constructor.

tokenizeNext()

bool SC::StringViewTokenizer::tokenizeNext ( Span< const StringCodePoint > separators, Options options = Options::SkipEmpty )

nodiscard

Splits the string along a list of separators.

Parameters

separators	List of separators
options	If to skip empty tokens or not

Returns

true if there are additional tokens to parse
Example:

StringViewTokenizer tokenizer("bring,me,the,horizon");
while (tokenizer.tokenizeNext(',', StringViewTokenizer::SkipEmpty))
{
    console.printLine(tokenizer.component);
}

tokenizeNextLine()

bool SC::StringViewTokenizer::tokenizeNextLine ( )

inlinenodiscard

Tokenizes from current position to first newline.

Returns

true if a new line has been found
Example:

StringViewTokenizer lines("Line1\nLine2\nLine3\n");
SC_TEST_EXPECT(lines.tokenizeNextLine());
SC_TEST_EXPECT(lines.component == "Line1");
SC_TEST_EXPECT(lines.tokenizeNextLine());
SC_TEST_EXPECT(lines.component == "Line2");
SC_TEST_EXPECT(lines.tokenizeNextLine());
SC_TEST_EXPECT(lines.component == "Line3");
SC_TEST_EXPECT(not lines.tokenizeNextLine());

Member Data Documentation

component

StringView SC::StringViewTokenizer::component

Current component that has been tokenized by tokenizeNext.

numSplitsNonEmpty

size_t SC::StringViewTokenizer::numSplitsNonEmpty = 0

How many non-empty splits have occurred in current tokenization.

numSplitsTotal

size_t SC::StringViewTokenizer::numSplitsTotal = 0

How many total splits have occurred in current tokenization.

processed

StringView SC::StringViewTokenizer::processed

Substring of original string passed in constructor processed so far.

remaining

StringView SC::StringViewTokenizer::remaining

Substring from current position until the end of original text.

splittingCharacter

StringCodePoint SC::StringViewTokenizer::splittingCharacter = 0

The last splitting character matched in current tokenization.

---

The documentation for this struct was generated from the following file:

• StringView.h