Purpose

TOKEN is used to tokenize a string delimited by various delimiters.

Return Value - This function returns the next token found in the string. 

Syntax

token = TOKEN (InString,delimiter_list,delimiter,return_code)

Operation

A sequence of calls to this function splits string into tokens, which are sequences of contiguous characters separated by any of the characters that are part of delimiter_list.

On a first call, the function expects a string as argument for InString, whose first character is used as the starting location to scan for tokens. In subsequent calls, the function expects a null string as the first argument and uses the position right after the end of the last token as the new starting location for scanning.

arguments:

- token is the resulting string after tokenize operation

- InString is any valid alphanumeric string

- delimiter_list is any valid alphanumeric literal or variable specifying the delimiter characters based on which string has to be tokenized

- delimiter is an alphanumeric which will be set to the delimiter used from delimiters_list automatically

- return_code is a numeric variable which holds the return value. This is set to TRUE when the string to be tokenized is completely scanned.

Remarks
 

This function determines the beginning and the end of a token by first scanning from the starting location for the first character not contained in delimiter_list (which becomes the beginning of the token); then scans starting from beginning of the token for the first character contained in delimiter_list, which becomes the end of the token. The scan stops if the end of string is encountered.

This end of the token is automatically replaced by a null-character, and the beginning of the token is returned by the function.

Once the terminating null character of InString is found in the first call to TOKEN, we make subsequent calls to this function (with an empty string as the first argument) till return_code is set to TRUE.

If no delimiters are found, the entire string is returned with return_code set to

TRUE.

Example
 

$STR = "ONE,TWO:THREE;FOUR,FIVE:SIX,SEVEN,EIGHT,NINE;TEN"

// specify the delimiters

$delimiter_list = ",:;"

$resulting_str = TOKEN ($STR, $delimiter_list, $delimiter, #return_code)

UMSG("The resulting string is " + $resulting_str, -1)

// iterate to the end of string till return code is TRUE

WHILE #return_code # TRUE

$resulting_str = TOKEN ("", $delimiter_list, $delimiter, #return_code)

// List the tokens output via the UMSG

UMSG ("The resulting string is " + $resulting_str, -1)

ENDWHILE

Output:

The resulting string is  ONE

The resulting string is  TWO

The resulting string is  THREE

The resulting string is  FOUR

The resulting string is  FIVE

The resulting string is  SIX

The resulting string is  SEVEN

The resulting string is  EIGHT

The resulting string is  NINE

The resulting string is  TEN