Module rnix::tokenizer

source ·
Expand description

The tokenizer: turns a string into tokens, such as numbers, strings, and keywords

Structs§

  • The tokenizer. You may want to use the tokenize convenience function from this module instead.

Functions§

  • A convenience function for tokenizing the given input

Type Aliases§