Module rnix::tokenizer

The tokenizer: turns a string into tokens, such as numbers, strings, and keywords

Structs

• Tokenizer
  The tokenizer. You may want to use the tokenize convenience function from this module instead.

Functions

• tokenize
  A convenience function for tokenizing the given input

Type Aliases

• Token