tokenize_example

Print each token id and its string piece for the given text.

Usage: tokenize -m model.gguf [-s] [text] - -s include BOS/EOS special tokens (off by default)

Members

Functions

main int main(string[] args): Undocumented in source. Be warned that the author may not have intended to support it.
printUsage int printUsage(string prog): Undocumented in source. Be warned that the author may not have intended to support it.

Meta

Source

See Source File

modules