- bosToken
llama_token bosToken(const(llama_vocab)* vocab)
Undocumented in source. Be warned that the author may not have intended to support it.
- detokenize
string detokenize(const(llama_vocab)* vocab, const(llama_token)[] tokens, bool removeSpecial, bool unparseSpecial)
Decode a token sequence back into text.
- eosToken
llama_token eosToken(const(llama_vocab)* vocab)
Undocumented in source. Be warned that the author may not have intended to support it.
- eotToken
llama_token eotToken(const(llama_vocab)* vocab)
Undocumented in source. Be warned that the author may not have intended to support it.
- fimMidToken
llama_token fimMidToken(const(llama_vocab)* vocab)
Undocumented in source. Be warned that the author may not have intended to support it.
- fimPadToken
llama_token fimPadToken(const(llama_vocab)* vocab)
Undocumented in source. Be warned that the author may not have intended to support it.
- fimPreToken
llama_token fimPreToken(const(llama_vocab)* vocab)
Undocumented in source. Be warned that the author may not have intended to support it.
- fimRepToken
llama_token fimRepToken(const(llama_vocab)* vocab)
Undocumented in source. Be warned that the author may not have intended to support it.
- fimSepToken
llama_token fimSepToken(const(llama_vocab)* vocab)
Undocumented in source. Be warned that the author may not have intended to support it.
- fimSufToken
llama_token fimSufToken(const(llama_vocab)* vocab)
Undocumented in source. Be warned that the author may not have intended to support it.
- isControl
bool isControl(const(llama_vocab)* vocab, llama_token token)
True if the token is a control token (not renderable text).
- isEog
bool isEog(const(llama_vocab)* vocab, llama_token token)
True if the token signals end of generation.
- nlToken
llama_token nlToken(const(llama_vocab)* vocab)
Undocumented in source. Be warned that the author may not have intended to support it.
- padToken
llama_token padToken(const(llama_vocab)* vocab)
Undocumented in source. Be warned that the author may not have intended to support it.
- sepToken
llama_token sepToken(const(llama_vocab)* vocab)
Undocumented in source. Be warned that the author may not have intended to support it.
- tokenAttr
llama_token_attr tokenAttr(const(llama_vocab)* vocab, llama_token token)
Token attribute flags (control, normal, byte, etc.).
- tokenScore
float tokenScore(const(llama_vocab)* vocab, llama_token token)
Log-probability score stored for a token in the vocab.
- tokenText
const(char)* tokenText(const(llama_vocab)* vocab, llama_token token)
Raw text piece for a token (pointer into model memory; do not free).
- tokenToString
string tokenToString(const(llama_vocab)* vocab, llama_token token)
The string piece for a single token.
- tokenize
llama_token[] tokenize(const(llama_vocab)* vocab, const(char)[] text, bool addSpecial, bool parseSpecial)
Split text into tokens. Returns a GC-allocated slice.
- vocabType
int vocabType(const(llama_vocab)* vocab)
Vocabulary type as int (compare to LLAMA_VOCAB_TYPE_* constants).