tokenize

Split text into tokens. Returns a GC-allocated slice.

@trusted

llama_token[]

tokenize

(

const(llama_vocab)* vocab

,

const(char)[] text

,

bool addSpecial = true

,

bool parseSpecial = true

)

Meta

Source

See Implementation