MtmdContext

A multimodal projector context loaded from a GGUF file. Encodes images and audio into embeddings for the paired language model. Check if (ctx) after construction.

extern (D) @nogc nothrow

struct MtmdContext {

mixin Owned!(mtmd_context, mtmd_free) ;

static MtmdContext initFromFile(string mmproj, const(llama_model)* model, mtmd_context_params params);

static MtmdContext initFromFile(string mmproj, const(llama_model)* model);

bool supportsVision [@property getter];

bool supportsAudio [@property getter];

bool useNonCausal [@property getter];

bool useMrope [@property getter];

int audioSampleRate [@property getter];

MtmdBitmap loadBitmap(string path);

MtmdBitmap loadBitmapFromBuf(const(ubyte)[] buf);

int tokenize(InputChunks output, mtmd_input_text text, const(mtmd_bitmap*)[] bitmaps);

int evalChunks(llama_context* lctx, InputChunks chunks, llama_pos nPast, llama_seq_id seqId, int nBatch, bool logitsLast, llama_pos newNPast);

int encodeChunk(const(mtmd_input_chunk)* chunk);

float* outputEmbd();

}

Members

Functions

encodeChunk int encodeChunk(const(mtmd_input_chunk)* chunk): Encode a single image chunk. Returns 0 on success; the embedding pointer is valid until the next encode call.
evalChunks int evalChunks(llama_context* lctx, InputChunks chunks, llama_pos nPast, llama_seq_id seqId, int nBatch, bool logitsLast, llama_pos newNPast): Evaluate all chunks against the language-model context. Advances newNPast to the position after the last evaluated token. Returns 0 on success.
loadBitmap MtmdBitmap loadBitmap(string path): Load an image or audio file into an owned bitmap. Returns falsy bitmap on failure.
loadBitmapFromBuf MtmdBitmap loadBitmapFromBuf(const(ubyte)[] buf): Load a bitmap from an in-memory byte buffer.
outputEmbd float* outputEmbd(): Pointer to the most recently encoded embeddings.
tokenize int tokenize(InputChunks output, mtmd_input_text text, const(mtmd_bitmap*)[] bitmaps): Tokenise a prompt string that contains mtmd_default_marker() placeholders. bitmaps must have exactly as many entries as markers in text.text. Returns 0 on success, 1 on count mismatch, 2 on preprocessing error.

Mixins

__anonymous mixin Owned!(mtmd_context, mtmd_free): Undocumented in source.

Properties

audioSampleRate int audioSampleRate [@property getter]: Undocumented in source. Be warned that the author may not have intended to support it.
supportsAudio bool supportsAudio [@property getter]: Undocumented in source. Be warned that the author may not have intended to support it.
supportsVision bool supportsVision [@property getter]: Undocumented in source. Be warned that the author may not have intended to support it.
useMrope bool useMrope [@property getter]: Undocumented in source. Be warned that the author may not have intended to support it.
useNonCausal bool useNonCausal [@property getter]: Undocumented in source. Be warned that the author may not have intended to support it.

Static functions

initFromFile MtmdContext initFromFile(string mmproj, const(llama_model)* model, mtmd_context_params params): Load a projector from a GGUF file. Returns a falsy context on failure or null model.
initFromFile MtmdContext initFromFile(string mmproj, const(llama_model)* model): Overload using default params.

Mixed In Members

From `mixin Owned!(mtmd_context, mtmd_free)`

this this(): Undocumented in source.
this(this) this(this): Undocumented in source.
~this ~this(): Undocumented in source.
opCast bool opCast(): True when the handle holds a non-null pointer.
ptr T* ptr [@property getter]: Raw C pointer (mutable).
ptr const(T)* ptr [@property getter]: Raw C pointer (const view).

MtmdContext

Members

Functions

Mixins

Properties

Static functions

Mixed In Members

From `mixin Owned!(mtmd_context, mtmd_free)`

Meta

Source

MtmdContext

From mixin Owned!(mtmd_context, mtmd_free)

From `mixin Owned!(mtmd_context, mtmd_free)`