voice_audio_multi/docs/proposals/omtd.md

Output Multimodal Transfer/Decoder

OMTD is a proposed output-side companion library for llama.cpp.

It mirrors MTMD conceptually, but runs in the opposite direction:

MTMD: media input -> embeddings/tokens -> language model
OMTD: language model output -> output companion -> generated media

This framework patch adds only the library boundary. Model-specific output implementations, such as audio, image, or video generators, should be layered on top in separate patches.

Intended Frontend Support

The intended frontend path is:

llama-cli
llama-server
llama-tts
    |
    v
  libomtd
    |
    v
concrete output backend

Planned frontend responsibilities:

• llama-cli: accept --omtd FILE for local output generation from a prompt.
• llama-server: accept --omtd FILE and expose output-generation endpoints.
• llama-tts: use OMTD as the shared TTS/audio-output backend path.

This OMTD-only patch does not wire those flags yet. It creates the stable shared library and API first. The frontend wiring should be a separate patch, followed by model-specific backend patches.

Review Path

Recommended patch order:

1. OMTD core library and docs
2. Frontend wiring for llama-cli, llama-server, and llama-tts
3. First concrete OMTD backend

This keeps the generic output-runtime boundary reviewable without tying it to a specific model implementation.

Responsibilities

OMTD owns:

• A public C API for output companion detection.
• A public C API for structured output generation requests.
• Common status/error reporting for output generation backends.
• Placeholder modality and model-type enums for future backends.

OMTD does not own:

• Input media processing. That remains MTMD.
• --mmproj handling.
• A concrete output model in this framework-only patch.
• Application-specific file or HTTP response handling.

Current API

Detection:

omtd_is_output_companion_gguf(path)
omtd_get_model_type(path)
omtd_get_model_modality(path)

Audio generation boundary:

omtd_audio_generate_file(params, error, error_size)

The audio generation entry point validates required fields and returns OMTD_STATUS_UNSUPPORTED until a concrete audio backend is added.

Diagram

llama.cpp frontend
        |
        v
  OMTD public API
        |
        +--> companion detection
        |
        +--> structured output request
        |
        v
 model-specific output backend
   added by later patches
        |
        v
generated media output

Relationship To MTMD

Area	MTMD	OMTD
Direction	input -> model	model -> output
Media role	user-provided input media	generated output media
File role	input projector	output companion
API style	tokenize/encode input	generate/decode output