Fabio
c36510c21e
Some checks failed
Linux Build / Linux cpu (push) Has been cancelled
Linux Build / Linux cpu-clang (push) Has been cancelled
Linux Build / Linux vulkan (push) Has been cancelled
mac-build / macOS CPU (push) Has been cancelled
Nix Build / Nix (cpu) (push) Has been cancelled
Nix Build / Nix (vulkan) (push) Has been cancelled
windows-build / Windows CPU (push) Has been cancelled
117 lines
2.8 KiB
Markdown
117 lines
2.8 KiB
Markdown
# Output Multimodal Transfer/Decoder
|
|
|
|
OMTD is a proposed output-side companion library for llama.cpp.
|
|
|
|
It mirrors MTMD conceptually, but runs in the opposite direction:
|
|
|
|
```text
|
|
MTMD: media input -> embeddings/tokens -> language model
|
|
OMTD: language model output -> output companion -> generated media
|
|
```
|
|
|
|
This framework patch adds only the library boundary. Model-specific output
|
|
implementations, such as audio, image, or video generators, should be layered on
|
|
top in separate patches.
|
|
|
|
## Intended Frontend Support
|
|
|
|
The intended frontend path is:
|
|
|
|
```text
|
|
llama-cli
|
|
llama-server
|
|
llama-tts
|
|
|
|
|
v
|
|
libomtd
|
|
|
|
|
v
|
|
concrete output backend
|
|
```
|
|
|
|
Planned frontend responsibilities:
|
|
|
|
- `llama-cli`: accept `--omtd FILE` for local output generation from a prompt.
|
|
- `llama-server`: accept `--omtd FILE` and expose output-generation endpoints.
|
|
- `llama-tts`: use OMTD as the shared TTS/audio-output backend path.
|
|
|
|
This OMTD-only patch does not wire those flags yet. It creates the stable shared
|
|
library and API first. The frontend wiring should be a separate patch, followed
|
|
by model-specific backend patches.
|
|
|
|
## Review Path
|
|
|
|
Recommended patch order:
|
|
|
|
```text
|
|
1. OMTD core library and docs
|
|
2. Frontend wiring for llama-cli, llama-server, and llama-tts
|
|
3. First concrete OMTD backend
|
|
```
|
|
|
|
This keeps the generic output-runtime boundary reviewable without tying it to a
|
|
specific model implementation.
|
|
|
|
## Responsibilities
|
|
|
|
OMTD owns:
|
|
|
|
- A public C API for output companion detection.
|
|
- A public C API for structured output generation requests.
|
|
- Common status/error reporting for output generation backends.
|
|
- Placeholder modality and model-type enums for future backends.
|
|
|
|
OMTD does not own:
|
|
|
|
- Input media processing. That remains MTMD.
|
|
- `--mmproj` handling.
|
|
- A concrete output model in this framework-only patch.
|
|
- Application-specific file or HTTP response handling.
|
|
|
|
## Current API
|
|
|
|
Detection:
|
|
|
|
```text
|
|
omtd_is_output_companion_gguf(path)
|
|
omtd_get_model_type(path)
|
|
omtd_get_model_modality(path)
|
|
```
|
|
|
|
Audio generation boundary:
|
|
|
|
```text
|
|
omtd_audio_generate_file(params, error, error_size)
|
|
```
|
|
|
|
The audio generation entry point validates required fields and returns
|
|
`OMTD_STATUS_UNSUPPORTED` until a concrete audio backend is added.
|
|
|
|
## Diagram
|
|
|
|
```text
|
|
llama.cpp frontend
|
|
|
|
|
v
|
|
OMTD public API
|
|
|
|
|
+--> companion detection
|
|
|
|
|
+--> structured output request
|
|
|
|
|
v
|
|
model-specific output backend
|
|
added by later patches
|
|
|
|
|
v
|
|
generated media output
|
|
```
|
|
|
|
## Relationship To MTMD
|
|
|
|
| Area | MTMD | OMTD |
|
|
| --- | --- | --- |
|
|
| Direction | input -> model | model -> output |
|
|
| Media role | user-provided input media | generated output media |
|
|
| File role | input projector | output companion |
|
|
| API style | tokenize/encode input | generate/decode output |
|