Fabio
c36510c21e
Some checks failed
Linux Build / Linux cpu (push) Has been cancelled
Linux Build / Linux cpu-clang (push) Has been cancelled
Linux Build / Linux vulkan (push) Has been cancelled
mac-build / macOS CPU (push) Has been cancelled
Nix Build / Nix (cpu) (push) Has been cancelled
Nix Build / Nix (vulkan) (push) Has been cancelled
windows-build / Windows CPU (push) Has been cancelled
155 lines
8.1 KiB
Markdown
155 lines
8.1 KiB
Markdown
# Ming 3.0 Audio Tiny
|
|
|
|
Ming 3.0 Audio combines a Whisper audio encoder, a hybrid KDA/MLA
|
|
mixture-of-experts Thinker, and an independent continuous-latent Talker with
|
|
an audio VAE. The Thinker produces text; the Talker synthesizes that text.
|
|
|
|
Upstream checkpoint: [inclusionAI/Ming-3.0-audio-tiny](https://huggingface.co/inclusionAI/Ming-3.0-audio-tiny/tree/25b96e30be3739849bf76c0f48ac11db4a262fb3).
|
|
Code: [inclusionAI/Ming-3.0-audio-flash](https://github.com/inclusionAI/Ming-3.0-audio-flash).
|
|
GGUF: [audio-cpp/Ming-3.0-Audio-Tiny-GGUF](https://huggingface.co/audio-cpp/Ming-3.0-Audio-Tiny-GGUF).
|
|
|
|
The standalone GGUF includes the Thinker, Talker, audio VAE, CAMPPlus speaker
|
|
encoder, tokenizers, and configuration. No separate Python runtime or codec
|
|
download is required. Speech output is mono 44.1 kHz. Only offline sessions
|
|
are exposed.
|
|
|
|
## Usage
|
|
|
|
### Transcription And Audio Understanding
|
|
|
|
```bash
|
|
audiocpp_cli --family ming3_audio --task asr --backend cuda \
|
|
--model /path/to/ming-3.0-audio-tiny-bf16.gguf \
|
|
--audio recording.wav --log
|
|
```
|
|
|
|
The default prompt returns a language label followed by a tab and the transcript.
|
|
Use `--request-option prompt="Describe the sounds in this recording."` for
|
|
audio understanding instead of transcription.
|
|
|
|
### Named Voice
|
|
|
|
```bash
|
|
audiocpp_cli --family ming3_audio --task tts --backend cuda \
|
|
--model /path/to/ming-3.0-audio-tiny-bf16.gguf \
|
|
--text "The sun warms the earth, and a soft breeze moves through the garden." \
|
|
--request-option voice_id=US_James --request-option language=en \
|
|
--out speech.wav --log
|
|
```
|
|
|
|
### Reference Voice
|
|
|
|
```bash
|
|
audiocpp_cli --family ming3_audio --task clon --backend cuda \
|
|
--model /path/to/ming-3.0-audio-tiny-bf16.gguf \
|
|
--voice-ref reference.wav --text "Please leave the lamp beside the window." \
|
|
--request-option language=en --out clone.wav --log
|
|
```
|
|
|
|
Reference audio supplies speaker identity. Adding
|
|
`--request-option reference_text="Words spoken in the reference."` also supplies
|
|
an acoustic prefix. Named voices are used only when reference audio is absent.
|
|
Both `tts` and `clon` accept reference audio.
|
|
|
|
Prefer a reference in the target language: cross-language cloned pronunciation
|
|
can drift even when the generated text is correct.
|
|
|
|
### Spoken Response
|
|
|
|
Use `--task s2s --audio question.wav --out response.wav` with the same model
|
|
and backend. The Thinker answers the question in the recording, and the Talker
|
|
speaks its answer. `prompt` overrides the Thinker instruction; `voice_id` or
|
|
`--voice-ref` controls the response voice.
|
|
|
|
## Controls
|
|
|
|
Use `--request-option name=value`, or the server request's `options` map.
|
|
|
|
| Option | Default | Meaning |
|
|
| --- | --- | --- |
|
|
| `prompt` | Task-dependent | Thinker instruction for input audio. |
|
|
| `system_prompt` | Empty | Thinker system instruction; selects the official chat-style text-before-audio message layout. |
|
|
| `enable_thinking` | `false` | Enable detailed thinking in the Thinker chat template. |
|
|
| `max_tokens` | `256` | Maximum generated Thinker text tokens. |
|
|
| `max_audio_tokens` | `1000` | Maximum generated Talker acoustic patches per fragment. |
|
|
| `voice_id` | `灵小柔` | Official named voice, used without reference audio; see the spec for accepted names. |
|
|
| `language` | Unset | Target Talker language; Chinese and English are the primary supported languages. |
|
|
| `reference_text` | Empty | Reference transcript enabling acoustic-prefix conditioning. |
|
|
| `seed` | `42` | Talker and reference-posterior sampling seed; Thinker decoding is greedy. |
|
|
| `guidance_scale` | `2` | Conditioned flow prediction plus this scale times its difference from the null prediction. |
|
|
| `sigma` | `0.25` | Flow SDE noise scale. |
|
|
| `temperature` | `0` | Flow sampling temperature; zero still uses seeded initial noise. |
|
|
| `raw_text` | `false` | Synthesize one fragment without text chunking. |
|
|
| `text_chunk_size` | `100` | Framework chunk budget in Unicode codepoints. |
|
|
| `text_chunk_mode` | `tag_aware` | `default`, `tag_aware`, `japanese`, or `endline`. |
|
|
| `trim_silence` | `true` | Trim trailing low-energy frames, retaining a short tail. |
|
|
|
|
Session option `ming3_audio.reference_cache=true` caches prepared conditioning
|
|
for the most recent reference waveform. There are no model-specific load options.
|
|
|
|
`--session-option ming3_audio.disk_experts=true` keeps Thinker expert weights in
|
|
host memory and stages selected experts into a bounded GPU cache. Inference
|
|
still runs on the selected GPU. `ming3_audio.pin_experts=true` retains expert
|
|
payloads in pinned RAM; set it to `false` for range reads from the GGUF instead.
|
|
It defaults to `false` on Metal, which uses ordinary host staging.
|
|
`ming3_audio.expert_cache_size` selects 8 to 128 slots per layer (default 8).
|
|
`ming3_audio.disk_expert_read_threads` selects 1 to 64 host workers for copying
|
|
selected GGUF experts into staging. The default is 1 (serial); try 4 using
|
|
OpenMP or Apple dispatch. This applies only with `disk_experts=true` and `pin_experts=false`.
|
|
The host reader is backend-independent; Metal performance has not been validated.
|
|
Spoken-response sessions release Thinker workspaces before allocating the speech
|
|
stage in this mode. The offloaded Thinker allocates its KV cache in growing
|
|
buckets rather than reserving the entire `max_tokens` allowance upfront.
|
|
Growth preserves the existing KV and recurrent state without changing precision
|
|
or reducing the token limit. Memory still grows with actual context length.
|
|
The normal resident path remains the default.
|
|
|
|
Thinker MLA attention uses F16 KV storage; KDA recurrent state and convolution
|
|
tails remain F32. CUDA decode reuses session-owned attention graphs sized to
|
|
the live prefix, retaining the full context. KDA input projections and shared
|
|
MLP gate/up projections are packed at load without changing weight precision.
|
|
These optimizations are enabled by default and do not change the Talker or VAE.
|
|
|
|
The native frontend uses the framework text chunker rather than Python's
|
|
word/character buffering and does not run the optional Python FST normalizer.
|
|
Spell out numbers and abbreviations when pronunciation matters. Long text is
|
|
synthesized as independent fragments. Reference conditioning is retained across
|
|
fragments; generated conversation history is not.
|
|
|
|
Long clones can omit words within a fragment. The default chunk budget is 100;
|
|
any positive integer remains supported. For example,
|
|
`--request-option text_chunk_size=300` selects the previous budget. On a
|
|
1,518-character English clone with the same reference audio and transcript,
|
|
reducing the budget from 300 to 100 lowered Whisper turbo WER from 4.35% to
|
|
0.40% with seed 42 and from 3.95% to 0% with seed 43. Whisper medium independently
|
|
confirmed the improvement. Smaller chunks reduce this risk but do not guarantee
|
|
word-perfect output.
|
|
Chunk boundaries and numerical execution can change generated lengths and the
|
|
random stream consumed by later fragments; a fixed seed does not guarantee
|
|
matching words across implementations. The tested BF16 and Q8_0 packages produce
|
|
identical clone WAVs, so this omission is not caused by Thinker quantization.
|
|
|
|
## Conversion
|
|
|
|
Build the Debug `audiocpp_gguf` target, then convert the complete official snapshot:
|
|
|
|
```bash
|
|
python tools/models/convert_ming3_audio_gguf.py \
|
|
--source /path/to/Ming-3.0-audio-tiny \
|
|
--output /path/to/ming-3.0-audio-tiny-bf16.gguf \
|
|
--type orig --temporary-dir /path/to/conversion-work --log
|
|
```
|
|
|
|
The converter stacks MoE expert banks without changing their BF16 values and
|
|
keeps expert routers in F32. Training-only next-token prediction weights are
|
|
not included. Conversion requires PyTorch, NumPy, ONNX, Safetensors, and GGUF
|
|
Python packages; inference does not require them. The checkpoint is MIT-licensed.
|
|
|
|
`--type q8_0` quantizes only Thinker expert banks. `--type q4_k` uses Q4_K
|
|
experts plus a Q8_0 text output head to reduce GPU-resident weights during
|
|
expert offload. Both retain source precision for token embeddings, attention and
|
|
dense/shared MLP projections: broader projection quantization caused repetition
|
|
or prolonged generation on the full meeting examples. Both
|
|
preserve source precision for the audio encoder, Talker, VAE and speaker
|
|
encoder, and keep routers F32. TTS-only memory usage therefore does not shrink
|
|
with these Thinker quantizations.
|