voice_audio_multi/docs/models/ming3_audio.md

Ming 3.0 Audio Tiny

Ming 3.0 Audio combines a Whisper audio encoder, a hybrid KDA/MLA mixture-of-experts Thinker, and an independent continuous-latent Talker with an audio VAE. The Thinker produces text; the Talker synthesizes that text.

Upstream checkpoint: inclusionAI/Ming-3.0-audio-tiny. Code: inclusionAI/Ming-3.0-audio-flash. GGUF: audio-cpp/Ming-3.0-Audio-Tiny-GGUF.

The standalone GGUF includes the Thinker, Talker, audio VAE, CAMPPlus speaker encoder, tokenizers, and configuration. No separate Python runtime or codec download is required. Speech output is mono 44.1 kHz. Only offline sessions are exposed.

Usage

Transcription And Audio Understanding

audiocpp_cli --family ming3_audio --task asr --backend cuda \
  --model /path/to/ming-3.0-audio-tiny-bf16.gguf \
  --audio recording.wav --log

The default prompt returns a language label followed by a tab and the transcript. Use --request-option prompt="Describe the sounds in this recording." for audio understanding instead of transcription.

Named Voice

audiocpp_cli --family ming3_audio --task tts --backend cuda \
  --model /path/to/ming-3.0-audio-tiny-bf16.gguf \
  --text "The sun warms the earth, and a soft breeze moves through the garden." \
  --request-option voice_id=US_James --request-option language=en \
  --out speech.wav --log

Reference Voice

audiocpp_cli --family ming3_audio --task clon --backend cuda \
  --model /path/to/ming-3.0-audio-tiny-bf16.gguf \
  --voice-ref reference.wav --text "Please leave the lamp beside the window." \
  --request-option language=en --out clone.wav --log

Reference audio supplies speaker identity. Adding --request-option reference_text="Words spoken in the reference." also supplies an acoustic prefix. Named voices are used only when reference audio is absent. Both tts and clon accept reference audio.

Prefer a reference in the target language: cross-language cloned pronunciation can drift even when the generated text is correct.

Spoken Response

Use --task s2s --audio question.wav --out response.wav with the same model and backend. The Thinker answers the question in the recording, and the Talker speaks its answer. prompt overrides the Thinker instruction; voice_id or --voice-ref controls the response voice.

Controls

Use --request-option name=value, or the server request's options map.

Option	Default	Meaning
prompt	Task-dependent	Thinker instruction for input audio.
system_prompt	Empty	Thinker system instruction; selects the official chat-style text-before-audio message layout.
enable_thinking	false	Enable detailed thinking in the Thinker chat template.
max_tokens	256	Maximum generated Thinker text tokens.
max_audio_tokens	1000	Maximum generated Talker acoustic patches per fragment.
voice_id	灵小柔	Official named voice, used without reference audio; see the spec for accepted names.
language	Unset	Target Talker language; Chinese and English are the primary supported languages.
reference_text	Empty	Reference transcript enabling acoustic-prefix conditioning.
seed	42	Talker and reference-posterior sampling seed; Thinker decoding is greedy.
guidance_scale	2	Conditioned flow prediction plus this scale times its difference from the null prediction.
sigma	0.25	Flow SDE noise scale.
temperature	0	Flow sampling temperature; zero still uses seeded initial noise.
raw_text	false	Synthesize one fragment without text chunking.
text_chunk_size	100	Framework chunk budget in Unicode codepoints.
text_chunk_mode	tag_aware	default, tag_aware, japanese, or endline.
trim_silence	true	Trim trailing low-energy frames, retaining a short tail.

Session option ming3_audio.reference_cache=true caches prepared conditioning for the most recent reference waveform. There are no model-specific load options.

--session-option ming3_audio.disk_experts=true keeps Thinker expert weights in host memory and stages selected experts into a bounded GPU cache. Inference still runs on the selected GPU. ming3_audio.pin_experts=true retains expert payloads in pinned RAM; set it to false for range reads from the GGUF instead. It defaults to false on Metal, which uses ordinary host staging. ming3_audio.expert_cache_size selects 8 to 128 slots per layer (default 8). ming3_audio.disk_expert_read_threads selects 1 to 64 host workers for copying selected GGUF experts into staging. The default is 1 (serial); try 4 using OpenMP or Apple dispatch. This applies only with disk_experts=true and pin_experts=false. The host reader is backend-independent; Metal performance has not been validated. Spoken-response sessions release Thinker workspaces before allocating the speech stage in this mode. The offloaded Thinker allocates its KV cache in growing buckets rather than reserving the entire max_tokens allowance upfront. Growth preserves the existing KV and recurrent state without changing precision or reducing the token limit. Memory still grows with actual context length. The normal resident path remains the default.

Thinker MLA attention uses F16 KV storage; KDA recurrent state and convolution tails remain F32. CUDA decode reuses session-owned attention graphs sized to the live prefix, retaining the full context. KDA input projections and shared MLP gate/up projections are packed at load without changing weight precision. These optimizations are enabled by default and do not change the Talker or VAE.

The native frontend uses the framework text chunker rather than Python's word/character buffering and does not run the optional Python FST normalizer. Spell out numbers and abbreviations when pronunciation matters. Long text is synthesized as independent fragments. Reference conditioning is retained across fragments; generated conversation history is not.

Long clones can omit words within a fragment. The default chunk budget is 100; any positive integer remains supported. For example, --request-option text_chunk_size=300 selects the previous budget. On a 1,518-character English clone with the same reference audio and transcript, reducing the budget from 300 to 100 lowered Whisper turbo WER from 4.35% to 0.40% with seed 42 and from 3.95% to 0% with seed 43. Whisper medium independently confirmed the improvement. Smaller chunks reduce this risk but do not guarantee word-perfect output. Chunk boundaries and numerical execution can change generated lengths and the random stream consumed by later fragments; a fixed seed does not guarantee matching words across implementations. The tested BF16 and Q8_0 packages produce identical clone WAVs, so this omission is not caused by Thinker quantization.

Conversion

Build the Debug audiocpp_gguf target, then convert the complete official snapshot:

python tools/models/convert_ming3_audio_gguf.py \
  --source /path/to/Ming-3.0-audio-tiny \
  --output /path/to/ming-3.0-audio-tiny-bf16.gguf \
  --type orig --temporary-dir /path/to/conversion-work --log

The converter stacks MoE expert banks without changing their BF16 values and keeps expert routers in F32. Training-only next-token prediction weights are not included. Conversion requires PyTorch, NumPy, ONNX, Safetensors, and GGUF Python packages; inference does not require them. The checkpoint is MIT-licensed.

--type q8_0 quantizes only Thinker expert banks. --type q4_k uses Q4_K experts plus a Q8_0 text output head to reduce GPU-resident weights during expert offload. Both retain source precision for token embeddings, attention and dense/shared MLP projections: broader projection quantization caused repetition or prolonged generation on the full meeting examples. Both preserve source precision for the audio encoder, Talker, VAE and speaker encoder, and keep routers F32. TTS-only memory usage therefore does not shrink with these Thinker quantizations.