voice_audio_multi/docs/models/ming3_tts.md

Ming 3.0 TTS 0.5B

Ming uses a Qwen backbone, a continuous-latent flow-matching head, a waveform VAE and CAMPPlus speaker conditioning. It supports Chinese, English, Japanese, Korean, German, French and Spanish, with voice cloning and voice design.

GGUF packages: audio-cpp/Ming-3.0-TTS-0.5B-GGUF.

Upstream: inclusionAI/Ming-3.0-tts. Output is mono 44.1 kHz. Inference supports CUDA, Vulkan and CPU, with offline and streaming TTS, voice cloning, and voice design.

Usage

audiocpp_cli --family ming3_tts --task tts --backend cuda \
  --model /path/to/ming-3.0-tts-0.5b-bf16.gguf \
  --text "Good evening. The harbor is quiet tonight." \
  --request-option language=en --out speech.wav --log

The GGUF includes the backbone, flow head, audio VAE, CAMPPlus speaker encoder, configuration and tokenizer. No separate codec or Python runtime is needed. BF16 preserves the original checkpoint dtypes. Q8 retains the small stop head in BF16 and the speaker encoder in F32.

Streaming

Add --mode streaming to a generation command:

audiocpp_cli --family ming3_tts --task tts --mode streaming --backend cuda \
  --model /path/to/ming-3.0-tts-0.5b-q8_0.gguf \
  --text "Good evening. The harbor is quiet tonight." \
  --request-option language=en --out speech.wav --log

Streaming decodes latent patches while the acoustic generator is running. The VAE retains enough left context for its causal attention and one patch of lookahead for interpolation and waveform reconstruction. The default schedule grows from one patch to four patches per decode. Set --request-option stream_chunk_policy=fixed for fixed-size groups or --request-option stream_frames_per_event=8 to change the maximum group size. The codec retains at most six streaming graph capacities, sharing one set of weights. Trailing-silence trimming buffers uncommitted audio; emitted audio is never retracted. The streaming window can produce numerical differences from full offline decoding.

For HTTP streaming, configure the model with "mode": "streaming" and use POST /v1/audio/speech with "response_format": "pcm" and "stream_format": "sse"; the same voice and generation options apply.

Voice Cloning

audiocpp_cli --family ming3_tts --task clon --backend cuda \
  --model /path/to/ming-3.0-tts-0.5b-bf16.gguf \
  --voice-ref reference.wav \
  --request-option reference_text="Words spoken in the reference." \
  --text "Please leave the lamp beside the window." \
  --request-option language=en --out clone.wav --log

With reference_text, the reference supplies both speaker identity and an acoustic latent prefix. Omit the transcript to use speaker conditioning only. Reference encoding samples the VAE posterior, matching the official pipeline's RNG consumption.

Voice Design

audiocpp_cli --family ming3_tts --task vdes --backend cuda \
  --model /path/to/ming-3.0-tts-0.5b-bf16.gguf \
  --text "Good evening. The harbor is quiet tonight." \
  --request-option language=en \
  --request-option instruct="A calm adult man with a clear, warm voice." \
  --out designed.wav --log

Request Options

Use --request-option name=value, or the server request's options map.

Option	Default	Description
template_name	tts; voice_design on vdes	tts, multilingual, basic, emotion, dialect, ip, voice_design, or freeform.
language	The request's language	Target language name or code: zh, en, ja, ko, de, fr, es. Without the option, the server's top-level language field or the CLI's --language is used. Required for multilingual; ignored by dialect.
instruct	Empty	Voice description for voice_design, required raw instruction for freeform, or optional raw instruction for tts.
instruction_schema	en for dialect, otherwise zh	Language of the structured instruction field names, not the spoken language. Values: en, zh.
speed	Unset	basic template: 慢速, 中速, 快速. Categorical conditioning, not waveform resampling.
pitch	Unset	basic template: 低, 中, 高.
volume	Unset	basic template: 低, 中, 高.
emotion	Unset	Emotion label for the emotion template.
dialect	Unset	dialect template: chuanyu dialect, cantonese.
voice_id	Unset	ip template: US_Emma, US_James, UK_George, UK_Charlotte.
reference_text	Empty	Transcript accompanying voice_ref, enabling its acoustic prefix.
speaker_embedding_file	Unset	Precomputed CAMPPlus embedding file or list; see below.
export_speaker_embedding_file	Unset	Export the reference speaker embedding as F32 Safetensors.
use_speaker_embedding	true	Enable reference-speaker embedding conditioning.
zero_speaker_embedding	false	Use a zero projected speaker row, overriding file and reference conditioning.
max_tokens	1000	Maximum generated acoustic patches per text chunk, also bounded by the 512-position context.
guidance_scale	2	Conditioned flow prediction plus this scale times its difference from the null prediction.
sigma	0.25	Nonnegative flow SDE noise scale.
temperature	0	Nonnegative flow sampling temperature; zero still uses seeded initial noise.
seed	1895	Nonnegative reference-posterior and generation noise seed.
raw_text	false	Bypass text chunking and synthesize one fragment.
text_chunk_size	50, 120 or 300	Framework text chunk budget in Unicode codepoints: 50 when hanzi, kana and Hangul outnumber other letters and digits, otherwise 300; multi-chunk generation without speaker conditioning uses at most 120, including explicit overrides, and anchors later chunks to the first generated voice.
text_chunk_mode	tag_aware	default, tag_aware, japanese, or endline.
trim_silence	true	Trim trailing low-energy 100-ms frames per fragment, retaining a 300-ms tail.
stream_frames_per_event	4	Maximum latent patches per streaming decode, from 1 to 64.
stream_chunk_policy	grow	grow starts at one patch and doubles to the maximum; fixed uses the maximum throughout.

Structured templates accept only their matching controls: basic uses speed/pitch/volume, emotion uses emotion, dialect uses dialect, and ip uses voice_id. Supply at least one matching control. Do not combine these templates with raw instruct; use voice_design for a voice description.

Speaker Embedding Files

speaker_embedding_file accepts a 192-dimensional CAMPPlus embedding instead of reference audio: a Torch checkpoint containing a root tensor or a dictionary with spk_emb, speaker_embedding, embedding or emb, or the same named tensor in Safetensors. Shapes [192] and [1, 192] are supported.

Supply one path with --request-option speaker_embedding_file=embedding.pt, or a list in the server's options map. Files must be accessible to the server process. Files supersede reference audio and its transcript; zero_speaker_embedding=true supersedes both. With files provided, use_speaker_embedding=false disables their conditioning without falling back to reference audio.

Export a reference embedding while generating audio:

audiocpp_cli --family ming3_tts --task clon --backend cuda \
  --model /path/to/ming-3.0-tts-0.5b-q8_0.gguf \
  --voice-ref reference.wav --text "Good morning." \
  --request-option export_speaker_embedding_file=speaker.safetensors \
  --out clone.wav --log

Reuse it with --request-option speaker_embedding_file=speaker.safetensors. Export requires reference audio and enabled speaker conditioning, without a speaker-embedding input file or zero embedding. It saves only the speaker embedding, not the acoustic prompt or transcript; loading it is equivalent to reference-only cloning without reference_text.

Session Options

Use --session-option ming3_tts.name=value, or the server model's session_options map with the same family-prefixed key.

Option	Default	Description
reference_cache	true	Cache prepared conditioning for the most recently used reference waveform.

There are no model-specific load options.

Long Text

Reference-conditioned synthesis uses the framework text chunker with a 300-codepoint budget, or 50 codepoints when hanzi, kana and Hangul outnumber other letters and digits. The official Python chunker cuts fragments of 50 semantic units, counting a CJK character like a Latin word; a 300-codepoint CJK chunk does not leave its speech enough of the 512-position context, and its end would be dropped. Multi-chunk generation without speaker conditioning anchors later chunks to the first generated voice, regardless of the prompt template. Every chunk after the first takes the first generated chunk as its reference (speaker embedding, acoustic prompt and transcript), so the voice stays the same across chunks. The reference shares the 512-position context, so automatically anchored chunks are capped at 120 codepoints, including when text_chunk_size requests a larger budget. Smaller overrides remain honored. This is close to the official 24-unit fragments. Inputs that fit in one original chunk keep their existing chunk size. raw_text=true bypasses chunking. The native frontend does not run the optional official FST text normalizer; spell out numbers and abbreviations when pronunciation matters.

Explicit reference audio, speaker-embedding files, built-in voices, and disabled or zero speaker embeddings retain their existing TTS behavior. Automatic TTS anchoring applies in both offline and streaming modes.

Generation has a 512-position context shared between the prompt and emitted acoustic patches. Reduce text_chunk_size or reference duration if the conditioning leaves insufficient generation space. Each acoustic patch represents approximately 80 ms. The default max_tokens=1000 is additionally bounded by that context. trim_silence=true trims each fragment's trailing low-energy audio and retains a short tail.

The flow sampler remains stochastic at temperature=0; use seed for controlled comparisons. guidance_scale controls flow guidance (default 2). New and cached references consume different amounts of posterior noise, as in the official implementation.

Conversion

Use the complete official checkpoint directory, including campplus.onnx, model.safetensors and the tokenizer files. Build audiocpp_gguf first. The Python converter requires NumPy, ONNX and Safetensors; inference does not require ONNX Runtime.

python tools/models/convert_ming3_tts_gguf.py \
  --source /path/to/Ming-3.0-tts-0.5B \
  --output /path/to/ming-3.0-tts-0.5b-bf16.gguf --type orig

python tools/models/convert_ming3_tts_gguf.py \
  --source /path/to/Ming-3.0-tts-0.5B \
  --output /path/to/ming-3.0-tts-0.5b-q8_0.gguf --type q8_0

Conversion extracts CAMPPlus parameters and embeds all required assets in each GGUF. Model weights retain their upstream MIT license; see model licenses.