# Ming 3.0 TTS 0.5B

Ming uses a Qwen backbone, a continuous-latent flow-matching head, a waveform
VAE and CAMPPlus speaker conditioning. It supports Chinese, English, Japanese,
Korean, German, French and Spanish, with voice cloning and voice design.

GGUF packages: [audio-cpp/Ming-3.0-TTS-0.5B-GGUF](https://huggingface.co/audio-cpp/Ming-3.0-TTS-0.5B-GGUF).

Upstream: [inclusionAI/Ming-3.0-tts](https://github.com/inclusionAI/Ming-3.0-tts).
Output is mono 44.1 kHz. Inference supports CUDA, Vulkan and CPU, with offline
and streaming TTS, voice cloning, and voice design.

## Usage

```bash
audiocpp_cli --family ming3_tts --task tts --backend cuda \
  --model /path/to/ming-3.0-tts-0.5b-bf16.gguf \
  --text "Good evening. The harbor is quiet tonight." \
  --request-option language=en --out speech.wav --log
```

The GGUF includes the backbone, flow head, audio VAE, CAMPPlus speaker encoder,
configuration and tokenizer. No separate codec or Python runtime is needed.
BF16 preserves the original checkpoint dtypes. Q8 retains the small stop head
in BF16 and the speaker encoder in F32.

### Streaming

Add `--mode streaming` to a generation command:

```bash
audiocpp_cli --family ming3_tts --task tts --mode streaming --backend cuda \
  --model /path/to/ming-3.0-tts-0.5b-q8_0.gguf \
  --text "Good evening. The harbor is quiet tonight." \
  --request-option language=en --out speech.wav --log
```

Streaming decodes latent patches while the acoustic generator is running.
The VAE retains enough left context for its causal attention and one patch of
lookahead for interpolation and waveform reconstruction. The default schedule
grows from one patch to four patches per decode. Set
`--request-option stream_chunk_policy=fixed` for fixed-size groups or
`--request-option stream_frames_per_event=8` to change the maximum group size.
The codec retains at most six streaming graph capacities, sharing one set of weights.
Trailing-silence trimming buffers uncommitted audio; emitted audio is never retracted.
The streaming window can produce numerical differences from full offline decoding.

For HTTP streaming, configure the model with `"mode": "streaming"` and use
`POST /v1/audio/speech` with `"response_format": "pcm"` and
`"stream_format": "sse"`; the same voice and generation options apply.

### Voice Cloning

```bash
audiocpp_cli --family ming3_tts --task clon --backend cuda \
  --model /path/to/ming-3.0-tts-0.5b-bf16.gguf \
  --voice-ref reference.wav \
  --request-option reference_text="Words spoken in the reference." \
  --text "Please leave the lamp beside the window." \
  --request-option language=en --out clone.wav --log
```

With `reference_text`, the reference supplies both speaker identity and an
acoustic latent prefix. Omit the transcript to use speaker conditioning only.
Reference encoding samples the VAE posterior, matching the official pipeline's
RNG consumption.

### Voice Design

```bash
audiocpp_cli --family ming3_tts --task vdes --backend cuda \
  --model /path/to/ming-3.0-tts-0.5b-bf16.gguf \
  --text "Good evening. The harbor is quiet tonight." \
  --request-option language=en \
  --request-option instruct="A calm adult man with a clear, warm voice." \
  --out designed.wav --log
```

## Request Options

Use `--request-option name=value`, or the server request's `options` map.

| Option | Default | Description |
| --- | --- | --- |
| `template_name` | `tts`; `voice_design` on `vdes` | `tts`, `multilingual`, `basic`, `emotion`, `dialect`, `ip`, `voice_design`, or `freeform`. |
| `language` | The request's language | Target language name or code: `zh`, `en`, `ja`, `ko`, `de`, `fr`, `es`. Without the option, the server's top-level `language` field or the CLI's `--language` is used. Required for `multilingual`; ignored by `dialect`. |
| `instruct` | Empty | Voice description for `voice_design`, required raw instruction for `freeform`, or optional raw instruction for `tts`. |
| `instruction_schema` | `en` for `dialect`, otherwise `zh` | Language of the structured instruction field names, not the spoken language. Values: `en`, `zh`. |
| `speed` | Unset | `basic` template: `慢速`, `中速`, `快速`. Categorical conditioning, not waveform resampling. |
| `pitch` | Unset | `basic` template: `低`, `中`, `高`. |
| `volume` | Unset | `basic` template: `低`, `中`, `高`. |
| `emotion` | Unset | Emotion label for the `emotion` template. |
| `dialect` | Unset | `dialect` template: `chuanyu dialect`, `cantonese`. |
| `voice_id` | Unset | `ip` template: `US_Emma`, `US_James`, `UK_George`, `UK_Charlotte`. |
| `reference_text` | Empty | Transcript accompanying `voice_ref`, enabling its acoustic prefix. |
| `speaker_embedding_file` | Unset | Precomputed CAMPPlus embedding file or list; see below. |
| `export_speaker_embedding_file` | Unset | Export the reference speaker embedding as F32 Safetensors. |
| `use_speaker_embedding` | `true` | Enable reference-speaker embedding conditioning. |
| `zero_speaker_embedding` | `false` | Use a zero projected speaker row, overriding file and reference conditioning. |
| `max_tokens` | `1000` | Maximum generated acoustic patches per text chunk, also bounded by the 512-position context. |
| `guidance_scale` | `2` | Conditioned flow prediction plus this scale times its difference from the null prediction. |
| `sigma` | `0.25` | Nonnegative flow SDE noise scale. |
| `temperature` | `0` | Nonnegative flow sampling temperature; zero still uses seeded initial noise. |
| `seed` | `1895` | Nonnegative reference-posterior and generation noise seed. |
| `raw_text` | `false` | Bypass text chunking and synthesize one fragment. |
| `text_chunk_size` | `50`, `120` or `300` | Framework text chunk budget in Unicode codepoints: `50` when hanzi, kana and Hangul outnumber other letters and digits, otherwise `300`; multi-chunk generation without speaker conditioning uses at most `120`, including explicit overrides, and anchors later chunks to the first generated voice. |
| `text_chunk_mode` | `tag_aware` | `default`, `tag_aware`, `japanese`, or `endline`. |
| `trim_silence` | `true` | Trim trailing low-energy 100-ms frames per fragment, retaining a 300-ms tail. |
| `stream_frames_per_event` | `4` | Maximum latent patches per streaming decode, from 1 to 64. |
| `stream_chunk_policy` | `grow` | `grow` starts at one patch and doubles to the maximum; `fixed` uses the maximum throughout. |

Structured templates accept only their matching controls: `basic` uses
`speed`/`pitch`/`volume`, `emotion` uses `emotion`, `dialect` uses `dialect`,
and `ip` uses `voice_id`. Supply at least one matching control. Do not combine
these templates with raw `instruct`; use `voice_design` for a voice description.

### Speaker Embedding Files

`speaker_embedding_file` accepts a 192-dimensional CAMPPlus embedding instead
of reference audio: a Torch checkpoint containing a root tensor or a dictionary
with `spk_emb`, `speaker_embedding`, `embedding` or `emb`, or the same named tensor
in Safetensors. Shapes `[192]` and `[1, 192]` are supported.

Supply one path with `--request-option speaker_embedding_file=embedding.pt`,
or a list in the server's `options` map. Files must be accessible to the server
process. Files supersede reference audio and its transcript;
`zero_speaker_embedding=true` supersedes both. With files provided,
`use_speaker_embedding=false` disables their conditioning without falling back
to reference audio.

Export a reference embedding while generating audio:

```bash
audiocpp_cli --family ming3_tts --task clon --backend cuda \
  --model /path/to/ming-3.0-tts-0.5b-q8_0.gguf \
  --voice-ref reference.wav --text "Good morning." \
  --request-option export_speaker_embedding_file=speaker.safetensors \
  --out clone.wav --log
```

Reuse it with `--request-option speaker_embedding_file=speaker.safetensors`.
Export requires reference audio and enabled speaker conditioning, without a
speaker-embedding input file or zero embedding. It saves only the speaker
embedding, not the acoustic prompt or transcript; loading it is equivalent to
reference-only cloning without `reference_text`.

## Session Options

Use `--session-option ming3_tts.name=value`, or the server model's
`session_options` map with the same family-prefixed key.

| Option | Default | Description |
| --- | --- | --- |
| `reference_cache` | `true` | Cache prepared conditioning for the most recently used reference waveform. |

There are no model-specific load options.

## Long Text

Reference-conditioned synthesis uses the framework text chunker with a 300-codepoint budget,
or 50 codepoints when hanzi, kana and Hangul outnumber other letters and
digits. The official Python chunker cuts fragments of 50 semantic units,
counting a CJK character like a Latin word; a 300-codepoint CJK chunk does not
leave its speech enough of the 512-position context, and its end would be
dropped.
Multi-chunk generation without speaker conditioning anchors later chunks to the
first generated voice, regardless of the prompt template. Every chunk after the first
takes the first generated chunk as its reference
(speaker embedding, acoustic prompt and transcript), so the voice
stays the same across chunks. The reference shares the 512-position context,
so automatically anchored chunks are capped at 120 codepoints, including when
`text_chunk_size` requests a larger budget. Smaller overrides remain honored.
This is close to the official 24-unit fragments. Inputs that fit in one original
chunk keep their existing chunk size. `raw_text=true` bypasses chunking. The native frontend
does not run the optional official FST text normalizer; spell out numbers
and abbreviations when pronunciation matters.

Explicit reference audio, speaker-embedding files, built-in voices, and disabled
or zero speaker embeddings retain their existing TTS behavior. Automatic TTS
anchoring applies in both offline and streaming modes.

Generation has a 512-position context shared between the prompt and emitted
acoustic patches. Reduce `text_chunk_size` or reference duration if the
conditioning leaves insufficient generation space. Each acoustic patch
represents approximately 80 ms. The default `max_tokens=1000` is additionally
bounded by that context. `trim_silence=true` trims each fragment's trailing
low-energy audio and retains a short tail.

The flow sampler remains stochastic at `temperature=0`; use `seed` for
controlled comparisons. `guidance_scale` controls flow guidance (default 2).
New and cached references consume different amounts of posterior noise,
as in the official implementation.

## Conversion

Use the complete official checkpoint directory, including `campplus.onnx`,
`model.safetensors` and the tokenizer files. Build `audiocpp_gguf` first.
The Python converter requires NumPy, ONNX and Safetensors; inference does not
require ONNX Runtime.

```bash
python tools/models/convert_ming3_tts_gguf.py \
  --source /path/to/Ming-3.0-tts-0.5B \
  --output /path/to/ming-3.0-tts-0.5b-bf16.gguf --type orig

python tools/models/convert_ming3_tts_gguf.py \
  --source /path/to/Ming-3.0-tts-0.5B \
  --output /path/to/ming-3.0-tts-0.5b-q8_0.gguf --type q8_0
```

Conversion extracts CAMPPlus parameters and embeds all required assets in each
GGUF. Model weights retain their upstream MIT license; see
[model licenses](../model_licenses.md).