voice_audio_multi/docs/models/kokoro_tts.md

# Kokoro 82M

Kokoro 82M is a compact multilingual text-to-speech model exposed as
`--family kokoro_tts`. audio.cpp packages the model as standalone GGUF files
with all 54 upstream voice packs.

## Install

For server speed control, use the Kokoro GGUF version published on
[Hugging Face on September 22, 2026 (UTC)](https://huggingface.co/audio-cpp/audio.cpp-gguf/commit/406756ee8e3b16e902ce40112986c1010775f888)
or a later published version. Its embedded spec declares
`speed`. Older GGUFs missing that declaration must be updated to use non-default
speed through `/v1/audio/speech`; `speed: 1` remains accepted when the declaration is absent.

```bash
python3 tools/model_manager_v2.py install kokoro_82m_q8_0
```

The default package installs:

```text
models/Kokoro-82M-GGUF/kokoro-82m-q8_0.gguf
```

BF16 is also available:

```bash
python3 tools/model_manager_v2.py install kokoro_82m_bf16
```

## Quick Start

```bash
audiocpp_cli --task tts --family kokoro_tts \
  --model models/Kokoro-82M-GGUF/kokoro-82m-q8_0.gguf \
  --backend cpu \
  --language en-us \
  --voice-id af_heart \
  --text "Hello from Kokoro." \
  --out out.wav
```

Chinese example:

```bash
audiocpp_cli --task tts --family kokoro_tts \
  --model models/Kokoro-82M-GGUF/kokoro-82m-q8_0.gguf \
  --backend cpu \
  --language zh \
  --voice-id zf_xiaobei \
  --text "你好，这是中文语音测试。" \
  --out out_zh.wav
```

## Streaming

Use `--mode streaming` to emit each synthesized text chunk immediately. Text
chunking uses the same `text_chunk_size` control as offline
generation; it does not split an already generated waveform into fake streaming
events. A short input that fits in one chunk produces one audio event. Longer
inputs produce incremental, non-cumulative audio chunks and a final merged WAV.

## Model
| Field | Value |
|---|---|
| Family | `kokoro_tts` |
| Task | `tts` |
| Modes | `offline`, `streaming` |
| Default package | `kokoro_82m_q8_0` |
| Other package | `kokoro_82m_bf16` |
| Model file | `models/Kokoro-82M-GGUF/kokoro-82m-q8_0.gguf` |

## Voices and Languages

Use `--voice-id <id>` to select one of the packaged voices. The voice prefix
selects the language family:

| Prefix | Language | Voice IDs |
|---|---|---|
| `af`, `am` | American English | `af_alloy`, `af_aoede`, `af_bella`, `af_heart`, `af_jessica`, `af_kore`, `af_nicole`, `af_nova`, `af_river`, `af_sarah`, `af_sky`, `am_adam`, `am_echo`, `am_eric`, `am_fenrir`, `am_liam`, `am_michael`, `am_onyx`, `am_puck`, `am_santa` |
| `bf`, `bm` | British English | `bf_alice`, `bf_emma`, `bf_isabella`, `bf_lily`, `bm_daniel`, `bm_fable`, `bm_george`, `bm_lewis` |
| `ef`, `em` | Spanish | `ef_dora`, `em_alex`, `em_santa` |
| `ff` | French | `ff_siwis` |
| `hf`, `hm` | Hindi | `hf_alpha`, `hf_beta`, `hm_omega`, `hm_psi` |
| `if`, `im` | Italian | `if_sara`, `im_nicola` |
| `jf`, `jm` | Japanese | `jf_alpha`, `jf_gongitsune`, `jf_nezumi`, `jf_tebukuro`, `jm_kumo` |
| `pf`, `pm` | Brazilian Portuguese | `pf_dora`, `pm_alex`, `pm_santa` |
| `zf`, `zm` | Mandarin Chinese | `zf_xiaobei`, `zf_xiaoni`, `zf_xiaoxiao`, `zf_xiaoyi`, `zm_yunjian`, `zm_yunxi`, `zm_yunxia`, `zm_yunyang` |

The request language must match the selected voice. For example, use
`--language zh` with `zf_*` or `zm_*` voices.

### Voice Blending

Use `af_heart+af_bella` for equal weights, or `af_heart(1)+jf_alpha(0.5)`
for a weighted blend. Weights are finite, nonnegative, and normalized by their
sum, which must be positive. Blending averages voice style embeddings, not WAVs.
The first voice selects the phonemizer language, including cross-language blends;
an explicit language must match that first voice.

```bash
audiocpp_cli --task tts --family kokoro_tts \
  --model models/Kokoro-82M-GGUF/kokoro-82m-q8_0.gguf --backend metal \
  --voice-id 'af_heart(1)+jf_alpha(0.5)' \
  --text "Hello, testing combined Kokoro voices." --out blended.wav
```

The same expression works through `--request-option voice_id=...` or the
`voice` field of `/v1/audio/speech`, with existing GGUFs:

```json
{
  "model": "kokoro-audiocpp",
  "input": "Hello, testing combined Kokoro voices.",
  "voice": "af_heart(1)+jf_alpha(0.5)",
  "response_format": "wav",
  "speed": 1
}
```

MP3 output uses the existing optional `mp3_encode`
[server frontend](../../app/server/api.md).

## Runtime Resources

The release GGUF includes model weights, config, vocabulary, all voice packs,
and the generated `g2p/ja.json` and `g2p/zh.json` tables.

English, Spanish, French, Hindi, Italian, and Portuguese use the shared eSpeak
runtime. Install or package eSpeak data as described in
[`docs/espeak_phonemizer.md`](../espeak_phonemizer.md).

Chinese works from the release GGUF because `g2p/zh.json` is bundled. Avoid
mixed Latin words inside Chinese text unless they are known to map to Kokoro's
vocabulary.

Japanese also needs MeCab and UniDic. The small release GGUF does not include
UniDic because it is large. Export a local full multilingual GGUF with
`--embed-multilingual-resources` if you need Japanese to work from a bundled
package.

## Common Options (use directly)

| Option | Values | Default | Meaning |
|---|---|---:|---|
| `--language` | language code | voice prefix | Language selection precedence: voice-style language, transcript language (CLI --language), request-option language, then voice prefix. Must match the selected voice. |
| `--voice-id` | voice ID or blend expression | `af_heart` | Built-in voice pack or normalized blend. |
| `--seed` | integer | random | Decoder noise seed for the current request; omitted requests choose a fresh random seed. |
| `--speaking-rate` | positive float | `1.0` | Speech speed multiplier. |
| `--text-chunk-size` | integer chars | `240`; `100` for Chinese and Japanese | Long-form chunk size. Chinese and Japanese chunks phonemize to more symbols per character and must stay under the 510-symbol input limit. |

## Request Options (use with `--request-option`)

`--voice-id` takes precedence over `--request-option voice_id=...`.

| Option | Values | Default | Meaning |
|---|---|---:|---|
| `voice_id` | packaged voice ID or blend expression | `af_heart` | Built-in voice pack or normalized blend, also selectable with `--voice-id`. |
| `language` | language code | voice prefix | Language selection precedence: voice-style language, transcript language (CLI --language), request-option language, then voice prefix. Must match the selected voice. |
| `seed` | integer | random | Decoder noise seed for the current request; omitted requests choose a fresh random seed. |
| `speed` | positive float | `1.0` | Speech speed multiplier; default 1.0. Must be finite and positive. |
| `text_chunk_size` | integer chars | `240`; `100` for Chinese and Japanese | Long-form chunk size. |

## Session Options (use with `--session-option`)

| Option | Values | Default | Meaning |
|---|---|---:|---|
| `kokoro_tts.weight_type` | `native`, `f32`, `f16`, `bf16`, `q8_0` | `native` | Matmul weight storage type. |
| `kokoro_tts.conv_weight_type` | `native`, `f32`, `f16` | `native` | Convolution weight storage type. |
| `kokoro_tts.predictor_duration_graph_arena_mb` | positive integer MiB | `384` | Duration predictor host graph-metadata arena. |
| `kokoro_tts.predictor_text_graph_arena_mb` | positive integer MiB | `256` | Text predictor host graph-metadata arena. |
| `kokoro_tts.predictor_tail_graph_arena_mb` | positive integer MiB | `640` | Tail predictor host graph-metadata arena. |
| `kokoro_tts.exec_plan` | JSON object of component plans, encoded as a string | `{}` | Select the CPU NEON generator with `{"generator":{"impl":"neon"}}` (`ENGINE_ENABLE_NEON=ON`, AArch64 CPU session). The complete generator uses a NEON execution plan; harmonic conditioning and iSTFT are unchanged. Select RKNN with `{"generator":{"impl":"rknn","artifacts":{"main":"/models/generator.rknn"},"params":{"overlap":0,"gain":false}}}` (`ENGINE_ENABLE_RKNN=ON`); see [component execution plans](../build/rockchip-npu.md#component-execution-plans). Default retains GGML. |

The RKNN generator accepts these fields inside the plan's `generator.params` object, not as
separate session options. GGML accepts no parameters.

| Parameter | Values | Default | Meaning |
|---|---|---:|---|
| `overlap` | integer latent frames, `0` .. emitted-window-1 | `0` | Advance the window walk by `emitted - overlap` frames and blend the shared spectrogram columns with a raised-cosine ramp instead of hard-writing them. Measured on RK3588: 1.54x wall for no change in the per-window loudness spread (7.23 → 7.14 dB), so it is a seam-smoothing knob, not a loudness one. |
| `gain` | boolean | `false` | Correct each NPU window's loudness toward a full-length CPU decode of the same latent (additive gain on the log-magnitude rows), and cap the top five log-magnitude rows at that reference's level at every column, which is what removes the near-Nyquist clicks (see below). Measured on RK3588: per-window loudness spread 7.23 → 3.27 dB and the worst of the top eight 9-12 kHz columns 14.7x → 2.8x the CPU render's energy there, for 1.9x wall and +601 MB RSS, because the ggml generator graph stays loaded and runs once per utterance. |

The graph-arena controls do not size device workspace. Their former
`predictor_duration_graph_mb`, `predictor_text_graph_mb`, and
`predictor_tail_graph_mb` names remain accepted with the `kokoro_tts.` prefix,
including with older embedded specs.

On RK3588 the generator is 83% of the runtime, so the NPU path is worth the extra file: measured
RTF 1.415 → 0.867 and peak RSS 2777 → 2334 MB on `kokoro-82m-q8_0` (185 Russian characters,
`--threads 4`, clipped-sine engine; the range-reduced engine the producer now exports by default
costs +21..36% generator time depending on the utterance, and buys 63 dB of sine accuracy, see the
RKNN document). The engine is compiled for a 128-latent-frame window and the runtime walks it over
the latent; see the RKNN document for the window limit and the per-window normalization caveat. The
NPU render also carries a few content-locked near-Nyquist clicks per utterance (2 to 3 on a 26 s
render): at the columns where the graph's activations peak, the top iSTFT bins run 10-24x the CPU
render's 9-12 kHz energy there. They are fp16 rounding in the convolutional accumulation, not the
sine approximation and not the window seams: the same columns at the same magnitude for the clip
sine, the range-reduced sine, the exact `Sin` node, the unfolded Snake (`--no-snake-fold`, which
bounds the convolution input) and the w128/w136 grids, and RKNN 2.3.2 has no per-layer dtype knob.
`generator.params.gain=true` in the execution plan caps them (14.7x → 2.8x at the worst of the top eight columns,
residual 1.4-2.8x is the iSTFT Hann-overlap smear into the bins the cap does not touch).

A `--global-stats` variant of the engine — the per-channel AdaIN mean/variance of a full-length
decode fed into the engine as graph I/O, so every window normalizes with the whole utterance's
statistics — is not in the build. In fp32 it is exact (the stitched windowed decode moves from
9.7 dB to 135.4 dB against the full-length decode), and that is exactly why it was dropped: on the
RK3588 the fed statistics scale the window activations past the engine's fp16 range and the graph
collapses — every window after the first renders as a constant-energy tone (rms flat at 51.5 dB,
spectral flatness 1.7e-17, energy at DC, 4800 Hz and 9600 Hz), for 1.8x the wall time and +1.5 GB
RSS. The producer, the runtime and the engine contract carry no code for it.

## Conversion

Convert from the official `hexgrad/Kokoro-82M` source checkout:

```bash
python tools/models/prepare_kokoro_gguf.py \
  --source /path/to/Kokoro-82M \
  --output-dir /path/to/Kokoro-82M-GGUF \
  --type both \
  --overwrite
```

For a fully bundled local multilingual package with eSpeak data and UniDic:

```bash
python tools/models/prepare_kokoro_gguf.py \
  --source /path/to/Kokoro-82M \
  --output-dir /path/to/Kokoro-82M-GGUF \
  --type both \
  --embed-multilingual-resources \
  --overwrite
```

The detailed validation notes live in
[`tests/kokoro_tts/MULTILINGUAL_GGUF.md`](../../tests/kokoro_tts/MULTILINGUAL_GGUF.md).

## Phoneme-group timings

Kokoro predicts a per-token frame count before the decoder runs, and the decoder upsamples by
exactly those counts, so where each unit lands in the output is known rather than estimated. Ask
for it with the `return_timestamps` request option, which is off by default:

```bash
audiocpp_cli --task tts --family kokoro_tts --model /path/to/Kokoro-82M-GGUF --backend cpu \
  --language en --text "The button was forgotten on the cotton coat." --voice-id af_heart \
  --out out.wav --words-out words.json
```

`--words-out` sets the option for you. The entries arrive in `word_timestamps`
(`audiocpp_result_word()` on the C API).

⚠ **These are phoneme groups, not written words, and they do not map one-to-one onto the input
text.** Read the next section before joining them to anything.

### What a group is, and what it is not

A group is a run of tokens between the space tokens Kokoro's vocabulary carries, labelled with its
own phonemes. Nothing in this family maps tokens back to the input text: the built-in G2P keeps no
span, and on the supplied-phoneme path there is no text being spoken at all. So the boundaries are
the model's, and whose they are depends on which path produced them:

- **Supplied phonemes.** The caller's own G2P chose the spacing, so the caller already knows which
  of its words became which group and can join the two in order.
- **Built-in G2P (the text path).** eSpeak-ng chose the spacing, and *it merges function words*.
  `on the` becomes the single group `ɔnðə`, `at a` becomes `æTə`, `in the` becomes `ɪnðə`. Three of
  five ordinary English sentences tested this way produced fewer groups than words:

  | text | words | groups |
  |---|---|---|
  | The button was forgotten **on the** cotton coat. | 8 | 7 |
  | She read the schedule aloud **at a** quarter past three. | 10 | 9 |
  | He said it was **in the** box under the table. | 10 | 9 |
  | Uranium and aluminium are both elements. | 6 | 6 |
  | I went to the store and bought a loaf of bread. | 11 | 11 |

  **Zipping these onto whitespace-split words is therefore wrong**, and wrong silently — the counts
  differ only sometimes, and where they do every later word is off by one. A caller that needs a
  written-word timeline from the text path needs its own G2P and the supplied-phoneme path.

The spans follow output order but need not be contiguous or cover the full buffer.
Space tokens, standalone punctuation, and padding can leave gaps between or around groups.

### Details

- **Punctuation the G2P spaced off is not reported.** An opening quote or a standalone dash gets no
  entry, because reporting one would shift a caller joining in order. Its duration is still
  consumed, so the following group starts after it. A mark attached to a word (`lˈɛft.`) stays in
  that word's span, which means a sentence-final pause falls inside the last word rather than after
  it — matching the reference implementation.
- **Spans are in output samples** at the result's own sample rate, offset across chunks so a
  multi-chunk render is one continuous timeline.
- **`confidence` is always 0.** The model does not score its own duration prediction and any number
  there would be read as one.
- **`audiocpp_model_supports_timestamps()` reports 0** for this family, and that is deliberate: the
  capability describes a written-word timeline, which this is not. Pass the option and read the
  result.