Fabio
c36510c21e
Some checks failed
Linux Build / Linux cpu (push) Has been cancelled
Linux Build / Linux cpu-clang (push) Has been cancelled
Linux Build / Linux vulkan (push) Has been cancelled
mac-build / macOS CPU (push) Has been cancelled
Nix Build / Nix (cpu) (push) Has been cancelled
Nix Build / Nix (vulkan) (push) Has been cancelled
windows-build / Windows CPU (push) Has been cancelled
289 lines
16 KiB
Markdown
289 lines
16 KiB
Markdown
# IndexTTS
|
|
|
|
IndexTTS is wired as `--family index_tts2` for offline text to speech and voice
|
|
cloning. The same model family handles IndexTTS2 and IndexTTS2.5; the runtime
|
|
selects the variant from the model config.
|
|
|
|
## IndexTTS2
|
|
|
|
IndexTTS2 is a Chinese and English TTS model with voice cloning and expressive
|
|
emotion controls. It requires a speaker reference through the framework
|
|
`--voice-ref` path.
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Family | `index_tts2` |
|
|
| Model directory | `models/IndexTTS-2` |
|
|
| Task | `tts`, `clon` |
|
|
| Modes | `offline` |
|
|
| Languages | `zh`, `en` |
|
|
| Voice input | Required reference WAV through `--voice-ref` |
|
|
| Built-in voices | Not exposed |
|
|
|
|
Voice clone:
|
|
|
|
```bash
|
|
audiocpp_cli --task clon --family index_tts2 \
|
|
--model /path/to/IndexTTS-2 \
|
|
--backend cuda \
|
|
--language en \
|
|
--text "Hello from IndexTTS2." \
|
|
--voice-ref /path/to/reference.wav \
|
|
--out out.wav
|
|
```
|
|
|
|
Emotion text:
|
|
|
|
```bash
|
|
audiocpp_cli --task tts --family index_tts2 \
|
|
--model /path/to/IndexTTS-2 \
|
|
--backend cuda \
|
|
--language zh \
|
|
--text "今天的演示会更有情绪。" \
|
|
--voice-ref /path/to/reference.wav \
|
|
--emotion "你吓死我了！你是鬼吗？" \
|
|
--request-option emotion_alpha=0.6 \
|
|
--out out.wav
|
|
```
|
|
|
|
### IndexTTS2 Common Options (use directly)
|
|
|
|
| Option | Values | Default | Meaning |
|
|
|---|---|---:|---|
|
|
| `--voice-ref` | WAV path | required | Reference speaker audio. |
|
|
| `--audio` | WAV path | not set | Emotion-reference audio. |
|
|
| `--language` | `zh`, `en` | empty | Text language label. |
|
|
| `--emotion` | text | not set | Emotion-text conditioning through the framework style field. |
|
|
| `--text-chunk-size` | characters | not set | Optional framework long-form text chunk size; omitted keeps the full input together. Must be positive. |
|
|
| `--text-chunk-mode` | `default`, `tag_aware`, `japanese`, `endline` | `default` | Framework chunking mode used only when `--text-chunk-size` is set. |
|
|
| `--max-tokens` | integer | model default | Maximum generated GPT mel tokens. |
|
|
| `--temperature` | float | model default | Sampling temperature. Must be positive. |
|
|
| `--top-p` | float | model default | Nucleus sampling probability. Must be in (0, 1]. |
|
|
| `--top-k` | integer | model default | GPT top-k sampling limit. |
|
|
| `--repetition-penalty` | float | model default | GPT repetition penalty. |
|
|
| `--do-sample` | `true`, `false` | model default | Enable stochastic GPT sampling. |
|
|
| `--num-beams` | integer | `3` | GPT beam count. |
|
|
| `--seed` | integer | random when omitted | Generation seed. |
|
|
|
|
### IndexTTS2 Request Options (use with `--request-option`)
|
|
|
|
| Option | Values | Default | Meaning |
|
|
|---|---|---:|---|
|
|
| `text` | text | not set | Fallback target text when text_input is absent; prompt is also accepted. |
|
|
| `language` | text | not set | IndexTTS2.5 text language; empty or auto selects from text. |
|
|
| `emotion_alpha` | float in `[0, 1]` | `1` | Blend strength for explicit emotion conditioning. |
|
|
| `emotion_vector` | 8 floats | not set | Explicit emotion vector. |
|
|
| `use_emotion_text` | bool | `false` | Infer emotion from text. |
|
|
| `emotion_text` | text | not set | Text used when emotion-text conditioning is enabled. |
|
|
| `use_random_emotion` | bool | `false` | Use random emotion weights in the emotion mixer. |
|
|
| `interval_silence_ms` | milliseconds | `200` | Silence inserted between generated text chunks. |
|
|
| `do_sample` | bool | `true` | Enable sampling. |
|
|
| `top_p` | float | `0.8` | Nucleus sampling probability. Must be in (0, 1]. |
|
|
| `top_k` | integer >= 1 | `30` | Top-k sampling limit. |
|
|
| `temperature` | float | `0.8` | Sampling temperature. Must be positive. |
|
|
| `length_penalty` | float | `0` | GPT beam-search length penalty. |
|
|
| `num_beams` | integer >= 1 | `3` | Beam search width. |
|
|
| `repetition_penalty` | float | `10` | Sampling repetition penalty. |
|
|
| `max_tokens` | integer >= 1 | `1500` | Maximum generated mel tokens. |
|
|
| `duration_scale` | positive float | `1` | Positive waveform duration rescaling, supported by both IndexTTS2 and IndexTTS2.5. |
|
|
| `seed` | integer >= 0 | random | Sampling seed; omitted chooses a random seed. |
|
|
| `text_chunk_size` | integer | not set | Optional framework long-form text chunk size; omitted keeps the full input together. Must be positive. |
|
|
| `text_chunk_mode` | `default`, `tag_aware`, `japanese`, `endline` | `default` | Framework text chunking mode. |
|
|
|
|
### IndexTTS2 Session Options (use with `--session-option`)
|
|
|
|
| Option | Values | Default | Meaning |
|
|
|---|---|---:|---|
|
|
| `index_tts2.weight_type` | `native`, `f32`, `f16`, `bf16`, `q8_0` | not set | Matmul weight storage type. |
|
|
| `index_tts2.conv_weight_type` | `native`, `f32`, `f16` | `native` | Convolution weight storage type. |
|
|
| `index_tts2.mem_saver` | bool | `false` | Release staged reference and conditioning graphs after request phases. |
|
|
| `index_tts2.speaker_cache_slots` | integer slots | `1` | Prepared speaker-reference cache slots; set `0` to disable reuse. |
|
|
| `index_tts2.emotion_cache_slots` | integer slots | `1` | Prepared emotion-reference cache slots; set `0` to disable reuse. |
|
|
| `index_tts2.emotion_text_cache_slots` | integer slots | `1` | Emotion-text weight cache slots; set `0` to disable reuse. |
|
|
| `index_tts2.gpt_graph_arena_mb` | MB | `32` | GPT graph arena size. |
|
|
| `index_tts2.s2mel_graph_arena_mb` | MB | `32` | S2Mel graph arena size. |
|
|
| `index_tts2.reference_graph_arena_mb` | MB | `512` | Reference encoder and codec graph arena size. |
|
|
| `index_tts2.emotion_text_prefill_graph_arena_mb` | MB | `2048` | Emotion-text prefill graph arena size. |
|
|
| `index_tts2.emotion_text_decode_graph_arena_mb` | MB | `512` | Emotion-text cached-step graph arena size. |
|
|
| `index_tts2.weight_context_mb` | MB | `32` | Shared ggml weight metadata context size. |
|
|
| `index_tts2.emotion_text_max_tokens` | tokens | `256` | Maximum generated tokens for emotion-text classification; old name `index_tts2.emotion_text_max_new_tokens` is still accepted. |
|
|
|
|
Text normalization note: audio.cpp reimplements the official IndexTTS text
|
|
front-end (wetext-style number/date/symbol rules, pinyin-tone and name
|
|
protection) as lightweight C++ rules instead of running the official OpenFst
|
|
grammars. Alignment is tracked by a golden-corpus parity test
|
|
(`index_tts_tn_corpus_test`, which replays 50+ tricky cases against outputs
|
|
generated by the official Python normalizer), but a rule-based port cannot be
|
|
exhaustive: for unusual inputs (letter-attached digits, rare date/unit formats,
|
|
emails, URLs) the normalized text can still differ from the official pipeline,
|
|
so the same prompt may be read differently. This is a text-frontend difference,
|
|
not a model-quality problem; the GPT/codec/S2Mel stack itself matches the
|
|
official pipeline token-for-token on the validated cases.
|
|
|
|
## IndexTTS2.5
|
|
|
|
IndexTTS2.5 is IndexTeam/bilibili's multilingual zero-shot TTS model (released
|
|
2026-07): a 0.8B GPT autoregressive model + DiT CFM + BigVGAN stack that keeps
|
|
IndexTTS2's timbre-emotion decoupling and adds Japanese, Spanish, and Arabic on
|
|
top of Chinese and English. It requires a speaker reference through the
|
|
framework `--voice-ref` path. Inline `<文字|发音>` pronunciation overrides
|
|
(pinyin, CMU phonemes, or kana) are supported. Upstream weights live at
|
|
[IndexTeam/IndexTTS-2.5](https://huggingface.co/IndexTeam/IndexTTS-2.5); the
|
|
reference implementation is [index-tts/index-tts](https://github.com/index-tts/index-tts)
|
|
branch `indextts-2.5`.
|
|
|
|
IndexTTS2.5 is implemented as a variant of the `index_tts2` family rather than
|
|
a separate family: both variants share the audio features, wav2vec2bert, Qwen
|
|
emotion, style encoder, BigVGAN vocoder, S2Mel, and GPT decode/cache code. The
|
|
tokenizer, GPT speaker conditioning, and semantic-codec decode path are selected
|
|
per variant from the model config `version` field (`"2.5"`). All
|
|
`index_tts2.*` session options apply to both variants.
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Family | `index_tts2` |
|
|
| Variant | `2.5`, selected from the model config `version` field |
|
|
| Model directory | `models/IndexTTS2.5-GGUF` |
|
|
| Default package | `index_tts2_5_q8_0` |
|
|
| Other packages | `index_tts2_5_f16`, `index_tts2_5_orig` |
|
|
| Task | `tts`, `clon` |
|
|
| Modes | `offline` |
|
|
| Languages | `zh`, `en`, `ja`, `es`, `ar` |
|
|
| Voice input | Required reference WAV through `--voice-ref` |
|
|
| Built-in voices | Not exposed |
|
|
|
|
Voice clone:
|
|
|
|
```bash
|
|
audiocpp_cli --task clon --family index_tts2 \
|
|
--model /path/to/IndexTTS2.5-GGUF \
|
|
--backend cuda \
|
|
--text "Hello from IndexTTS2.5." \
|
|
--voice-ref /path/to/reference.wav \
|
|
--out out.wav
|
|
```
|
|
|
|
Emotion text:
|
|
|
|
```bash
|
|
audiocpp_cli --task tts --family index_tts2 \
|
|
--model /path/to/IndexTTS2.5-GGUF \
|
|
--backend cuda \
|
|
--text "今天的演示会更有情绪。" \
|
|
--voice-ref /path/to/reference.wav \
|
|
--emotion "你吓死我了！你是鬼吗？" \
|
|
--request-option emotion_alpha=0.6 \
|
|
--out out.wav
|
|
```
|
|
|
|
The `language` request option selects the text language (`auto`, `zh`, `en`,
|
|
`ja`, `es`, `ar`, or any tokenizer language code). The default `auto` picks
|
|
`zh` when the text contains Han characters and `en` otherwise, so mixed
|
|
Japanese, Spanish, or Arabic text should set
|
|
`--request-option language=ja|es|ar` explicitly.
|
|
|
|
Emotion conditioning supports all three IndexTTS2 paths: an emotion reference
|
|
WAV through `--audio`, an explicit `emotion_vector`, and Qwen-based emotion-text
|
|
classification through `--emotion` / `use_emotion_text`. See the IndexTTS2
|
|
text-normalization note above for the zh/en caveat. Known limitations: the
|
|
Spanish NeMo text normalizer is not ported, so es text passes through without
|
|
upstream-style normalization. The official ja path has no NeMo grammar, so NeMo
|
|
TN is a no-op for Japanese upstream; the official ja wakachi/fugashi word
|
|
segmentation is also not ported, so ja text is tokenized unsegmented. It is
|
|
intelligible, but not token-identical to the official pipeline.
|
|
|
|
License: IndexTTS-2.5 weights are distributed under the bilibili Model Use
|
|
License, which is not OSI-approved. It requires separate commercial
|
|
authorization when monthly active users exceed 100 million or annual revenue
|
|
exceeds 1 billion RMB, and it forbids using model outputs to improve other AI
|
|
models. Check the upstream repository for the full terms before redistribution
|
|
or commercial use.
|
|
|
|
### IndexTTS2.5 Common Options (use directly)
|
|
|
|
| Option | Values | Default | Meaning |
|
|
|---|---|---:|---|
|
|
| `--voice-ref` | WAV path | required | Reference speaker audio. |
|
|
| `--audio` | WAV path | not set | Emotion-reference audio. |
|
|
| `--emotion` | text | not set | Emotion-text conditioning through the framework style field. |
|
|
| `--text-chunk-size` | characters | not set | Optional framework long-form text chunk size; omitted keeps the full input together. Must be positive. |
|
|
| `--text-chunk-mode` | `default`, `tag_aware`, `japanese`, `endline` | `default` | Framework chunking mode used only when `--text-chunk-size` is set. |
|
|
| `--max-tokens` | integer | `1500` | Maximum generated GPT mel tokens. |
|
|
| `--temperature` | float | `0.8` | Sampling temperature. Must be positive. |
|
|
| `--top-p` | float | `0.8` | Nucleus sampling probability. Must be in (0, 1]. |
|
|
| `--top-k` | integer | `30` | GPT top-k sampling limit. |
|
|
| `--repetition-penalty` | float | `10.0` | GPT repetition penalty. |
|
|
| `--do-sample` | `true`, `false` | `true` | Enable stochastic GPT sampling. |
|
|
| `--seed` | integer | random when omitted | Generation seed. |
|
|
|
|
### IndexTTS2.5 Request Options (use with `--request-option`)
|
|
|
|
| Option | Values | Default | Meaning |
|
|
|---|---|---:|---|
|
|
| `text` | text | not set | Fallback target text when text_input is absent; prompt is also accepted. |
|
|
| `language` | `auto`, `zh`, `en`, `ja`, `es`, `ar`, ... | not set | Text language hint; `auto` infers `zh` when the text contains Han characters, otherwise `en`. |
|
|
| `emotion_alpha` | float in `[0, 1]` | `1` | Blend strength for explicit emotion conditioning. |
|
|
| `emotion_vector` | 8 floats | not set | Explicit emotion vector. |
|
|
| `use_emotion_text` | bool | `false` | Infer emotion from text. |
|
|
| `emotion_text` | text | not set | Text used when emotion-text conditioning is enabled. |
|
|
| `use_random_emotion` | bool | `false` | Use random emotion weights in the emotion mixer. |
|
|
| `interval_silence_ms` | milliseconds | `200` | Silence inserted between generated text chunks. |
|
|
| `do_sample` | bool | `true` | Enable sampling. |
|
|
| `top_p` | float | `0.8` | Nucleus sampling probability. Must be in (0, 1]. |
|
|
| `top_k` | integer >= 1 | `30` | Top-k sampling limit. |
|
|
| `temperature` | float | `0.8` | Sampling temperature. Must be positive. |
|
|
| `length_penalty` | float | `0` | GPT beam-search length penalty. |
|
|
| `num_beams` | integer | `3` | GPT beam count. |
|
|
| `repetition_penalty` | float | `10` | Sampling repetition penalty. |
|
|
| `max_tokens` | integer >= 1 | `1500` | Maximum generated mel tokens. |
|
|
| `duration_scale` | positive float | `1` | Positive waveform duration rescaling, supported by both IndexTTS2 and IndexTTS2.5. |
|
|
| `seed` | integer >= 0 | random | Sampling seed; omitted chooses a random seed. |
|
|
| `text_chunk_size` | integer | not set | Optional framework long-form text chunk size; omitted keeps the full input together. Must be positive. |
|
|
| `text_chunk_mode` | `default`, `tag_aware`, `japanese`, `endline` | `default` | Framework text chunking mode. |
|
|
|
|
### IndexTTS2.5 Session Options (use with `--session-option`)
|
|
|
|
| Option | Values | Default | Meaning |
|
|
|---|---|---:|---|
|
|
| `index_tts2.weight_type` | `native`, `f32`, `f16`, `bf16`, `q8_0` | not set | Matmul weight storage type. |
|
|
| `index_tts2.conv_weight_type` | `native`, `f32`, `f16` | `native` | Convolution weight storage type. |
|
|
| `index_tts2.mem_saver` | bool | `false` | Release staged reference and conditioning graphs after request phases. |
|
|
| `index_tts2.speaker_cache_slots` | integer slots | `1` | Prepared speaker-reference cache slots; set `0` to disable reuse. |
|
|
| `index_tts2.emotion_cache_slots` | integer slots | `1` | Prepared emotion-reference cache slots; set `0` to disable reuse. |
|
|
| `index_tts2.emotion_text_cache_slots` | integer slots | `1` | Emotion-text weight cache slots; set `0` to disable reuse. |
|
|
| `index_tts2.gpt_graph_arena_mb` | MB | `32` | GPT graph arena size. |
|
|
| `index_tts2.s2mel_graph_arena_mb` | MB | `32` | S2Mel graph arena size. |
|
|
| `index_tts2.reference_graph_arena_mb` | MB | `512` | Reference encoder and codec graph arena size. |
|
|
| `index_tts2.emotion_text_prefill_graph_arena_mb` | MB | `2048` | Emotion-text prefill graph arena size. |
|
|
| `index_tts2.emotion_text_decode_graph_arena_mb` | MB | `512` | Emotion-text cached-step graph arena size. |
|
|
| `index_tts2.weight_context_mb` | MB | `32` | Shared ggml weight metadata context size. |
|
|
| `index_tts2.emotion_text_max_tokens` | tokens | `256` | Maximum generated tokens for emotion-text classification; old name `index_tts2.emotion_text_max_new_tokens` is still accepted. |
|
|
|
|
## Converting IndexTTS2.5 From Upstream Weights
|
|
|
|
`tools/community_models/convert_index_tts2_5.py` turns an official `IndexTeam/IndexTTS-2.5`
|
|
snapshot (the `.pth` checkpoints) into the Safetensors staging layout the
|
|
engine expects, and prints or runs the matching `audiocpp_gguf` command. The
|
|
w2v-bert-2.0, CAMPPlus, and BigVGAN checkpoints are auto-detected under
|
|
`<model-dir>/hf_cache/` after running official inference once to populate it,
|
|
and each has an explicit override flag:
|
|
|
|
```bash
|
|
python tools/community_models/convert_index_tts2_5.py \
|
|
--model-dir /path/to/IndexTTS-2.5 \
|
|
--output-dir /path/to/staging \
|
|
--run-converter /path/to/audiocpp_gguf --type q8_0
|
|
```
|
|
|
|
Pass `--native-dir /path/to/IndexTTS-2.5-native` to also emit a directly
|
|
loadable native Safetensors model directory hardlinked from the staging files,
|
|
without GGUF conversion.
|
|
|
|
The script repackages the checkpoints the loader needs, unwraps the
|
|
`s2mel.pth`/`codec.pth` container keys, prefixes CAMPPlus tensors with
|
|
`speaker_encoder.`, strips BigVGAN's `generator.` prefix, wraps the
|
|
`feat1/feat2.pt` matrices as a single `tensor`, and assembles the sidecar
|
|
`root/` config/tokenizer resources that get embedded into the GGUF. The staged
|
|
`config.yaml` has its `version` field normalized to `"2.5"` because the
|
|
official snapshot ships `version: 2.0`; the engine uses that field to select the
|
|
IndexTTS2 family variant.
|