Fabio
c36510c21e
Some checks failed
Linux Build / Linux cpu (push) Has been cancelled
Linux Build / Linux cpu-clang (push) Has been cancelled
Linux Build / Linux vulkan (push) Has been cancelled
mac-build / macOS CPU (push) Has been cancelled
Nix Build / Nix (cpu) (push) Has been cancelled
Nix Build / Nix (vulkan) (push) Has been cancelled
windows-build / Windows CPU (push) Has been cancelled
101 lines
4.1 KiB
Markdown
101 lines
4.1 KiB
Markdown
# Voxtral TTS
|
|
|
|
Voxtral 4B TTS generates 24 kHz mono speech using a Ministral AR backbone,
|
|
a per-frame acoustic flow-matching transformer, and Voxtral Codec. The
|
|
self-contained GGUF includes the tokenizer, codec decoder, and all 20 presets.
|
|
|
|
Upstream: [mistralai/Voxtral-4B-TTS-2603](https://huggingface.co/mistralai/Voxtral-4B-TTS-2603).
|
|
The released model and reference voices use **CC BY-NC 4.0**.
|
|
|
|
The open checkpoint does not include the codec encoder required for arbitrary
|
|
reference-audio voice cloning. Select an embedded preset instead.
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
audiocpp_cli --task tts --family voxtral_tts \
|
|
--model /path/to/voxtral-4b-tts-2603-bf16.gguf --backend cuda \
|
|
--text "The next train leaves in ten minutes." \
|
|
--request-option voice_id=casual_male --seed 42 \
|
|
--out speech.wav --log
|
|
```
|
|
|
|
No separate codec, tokenizer, or voice files are needed at inference time.
|
|
The BF16 package preserves upstream precision; Q8_0 reduces weight memory and
|
|
was also validated with the embedded presets. Both keep preset embeddings in
|
|
their original BF16 precision.
|
|
|
|
## Request Options
|
|
|
|
Use with `--request-option name=value`.
|
|
|
|
| Option | Default | Description |
|
|
| --- | --- | --- |
|
|
| `voice_id` | `casual_male` | Embedded preset, listed below. The CLI/server voice field can also select it. |
|
|
| `max_tokens` | `2048` | Maximum generated audio frames per text chunk, at 12.5 frames/s. |
|
|
| `num_inference_steps` | `7` | Euler integration steps per acoustic frame, matching the released reference implementation. |
|
|
| `guidance_scale` | `1.2` | Acoustic CFG scale; no CFG is applied to the AR backbone. |
|
|
| `seed` | Random | Acoustic noise seed, also settable through `--seed`. |
|
|
| `text_chunk_size` | `300` | Maximum Unicode codepoints per framework text chunk. |
|
|
| `text_chunk_mode` | `default` | `default`, `tag_aware`, `japanese`, or `endline`. |
|
|
|
|
Semantic tokens use greedy selection, matching the reference implementation.
|
|
|
|
## Embedded Voices
|
|
|
|
| Group | Voice IDs |
|
|
| --- | --- |
|
|
| General presets | `casual_female`, `casual_male`, `cheerful_female`, `neutral_female`, `neutral_male` |
|
|
| Portuguese | `pt_male`, `pt_female` |
|
|
| Dutch | `nl_male`, `nl_female` |
|
|
| Italian | `it_male`, `it_female` |
|
|
| French | `fr_male`, `fr_female` |
|
|
| Spanish | `es_male`, `es_female` |
|
|
| German | `de_male`, `de_female` |
|
|
| Arabic | `ar_male` |
|
|
| Hindi | `hi_male`, `hi_female` |
|
|
|
|
## Session Options
|
|
|
|
Use with `--session-option voxtral_tts.name=value`.
|
|
|
|
| Option | Default | Description |
|
|
| --- | --- | --- |
|
|
| `weight_type` | `native` | Weight storage type; preserves the GGUF tensor types by default. |
|
|
| `kv_cache_type` | `f16` | AR cache storage: `f32`, `f16`, or `bf16`. |
|
|
|
|
## Long-Form and Streaming
|
|
|
|
Long text uses the framework text chunker. Each text chunk uses the selected
|
|
preset and increments the acoustic seed. `--mode streaming` emits new audio
|
|
chunks as generation proceeds, not repeated accumulated waveforms.
|
|
|
|
Codec graphs have fixed initial and continuation capacities. Continuation
|
|
decoding retains bounded left context instead of building a graph for the
|
|
complete recording. Graphs and weights remain owned by the session.
|
|
The AR decode graph grows only when a request needs more cache capacity. The
|
|
runtime retains one prefill graph, replacing it when the prompt length changes.
|
|
|
|
The server uses the existing `/v1/audio/speech` and `/v1/audio/speech/live`
|
|
routes. Select a preset through `voice`, or `request_options.voice_id`.
|
|
Configure the model with `"mode": "streaming"` to use streaming speech:
|
|
|
|
```bash
|
|
curl http://127.0.0.1:8080/v1/audio/speech \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{"model":"voxtral-tts","input":"Paris is a beautiful city!","voice":"casual_male","stream":true,"seed":42}'
|
|
```
|
|
|
|
The SSE `speech.audio.delta` payloads contain new PCM16 mono samples, at 24 kHz.
|
|
Concatenate their decoded audio bytes in order; do not replace preceding chunks.
|
|
|
|
## Conversion
|
|
|
|
```bash
|
|
python tests/voxtral_tts/convert_gguf.py \
|
|
--model-dir /path/to/Voxtral-4B-TTS-2603 \
|
|
--output-dir /path/to/Voxtral-4B-TTS-2603-GGUF
|
|
```
|
|
|
|
The converter retains original-precision transformer weights, folds codec
|
|
weight normalization, and embeds the preset voice embeddings and text tokenizer.
|