Fabio
c36510c21e
4.1 KiB
Voxtral TTS
Voxtral 4B TTS generates 24 kHz mono speech using a Ministral AR backbone, a per-frame acoustic flow-matching transformer, and Voxtral Codec. The self-contained GGUF includes the tokenizer, codec decoder, and all 20 presets.
Upstream: mistralai/Voxtral-4B-TTS-2603. The released model and reference voices use CC BY-NC 4.0.
The open checkpoint does not include the codec encoder required for arbitrary reference-audio voice cloning. Select an embedded preset instead.
Quick Start
audiocpp_cli --task tts --family voxtral_tts \
--model /path/to/voxtral-4b-tts-2603-bf16.gguf --backend cuda \
--text "The next train leaves in ten minutes." \
--request-option voice_id=casual_male --seed 42 \
--out speech.wav --log
No separate codec, tokenizer, or voice files are needed at inference time. The BF16 package preserves upstream precision; Q8_0 reduces weight memory and was also validated with the embedded presets. Both keep preset embeddings in their original BF16 precision.
Request Options
Use with --request-option name=value.
| Option | Default | Description |
|---|---|---|
voice_id |
casual_male |
Embedded preset, listed below. The CLI/server voice field can also select it. |
max_tokens |
2048 |
Maximum generated audio frames per text chunk, at 12.5 frames/s. |
num_inference_steps |
7 |
Euler integration steps per acoustic frame, matching the released reference implementation. |
guidance_scale |
1.2 |
Acoustic CFG scale; no CFG is applied to the AR backbone. |
seed |
Random | Acoustic noise seed, also settable through --seed. |
text_chunk_size |
300 |
Maximum Unicode codepoints per framework text chunk. |
text_chunk_mode |
default |
default, tag_aware, japanese, or endline. |
Semantic tokens use greedy selection, matching the reference implementation.
Embedded Voices
| Group | Voice IDs |
|---|---|
| General presets | casual_female, casual_male, cheerful_female, neutral_female, neutral_male |
| Portuguese | pt_male, pt_female |
| Dutch | nl_male, nl_female |
| Italian | it_male, it_female |
| French | fr_male, fr_female |
| Spanish | es_male, es_female |
| German | de_male, de_female |
| Arabic | ar_male |
| Hindi | hi_male, hi_female |
Session Options
Use with --session-option voxtral_tts.name=value.
| Option | Default | Description |
|---|---|---|
weight_type |
native |
Weight storage type; preserves the GGUF tensor types by default. |
kv_cache_type |
f16 |
AR cache storage: f32, f16, or bf16. |
Long-Form and Streaming
Long text uses the framework text chunker. Each text chunk uses the selected
preset and increments the acoustic seed. --mode streaming emits new audio
chunks as generation proceeds, not repeated accumulated waveforms.
Codec graphs have fixed initial and continuation capacities. Continuation decoding retains bounded left context instead of building a graph for the complete recording. Graphs and weights remain owned by the session. The AR decode graph grows only when a request needs more cache capacity. The runtime retains one prefill graph, replacing it when the prompt length changes.
The server uses the existing /v1/audio/speech and /v1/audio/speech/live
routes. Select a preset through voice, or request_options.voice_id.
Configure the model with "mode": "streaming" to use streaming speech:
curl http://127.0.0.1:8080/v1/audio/speech \
-H 'Content-Type: application/json' \
-d '{"model":"voxtral-tts","input":"Paris is a beautiful city!","voice":"casual_male","stream":true,"seed":42}'
The SSE speech.audio.delta payloads contain new PCM16 mono samples, at 24 kHz.
Concatenate their decoded audio bytes in order; do not replace preceding chunks.
Conversion
python tests/voxtral_tts/convert_gguf.py \
--model-dir /path/to/Voxtral-4B-TTS-2603 \
--output-dir /path/to/Voxtral-4B-TTS-2603-GGUF
The converter retains original-precision transformer weights, folds codec weight normalization, and embeds the preset voice embeddings and text tokenizer.