voice_audio_multi/docs/models/voxtral_realtime.md

Voxtral Realtime

Voxtral Realtime is a Mistral realtime ASR model with offline and streaming sessions. The model manager installs the Q8_0 standalone GGUF package by default; native Hugging Face directories and other standalone GGUF variants can also be used when provided directly. A Q4_K GGUF package is available for lower memory use and faster CUDA runs; in a quick path check its transcripts matched Q8_0 except for one capitalization-only difference.

Model

Field	Value
Family	voxtral_realtime
Model path	models/Voxtral-Mini-4B-Realtime-2602-GGUF/voxtral-mini-4b-realtime-2602-q8_0.gguf when installed through the model manager
GGUF variants	bf16, q8_0, q4_k
Task	asr
Modes	offline, streaming
Output	Transcription text
Streaming input	Audio chunks
Timestamps	Not exposed

CLI

Offline:

audiocpp_cli --task asr --family voxtral_realtime --model models/Voxtral-Mini-4B-Realtime-2602-GGUF/voxtral-mini-4b-realtime-2602-q8_0.gguf --backend cuda --threads 8 --audio assets/resources/sample.wav --text-out transcript.txt

Sampling and token-cap options can be passed through request options:

audiocpp_cli --task asr --family voxtral_realtime --model models/Voxtral-Mini-4B-Realtime-2602-GGUF/voxtral-mini-4b-realtime-2602-q8_0.gguf --backend cuda --threads 8 --audio assets/resources/sample.wav --text-out transcript.txt --request-option max_new_tokens=256 --do-sample false --temperature 1.0 --top-p 1.0 --top-k 50 --seed 1234

Streaming:

audiocpp_cli --task asr --family voxtral_realtime --model models/Voxtral-Mini-4B-Realtime-2602-GGUF/voxtral-mini-4b-realtime-2602-q8_0.gguf --backend cuda --threads 8 --mode streaming --audio assets/resources/sample.wav --text-out transcript.txt

Live Streaming Input

--audio - reads raw, headerless interleaved PCM from stdin and feeds it to the model chunk by chunk as it arrives, so the audio is never buffered up front and does not have to exist as a file. Any capture tool that can write PCM to a pipe works as the source:

# Microphone (macOS; use -f alsa on Linux or -f dshow on Windows)
ffmpeg -f avfoundation -i ":0" -ar 16000 -ac 1 -f s16le - \
  | audiocpp_cli --task asr --family voxtral_realtime --model models/Voxtral-Mini-4B-Realtime-2602-GGUF/voxtral-mini-4b-realtime-2602-q8_0.gguf --backend cuda --threads 8 --mode streaming --audio -

# Any file or network stream, decoded to PCM on the fly
ffmpeg -i input.mp3 -ar 16000 -ac 1 -f s16le - \
  | audiocpp_cli --task asr --family voxtral_realtime --model models/Voxtral-Mini-4B-Realtime-2602-GGUF/voxtral-mini-4b-realtime-2602-q8_0.gguf --backend cuda --threads 8 --mode streaming --audio -

Stdin input requires --mode streaming, and the PCM format must be described up front because a live stream carries no header. The defaults (s16le, 16 kHz, mono) match what the model expects. The chosen interpretation is echoed as an audio_input=stdin line.

Each update carries only the text decoded since the last one, matching the other streaming ASR models, so the updates concatenate into the transcript. On a terminal they are appended unlabelled and the transcript scrolls like ordinary output. When stdout is redirected, each update is written as its own partial_text= line and flushed as it is produced, so pipes and logs stay parseable. The complete transcript is also printed once at the end as text_output=.

An update covers one decoded chunk, so stream_batch_tokens=<n> reports every nth token's worth of text in a single update rather than making the updates n times shorter. Whatever the batch size, concatenating the updates reproduces text_output= exactly.

Emitting deltas rather than restating the transcript matters for long runs, where the restated form is quadratic in the transcript length: a one-hour session writes roughly 364 MB restated against about 54 KB as deltas.

To capture the transcript itself rather than the update stream, use --text-out, which writes the complete transcript and nothing else:

ffmpeg -f avfoundation -i ":0" -ar 16000 -ac 1 -f s16le - \
  | audiocpp_cli --task asr --family voxtral_realtime --model models/Voxtral-Mini-4B-Realtime-2602-GGUF/voxtral-mini-4b-realtime-2602-q8_0.gguf --backend cuda --mode streaming --audio - --text-out transcript.txt

--text-out and the text_output= line are both written when the stream ends, so an interrupted session leaves neither. The partial_text= lines are flushed as they are produced, so a log of them survives an interrupted run and concatenates back into the transcript:

grep '^partial_text=' session.log | sed 's/^partial_text=//' | tr -d '\n' > transcript.txt

Live PCM Over HTTP

The same live source is available to an HTTP client through POST /v1/audio/transcriptions/live: raw PCM goes up in a chunked request body while transcript deltas come back as SSE on the same connection. This is the server equivalent of --audio -, and the only way to get capture-time partials without the CLI. See the server README for parameters and examples.

ffmpeg -f alsa -i default -ar 16000 -ac 1 -f s16le - \
  | curl -N -X POST -H 'Expect:' -T - \
      'http://127.0.0.1:8080/v1/audio/transcriptions/live?model=voxtral-realtime'

Use -T -, not --data-binary @-: the latter reads stdin to EOF before it connects, so a live capture would be uploaded as a finished file and no partial could arrive early.

Whether text appears while the speaker is still talking depends on the model's streaming policy rather than on the transport. voxtral_realtime decodes as audio arrives and emits throughout the utterance; nemotron_asr consumes the full utterance in its encoder first, so its deltas arrive only once the audio ends. Both are supported here; the difference is what the transcript looks like mid-sentence.

Streaming Throughput

A streaming step always advances 80 ms of audio, so a step has to cost under 80 ms to keep up with a realtime source. Measured on an Apple M3 Air (Metal, Q8_0):

Config	Short clip, cool	Sustained 7 min
default	78 ms/step (0.98x)	88 ms/step (1.10x)
stream_batch_tokens=4	60 ms/step (0.76x)	74 ms/step (0.92x)

The default splits roughly 48 ms for the text decoder and 30 ms for the audio encoder; batching takes the encoder to about 13 ms. The second column is what a long session actually gets on a fanless machine: a short clip run immediately after the seven-minute one still measured 88 ms/step, so the gap is the machine staying warm rather than anything that resets between sessions. Budget for the sustained column, and prefer stream_batch_tokens=4 if the source is realtime.

The decoder runs one step per 80 ms whether the audio holds speech or silence, so a session that falls behind stays behind. The lag is monotonic and does not recover during pauses. Measure your own hardware before relying on a live source.

Server

Streaming server config:

{
  "host": "127.0.0.1",
  "port": 8080,
  "backend": "cuda",
  "device": 0,
  "threads": 8,
  "lazy_load": true,
  "models": [
    {
      "id": "voxtral-stream",
      "family": "voxtral_realtime",
      "path": "/path/to/voxtral-mini-4b-realtime-2602-q8_0.gguf",
      "task": "asr",
      "mode": "streaming"
    }
  ]
}

Streaming server request:

curl -N http://127.0.0.1:8080/v1/audio/transcriptions \
  -F model=voxtral-stream \
  -F stream=true \
  -F file=@assets/resources/sample.wav

Common Options (use directly)

Option	Values	Default	Meaning
--audio	WAV path or -	required	Speech input. - streams raw PCM from stdin and requires --mode streaming.
--input-format	s16le, f32le	s16le	Sample format of raw PCM read from stdin. Ignored for file input.
--input-rate	integer Hz	16000	Sample rate of raw PCM read from stdin. Ignored for file input.
--input-channels	integer	1	Channel count of raw PCM read from stdin. Ignored for file input.
--mode	offline, streaming	offline	Full-context or streaming session.
--do-sample	bool	false	Enable sampling instead of greedy decode.
--temperature	float > 0	1.0	Sampling temperature.
--top-p	(0, 1]	1.0	Nucleus sampling limit.
--top-k	integer	50	Top-k sampling limit; 0 disables top-k.
--seed	integer	1234	Sampling seed.
--text-out	TXT path	not set	Transcript output. The transcript is also printed to stdout.

Request Options (use with --request-option)

Option	Values	Default	Meaning
max_tokens	integer	not set	Maximum generated transcript tokens; omitted uses the audio-derived limit. Offline decoding only; ignored by live streaming.
do_sample	bool	false	Sample rather than greedily decode.
temperature	float > 0	1	Positive sampling temperature.
top_p	(0, 1]	1	Nucleus sampling probability in (0, 1].
top_k	integer >= 0	50	Top-k sampling limit; zero disables filtering.
seed	integer >= 0	1234	Sampling seed.

The legacy request name max_new_tokens remains accepted. Use only one name per request.

Session Options (use with --session-option)

Option	Values	Default	Meaning
voxtral_realtime.weight_type	native, f32, f16, bf16, q4_0, q4_1, q5_0, q5_1, q2_k, q3_k, q4_k, q5_k, q6_k, q8_0, nvfp4	native	Shared matmul weight storage type accepted by the tensor loader; backend and shape support still apply.
voxtral_realtime.audio_encoder_weight_type	same as above	shared setting	Audio encoder matmul weight storage type. Leave at native for streaming: the encoder is not bandwidth-bound there, so quantizing it makes it slower.
voxtral_realtime.text_decoder_weight_type	same as above	shared setting	Text decoder matmul weight storage type. q4_k roughly halves the streaming decoder step cost.
voxtral_realtime.audio_encoder_graph_arena_mb	integer MiB	512	Audio encoder graph arena size.
voxtral_realtime.audio_encoder_weight_context_mb	integer MiB	128	Audio encoder weight context arena size.
voxtral_realtime.text_decoder_prefill_graph_arena_mb	integer MiB	512	Decoder prefill graph arena in MiB. Decoder uses the maximum of prefill and decode arena settings.
voxtral_realtime.text_decoder_decode_graph_arena_mb	integer MiB	512	Decoder step graph arena in MiB. Decoder uses the maximum of prefill and decode arena settings.
voxtral_realtime.text_decoder_weight_context_mb	integer MiB	128	Text decoder weight context arena size.
voxtral_realtime.stream_decode_cache_steps	integer >= 1	1024	Streaming decoder cache capacity in 80 ms steps. Must be positive and no greater than checkpoint sliding_window.
voxtral_realtime.stream_batch_tokens	integer >= 1	1	Audio tokens per encoder forward. The decoder still runs one step per 80 ms; batching amortizes the encoder's fixed per-forward cost at the price of delaying each partial by up to n * 80 ms.

Weight storage types are applied when the model loads, so requesting a type the GGUF does not already hold requantizes on the CPU before the first token. This takes around three minutes for q4_k from the shipped Q8_0 package. Prefer the published q4_k GGUF variant, which needs no load-time conversion. See GGUF.

For backend weight-type controls, use audiocpp_cli --inspect --model <model-dir> --family <family>.