voice_audio_multi/docs/models/vibevoice_asr.md

VibeVoice ASR Models

audio.cpp supports two VibeVoice ASR families:

Model	Family	Mode(s)	Recommended package
VibeVoice ASR	vibevoice_asr	offline	vibevoice_asr_q8_0
VibeVoice ASR Streaming 1.5B	vibevoice_asr_streaming	offline, streaming	vibevoice_asr_streaming_1_5b_q8_0
VibeVoice ASR Streaming 7B	vibevoice_asr_streaming	offline, streaming	vibevoice_asr_streaming_7b_q8_0

VibeVoice ASR

VibeVoice ASR is an offline ASR model with greedy, sampling, and beam-search decode paths. It can return transcription text and structured segment or speaker-turn output when the model produces timestamps.

A fully quantized port of the same model — INT8 activations through the encoder, ternary BitNet weights in the decoder — lives under community models as vibeasr: see VibeASR. It is not a separate model, only a CPU-only alternative numeric pipeline for the same weights.

Field	Value
Family	vibevoice_asr
Model directory	models/VibeVoice-ASR-GGUF
Task	asr
Modes	offline
Required tokenizer files	Embedded in the standalone GGUF
Output	Transcription text; optional segments through --segments-out; optional speaker turns through --turns-out
Streaming	Not supported
Timestamps	Segment and speaker-turn timestamps when produced

Install:

python3 tools/model_manager_v2.py install vibevoice_asr_q8_0

Offline CLI:

audiocpp_cli --task asr \
  --family vibevoice_asr \
  --model models/VibeVoice-ASR-GGUF/vibevoice-asr-q8_0.gguf \
  --backend cuda \
  --audio assets/resources/sample_16k.wav \
  --text-out transcript.txt \
  --metrics \
  --log

Structured output:

audiocpp_cli --task asr \
  --family vibevoice_asr \
  --model models/VibeVoice-ASR-GGUF/vibevoice-asr-q8_0.gguf \
  --backend cuda \
  --audio meeting.wav \
  --text "The recording is a meeting conversation." \
  --text-out transcript.txt \
  --segments-out segments.json \
  --turns-out turns.json \
  --metrics \
  --log

With VAD chunking, provide the bundled Silero VAD model:

audiocpp_cli --task asr \
  --family vibevoice_asr \
  --model models/VibeVoice-ASR-GGUF/vibevoice-asr-q8_0.gguf \
  --backend cuda \
  --audio assets/resources/sample_16k.wav \
  --audio-chunk-mode vad \
  --session-option vibevoice_asr.vad_model_path=assets/framework/models/silero_vad \
  --text-out transcript.txt \
  --metrics \
  --log

Convert from safetensors:

audiocpp_gguf.exe --input models\VibeVoice-ASR\model.safetensors.index.json --output models\VibeVoice-ASR-Q8_0\model.gguf --type q8_0

Configuration and tokenizer assets are embedded by default, so the output directory may contain only model.gguf.

Common Options (use directly)

Option	Values	Default	Meaning
--audio	WAV path	required	Speech input.
--text	text	empty string	Context prompt for the ASR request.
--language	language code	auto	ASR language label.
--max-tokens	integer	model default	Maximum generated transcript tokens.
--temperature	float	model default	Sampling temperature; 0 uses deterministic decoding.
--top-p	float	model default	Nucleus sampling probability.
--top-k	integer	model default	Top-k sampling limit; 0 disables top-k filtering.
--num-beams	integer	1	Beam count for deterministic beam search.
--repetition-penalty	float	model default	Generation repetition penalty.
--seed	integer	1234	Acoustic latent sampling seed.
--audio-chunk-mode	auto, fixed, vad, none	auto	Long-audio chunking mode. auto uses fixed chunks.
--audio-chunk-seconds	float seconds	1200	Fixed audio chunk duration.
--text-out	TXT path	not set	Transcript output. The transcript is also printed to stdout.
--segments-out	JSON path	not set	Write structured ASR segments when produced.
--turns-out	JSON path	not set	Write speaker turns when produced.

Request Options (use with --request-option)

Option	Values	Default	Meaning
context	text	not set	Context hint; context_info is also accepted.
language	text	not set	ASR language label.
max_tokens	integer >= 1	32768	Maximum generated text tokens.
temperature	float	0	Sampling temperature; zero uses deterministic decoding.
top_p	float	1	Nucleus probability in (0, 1].
top_k	integer >= 0	50	Top-k sampling limit; zero disables filtering.
num_beams	integer >= 1	1	Deterministic beam search width.
repetition_penalty	float	1	Positive generation repetition penalty.
seed	integer >= 0	1234	Acoustic latent sampling seed.
audio_chunk_mode	auto, fixed, vad, none	auto	Pre-encoder audio splitting; auto uses fixed chunks.
audio_chunk_duration_sec	float	1200	Positive audio chunk duration in seconds.

Session Options (use with --session-option)

Option	Values	Default	Meaning
vibevoice_asr.weight_type	native, f32, f16, bf16, q8_0	not set	Weight storage type; component-specific settings inherit weight_type.
vibevoice_asr.tokenizer_weight_type	native, f32, f16, bf16, q8_0	not set	Weight storage type; component-specific settings inherit weight_type.
vibevoice_asr.connector_weight_type	native, f32, f16, bf16, q8_0	not set	Weight storage type; component-specific settings inherit weight_type.
vibevoice_asr.decoder_weight_type	native, f32, f16, bf16, q8_0	not set	Weight storage type; component-specific settings inherit weight_type.
vibevoice_asr.tokenizer_weight_context_mb	integer >= 1	512	Context or graph arena size in MiB.
vibevoice_asr.connector_weight_context_mb	integer >= 1	128	Context or graph arena size in MiB.
vibevoice_asr.decoder_weight_context_mb	integer >= 1	not set	Decoder weight context in MiB; 4096 on 64-bit builds, 1024 on 32-bit builds.
vibevoice_asr.max_history_steps	integer	0	Maximum retained decoder history; zero uses the model limit. Cannot exceed checkpoint position capacity; zero uses that capacity.
vibevoice_asr.vad_model_path	model directory	assets/framework/models/silero_vad	Internal VAD model used by --audio-chunk-mode vad.

VibeVoice ASR Streaming 1.5B

The 1.5B checkpoint is the smaller sibling of the streaming 7B and runs through the same loader with no code changes: the layer count, hidden size, and head counts are all read from the checkpoint's own config.json, and the tensor names are identical. It is a drop-in smaller package, not a separate family.

Field	Value
Family	vibevoice_asr_streaming
Model package	vibevoice_asr_streaming_1_5b_q8_0
Model directory	models/VibeVoice-ASR-Streaming-1.5B-GGUF
GGUF repo	https://huggingface.co/christopherthompson81/VibeVoice-ASR-Streaming-1.5B-GGUF
Upstream weights	https://huggingface.co/microsoft/VibeVoice-ASR-Streaming-1.5B
Task, modes, output, timestamps	As streaming 7B

Sizes, and word error rate on the four LibriSpeech clips in assets/asr_validation/librispeech/, greedy decode:

Package	GGUF size	WER (CUDA)	WER (CPU)
vibevoice_asr_streaming_7b_q4_k	5.86 GB	4.35%	4.35%
vibevoice_asr_streaming_1_5b_bf16	5.64 GB	4.35%	4.35%
vibevoice_asr_streaming_1_5b_q8_0	3.34 GB	5.80%	4.35%
vibevoice_asr_streaming_1_5b_q4_k	2.12 GB	7.25%	5.80%

Note

A WER number for a quantized package is only meaningful with its backend. CPU and CUDA quantize activations differently in upstream ggml — Q4_K weights meet Q8_K activations on CPU (one scale per 256) and Q8_1 on CUDA (scale and sum per 32), and Q8_0 weights meet Q8_0 against Q8_1. The two backends therefore differ slightly but deterministically on every quantized matmul. BF16 quantizes no activations, which is why its two columns agree exactly.

On these clips the whole effect is one fragile word, where CPU hears the correct "cutter" and CUDA hears "country". This is expected upstream behavior, not an audio.cpp defect. The 7B not flipping here is four clips, not immunity.

Warning

Four clips is 69 words. One substitution moves the number by 1.4 points, so these separate "works and is in the right class" from "broken" and nothing finer. The like-for-like pair is the two q4_k rows: at equal quantization the 7B is ahead on both backends. Do not read the tie between 7B Q4_K and 1.5B BF16 as parity -- different clips happen to sum to the same total.

Install:

python3 tools/model_manager_v2.py install vibevoice_asr_streaming_1_5b_q8_0

Offline CLI, identical to the 7B apart from the model path:

audiocpp_cli --task asr \
  --family vibevoice_asr_streaming \
  --model models/VibeVoice-ASR-Streaming-1.5B-GGUF/vibevoice-asr-streaming-1.5b-q8_0.gguf \
  --backend cuda \
  --threads 8 \
  --audio assets/resources/sample_16k.wav \
  --text-out transcript.txt \
  --turns-out speaker_turns.json \
  --metrics \
  --log

Convert from safetensors:

audiocpp_gguf --input model.safetensors.index.json \
              --output vibevoice-asr-streaming-1.5b-q8_0.gguf \
              --type q8_0 --family vibevoice_asr_streaming --root sidecars

The padded vocabulary differs, and that is fine

embed_tokens is (151936, 1536) here against (152064, 3584) in the 7B: the 7B's vocabulary row count is padded, the 1.5B's is not. Nothing breaks, because the shape is validated against the vocab_size in the same checkpoint's config. It would only matter to code that treats the 7B's padded count as a constant -- a shared tokenizer bundle, or a logits slice sized for the 7B -- so keep that in mind when adding anything that spans both sizes.

VibeVoice ASR Streaming 7B

VibeVoice ASR Streaming 7B is the streaming VibeVoice ASR model. It keeps a persistent decoder state and can emit speaker-attributed transcript deltas as audio arrives. It can also run in offline mode through the same family.

The upstream streaming model emits speaker-attributed text such as Speaker 0: ..., but it does not emit timestamped segments. In audio.cpp, --turns-out can expose those speaker/text turns when the model emits speaker labels; --segments-out is not populated by this streaming model. Use the non-streaming vibevoice_asr family when timestamped segment output is needed.

Warning

Speaker-turn segmentation can differ from the offline VibeVoice ASR family. In local validation, the streaming 7B model merged adjacent speech into fewer speaker turns than the offline model; the same behavior was reproduced with the official Python streaming reference.

The recommended audio.cpp package is the Q8_0 GGUF in the dedicated model repo: https://huggingface.co/audio-cpp/VibeVoice-ASR-Streaming-7B-GGUF. BF16 and Q4_K GGUF variants are also available in the same repo.

Upstream lists ten supported language codes for this streaming model: en, zh, es, pt, de, ja, ko, fr, ru, and it.

Field	Value
Family	vibevoice_asr_streaming
Model package	vibevoice_asr_streaming_7b_q8_0
Model directory	models/VibeVoice-ASR-Streaming-7B-GGUF
Task	asr
Modes	offline, streaming
Required tokenizer files	Embedded in the standalone GGUF
Output	Transcript text; speaker/text turns when produced by the model
Timestamps	Not produced by the streaming model
Streaming	Live audio chunks over the /v1/audio/transcriptions/live endpoint

Install:

python3 tools/model_manager_v2.py install vibevoice_asr_streaming_7b_q8_0

Offline CLI:

audiocpp_cli --task asr \
  --family vibevoice_asr_streaming \
  --model models/VibeVoice-ASR-Streaming-7B-GGUF/vibevoice-asr-streaming-7b-q8_0.gguf \
  --backend cuda \
  --threads 8 \
  --audio assets/resources/sample_16k.wav \
  --text-out transcript.txt \
  --turns-out speaker_turns.json \
  --metrics \
  --log

Server config for live streaming:

{
  "host": "127.0.0.1",
  "port": 8080,
  "backend": "cuda",
  "threads": 8,
  "models": [
    {
      "id": "vibevoice-streaming-7b",
      "family": "vibevoice_asr_streaming",
      "path": "models/VibeVoice-ASR-Streaming-7B-GGUF/vibevoice-asr-streaming-7b-q8_0.gguf",
      "task": "asr",
      "mode": "streaming"
    }
  ]
}

Start the server:

audiocpp_server --config server.json --log

Live streaming request with 16 kHz mono signed 16-bit PCM:

ffmpeg -hide_banner -loglevel error -i input.wav -f s16le -ac 1 -ar 16000 - \
  | curl -N -X POST \
      -H 'Content-Type: application/octet-stream' \
      -H 'Transfer-Encoding: chunked' \
      -H 'Expect:' \
      -T - \
      'http://127.0.0.1:8080/v1/audio/transcriptions/live?model=vibevoice-streaming-7b&sample_rate=16000&channels=1&sample_format=s16le'

Common Options (use directly)

Option	Values	Default	Meaning
--language	language label	auto	ASR language label.
--max-tokens	integer	256	Maximum generated transcript tokens per chunk.
--temperature	float	0	Sampling temperature; 0 uses deterministic decoding.
--top-p	float	1	Nucleus sampling probability.
--top-k	integer	50	Top-k sampling limit; 0 disables top-k filtering.
--num-beams	integer	1	Accepted but ignored by the streaming checkpoint's incremental path, including offline mode.
--repetition-penalty	float	1	Generation repetition penalty.
--audio-chunk-mode	auto, fixed, vad, none	auto	Ignored by this checkpoint in both offline and streaming modes; processor chunk/lookahead settings determine segmentation.
--audio-chunk-seconds	float seconds	1200	Ignored by this checkpoint in both offline and streaming modes.

Request Options (use with --request-option)

Option	Values	Default	Meaning
context	text	empty string	Extra context or hotwords injected into the streaming prompt.

Session Options (use with --session-option)

Option	Values	Default	Meaning
vibevoice_asr_streaming.max_history_steps	integer	0 (checkpoint capacity)	Rolling window for decoder history, in steps. Refer to Long streams.

Long streams and the history window

With max_history_steps=0, history capacity comes from the checkpoint's max_position_embeddings; it is not unlimited. Memory use increases with stream length (approximately 0.4 MiB per step on Q8/CUDA) up to that capacity.

Set max_history_steps to prevent this. The model then stores only the most recent steps. Memory use remains constant for streams of any length.

audiocpp_cli --task asr --family vibevoice_asr_streaming \
  --model models/VibeVoice-ASR-Streaming-7B-GGUF/vibevoice-asr-streaming-7b-q8_0.gguf \
  --backend cuda --mode streaming --audio - --input-format s16le \
  --session-option vibevoice_asr_streaming.max_history_steps=4096

The model always stores the streaming prompt. It never removes the prompt. New steps replace the oldest stored steps, but not the prompt.

Each step is approximately one audio frame or one generated word part. Speech contains approximately ten steps per second. Thus 4096 stores approximately ten minutes of speech. At 4096 on Q8/CUDA, total memory use is stable at approximately 10.0 GB. Larger windows use more memory. The value must not exceed the model position capacity (131072 for the 7B model). The program rejects larger values at startup.

Note: A window changes the transcription when compared to full history because removed context is not available. The window size also causes very small differences in results because the model calculates in a different sequence. For reproducible results, do not change the binary or the window size.