Fabio
c36510c21e
Some checks failed
Linux Build / Linux cpu (push) Has been cancelled
Linux Build / Linux cpu-clang (push) Has been cancelled
Linux Build / Linux vulkan (push) Has been cancelled
mac-build / macOS CPU (push) Has been cancelled
Nix Build / Nix (cpu) (push) Has been cancelled
Nix Build / Nix (vulkan) (push) Has been cancelled
windows-build / Windows CPU (push) Has been cancelled
137 lines
5.9 KiB
Markdown
137 lines
5.9 KiB
Markdown
# FireRed VAD
|
|
|
|
Native GGML inference for the [FireRedTeam/FireRedVAD](https://github.com/FireRedTeam/FireRedVAD)
|
|
offline VAD and causal Stream-VAD checkpoints. Both models take 16 kHz mono audio
|
|
and return speech segments. Stream-VAD also emits speech-start and speech-end events.
|
|
The original code and weights are Apache-2.0 licensed.
|
|
|
|
## Build and Convert
|
|
|
|
Run these commands from the repository root:
|
|
|
|
```bash
|
|
scripts/build_metal.sh --build-dir build/firered-vad --build-type Release \
|
|
--model-set custom --models firered_vad --with-tests \
|
|
--target audiocpp_cli --target audiocpp_gguf --target firered_vad_probe
|
|
```
|
|
|
|
Download `VAD/model.pth.tar`, `VAD/cmvn.ark`, `Stream-VAD/model.pth.tar` and
|
|
`Stream-VAD/cmvn.ark` from [FireRedTeam/FireRedVAD on Hugging Face](https://huggingface.co/FireRedTeam/FireRedVAD).
|
|
Preserve the `VAD/` and `Stream-VAD/` directories under `<source-dir>`.
|
|
The converter needs `torch`, `numpy`, `safetensors` and `kaldiio`.
|
|
|
|
```bash
|
|
python tests/firered_vad/convert_gguf.py \
|
|
--source <source-dir>/VAD --type f32 \
|
|
--converter build/firered-vad/bin/audiocpp_gguf \
|
|
--output models/FireRedVAD-GGUF/firered-vad-f32.gguf
|
|
|
|
python tests/firered_vad/convert_gguf.py \
|
|
--source <source-dir>/Stream-VAD --type f32 \
|
|
--converter build/firered-vad/bin/audiocpp_gguf \
|
|
--output models/FireRedVAD-GGUF/firered-stream-vad-f32.gguf
|
|
```
|
|
|
|
Only F32 packages are provided: F16 and Q8 offer no meaningful performance or
|
|
peak-VRAM benefit for these small models, and Q8 adds numerical drift.
|
|
The converter still accepts `f32`, `f16` and `q8_0` for experimentation.
|
|
Each GGUF contains its weights, FP32 CMVN
|
|
coefficients, architecture configuration and v1 model spec. The family is marked
|
|
experimental while converted packages are prepared for publication.
|
|
|
|
## Run
|
|
|
|
Offline VAD:
|
|
|
|
```bash
|
|
build/firered-vad/bin/audiocpp_cli --task vad --family firered_vad \
|
|
--model models/FireRedVAD-GGUF/firered-vad-f32.gguf \
|
|
--backend metal --audio assets/resources/sample_16k.wav --metrics --log
|
|
```
|
|
|
|
Streaming with the causal checkpoint:
|
|
|
|
```bash
|
|
build/firered-vad/bin/audiocpp_cli --task vad --mode streaming --family firered_vad \
|
|
--model models/FireRedVAD-GGUF/firered-stream-vad-f32.gguf \
|
|
--backend metal --audio assets/resources/sample_16k.wav --log
|
|
```
|
|
|
|
Use `--backend cpu --threads <count>` for CPU inference. Thread count controls
|
|
parallelism; the CPU graph and cache layout are the same across thread counts.
|
|
CPU depthwise convolution uses channel-contiguous time blocks, and eligible matrix
|
|
multiplications use GGML BLAS when available, following Moonshine's scheduler pattern.
|
|
The requested thread count is passed to both backends; vendor BLAS libraries may
|
|
manage their own worker threads. Input chunks use 160-sample hops with a retained 240-sample overlap for
|
|
the 400-sample Kaldi window. The default transport chunk is 2560 samples.
|
|
|
|
The offline checkpoint includes lookahead; the causal checkpoint supports both
|
|
full-input and streaming runs. Streaming resets caches and segmentation state at
|
|
the start of each request.
|
|
|
|
## Options
|
|
|
|
Pass options with `--request-option name=value`. Duration options are rounded to
|
|
the nearest 10 ms model frame; halfway values round up.
|
|
|
|
| Option | VAD default | Stream-VAD default |
|
|
|---|---:|---:|
|
|
| `threshold` | 0.4 | 0.5 |
|
|
| `smooth_window_ms` | 50 | 50 |
|
|
| `min_speech_duration_ms` | 200 | 80 |
|
|
| `max_speech_duration_sec` | 20 | 20 |
|
|
| `min_silence_duration_ms` | 200 | 200 |
|
|
| `merge_silence_duration_ms` | 0 | Offline VAD only |
|
|
| `speech_pad_ms` | 0 | Offline VAD only |
|
|
| `speech_start_pad_ms` | Stream-VAD only | 50 |
|
|
| `audio_chunk_duration_sec` | 300 | 300 for full-input runs |
|
|
|
|
Full-input runs split network input at `audio_chunk_duration_sec`, resetting network history
|
|
between chunks as upstream does. Live streaming retains network history.
|
|
|
|
## Validation
|
|
|
|
The reference tools use the official Python implementation, including Kaldi
|
|
features, CMVN, frame probabilities and segment rules. Install `soundfile` and
|
|
`kaldi-native-fbank` as well as the conversion dependencies. `--gguf` additionally
|
|
requires the Python `gguf` package and compares against dequantized package weights
|
|
to separate quantization drift from backend numerical error.
|
|
|
|
```bash
|
|
python tests/firered_vad/python_reference.py \
|
|
--reference <FireRedVAD-source-repo> --source <source-dir>/VAD \
|
|
--audio assets/resources/sample_16k.wav --threads 1 \
|
|
--output-prefix build/firered-vad/reference --device cpu --log
|
|
|
|
build/firered-vad/bin/firered_vad_probe \
|
|
models/FireRedVAD-GGUF/firered-vad-f32.gguf \
|
|
assets/resources/sample_16k.wav metal 8 \
|
|
build/firered-vad/native build/firered-vad/reference --log
|
|
|
|
python tests/firered_vad/check_parity.py \
|
|
--reference build/firered-vad/reference --native build/firered-vad/native
|
|
```
|
|
|
|
The probe checks streaming/full-input agreement for causal checkpoints and exports
|
|
features, probabilities and timestamps. It measures frontend, network and complete
|
|
detection separately, with five warmups and the median of 21 measured runs.
|
|
It accepts `cpu`, `cuda`, `vulkan`, and `metal`; the Python reference accepts
|
|
`--device cpu` or `--device cuda` and synchronizes CUDA timing. For causal
|
|
regression coverage, also run the official `TEST_MEETING_T0000000001_S00000.wav`
|
|
example with the Stream-VAD checkpoint: its near-threshold speech boundary exposes
|
|
small-matrix precision loss that the shorter examples do not.
|
|
The default parity checks bound normalized frontend RMSE by `1e-5`, frontend maximum
|
|
error by `1e-3`, frame-probability error by `0.002`, and require exact timestamps.
|
|
|
|
Unit tests cover upstream postprocessing fixtures and sparse/dense Kaldi frontend
|
|
equivalence. Sparse projection is opt-in and disabled by default for existing callers.
|
|
|
|
```bash
|
|
cmake --build build/firered-vad --target \
|
|
firered_vad_rules_test kaldi_fbank_sparse_test
|
|
ctest --test-dir build/firered-vad \
|
|
-R 'firered_vad_rules_test|kaldi_fbank_sparse_test' \
|
|
--output-on-failure
|
|
```
|
|
|
|
Measured results are recorded in [validation.md](../../tests/firered_vad/validation.md).
|