voice_audio_multi/docs/models/whistle_asr.md

# Whistle ASR (experimental)

The `whistle_asr` family runs [Cactus Compute Whistle](https://huggingface.co/Cactus-Compute/whistle)
locally through audio.cpp. The current implementation supports **offline**
transcription of 16 kHz mono WAV files up to 30 seconds. The encoder uses the
selected backend and the decoder runs on the host CPU. It detects one of the
model's seven languages (English, German, French, Spanish, Italian, Dutch, or
Polish); `--language en`, `de`, `fr`, `es`, `it`, `nl`, or `pl` forces a language.
These are supported language codes, not seven-language validation evidence.
The focused validation report records the English and German checks separately.
French, Spanish, Italian, Dutch, and Polish each have one ElevenLabs synthetic
clip checked through automatic and forced-language CLI requests. See the
[five-language spot checks](../../tests/whistle_asr/MULTILINGUAL_VALIDATION.md).
This limited coverage does not qualify general accuracy in all seven languages.

## GGUF package

The default package is `whistle_asr_f32`, containing
`Whistle-GGUF/whistle-f32.gguf` in
[`audio-cpp/audio.cpp-gguf`](https://huggingface.co/audio-cpp/audio.cpp-gguf/tree/main/Whistle-GGUF)
(about 211 MiB).

```bash
audiocpp_cli --task asr --family whistle_asr --backend cuda --threads 4 \
  --model models/Whistle-GGUF/whistle-f32.gguf \
  --audio assets/resources/sample_16k.wav --log --metrics
```

Use `--backend vulkan`, `cpu`, or `metal` for those backends. The decoder
runs on the host CPU in every case.

## Local conversion

To convert the official model locally, use the exact revision
`b358ddadd89b7a713b5aa131f23032d3cca1b251` and put
`checkpoints/whistle.safetensors` in a dedicated directory as
`whistle.safetensors`, together with `config.json` and `LICENSE`. Keep
`whistle.cact` outside that directory. The converter checks their byte lengths
and SHA-256 hashes, extracts the tokenizer, mel filterbank, and Hadamard
permutations from the official `.cact`, and embeds those six sidecars plus the
Whistle model spec in the GGUF:

```bash
cmake --build build/release --target audiocpp_gguf audiocpp_cli
python3 tools/community_models/whistle_asr/convert.py \
  --source /path/to/whistle-source \
  --cact /path/to/whistle.cact \
  --converter build/release/bin/audiocpp_gguf \
  --output /path/to/whistle-f32.gguf
audiocpp_cli --task asr --family whistle_asr --backend cpu \
  --model /path/to/whistle-f32.gguf \
  --audio assets/resources/sample_16k.wav --log --metrics
```

The official `.cact` uses Cactus Quants. The GGUF stores the published
FP32 Safetensors checkpoint instead. The two weight sets
can yield different spellings or number formatting for the same speech.
The runtime accepts FP32 tensors only, including for an extracted Safetensors
directory with the same five required assets. Raw upstream Safetensors alone
are insufficient. Other GGUF precisions are rejected.
Conversion and inference do not require the Cactus engine, ONNX Runtime, or
Python at recognition time.

Streaming, keyword biasing, word timestamps, speech embeddings, resampling, and
longer recordings are not supported by this family. CPU, Metal, CUDA, and Vulkan
are supported encoder backends. HIP has not been validated.
The runtime rejects unsupported sample rates, channel counts, and durations
rather than silently changing the input.
Decoding uses the checkpoint's 320-token ceiling; a non-terminating decode
fails explicitly instead of reporting a truncated transcript as complete.
No model is downloaded automatically.

Constant-RMS speech is passed to the model rather than rejected by an
amplitude-based noise gate. Digital silence returns an empty transcript.
Inputs shorter than 40 milliseconds also return an empty transcript.
This family has no speech activity detector, so noise-only recordings may
produce words.

## Runtime structure

The mel frontend reuses the shared `NemoMelFrontend` (512-point FFT, 400-sample
symmetric Hann window, 160-sample hop, constant padding, natural-log mel energies,
per-bin mean and unbiased-variance normalization). Whistle's 99th-percentile block
RMS gain normalization is applied locally before it.

The encoder runs as one GGML graph on the selected backend. It is built from the
framework `Conv2dModule`, `DepthwiseConv2dModule`, `DepthwiseConv1dModule`,
`LinearModule`, `RMSNormModule`, `GemmaRMSNormModule`, `RoPEModule`, `SiluModule`,
`SigmoidModule`, `SoftmaxModule`, and `GLUModule`. The stem does not use the shared
`DepthwiseConvSubsamplingModule` because that module hardcodes ReLU and Whistle's
stem uses SiLU. Whistle-specific math stays model-local in `graph.cpp`, shared by
the encoder graph: the four-lane manifold hyper-connection mixing with
its Sinkhorn normalization, the conditioned Kronecker-factored Hadamard MLP with its
two fixed permutations, and the per-head RMS norm and rotary embedding over 48-wide
heads. Checkpoint tensors are JAX `[in, out]` kernels; the loader transposes them
once into the backend weight store.

The autoregressive decoder uses the original host implementation from PR #824,
with OpenMP workers bounded by `--threads`. Its weights are retained across
requests; self-attention caches and three-tap projection history reset for each
recording. The encoder graph computes cross-attention keys and values, which
are transferred to the host decoder once per request. Grouped-query and
cross-attention preserve the model's 48-wide queries/keys and 64-wide values.
Engram lookup, projections, gates and greedy argmax also run on the host.

This avoids thousands of small GGML decoder nodes per token. Four-thread CPU
and Metal sessions benefit on the tested M4 Pro; one-thread decoding is slower
than the previous one-thread GGML graph.

## Build and request options

Follow the platform [build instructions](../build/windows.md) on Windows.
For a focused build, select `AUDIOCPP_MODEL_SET=custom` and
`AUDIOCPP_MODELS=whistle_asr`. A Windows helper example is:

```powershell
.\scripts\build_windows.ps1 -Preset windows-cpu-release -ModelSet custom -Models "whistle_asr" -Target audiocpp_cli
cmake --build build/windows-cpu-release --target audiocpp_gguf
```

On macOS the Metal helper configures the same model set:

```bash
scripts/build_metal.sh --build-dir build/whistle --build-type Release --openmp auto --with-tests --deployment-build --model-set custom --models whistle_asr --target audiocpp_cli --target audiocpp_gguf
```

Use `--backend` to select an available backend: `cpu`, `metal`, `cuda`, or `vulkan`.
`--threads` sets the CPU encoder's
thread count and bounds the host decoder's OpenMP workers; the runtime accepts
1 to 64. Build with OpenMP enabled for parallel host decoding. Without OpenMP,
the host decoder runs serially. An experimental decoder implementation reuses
the framework NEON library for the FP32 vocabulary projection, retaining the
same host attention, mHC, Hadamard and cache logic. Build with
`-DENGINE_ENABLE_NEON=ON` on AArch64 and opt in with:

```bash
--session-option 'whistle_asr.exec_plan={"decoder":{"impl":"neon"}}'
```

The default is `native`; `impl: "default"` also selects it. Both implementations
run on CPU independently of the encoder backend and use the session thread
count. No artifacts or parameters are accepted. Existing embedded GGUF specs
can use this option without reconversion. The spec exposes the optional `language`
request option and `exec_plan` session option. Use `--language de` or
`--request-option language=de` to force German. Omit the language option for model
language selection. Decoding is greedy, with at most 320 text tokens.

## Provenance and licensing

Use the pinned [upstream revision](https://huggingface.co/Cactus-Compute/whistle/tree/b358ddadd89b7a713b5aa131f23032d3cca1b251).
The converter preserves the upstream `LICENSE` inside the GGUF. The repository
license does not grant rights to the weights or extracted assets. See the
[provenance record](../../tools/community_models/whistle_asr/PROVENANCE.md) and
[model license table](../model_licenses.md) for source-specific terms and
unresolved questions. No weights or extracted assets are included in this port.

## Validation references

- [Official model, checkpoint, and license](https://huggingface.co/Cactus-Compute/whistle)
- [Published Whistle reference reconstruction](https://huggingface.co/mrfakename/whistle-ONNX)
- `whistle_frontend_test` checks mel features against independently computed values.
- `whistle_tokenizer_test` checks byte-fallback UTF-8 replacement and metaspace decoding without model weights.
- `whistle_assets_test <gguf> assets/resources/sample_16k.wav
  "Some call me Nature."` checks a converted GGUF against the public
  sample's first three seconds. `--backend cpu|metal|cuda|vulkan` and `--threads n` select
  the backend; `--full` uses the whole recording.
- `whistle_assets_test <gguf> <wav> --reference <dir> [--tolerance f]` compares
  the mel features, encoder memory, and cross-attention keys/values against
  `.f32` reference dumps in `<dir>` and reports the largest difference as a
  fraction of each reference tensor's largest magnitude (default tolerance 2e-3;
  Metal's half-precision matmul operand staging needs `--tolerance 5e-3`).
  The same modes also run the decoder on the full recording.
  `--dump` writes `decoder_logits_<position>.f32` (the full 8199-wide logits of
  each step), `decoder_tokens.i32` (input tokens, BOS and language token first)
  and `decoder_transcript.txt`. `--reference` compares every logits file with the
  same scaled metric and requires identical tokens and transcript.
  In [the validation report](../../tests/whistle_asr/VALIDATION.md), the encoder
  dumps came from the earlier host-only implementation at commit `6e0a40c4`. The
  decoder dumps came from the host decoder, before the step graph replaced it.

To run the native tests through CTest, configure with
`-DENGINE_BUILD_TESTS=ON` and
`-DAUDIOCPP_WHISTLE_TEST_MODEL=<absolute-path-to-converted-GGUF>`, then build
`whistle_frontend_test`, `whistle_tokenizer_test`, and `whistle_assets_test`, then run
`ctest --test-dir <build-directory> -R whistle_ --output-on-failure`. The
opt-in tests check both the three-second excerpt and the entire public WAV,
plus quiet and constant-RMS speech, silence, the 30-second limit, and decoder reuse.

See [validation evidence](../../tests/whistle_asr/VALIDATION.md)
for exact local commands, observed results, performance, and readiness blockers.
Comparisons with the quantized official runtime are transcript checks, not exact
FP32 numerical parity.