Fabio
c36510c21e
Some checks failed
Linux Build / Linux cpu (push) Has been cancelled
Linux Build / Linux cpu-clang (push) Has been cancelled
Linux Build / Linux vulkan (push) Has been cancelled
mac-build / macOS CPU (push) Has been cancelled
Nix Build / Nix (cpu) (push) Has been cancelled
Nix Build / Nix (vulkan) (push) Has been cancelled
windows-build / Windows CPU (push) Has been cancelled
79 lines
3.6 KiB
Markdown
79 lines
3.6 KiB
Markdown
# R2T2 ASR verification
|
|
|
|
These files verify the `confucius4_r2t2` family against the macOS MPS reference
|
|
implementation in the Confucius4-R2T2 repository (see `docs/models/r2t2.md`).
|
|
|
|
## Files
|
|
|
|
| File | Purpose |
|
|
|---|---|
|
|
| `make_golden.py` | Runs the Python reference (`R2T2ASRModel` on MPS) for an audio file and writes offline text plus per-chunk streaming `fixed_text`, `raw_decoded`, and `text` to a golden JSON. |
|
|
| `compare.py` | Runs `audiocpp_cli` offline and streaming with `--log-file`, parses the per-chunk trace, and diffs everything against a golden. |
|
|
| `test_confucius4_r2t2_transcription.cpp` | Repo-native smoke test: offline + final streaming transcripts against the golden, plus a check that Auto-language deltas form a nonempty prefix of the expected transcript for `assets/resources/sample_16k.wav`. Skips with exit code 125 when the model or audio is missing. Also exposes `--encode <text>` to dump token ids for tokenizer diffing. |
|
|
| `golden*.json` | Recorded reference outputs. |
|
|
|
|
## Goldens
|
|
|
|
| Golden | Audio | Reference run |
|
|
|---|---|---|
|
|
| `golden.json` | upstream `resources/test.wav` (Chinese) | MPS fp16, auto language, 320 ms chunks |
|
|
| `golden_zh.json` | same | MPS, forced `--language Chinese` |
|
|
| `golden_sample16k.json` | repo `assets/resources/sample_16k.wav` (English) | MPS fp16, auto language |
|
|
| `golden_sample16k_bf16.json` | same | MPS bf16 (isolates reference dtype) |
|
|
|
|
Streaming parameters are fixed across goldens so comparisons are deterministic:
|
|
`chunk_size_ms=320`, `unfixed_chunk_num=2`, `unfixed_token_num=5`,
|
|
`max_new_tokens=32`, `rollback_punctuation=false`.
|
|
|
|
## Regenerating a golden
|
|
|
|
Run from the Confucius4-R2T2 checkout (its `uv` environment has torch/MPS):
|
|
|
|
```bash
|
|
cd /path/to/Confucius4-R2T2
|
|
PYTHONPATH=. uv run python /path/to/audio.cpp/tests/confucius4_r2t2/make_golden.py \
|
|
--model_path /path/to/audio.cpp/models/Confucius4-R2T2 \
|
|
--audio resources/test.wav \
|
|
--out /path/to/audio.cpp/tests/confucius4_r2t2/golden.json
|
|
```
|
|
|
|
## Comparing
|
|
|
|
```bash
|
|
cd /path/to/audio.cpp
|
|
python3 tests/confucius4_r2t2/compare.py \
|
|
--cli build/macos-metal-release/bin/audiocpp_cli \
|
|
--model models/Confucius4-R2T2 \
|
|
--audio assets/resources/sample_16k.wav \
|
|
--golden tests/confucius4_r2t2/golden_sample16k.json \
|
|
--backend metal
|
|
```
|
|
|
|
The comparison checks four things: the offline transcript, the committed delta
|
|
stream, every per-chunk committed `fixed_text`, and the final streaming
|
|
transcript. Metadata-only rollback prefixes are filtered from the unmodified
|
|
reference goldens before comparing committed text; `language` fragments must
|
|
not appear in the emitted deltas. See the results table in
|
|
`docs/models/r2t2.md` for what is exact and the two documented internal
|
|
(non-observable) differences on the English clip.
|
|
|
|
The same binary can dump the family tokenizer for diffing against Hugging Face:
|
|
|
|
```bash
|
|
build/macos-metal-release/bin/test_confucius4_r2t2_transcription \
|
|
--encode "language English<asr_text>Some text 22,500"
|
|
```
|
|
|
|
## Graph reuse regression
|
|
|
|
```bash
|
|
build/macos-metal-release/bin/test_confucius4_r2t2_graph_reuse --backend metal
|
|
```
|
|
|
|
This model-backed test compares actual encoder capacities after growth and
|
|
shrink, checks the Metal attention precision boundary, and compares exact
|
|
encoder + exact decoder against reused encoder + reused decoder on real audio
|
|
prefixes (automatic language and forced English). Decoder-only synthetic
|
|
injection tests remain separate. The embedding RMSE bound is a drift alarm;
|
|
observable token equality is checked independently. The test defaults to CPU
|
|
and accepts `--model` for a local safetensors directory or GGUF file.
|