Fabio
c36510c21e
Some checks failed
Linux Build / Linux cpu (push) Has been cancelled
Linux Build / Linux cpu-clang (push) Has been cancelled
Linux Build / Linux vulkan (push) Has been cancelled
mac-build / macOS CPU (push) Has been cancelled
Nix Build / Nix (cpu) (push) Has been cancelled
Nix Build / Nix (vulkan) (push) Has been cancelled
windows-build / Windows CPU (push) Has been cancelled
133 lines
6.2 KiB
Markdown
133 lines
6.2 KiB
Markdown
# Parakeet TDT numerical parity harness
|
|
|
|
A heavier, more granular companion to `test_golden_transcription.cpp`
|
|
(the "Tier 1" golden-transcription regression test): instead of asserting
|
|
the final decoded text matches, this compares intermediate encoder
|
|
activations directly against a real NeMo reference run, stage by stage.
|
|
|
|
This exists because greedy/argmax decoding can be robust to a fair amount
|
|
of numerical drift without the final transcription changing — the golden
|
|
transcription test alone did **not** catch a real bug (the depthwise conv's
|
|
folded batch-norm bias being silently dropped — `use_bias=false` where it
|
|
needed to be `true`) that measurably corrupted every encoder layer's output,
|
|
because the greedy decode happened to pick the same tokens anyway on the
|
|
one test clip. This harness is what actually found and confirmed that bug.
|
|
|
|
## What it checks
|
|
|
|
Three stages, run through two independent code paths (real production
|
|
entry points where possible, an isolated single-layer build where not — see
|
|
the comment at the top of `dump_cpp_reference.cpp` for why intermediate
|
|
production-graph debug taps are deliberately avoided here):
|
|
|
|
1. `mel_features` — `ParakeetFrontend::extract(...)`, straight through the
|
|
real, unmodified frontend entry point.
|
|
2. `layer_0` — `build_encoder_layer(...)` (the exported, real production
|
|
per-layer function — see `encoder.h`), invoked once in a small isolated
|
|
graph, fed NeMo's own captured `pre_encode`/`pos_emb` for layer 0 and the
|
|
real layer-0 weights.
|
|
3. `enc_out` — `ParakeetFastConformerEncoderRuntime::encode(...)`, straight through the
|
|
real, unmodified full-encoder entry point (all 24 layers + positional
|
|
encoding, one large production graph).
|
|
|
|
`compare_parity.py` computes cosine similarity and a relative-std ratio for
|
|
each and fails loudly (naming the stage) if anything drops out of
|
|
tolerance. `enc_out`'s tolerance is deliberately looser than the other two
|
|
— see the comment above its threshold in `compare_parity.py` for why (short
|
|
version: float32 summation order isn't required to match bit-for-bit
|
|
between a small isolated single-layer graph and the real ~2M-node
|
|
24-layer production graph, and per-op rounding differences compound
|
|
through 24 layers of softmax/normalization even when the underlying math is
|
|
identical — verified directly by chaining 24 isolated single-layer builds
|
|
end to end, which stays at cosine >= 0.999999 the entire way through).
|
|
|
|
## Setup
|
|
|
|
### C++ side (no extra dependencies beyond the normal build)
|
|
|
|
```
|
|
cmake --build build/<preset> --target parakeet_parity_dump
|
|
```
|
|
|
|
### Python/NeMo side
|
|
|
|
NeMo's dependency chain is heavy and somewhat fragile outside a normal
|
|
desktop Python environment; a dedicated venv is recommended:
|
|
|
|
```
|
|
python3 -m venv /tmp/parakeet_parity_venv
|
|
/tmp/parakeet_parity_venv/bin/pip install torch nemo_toolkit[asr] librosa numpy pysqlite3-binary
|
|
```
|
|
|
|
If your system Python lacks a usable `sqlite3` module (common in minimal
|
|
containers — NeMo's dependency chain pulls it in transitively via
|
|
IPython's history feature even though nothing here needs it), install
|
|
`pysqlite3-binary` and let `dump_nemo_reference.py`'s
|
|
`sys.modules['sqlite3'] = __import__('pysqlite3')` shim handle it — it
|
|
already does this automatically when a real `sqlite3` import fails.
|
|
|
|
The first run downloads the `nvidia/parakeet-tdt-0.6b-v3` NeMo checkpoint
|
|
(~2.5GB) to the standard HuggingFace cache.
|
|
|
|
## Running it
|
|
|
|
```
|
|
# 1. NeMo reference dump
|
|
/tmp/parakeet_parity_venv/bin/python3 tests/parakeet_tdt/parity/dump_nemo_reference.py \
|
|
--audio tests/parakeet_tdt/assets/2086-149220-0033.wav \
|
|
--output-dir /tmp/parakeet_nemo_dump
|
|
|
|
# 2. C++ dump (accepts the installed safetensors directory or standalone GGUF).
|
|
# --matmul-weight-type defaults to "native" (F32); pass f16/bf16/q8_0 to
|
|
# numerically quantify the accuracy cost of a reduced-precision weight
|
|
# storage type against the same NeMo reference. --flash-attention 1 swaps
|
|
# the encoder's relative-position self-attention onto
|
|
# ggml_flash_attn_ext_with_bias_mask instead of the default explicit
|
|
# QK^T + soft_max_ext path — see the Performance section in
|
|
# docs/models/parakeet_tdt.md for measured numbers (and why it's
|
|
# not the default despite being numerically validated: it measured
|
|
# slower, not faster, on the hardware tested).
|
|
build/<preset>/bin/parakeet_parity_dump \
|
|
--model models/parakeet-tdt-0.6b-v3 \
|
|
--audio tests/parakeet_tdt/assets/2086-149220-0033.wav \
|
|
--nemo-dir /tmp/parakeet_nemo_dump \
|
|
--output-dir /tmp/parakeet_cpp_dump \
|
|
--matmul-weight-type native \
|
|
--backend cpu # or: --backend cuda / vulkan
|
|
|
|
# 3. Compare (numpy only, no NeMo/torch needed — can run anywhere the two
|
|
# dump directories are available, e.g. copied off a NeMo-enabled machine)
|
|
python3 tests/parakeet_tdt/parity/compare_parity.py \
|
|
--nemo-dir /tmp/parakeet_nemo_dump --cpp-dir /tmp/parakeet_cpp_dump
|
|
```
|
|
|
|
### Cross-backend comparison (no NeMo needed)
|
|
|
|
`--backend` also makes this harness answer a second question the NeMo
|
|
comparison cannot: does an accelerator still agree with the CPU? Dump twice
|
|
and diff the two `enc_out.npy` files directly. This is the check that catches
|
|
a change which is correct on CPU but silently wrong on a GPU — a strided view
|
|
that a CPU op materializes and a GPU kernel reads differently, for example.
|
|
|
|
Expect agreement to float32 accumulation noise, not bit-exactness: different
|
|
kernels sum in different orders. Currently CPU vs CUDA is cosine 0.99999988,
|
|
relative RMS 3.5e-06, with bit-identical mel features.
|
|
|
|
Expected output on a healthy build:
|
|
|
|
```
|
|
PASS mel_features: cosine=1.000000 (>= 0.999) ...
|
|
PASS layer_0: cosine=1.000000 (>= 0.999) ...
|
|
PASS enc_out: cosine=0.97xxxx (>= 0.97) ...
|
|
```
|
|
|
|
## Known limitation
|
|
|
|
This only covers the offline full-context encoder path with one test clip. It
|
|
does not exercise the decoder/joint network numerically; the golden and
|
|
streaming transcription tests cover those paths end to end, but not stage by
|
|
stage. The implemented streaming mode deliberately re-encodes bounded
|
|
bidirectional windows, so its encoder output is not expected to concatenate
|
|
into the utterance-wide `enc_out` tensor frame-for-frame. A cache-aware parity
|
|
test would require a checkpoint trained for chunked-limited attention; this
|
|
checkpoint is not one.
|