voice_audio_multi/docs/archive/2026-07-29-fun-asr-nano-plan.md

# Fun-ASR-Nano Native Runtime Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Add complete offline Fun-ASR-Nano transcription to audio.cpp for official HF safetensors and audio.cpp GGUF on CPU and CUDA.

**Architecture:** Parse the official Transformers checkpoint into typed assets, run a shared Kaldi-LFR frontend and SAN-M encoder, project through the bidirectional adaptor, and reuse audio.cpp's Qwen causal decoder for generation. Register the completed family through the existing loader, model-spec, CLI, server, and GGUF systems.

**Tech Stack:** C++20, CMake, GGML, audio.cpp runtime/model-spec APIs, Python reference probes, safetensors, GGUF, Hugging Face Hub.

## Global Constraints

- Family id is exactly `fun_asr_nano`; first supported variant is `FunAudioLLM/Fun-ASR-Nano-2512-hf`.
- Support offline ASR only; do not advertise native streaming, timestamps, translation, diarization, or CTC decoding.
- CPU and CUDA are release gates; Metal remains supported through generic GGML operations but is not a first-release gate.
- Official safetensors and audio.cpp-owned F16, Q8_0, and Q4_K_M GGUF use one logical tensor-name layout.
- Preserve MIT attribution for math adapted from transcribe.cpp commit `c87109304c42707867f926fb7b9c378d9f46df8a`.
- Preserve the FunASR Model Open Source License Agreement v1.1 in model package metadata and docs.
- Use TDD for every task and keep `.mcp-tasks/` untracked and untouched.
- Back up the worktree with a git bundle and patch before every commit.

---

## File Map

- `include/engine/framework/audio/kaldi_fbank.h`, `src/framework/audio/kaldi_fbank.cpp`: parameterized Kaldi fbank and LFR frontend.
- `include/engine/framework/modules/speech_encoders/sanm.h`, `src/framework/modules/speech_encoders/sanm.cpp`: reusable SAN-M graph primitives.
- `include/engine/models/fun_asr_nano/assets.h`, `src/models/fun_asr_nano/assets.cpp`: typed config, resources, and tensor source.
- `include/engine/models/fun_asr_nano/types.h`: frontend, embedding, prompt, generation, and result value types.
- `include/engine/models/fun_asr_nano/frontend.h`, `src/models/fun_asr_nano/frontend.cpp`: fixed Fun-ASR frontend wrapper.
- `include/engine/models/fun_asr_nano/encoder.h`, `src/models/fun_asr_nano/encoder.cpp`: 70-block SAN-M runtime.
- `include/engine/models/fun_asr_nano/adaptor.h`, `src/models/fun_asr_nano/adaptor.cpp`: projector and two-layer bidirectional adaptor.
- `include/engine/models/fun_asr_nano/tokenizer_text.h`, `src/models/fun_asr_nano/tokenizer_text.cpp`: Qwen tokenizer wrapper.
- `include/engine/models/fun_asr_nano/prompt.h`, `src/models/fun_asr_nano/prompt.cpp`: prompt construction and ITN selection.
- `include/engine/models/fun_asr_nano/decoder.h`, `src/models/fun_asr_nano/decoder.cpp`: Qwen prefill/decode runtime.
- `include/engine/models/fun_asr_nano/session.h`, `src/models/fun_asr_nano/session.cpp`: offline task orchestration and timings.
- `include/engine/models/fun_asr_nano/loader.h`, `src/models/fun_asr_nano/loader.cpp`: runtime registration and capabilities.
- `model_specs/fun_asr_nano.json`: packages, sources, capabilities, and options.
- `tests/fun_asr_nano/`: config, frontend, stage parity, warm bench, and reference scripts.
- `docs/models/fun_asr_nano.md`, `docs/asr.md`, `docs/gguf.md`: user-facing install, CLI, server, and GGUF instructions.

### Task 1: Typed Assets And Model Spec

**Files:**
- Create: `include/engine/models/fun_asr_nano/assets.h`
- Create: `src/models/fun_asr_nano/assets.cpp`
- Create: `model_specs/fun_asr_nano.json`
- Create: `tests/unittests/test_fun_asr_nano_assets.cpp`
- Modify: `CMakeLists.txt`

**Interfaces:**
- Produces: `FunAsrNanoConfig`, `FunAsrNanoAssets`, and `load_fun_asr_nano_assets(path)`.
- Consumes: `ResourceBundle`, JSON helpers, `TensorSource`, and the existing model-spec loader.

- [ ] **Step 1: Write failing config and spec tests**

Create fixtures with nested current config and legacy flat adaptor fields. Assert the parser yields encoder `(560, 512, 50, 20, 4, 2048, 11)`, adaptor `(1024, 8, 256, 2)`, projector hidden size `2048`, and Qwen `(1024, 3072, 28, 16, 8, 128, 151936)`.

```cpp
engine::test::require(config.encoder.input_size == 560, "Fun-ASR input size");
engine::test::require(config.adaptor.ffn_dim == 256, "Fun-ASR adaptor FFN");
engine::test::require(config.text.rope_theta == 1000000.0F, "Fun-ASR RoPE");
```

- [ ] **Step 2: Run the focused test and confirm it fails**

Run: `cmake --build build/debug --target test_fun_asr_nano_assets -j 8 && build/debug/bin/test_fun_asr_nano_assets`

Expected: compilation fails because `FunAsrNanoConfig` is not defined.

- [ ] **Step 3: Implement strict config parsing and the typed spec**

Reject dimension mismatches, unsupported activation functions, untied embeddings, absent tokenizer resources, and a tensor source missing the stem or token embedding. Define the HF package and safetensors source in the typed schema.

- [ ] **Step 4: Run asset and model-spec checks**

Run: `cmake --build build/debug --target test_fun_asr_nano_assets model_spec_demo -j 8`

Run: `build/debug/bin/model_spec_demo model_specs/fun_asr_nano.json tests/fixtures/fun_asr_nano_package`

Expected: both commands pass and report family `fun_asr_nano`.

- [ ] **Step 5: Back up and commit**

```bash
git add CMakeLists.txt include/engine/models/fun_asr_nano/assets.h \
  src/models/fun_asr_nano/assets.cpp model_specs/fun_asr_nano.json \
  tests/unittests/test_fun_asr_nano_assets.cpp
git commit -s -m "Add Fun-ASR-Nano assets and model spec"
```

### Task 2: Shared Kaldi Fbank And LFR Frontend

**Files:**
- Create: `include/engine/framework/audio/kaldi_fbank.h`
- Create: `src/framework/audio/kaldi_fbank.cpp`
- Create: `tests/fun_asr_nano/reference_frontend.py`
- Create: `tests/fun_asr_nano/fun_asr_nano_frontend_probe.cpp`
- Modify: `CMakeLists.txt`

**Interfaces:**
- Produces: `KaldiFbankOptions`, `KaldiFbankFeatures`, and `extract_kaldi_fbank(audio, options)`.
- Output layout: time-major `frames * (num_mels * lfr_m)` float values and a valid-frame count.

- [ ] **Step 1: Generate pinned reference values**

The Python probe must use the Transformers `FunAsrNanoFeatureExtractor` from PR #46180 and print JSON summaries plus the first and last 32 values for silence, impulse, 440 Hz sine, and `assets/resources/sample_16k.wav`.

- [ ] **Step 2: Write the C++ probe assertions**

Assert frame counts, output dimension `560`, finite values, and per-fixture numerical parity at `atol=2e-4`, `rtol=2e-4`.

- [ ] **Step 3: Run and confirm the frontend probe fails**

Run: `cmake --build build/debug --target fun_asr_nano_frontend_probe -j 8`

Expected: compilation fails because `extract_kaldi_fbank` is not defined.

- [ ] **Step 4: Implement the frontend**

Implement pre-emphasis `0.97`, 25 ms Hamming frames, 10 ms shift, 80 HTK mel bins, natural log, centered LFR `7/6`, replicated edge frames, and optional CMVN. Keep Fun-ASR's CMVN flag off.

- [ ] **Step 5: Run parity and existing audio tests**

Run: `build/debug/bin/fun_asr_nano_frontend_probe tests/fun_asr_nano/frontend_reference.json`

Run: `ctest --test-dir build/debug -R "audio|frontend" --output-on-failure`

Expected: all pass.

- [ ] **Step 6: Back up and commit**

```bash
git add CMakeLists.txt include/engine/framework/audio/kaldi_fbank.h \
  src/framework/audio/kaldi_fbank.cpp tests/fun_asr_nano
git commit -s -m "Add Kaldi LFR audio frontend"
```

### Task 3: Shared SAN-M Graph Primitives

**Files:**
- Create: `include/engine/framework/modules/speech_encoders/sanm.h`
- Create: `src/framework/modules/speech_encoders/sanm.cpp`
- Create: `tests/fun_asr_nano/sanm_probe.cpp`
- Create: `tests/fun_asr_nano/reference_sanm.py`
- Modify: `CMakeLists.txt`

**Interfaces:**
- Produces: `SanmBlockWeightsView`, `SanmBlockConfig`, `sanm_projection_block`, `sanm_residual_block`, and `sanm_layer_norm`.
- Consumes: GGML graph context and non-owning weight tensors.

- [ ] **Step 1: Write one-block reference and failing probe**

Use deterministic small tensors with `d_input=12`, `d_model=8`, two heads, FFN 16, and kernel 3. Compare projection and residual variants separately.

- [ ] **Step 2: Run the probe and confirm missing symbols**

Run: `cmake --build build/debug --target fun_asr_nano_sanm_probe -j 8`

Expected: link or compilation failure for the SAN-M API.

- [ ] **Step 3: Implement SAN-M math**

Implement Q/K/V projection, scaled dot-product attention, depthwise FSMN over V, residual addition, affine layer norm, and ReLU FFN. Request F32 accumulation for F16 matrix multiplication.

- [ ] **Step 4: Run CPU and CUDA probe variants**

Run: `build/debug/bin/fun_asr_nano_sanm_probe --backend cpu`

Run: `build/debug/bin/fun_asr_nano_sanm_probe --backend cuda`

Expected: both pass their configured tolerances.

- [ ] **Step 5: Back up and commit**

```bash
git add CMakeLists.txt include/engine/framework/modules/speech_encoders/sanm.h \
  src/framework/modules/speech_encoders/sanm.cpp tests/fun_asr_nano
git commit -s -m "Add reusable SAN-M encoder blocks"
```

### Task 4: Fun-ASR Encoder Runtime

**Files:**
- Create: `include/engine/models/fun_asr_nano/types.h`
- Create: `include/engine/models/fun_asr_nano/frontend.h`
- Create: `src/models/fun_asr_nano/frontend.cpp`
- Create: `include/engine/models/fun_asr_nano/encoder.h`
- Create: `src/models/fun_asr_nano/encoder.cpp`
- Create: `tests/fun_asr_nano/fun_asr_nano_encoder_probe.cpp`
- Create: `tests/fun_asr_nano/reference_encoder.py`
- Modify: `CMakeLists.txt`

**Interfaces:**
- Produces: `FunAsrNanoAudioFeatures` and `FunAsrNanoEncoderRuntime::encode(features)` returning `[frames, 512]`.
- Consumes: Tasks 1-3.

- [ ] **Step 1: Write tensor catalog and stage-parity failures**

Assert required tensor names and shapes. Capture stem, main blocks 0/24/48, timestamp blocks 0/10/19, and final normalization from the HF model.

- [ ] **Step 2: Run the probe and confirm the runtime is absent**

Run: `cmake --build build/debug --target fun_asr_nano_encoder_probe -j 8`

Expected: compilation fails for `FunAsrNanoEncoderRuntime`.

- [ ] **Step 3: Implement weight loading and shape-keyed graph caching**

Load split Q/K/V tensors from the HF namespaces. Build `sqrt(512)` scaling, sinusoidal positions, one 560 -> 512 stem, 49 main blocks, normalization, 20 timestamp blocks, and final normalization.

- [ ] **Step 4: Verify stage parity on CPU and CUDA**

Run: `build/debug/bin/fun_asr_nano_encoder_probe --backend cpu --reference tests/fun_asr_nano/encoder_reference`

Run: `build/debug/bin/fun_asr_nano_encoder_probe --backend cuda --reference tests/fun_asr_nano/encoder_reference`

Expected: every captured stage passes `atol=2e-3`, `rtol=2e-3`.

- [ ] **Step 5: Back up and commit**

```bash
git add CMakeLists.txt include/engine/models/fun_asr_nano \
  src/models/fun_asr_nano tests/fun_asr_nano
git commit -s -m "Implement Fun-ASR-Nano SAN-M encoder"
```

### Task 5: Projector And Bidirectional Adaptor

**Files:**
- Create: `include/engine/models/fun_asr_nano/adaptor.h`
- Create: `src/models/fun_asr_nano/adaptor.cpp`
- Create: `tests/fun_asr_nano/fun_asr_nano_adaptor_probe.cpp`
- Create: `tests/fun_asr_nano/reference_adaptor.py`
- Modify: `CMakeLists.txt`

**Interfaces:**
- Produces: `FunAsrNanoAdaptorRuntime::adapt(encoder_embeddings, mask)` returning packed valid `[tokens, 1024]` audio embeddings.
- Consumes: `FunAsrNanoEncoderEmbeddings` and Task 1 config.

- [ ] **Step 1: Write projector and block parity tests**

Capture `linear_1`, `linear_2`, adaptor block 0, adaptor block 1, and packed valid embeddings from Transformers.

- [ ] **Step 2: Confirm the probe fails before implementation**

Run: `cmake --build build/debug --target fun_asr_nano_adaptor_probe -j 8`

- [ ] **Step 3: Implement projector, mask, and two pre-norm blocks**

Use 512 -> 2048 -> 1024 projector linears with ReLU, eight-head bidirectional attention, 256-wide ReLU FFN, residuals, and padding removal.

- [ ] **Step 4: Run CPU/CUDA parity and padding cases**

Expected: stage tensors pass `atol=2e-3`, `rtol=2e-3`; padded batches match individual inference.

- [ ] **Step 5: Back up and commit**

```bash
git add CMakeLists.txt include/engine/models/fun_asr_nano/adaptor.h \
  src/models/fun_asr_nano/adaptor.cpp tests/fun_asr_nano
git commit -s -m "Implement Fun-ASR-Nano audio adaptor"
```

### Task 6: Tokenizer, Prompt, And Qwen Decoder

**Files:**
- Create: `include/engine/models/fun_asr_nano/tokenizer_text.h`
- Create: `src/models/fun_asr_nano/tokenizer_text.cpp`
- Create: `include/engine/models/fun_asr_nano/prompt.h`
- Create: `src/models/fun_asr_nano/prompt.cpp`
- Create: `include/engine/models/fun_asr_nano/decoder.h`
- Create: `src/models/fun_asr_nano/decoder.cpp`
- Create: `tests/fun_asr_nano/fun_asr_nano_decoder_probe.cpp`
- Create: `tests/fun_asr_nano/reference_decoder.py`
- Modify: `CMakeLists.txt`

**Interfaces:**
- Produces: `FunAsrNanoPromptBuilder::build(request, audio_tokens)` and `FunAsrNanoDecoderRuntime::generate(prompt, audio_embeddings, options)`.
- Consumes: `HfTokenizerJson`, `QwenCausalDecoderModule`, generic HF sampler, and Task 5 embeddings.

- [ ] **Step 1: Write prompt token and logit parity tests**

Assert the exact system/user/assistant token sequence, audio token positions, ITN prompt variant, prefill logits, and first three greedy decode steps.

- [ ] **Step 2: Run and confirm missing decoder APIs**

Run: `cmake --build build/debug --target fun_asr_nano_decoder_probe -j 8`

- [ ] **Step 3: Implement tokenizer and prompt splice**

Resolve `<|im_start|>`, `<|im_end|>`, and audio token id `151646` from tokenizer assets. Insert one placeholder per valid audio embedding.

- [ ] **Step 4: Wrap the existing Qwen causal decoder**

Configure hidden size 1024, 28 layers, 16 query heads, 8 KV heads, head dim 128, FFN 3072, RoPE theta `1e6`, and tied embeddings. Do not duplicate Qwen layer math.

- [ ] **Step 5: Run token, prefill, decode, and KV-cache parity**

Expected: prompt ids are exact and logits pass `atol=3e-3`, `rtol=3e-3`.

- [ ] **Step 6: Back up and commit**

```bash
git add CMakeLists.txt include/engine/models/fun_asr_nano \
  src/models/fun_asr_nano tests/fun_asr_nano
git commit -s -m "Add Fun-ASR-Nano Qwen decoding"
```

### Task 7: Offline Session And Loader Registration

**Files:**
- Create: `include/engine/models/fun_asr_nano/session.h`
- Create: `src/models/fun_asr_nano/session.cpp`
- Create: `include/engine/models/fun_asr_nano/loader.h`
- Create: `src/models/fun_asr_nano/loader.cpp`
- Create: `tests/fun_asr_nano/fun_asr_nano_warm_bench.cpp`
- Create: `tests/fun_asr_nano/fun_asr_nano_warm_bench_cases.json`
- Modify: `CMakeLists.txt`

**Interfaces:**
- Produces: `make_fun_asr_nano_loader()` and an `IOfflineVoiceTaskSession` for task `asr`.
- Consumes: Tasks 1-6 and framework audio chunking.

- [ ] **Step 1: Write failing loader, option, and transcript tests**

Assert family/task/mode, rejection of streaming and timestamps, `fun_asr_nano.itn`, unknown-option errors, and deterministic transcripts for zh/en/ja fixtures.

- [ ] **Step 2: Confirm the family is unavailable**

Run: `build/debug/bin/audiocpp_cli --inspect --family fun_asr_nano --model tests/fixtures/fun_asr_nano_package`

Expected: loader registry reports unknown family.

- [ ] **Step 3: Implement session orchestration and timings**

Execute resample -> frontend -> encoder -> adaptor -> prompt -> decoder -> decode. Emit `fun_asr_nano.frontend_ms`, `encoder_ms`, `adaptor_ms`, `decoder_ms`, token count, audio frame count, and `session.wall_ms`.

- [ ] **Step 4: Register only the complete family**

Add an `audiocpp_add_model(fun_asr_nano ...)` entry with loader include and `make_fun_asr_nano_loader`.

- [ ] **Step 5: Run CLI and server smoke tests**

Run: `build/debug/bin/audiocpp_cli --task asr --family fun_asr_nano --model "$FUN_ASR_NANO_MODEL" --audio assets/resources/sample_16k.wav --text-out /tmp/fun_asr.txt`

Run the existing OpenAI transcription route test with family `fun_asr_nano` and assert the JSON transcript equals the CLI result.

- [ ] **Step 6: Back up and commit**

```bash
git add CMakeLists.txt include/engine/models/fun_asr_nano \
  src/models/fun_asr_nano tests/fun_asr_nano
git commit -s -m "Register Fun-ASR-Nano offline ASR"
```

### Task 8: GGUF Conversion And Package Validation

**Files:**
- Modify: `model_specs/fun_asr_nano.json`
- Create: `tests/fun_asr_nano/test_gguf_roundtrip.cpp`
- Create: `tests/fun_asr_nano/compare_transcripts.py`
- Modify: `tools/model_manager_deprecated.py`
- Modify: `docs/gguf.md`

**Interfaces:**
- Produces: installable F16, Q8_0, and Q4_K_M packages with embedded sidecars and spec.
- Consumes: generic `audiocpp_gguf` and the safetensors loader.

- [ ] **Step 1: Write a failing round-trip test**

Convert a tiny fixture, reopen it as a standalone GGUF, and assert exact config, tokenizer sidecars, tensor logical names, and family selection.

- [ ] **Step 2: Run the generic converter on the official checkpoint**

```bash
build/release/bin/audiocpp_gguf --family fun_asr_nano \
  --model "$FUN_ASR_NANO_MODEL" --type f16 --output fun-asr-nano-f16.gguf
```

Repeat with `q8_0` and `q4_k_m` after F16 parity passes.

- [ ] **Step 3: Validate transcript and logit drift**

Compare safetensors, F16, Q8_0, and Q4_K_M on the zh/en/ja fixture matrix. Require exact greedy transcripts for F16 and Q8_0; record any Q4_K_M text differences.

- [ ] **Step 4: Add package records after artifacts exist**

Use `audio-cpp/audio.cpp-gguf`, exact filenames, SHA-backed Hub revisions, model license metadata, and Q8_0 as the default package.

- [ ] **Step 5: Back up and commit**

```bash
git add model_specs/fun_asr_nano.json tools/model_manager_deprecated.py \
  tests/fun_asr_nano docs/gguf.md
git commit -s -m "Add Fun-ASR-Nano GGUF packages"
```

### Task 9: Documentation, Full Verification, And Draft PR

**Files:**
- Create: `docs/models/fun_asr_nano.md`
- Modify: `docs/asr.md`
- Modify: `docs/model_manager.md`
- Modify: `README.md`
- Modify: `tools/audiocpp_cli/audiocpp_cli_path_cases.json`

**Interfaces:**
- Produces: complete install, CLI, server, GGUF, limitations, licensing, and validation documentation.

- [ ] **Step 1: Add executable CLI path coverage**

Add `fun_asr_nano_offline` with `task=asr`, `mode=offline`, `family=fun_asr_nano`, a 16 kHz fixture, text output, and expected non-empty transcript.

- [ ] **Step 2: Document user workflows and limits**

Include model-manager install, safetensors and GGUF commands, OpenAI-compatible transcription endpoint, ITN option, CPU/CUDA selection, 30-second chunking, language coverage, and unsupported timestamps/streaming.

- [ ] **Step 3: Run repository verification**

```bash
cmake -S . -B build/release -DCMAKE_BUILD_TYPE=Release
cmake --build build/release --target audiocpp_cli audiocpp_server audiocpp_gguf \
  fun_asr_nano_frontend_probe fun_asr_nano_encoder_probe \
  fun_asr_nano_adaptor_probe fun_asr_nano_decoder_probe \
  fun_asr_nano_warm_bench -j 8
ctest --test-dir build/release --output-on-failure
python3 tools/check_loader_catalog_sync.py
python3 tools/audiocpp_cli/run_audiocpp_cli_path_tests.py \
  --binary build/release/bin/audiocpp_cli --case fun_asr_nano_offline
```

Expected: every command passes.

- [ ] **Step 4: Run release parity matrix**

Run zh/en/ja transcripts on CPU and CUDA for safetensors, F16, Q8_0, and Q4_K_M. Record timing, peak memory, transcript, and known limitations in `docs/reports/fun_asr_nano_validation.md`.

- [ ] **Step 5: Self-review scope and attribution**

Confirm no incomplete CTC/timestamp/streaming claims, no unrelated refactor, transcribe.cpp MIT attribution is present, and model license text is linked from every package.

- [ ] **Step 6: Back up, commit, push, and open a draft PR**

```bash
git add README.md docs tools tests
git commit -s -m "Document and validate Fun-ASR-Nano support"
git push -u origin codex/add-fun-asr-nano-audiocpp-20260729
```

Open a draft PR linked to modelscope/FunASR#3439 with exact local verification results and remaining release gates. Mark ready only after CPU/CUDA and GGUF parity are green.

## Self-Review

- Spec coverage: assets, official HF loading, frontend, SAN-M, adaptor, Qwen decoding, session, CLI/server, GGUF, CPU/CUDA, parity, docs, licensing, and rollback each map to a task.
- Placeholder scan: all steps name concrete files, interfaces, commands, and expected outcomes.
- Type consistency: frontend emits `FunAsrNanoAudioFeatures`; encoder emits 512-wide embeddings; adaptor emits packed 1024-wide embeddings; decoder consumes those embeddings and returns token ids; session returns a framework transcript.