voice_audio_multi/docs/archive/2026-07-29-fun-asr-nano-design.md

# Fun-ASR-Nano Native Runtime Design

## Goal

Add production-grade offline transcription for `FunAudioLLM/Fun-ASR-Nano-2512`
to audio.cpp under the family id `fun_asr_nano`. The first complete release must
load the official Hugging Face safetensors checkpoint, convert it with the normal
audio.cpp GGUF path, run on CPU and CUDA, and serve through the existing CLI and
OpenAI-compatible server.

## Scope

The first release includes:

- 16 kHz mono audio input with Kaldi-compatible 80-bin fbank extraction;
- pre-emphasis `0.97`, 25 ms frames, 10 ms shift, and LFR `m=7`, `n=6`;
- the 70-block SenseVoice SAN-M encoder (1 projection stem, 49 main blocks,
  20 timestamp-prediction blocks);
- the 512 -> 2048 -> 1024 projector and two-layer bidirectional adaptor;
- the bundled Qwen3-0.6B causal decoder with KV cache;
- Chinese, English, and Japanese transcription plus the model's Chinese
  dialect and regional-accent coverage;
- prompt-controlled inverse text normalization through `fun_asr_nano.itn`;
- native safetensors and audio.cpp-owned F16, Q8_0, and Q4_K_M GGUF packages;
- deterministic greedy decoding, timing metrics, and explicit input limits;
- reference parity against Transformers and the MIT-licensed transcribe.cpp
  implementation pinned at `c87109304c42707867f926fb7b9c378d9f46df8a`.

Streaming, built-in timestamps, translation, diarization, and the unpublished
CTC path are outside the first release. The published checkpoint contains no
complete CTC decoder, so the runtime is intentionally LLM-only.

## Source Of Truth

The implementation uses these pinned references:

- official HF checkpoint: `FunAudioLLM/Fun-ASR-Nano-2512-hf` at
  `854d88f94205cd17d2afdb24332130d86fbe654a`;
- Transformers integration PR #46180 and its `fun_asr_nano` config, frontend,
  model, and conversion mapping;
- transcribe.cpp at `c87109304c42707867f926fb7b9c378d9f46df8a`
  for independently validated SAN-M, adaptor, and frontend math;
- transcribe.cpp BF16 result of 1.78% WER on LibriSpeech test-clean as an
  external acceptance reference, not as a replacement for audio.cpp parity.

Adapted MIT code must retain attribution in the affected source files and in
the repository's third-party notices. Model packages retain the FunASR Model
Open Source License Agreement v1.1.

## Architecture

### Assets And Configuration

`FunAsrNanoAssets` owns a `ResourceBundle`, parsed configuration, tokenizer
assets, and one tensor source. The parser accepts the current nested
Transformers config and the legacy flat adaptor fields already published by
the HF checkpoint. Validation rejects mismatched dimensions before allocating
weights.

The runtime reads the converted HF tensor namespaces directly:

- `model.audio_tower.stem`;
- `model.audio_tower.layers`;
- `model.audio_tower.timestamp_prediction_layers`;
- `model.audio_tower.layer_norm` and
  `model.audio_tower.timestamp_prediction_layer_norm`;
- `model.multi_modal_projector` and `model.audio_adaptor.blocks`;
- `model.language_model`, `lm_head`, and tied token embeddings.

The audio.cpp GGUF converter preserves these logical names and embeds config,
processor config, generation config, tokenizer JSON, chat template, and the
typed `fun_asr_nano` model spec. This keeps safetensors and GGUF on one loader
path.

### Shared Frontend

A parameterized Kaldi fbank frontend belongs in `engine/framework/audio`, not
inside the model. It supports the exact options needed by Fun-ASR-Nano and can
later replace duplicated SenseVoice-style frontend code:

- pre-emphasis and Hamming window;
- HTK mel scale and natural-log output;
- configurable snip-edges behavior;
- centered LFR stacking with first/last frame replication;
- optional CMVN, disabled for Fun-ASR-Nano.

The model wrapper fixes the published values and returns time-major LFR frames,
an attention mask, and valid lengths.

### Shared SAN-M Module

The SAN-M graph primitives belong in
`engine/framework/modules/speech_encoders/sanm`. A non-owning weight view feeds
three operations:

- the 560 -> 512 projection stem;
- a residual SAN-M block;
- affine layer normalization.

Each block computes projected multi-head self-attention, the depthwise FSMN
branch over value states with kernel size 11, a residual connection, and the
ReLU feed-forward branch. F16 matrix multiplication requests F32 accumulation
where the validated reference requires it.

### Encoder And Adaptor

`FunAsrNanoEncoderRuntime` owns typed weight storage and a shape-keyed graph
cache. It applies `sqrt(512)` scaling, sinusoidal positions, the stem, 49 main
blocks, normalization, 20 timestamp-prediction blocks, and final normalization.

`FunAsrNanoAdaptorRuntime` applies the two projector linears and two pre-norm
bidirectional Transformer blocks. The adaptor preserves one output embedding
per valid LFR frame; padded frames are masked and removed before prompt splice.

### Decoder And Prompt

The text path reuses audio.cpp's `QwenCausalDecoderModule`, Qwen3 tokenizer,
sampling code, and KV-cache conventions. The Fun-ASR wrapper supplies the
1024-hidden, 28-layer, 16-query/8-KV-head configuration with RoPE theta
`1e6` and tied embeddings.

The prompt builder emits the checkpoint chat template, inserts one audio token
per adaptor output, and supports the model's two Chinese normalization prompts.
Greedy decoding is the default. Existing request sampling fields remain
available without creating model-specific duplicates.

### Session And Serving

`FunAsrNanoSession` implements `IOfflineVoiceTaskSession`. It validates task and
run mode, resamples input to 16 kHz, executes frontend -> encoder -> adaptor ->
decoder -> tokenizer, and publishes per-stage timings. It uses the framework's
existing audio chunking for long files, with 30-second default chunks and no
claim of model-native streaming.

Registration through `audiocpp_add_model` automatically exposes the family to
`audiocpp_cli`, `audiocpp_server`, loader inspection, and generic GGUF tooling.

## Model Spec And Packages

`model_specs/fun_asr_nano.json` is the source of truth. It advertises:

- task `asr`, mode `offline`, languages `zh`, `en`, and `ja`;
- official HF safetensors package from
  `FunAudioLLM/Fun-ASR-Nano-2512-hf`;
- audio.cpp-hosted F16, Q8_0, and Q4_K_M packages after parity validation;
- request option `language` and session option `itn`;
- explicit lack of timestamps and streaming.

The package license and attribution are displayed in the model documentation
and release metadata.

## Verification

Tests proceed in increasing cost:

1. config, tensor-name, tokenizer, and model-spec unit tests;
2. frontend parity on fixed waveforms, including short and padded audio;
3. encoder stage parity at stem, main block 0/24/48, and timestamp block
   0/10/19;
4. adaptor parity after each projector and adaptor block;
5. decoder prefill and first three decode-step logits parity;
6. end-to-end transcript parity on Chinese, English, and Japanese fixtures;
7. CPU and CUDA smoke tests for safetensors and GGUF;
8. WER evaluation on LibriSpeech test-clean before publishing quantized files.

F16/BF16 stage tensors target `atol=2e-3`, `rtol=2e-3`; end-to-end greedy
transcripts must match the HF reference on the fixture set. Quantized packages
must remain within 0.20 absolute WER percentage points of the F16 audio.cpp
baseline.

## Rollback

Work stays on `codex/add-fun-asr-nano-audiocpp-20260729` with a bundle and
patch backup before each commit. Each commit is independently buildable. The
model is registered only in the final end-to-end commit, so earlier commits do
not expose an incomplete family to users.