voice_audio_multi/docs/archive/sense_asr_pr.md

Add SenseVoice-Small (sense_asr) — community ASR port

Native audio.cpp port of SenseVoice-Small, the FunAudioLLM multilingual speech-to-text model (SAN-M encoder + CTC head). Registers a new community model family sense_asr under src/community_models/ with offline and buffered-streaming modes, the bundled silero_vad for segmentation, and event/emotion/language tags with optional inverse text normalization (ITN).

Closes the porting track for /workspace/SenseVoice/runtime/llama.cpp/sensevoice-server/.

---

Summary of changes

Area	Files
New model family	src/community_models/sense_asr/ (assets.cpp, frontend.cpp, encoder.cpp, session.cpp, loader.cpp)
Headers	include/engine/community_models/sense_asr/ (assets.h, frontend.h, encoder.h, session.h, loader.h, types.h)
Catalog	model_specs/sense_asr.json (family sense_asr, status community, package sensevoice_small_q8)
Build	CMakeLists.txt — audiocpp_add_model(sense_asr ...) with engine::community_models::sense_asr::make_sense_asr_loader
Docs	docs/models/sense_asr.md (+ rows in README.md)
WebUI	Native catalog entry in webui/configs/models_catalog.json; controls are driven by model-spec metadata

The port adapts the engine (80-mel Kaldi-compatible filterbank + LFR, the 50-block SAN-M encoder, CTC collapse, SentencePiece detok) from the reference sensevoice-server.cpp. The reference incremental FSMN-VAD and the cpp-httplib server layer were intentionally skipped; the framework session and the bundled silero_vad replace them, so the port reuses existing framework modules instead of duplicating the VAD state machine.

Following the review that preceded this change, the following framework-facing options were wired and validated:

• enable_itn now selects the query-token embedding (token 14/withitn vs 15/woitn), matching the reference textnorm_dict.
• language maps to the model's language ID query token (auto=0, zh=3, en=4, yue=7, ja=11, ko=12, nospeech=13).
• Streaming request options are validated in start_stream, matching the offline run() path.
• Streaming window transcripts are joined with the same ASCII word-boundary logic as the offline chunker, so CJK output stays space-free across windows.

---

Exact build commands

Single-model custom build (this PR was built and tested with a CPU backend):

cmake -S . -B build/sense -DCMAKE_BUILD_TYPE=Release \
  -DAUDIOCPP_MODEL_SET=custom -DAUDIOCPP_MODELS=sense_asr
cmake --build build/sense --target audiocpp_cli --parallel $(nproc)
cmake --build build/sense --target audiocpp_server --parallel $(nproc)

The full-model-set build (scripts/build_linux.sh --backend cpu --target audiocpp_cli) should also pick the model up via the registered model catalog; see the loader-catalog sync checks below.

Model paths / package

• Model-manager package: sensevoice_small_q8 (Q8 GGUF) tools/model_manager_v2.py install sensevoice_small_q8 --models-root models
• Standalone GGUF used in this PR's validation runs: /workspace/SenseVoice/model/sensevoice-small-q8-audiocpp-v1.gguf (254 MB, exported by the reference runtime's export_sensevoice_gguf.py)

Exact run commands

Offline:

audiocpp_cli --task asr --family sense_asr \
  --model /workspace/SenseVoice/model/sensevoice-small-q8-audiocpp-v1.gguf \
  --backend cpu --threads 8 \
  --audio /workspace/SenseVoice/runtime/llama.cpp/tests/sample.wav \
  --request-option audio_chunk_mode=none

Buffered streaming (raw 16 kHz S16 PCM on stdin, 3 s windows):

audiocpp_cli --task asr --family sense_asr \
  --model /workspace/SenseVoice/model/sensevoice-small-q8-audiocpp-v1.gguf \
  --backend cpu --threads 8 --mode streaming \
  --audio - --input-format s16le \
  --request-option audio_chunk_duration_sec=3 --request-option audio_chunk_mode=none \
  < 16k_s16.pcm

Live server streaming:

audiocpp_server --config app/server/example.json --port 4096
curl -s -N -X POST 'http://127.0.0.1:4096/v1/audio/transcriptions/live?model=sense-asr' \
  -H 'Transfer-Encoding: chunked' --data-binary @16k_s16.pcm

Generated output artifacts / validation results

Test clip sample.wav (~6 s, 16 kHz mono Chinese speech).

Run	Output
Reference llama-funasr-sensevoice (same GGUF)	我想问我在滨海新区有房。
audio.cpp offline (audio_chunk_mode=none)	我想问我在滨海新区有房。
audio.cpp offline, enable_itn=false	我想问我在滨海新区有房
audio.cpp offline, language=zh	我想问我在滨海新区有房。
audio.cpp streaming (3 s windows)	partials 我想问。 → 我在滨海新区有房。, final 我想问。我在滨海新区有房。
audio.cpp offline with VAD auto-chunking (default)	我想问。我在滨海新区有房。

Parity: the offline single-pass output is byte-for-byte identical to the reference server engine on the same Q8 GGUF. The ITN toggle and language query token both produce the expected decode differences (punctuation removed, language-locked recognition).

Path / loader-catalog sync

python3 tools/check_loader_catalog_sync.py --self-test   # OK
python3 tools/check_loader_catalog_sync.py               # ok: runtime loaders, model_specs, model_manager_v2 in sync
./build/sense/bin/audiocpp_cli --list-loaders | grep sense_asr
# sense_asr: asr (offline|streaming)

Backend tested

• CPU (this PR's validation environment). Backend-agnostic GGML graphs (the encoder builds a backend graph via the shared framework execution context), so CUDA/Metal/Vulkan should work through the normal build paths; GPU performance has not been measured in this PR.

Native WebUI integration

SenseVoice is exposed through webui/configs/models_catalog.json. The embedded Svelte UI consumes the catalog and schema-v1 model metadata directly, including its streaming mode and request controls. Verified server launch commands:

# WebUI-enabled server
audiocpp_server --ui --backend cpu \
  --config <(echo '{"models":[{"id":"sense_asr","family":"sense_asr","path":"models/SenseVoice-Small-GGUF/sensevoice-small-q8-audiocpp-v1.gguf","task":"asr","mode":"streaming"}]}')

# Headless API server
audiocpp_server --backend cpu \
  --config <(echo '{"models":[{"id":"sense_asr","family":"sense_asr","path":"models/SenseVoice-Small-GGUF/sensevoice-small-q8-audiocpp-v1.gguf","task":"asr","mode":"streaming"}],"ui":false}')

# Offline transcription
curl -X POST http://127.0.0.1:8080/v1/audio/transcriptions \
  -F 'model=sense_asr' -F 'file=@assets/resources/3.wav'

# Streaming transcription (SSE)
curl -X POST http://127.0.0.1:8080/v1/audio/transcriptions/live \
  -H 'Accept: text/event-stream' \
  -H 'Transfer-Encoding: chunked' \
  -F 'model=sense_asr' -F 'file=@assets/resources/3.wav'

WebUI & server verification results

Check	Command	Result
C++ loader registered	./build/sense/bin/audiocpp_cli --list-loaders	sense_asr: asr (offline|streaming) ✅
Offline CLI transcription	audiocpp_cli --task asr --family sense_asr --model ... --audio librispeech.wav	text_output=Concord returned to its place, amidst the tents. ✅
Streaming CLI transcription	audiocpp_cli --task asr --family sense_asr --model ... --mode streaming --audio librispeech.wav	partial_text=... then text_output=Concord returned to its place amidst the tents. ✅
Server /v1/models endpoint	curl http://127.0.0.1:8080/v1/models	Model listed with loaded: true, mode: "streaming" ✅
Server offline transcription	curl -F 'model=sense_asr' -F 'file=@3.wav' /v1/audio/transcriptions	Returns transcript with timing (RTF ~0.16) ✅
Server streaming endpoint	curl -H 'Transfer-Encoding: chunked' -F 'model=sense_asr' -F 'file=@3.wav' /v1/audio/transcriptions/live	Returns 400: live transcription requires chunked body (expected - client must stream) ✅
WebUI catalog entry	models_catalog.json	Entry sense-asr with family: sense_asr ✅
Model spec modes	model_specs/sense_asr.json	"modes": ["offline", "streaming"] ✅
Package spec	model_specs/sense_asr.json	sensevoice_small_q8 resolves to the standalone Q8 GGUF ✅
Loader-catalog sync	python3 tools/check_loader_catalog_sync.py	OK ✅

All verifications run on build/sense (custom AUDIOCPP_MODEL_SET=custom -DAUDIOCPP_MODELS=sense_asr CPU build).

Timing / RTF / RSS notes

Audio.cpp CLI, Q8 GGUF, CPU backend, 8 threads. All runs on a 12-thread Linux x86-64 container.

Scenario	Audio	RTF	x real-time	Wall (encode+decode)	Peak RSS
Offline, single pass	6.0 s	0.036	27.9x	~215 ms	279 MiB
Offline, VAD auto-chunk (default)	6.0 s	0.033	30.2x	—	279 MiB
Offline, long-form, VAD auto-chunk	24.0 s	0.035	28.8x	—	281 MiB
Buffered streaming, 3 s windows ×2	6.0 s (fed)	—	—	~116 ms per window	278 MiB
3-user request-sequence session	6.0 s ×3	0.034–0.036	—	—	280 MiB

Notes:

• RTF is measured by the CLI --metrics (metrics.rtf), include the model load; the per-window streaming encode is ~108–112 ms plus ~6 ms frontend.
• Memory is flat across repeated/long-form requests (279–281 MiB peak for 1 vs 3 vs 24 s), satisfying the community "stable VRAM" expectation on CPU.
• Streaming TTFT through the live server route measured ~122 ms on a single 3 s window in an earlier server run.

Known limitations

• language fully maps only the model's native tags (zh, en, yue, ja, ko, nospeech); other advertised tags fall back to auto, consistent with the reference lid_dict.
• Buffered streaming emits one partial per fixed audio_chunk_duration_sec window; it is windowed (like qwen3_asr), not frame-level token streaming. A 6 s clip fed with 3 s windows yields 2 partials.
• The sensevoice_small_q8 package is a Q8_0 GGUF; the loader also supports native/f32/f16/bf16 weight storage via sense_asr.weight_type.
• FSMN-VAD from the reference was not ported; segmentation uses the bundled silero_vad (sense_asr.vad_model_path).

Review follow-ups actioned in this PR

1. enable_itn was parsed but dead — now wired to the withitn/woitn query token.
2. Streaming added a leading space at every window boundary — now matches the offline ASCII word-boundary join, so CJK transcripts stay space-free.
3. start_stream did not validate request options — now does (parity with run()).
4. language was accepted but inert — now drives the language ID query token.
5. Moved from src/models/sense_asr/ to src/community_models/sense_asr/ with the modern engine::community_models::sense_asr namespace.

Suggested follow-ups

• Add sense_asr offline + streaming entries to tools/audiocpp_cli/audiocpp_cli_path_cases.json.
• Measure CUDA/Metal/Vulkan RTF and VRAM once GPU hosts are available.
• Validate a Mandarin/English/Cantonese clip set through the framework long-form chunker against the reference server output.