voice_audio_multi/tests/moss_transcribe_diarize/README.md

MOSS-Transcribe-Diarize Validation

The native family uses the framework Whisper component and Qwen AR runtime. Validation is CUDA-only; CPU, Vulkan, and Metal performance is not claimed.

Regenerated Packages and Quantization

Original checkpoints and conversion intermediates live outside the local audio.cpp-gguf repository. All variants are generated from the pinned original safetensors, with the lossless Whisper name mapping, not by requantizing GGUFs. The converter accepts --type orig|q8_0|q4_k|q4_0; original mode retains its byte-exact checks. Quantized mode checks shapes, retained BF16 bytes, finite dequantized values and reports quantization error, not a claim of output parity.

Before regeneration, run tools/check_loader_catalog_sync.py --self-test, tools/check_loader_catalog_sync.py, and the typed spec checks. Sync alone checks catalog drift, not schema correctness. Supported specs require package entries.

Sequential RTX 5090 CUDA Debug comparison, eight threads, default 5120-token budget, one cold plus three warm requests per recording and precision:

Precision	28 s warm median / RTF	327.6 s warm median / RTF
BF16	626.398 ms / 0.02237	8845.980 ms / 0.02700
Q8_0	587.946 ms / 0.02100	8402.070 ms / 0.02565
Q4_K	589.642 ms / 0.02106	8318.250 ms / 0.02539
Q4_0	896.753 ms / 0.03203	8215.950 ms / 0.02508

All repeated transcripts are identical within each precision. Timings exclude load/upload and are end-to-end, not an equal-generated-token kernel comparison. The comparison used the corrected spec override for every candidate. All three final packages subsequently passed standalone server loading from /tmp, with no spec override, and short/long inference using the embedded instruct spec.

Q8_0 passes the unchanged Python FP32 gate on both primary fixtures: zero word or speaker errors, no segment reflow, at most the pre-existing 30 ms timestamp drift. Q4_K passes the short gate but fails the long gate: words/speakers match, but segment boundaries change and matched timestamps differ by up to 680 ms. Q4_0 fails both gates: the short result has 14 segments instead of six, and the long result changes phrase to frays (WER 0.0011013) with segment reflow. No thresholds were relaxed. Q4_K is the only retained Q4 variant, based on its better short-input latency and word preservation; its long latency is about 1.2% slower than Q4_0. Q4_K must not be described as parity-safe.

Evidence: quant_*_{short,long}_{0,1,2,3}.json, quant_*_parity.json, quant_comparison_summary.json, and server_quant_compare.log under build/logs/moss_transcribe_diarize/. Quantized tensors use 342 matrices; 339 tensors remain BF16 and two lookup tables become F16. Both Q4 candidates have the same file size. The original-mode tensor checks remain byte-exact.

Final Package Regression Checks

The final BF16, Q8_0 and Q4_K conversions ran once each from the original safetensors after the instruct rename. No further regeneration was needed. All six final server results exactly match their corresponding precision's saved pre-regeneration text, segments, speaker turns and sample rate. BF16 and Q8_0 pass both Python gates; Q4_K retains the documented long-input failure above. Regeneration parity does not imply quantization parity.

The standalone BF16 CLI accepts --instruct in streaming mode. Its short output passes the Python gate. The long server SSE output passes the same gate and the delta/final-text and completion protocol checks. This is text-output streaming after complete input processing, not streaming audio input.

Evidence under build/logs/moss_transcribe_diarize/: instruct-*-conversion.{log,stdout}, final_*_load.json, final_*_{short,long}.json, final_regeneration_identity.jsonl, final_instruct_cli.{log,stdout}, final_bf16_long.sse, final_{bf16,q8_0,q4_k,stream}_parity.{log,json} and server_final_packages.log. The temporary validation server was stopped.

Pending Apollo and UniverSR regression checks also completed: Apollo music and held-out Jingle Bells, and UniverSR music, speech and held-out Jingle Bells are byte-identical to their saved native WAVs. Apollo music and all three UniverSR cases pass their Python waveform gates. Apollo's held-out check uses the saved native reference; its previously documented strict Python drift is unchanged, not claimed as a Python pass. See regenerated-final-* logs/WAVs under build/logs/{apollo-family,universr-family}/.

Pinned Inputs

• Weights: OpenMOSS-Team/MOSS-Transcribe-Diarize, revision 704aa4a9c304e8520be88901e0d1960158ef5b15.
• Reference code: OpenMOSS/MOSS-Transcribe-Diarize, revision 61bc29cd4120be7b5d3b761b64cd5dff57263642.
• Original checkpoint: 683 tensors, 908,513,280 BF16 parameters, Apache-2.0.
• Local upstream directories: /media/leo/Share/models/MOSS-Transcribe-Diarize and /media/leo/Share/models/MOSS-Transcribe-Diarize-upstream.
• Logs: build/logs/moss_transcribe_diarize/.

Reference

Run in conda qwen3-tts, sequentially. The runner uses upstream generate_transcription, SDPA, greedy decoding, eight CPU threads, and disables TF32. It includes audio loading/preprocessing and decoding in the measured time, but excludes model loading. Trial zero is cold; the median uses three warm trials. It rejects truncated generations and inconsistent repeated transcripts.

conda run --no-capture-output -n qwen3-tts python tests/moss_transcribe_diarize/reference.py \
  --model /media/leo/Share/models/MOSS-Transcribe-Diarize \
  --upstream /media/leo/Share/models/MOSS-Transcribe-Diarize-upstream \
  --audio assets/resources/four_speaker_short.wav assets/resources/qwen3_tts_longform_asr_input.wav \
  --max-tokens 4096 \
  --output build/logs/moss_transcribe_diarize/python_bf16.json \
  --log build/logs/moss_transcribe_diarize/python_bf16.log

RTX 5090, PyTorch 2.11.0+cu128, Transformers 5.12.1:

Input	Seconds	Warm inference	RTF	Peak allocated	Peak reserved
Four speakers	28.0	1.960 s	0.0700	1,918,658,560 B	2,000,683,008 B
Long narration	327.6	25.466 s	0.0777	2,638,412,800 B	2,921,332,736 B

Weights occupy 1,817,026,560 bytes. PyTorch's peak allocated memory is 1.06x and 1.45x that size respectively. These are allocator statistics, not total process VRAM including CUDA context/library overhead. Native validation must also measure process peak VRAM and account for weights, KV cache, and graph workspace. Native memory use must not be justified solely by fitting the GPU.

GGUF

convert_gguf.py calls the existing native converter with --type orig, embeds the model spec and tokenizer/configuration sidecars, and verifies all tensor shapes, BF16 types, and bytes. Only Whisper names change, to the existing OpenAI Whisper loader layout. No weights are quantized, folded, or numerically changed.

conda run --no-capture-output -n qwen3-tts python tests/moss_transcribe_diarize/convert_gguf.py \
  --model /media/leo/Share/models/MOSS-Transcribe-Diarize \
  --output /media/leo/Share/models/audio.cpp-gguf/MOSS-Transcribe-Diarize-GGUF/moss-transcribe-diarize-bf16.gguf \
  --log build/logs/moss_transcribe_diarize/conversion.log

Native Implementation Boundary

Reuse WhisperFrontendComponent::load_openai_layout with explicit 16-head, 24-layer configuration; WhisperLogMelExtractor; framework linear, activation, normalization, and embedding modules; LlamaBpeTokenizer with Qwen2 pretokenization; and CausalDecoderRuntime with managed KV cache. Model code owns audio chunk assembly, four-frame merging, prompt time anchors, the adaptor, and output parsing. Large/frequent graphs must be session-owned and reused.

Remaining framework candidates (not implemented):

• HF Whisper loading currently hardcodes 12 heads and accepts no explicit encoder config. An explicit config/prefix binding API would remove the need for conversion-time Whisper renaming.
• Whisper's reusable component exposes host-vector output, while the Qwen runtime accepts host-vector prefill embeddings. A device-resident handoff could remove encoder/adaptor/decoder transfers. Measure before proposing it.
• The original Qwen prefill API exports KV to host before decode. MOSS now uses the additive device-cache prefill API below, avoiding that transfer.
• A config-driven HF Qwen3 weight-binding API could replace the model-local layer binding loop. It currently uses framework norm/linear binding helpers; no public whole-Qwen3 binding helper was available in this checkout.

Native Results

Original BF16 GGUF weights; F32 graph activations and F16 managed decode KV storage. There is no quantized-weight path in this validation.

The checkpoint's processor config sets time markers every five seconds, overriding the Python class default of two. Its integer marker stride is 62 audio embeddings, not 62.5. Both details must be preserved. Native/Python prompt lengths are 445 for the short fixture and 4,356 for the long fixture.

Final-output comparison against Python FP32 (parity_f32.json):

• Four speakers: exact raw transcript, including timestamps and speaker labels.
• Long narration: all words and all 77 segment texts/speakers match. One shared segment boundary differs by 30 ms (146.59 s versus 146.56 s); all others match.
• Gate: zero word errors, no segment reflow, no speaker attribution changes, at most 50 ms timestamp difference. The tolerance is not a claim of bit parity.
• Held-out 14.07-second sample.wav: identical words, segment texts, and speaker labels to Python FP32, with at most 30 ms timestamp drift. Held-out LibriSpeech test_other_7902-96591-0001.wav: exact raw transcript. See parity_heldout_f32.json; the same zero-word-error/50-ms gate passes both.

Against upstream BF16, the short fixture has identical words/speakers and up to 40 ms timestamp differences. The long fixture has one homophone substitution (frays versus phrase) and a few segment reflows. Python FP32 makes the same word and segmentation choices as native, establishing their precision sensitivity rather than treating BF16 output as identical.

Warm server results (one cold plus three sequential warm requests, same 4,096 token limit, eight threads; model loading excluded):

Input	Native median	Native RTF	Python BF16 median	Speedup
Four speakers, 28 s	0.96931 s	0.03462	1.95993 s	2.02x
Narration, 327.6 s	9.69783 s	0.02960	25.46615 s	2.63x

Every repeated native output was identical. The framework log shows one graph build per input shape, reused by subsequent requests. Python timing includes file loading/preprocessing; native server timing starts with decoded waveform input and includes resampling/frontend/inference. HTTP upload time is excluded from this table. Curl's default Expect/Continue wait must not be counted as inference; use -H 'Expect:' for HTTP wall-time comparisons.

Memory

server_vram.csv sampled NVML process memory every 50 ms for PID 237546. The test server was stopped afterwards; unrelated running servers were untouched.

• BF16 weight payload: 1.69 GiB.
• Native short-case process peak: 3,242 MiB (3.17 GiB).
• Native long-case process peak: 5,144 MiB (5.02 GiB), stable over repeats.
• Returning to the short shape reduced residency to 3,718 MiB.

The memory overhead is explainable but not optimal: the 28-layer, eight-KV-head, 128-head-dimension decoder stores a separate F32 prefill state and F16 decode cache. For the long case, the 4,356-token prefill KV is about 0.93 GiB and the 8,452-slot decode KV about 0.90 GiB, in addition to weights, encoder/decoder workspaces, CUDA graphs, and driver/library allocations. This is reasonable for 5.5 minutes of retained context, not a minimal-memory implementation. It is roughly 3x the weight payload, and higher than Python's 2.46 GiB allocator peak; NVML process usage and PyTorch allocator-only usage are different measurements.

The detailed server endpoint /v1/audio/transcriptions/details was verified to return six segments, six speaker turns, and the original input sample rate for the four-speaker fixture. Those initial baseline results used no framework or GGML changes; the authorized opt-in extensions are documented below.

The final standalone GGUF was run from /tmp, without a model-spec override, on both primary fixtures. standalone_short.txt and standalone_long.txt are byte-identical to the native outputs used for the parity reports. The final conversion log verifies all 683 tensors again; package size is 1,833,016,992 bytes. The packaged default of 5,120 generated tokens was also tested without a request override; the short output remained byte-identical (default_options.txt).

Output Streaming Follow-Up

The model implements the framework pull-event streaming interface. Complete audio is still required before generation, matching the upstream generation order. Text deltas end at ASCII transcript delimiters to avoid splitting UTF-8 characters. Both modes share one decoder path and retain session-owned graphs.

CUDA CLI streaming runs with eight threads, max_tokens=4096, and the local spec override produced stream_short and stream_long logs/stdout/text under the same log directory. Both final text files compare byte-identically to the previous offline fixtures. Partial text events appear before the final result. The GGUF was subsequently repackaged with the streaming-capable spec. conversion_streaming.log verifies all 683 BF16 tensors byte-for-byte against the original checkpoint. stream_standalone was run from /tmp without a spec override; its final text is byte-identical to the short offline baseline.

The pinned Transformers runner uses full-context Qwen generation; its ProgressStreamer counts output tokens, not incoming audio chunks. The original native path retained a square prefill mask and separate prefill/decode KV. The default chunked implementation below replaces that execution path, without independent audio windows or sliding-window truncation.

The upstream-recommended SGLang-Omni serving implementation documents 4096-token chunked prefill and incremental output. This is an optimization reference distinct from the pinned Transformers runner. Chunked prefill limits the query dimension of temporary attention work; it does not eliminate full-context KV storage. Strict input-independent VRAM still requires a fixed-capacity policy or KV offloading, not audio-window truncation.

GPU-Resident Chunked Prefill

Chunked prefill is the only MOSS path; there is no memory-saver option. The rejected CPU KV-offload runtime has been removed. MOSS uses explicit prefill_embeddings_into_cache, build_with_static_cache_block, and FastKVSetRowsModule::build_block APIs. The original row writer is unchanged; the decoder shares its math implementation between the two entry points.

Prefill writes 128-token blocks directly into the same FP16 backend cache used by token decoding. A session-owned graph and allocator are reused for each block and subsequent requests with the same capacity and block size. The final partial block writes padding only into unused future slots, which remain masked during decode until overwritten. No history is exported to CPU, no separate full-prompt KV is retained, and no square prompt mask is allocated. Full-context KV and the block mask still grow with capacity; this does not promise constant VRAM or unlimited context.

The additive prefill_embeddings_into_cache API is opt-in. Existing runtime entry points retain their behavior. The special import optimization and its configuration flag have been removed. The existing import implementation is unchanged; the new path calls the additive clear_on_backend cache API. GGML source, weights, tokenizer rules, and sampling math are unchanged.

The earlier A/B validation below used RTX 5090, Debug, eight threads, original BF16 weights, sequential requests, and three warm requests after one cold request. The multipart max_tokens=4096 field in these HTTP commands is not consumed: the effective model default is 5120, confirmed by cache capacities 5565 (short) and 9476 (long). Both sides use those same effective settings.

Audio	Non-chunked median	Chunked median	Chunked RTF	Non-chunked peak	Chunked peak
28 s	660.857 ms	617.826 ms	0.0221	3390 MiB	3234 MiB
327.6 s	9405.42 ms	8846.91 ms	0.0270	5258 MiB	3664 MiB

Elapsed-time reductions are 6.5% and 5.9%; long-input peak VRAM falls 30.3%. Peaks are per-process NVML measurements, not post-request residency.

Evidence under build/logs/moss_transcribe_diarize/:

• server_device_{short,long}_{0,1,2,3}.json: all eight final transcripts are byte-identical to the saved native non-chunked baselines.
• parity_device.json: zero word or speaker errors and no segment reflow against Python FP32; only the pre-existing 30 ms long-recording boundary difference remains.
• server_device_prefill.log and server_device_prefill_vram.csv (PID 288882): peak process VRAM sampled every 50 ms.
• server_device_control_{short,long}_{0,1,2,3}.json, server_device_control.log, and server_device_control_vram.csv (PID 289555): fresh non-chunked control in the same build and environment.
• chunked_unit_{cuda,cpu}.log: partial blocks, repeated requests, decoding, cache growth, capacity and reset tests. Legacy and backend-zero import produce identical logits and F32/F16/BF16 cache bytes, including unused tails.
• conversion_device_prefill.log: all 683 BF16 tensors remain byte-exact against the pinned safetensors after refreshing the embedded spec. That package was 1,833,017,184 bytes, before removing the mode option.
• device_stream_{short,long}.sse, server_device_stream.log, and parity_device_stream.json: both fixtures pass the SSE protocol and Python output gates. Final transcripts equal offline byte-for-byte. The server ran from /tmp without a spec override, using the refreshed GGUF.

Default Path Validation

After removing the mode option and special import optimization, the refreshed GGUF was loaded from /tmp with no spec override or session options. RTX 5090, Debug, eight threads, default 5120 output-token budget; sequential requests with one cold request followed by three warm requests:

Audio	Warm median	RTF	Peak process VRAM
28 s	603.660 ms	0.0216	3234 MiB
327.6 s	8560.940 ms	0.0261	3664 MiB

All eight transcripts match the previous C++ results byte-for-byte. parity_default.json passes both Python FP32 gates with zero word/speaker errors and only the existing 30 ms long-recording boundary difference. These are current default-path timings, not a fresh non-chunked A/B run.

• server_default_{short,long}_{0,1,2,3}.json, server_default_chunked.log, and server_default_chunked_vram.csv (PID 307440) contain timing and 50 ms NVML sampling evidence.
• additive_unit_{cpu,cuda}.log covers block boundaries, repeated prefill, cache growth, decode parity, and F32/F16/BF16 cache clearing.
• conversion_default_chunked.log verifies all 683 BF16 tensors byte-for-byte after removing the option from the embedded spec. GGUF size: 1,833,016,992 bytes.
• default_stream_{short,long}.sse and parity_default_stream.json pass both streaming protocol and Python output gates without session options.
• The original FastKVSetRowsModule::build implementation and cache-import implementation were compared with HEAD and are unchanged. Decoder math is shared by explicit single-token and block entry points, without duplication.

Historical server_saver* / saver_* evidence describes the removed CPU offload experiment, not the current implementation.

conda run --no-capture-output -n qwen3-tts build/debug/bin/causal_decoder_chunked_prefill_test \
  --backend cuda --log build/logs/moss_transcribe_diarize/chunked_unit_cuda.log

Recheck the final-output gate against the saved FP32 reference:

conda run --no-capture-output -n qwen3-tts python tests/moss_transcribe_diarize/test_parity.py \
  --upstream /media/leo/Share/models/MOSS-Transcribe-Diarize-upstream \
  --reference build/logs/moss_transcribe_diarize/python_f32.json \
  --actual four_speaker_short=build/logs/moss_transcribe_diarize/standalone_short.stdout \
    qwen3_tts_longform_asr_input=build/logs/moss_transcribe_diarize/standalone_long.stdout \
  --max-timestamp-drift 0.05 \
  --log build/logs/moss_transcribe_diarize/parity_f32.log \
  --report build/logs/moss_transcribe_diarize/parity_f32.json