voice_audio_multi/docs/validation/underdog_bark3.md

Underdog Bark 3.0 Pro validation

Validated follow-up for PR #1130, tested on Windows 11, Ryzen 9 7950X3D (eight inference threads), RTX 3090 24 GiB, MSVC 14.43, CUDA 12.4 and Vulkan SDK 1.4.350.0. CPU-only and combined CUDA/Vulkan Release builds pass. Build, conversion and CLI/server commands are in the model guide. The initial original-runner comparison records the earlier CPU failure and Q8 profile; the fixes below supersede them. No shared framework, optimizer or backend/kernel behavior is changed.

CPU memory fix

CPU schedule pruning removed tensor-view nodes before ggml allocation. The allocator then lost last-use boundaries for large temporary backing buffers. The Bark-only CPU policy retains those nodes and releases the one-shot position projection producer after copying its outputs to the persistent graph inputs.

Five-minute full-context CPU graph	Before	After
Graph buffer	37,815,713,312 bytes (35.2 GiB)	2,812,205,088 bytes (2.62 GiB)
Allocation reduction		92.56%
Full original F32-reference transcript	Could not complete safely on this machine	Exact match, without chunking

This is the graph buffer, not total process RAM. Measured total peaks are 5.17 GiB for F32 and 4.45 GiB for protected Q8. A weight-free regression builds 32 dependent cropped views, measures allocation without allocating the old oversized layout, and executes the corrected graph to verify exact values.

Packages and Q8 precision

Source: ConwayResearch/Underdog-Bark-3.0-Pro, pinned revision 4b5c51e04aa8d16dda9ecdff71bc29f7360149e4, checkpoint SHA-256 6f84b33e63ebd1851e910e8b2fe299d29f5e0efc6f57e84d92d2f79af464e0e5. 699 inference tensors are prepared; 24 training-only counters are omitted. Tokenizer/configs, spec v1, conversion provenance, NOTICE and CC-BY-4.0 license are embedded. No external files or Python are needed for native ASR inference. VAD additionally loads the existing optional framework VAD model.

Standalone GGUF	Bytes	SHA-256
f32	2,509,681,888	68b4eef71c528ab4ebbf99478bbc7198bb84d3b7bee0ac0cfe171d7f242fb339
bf16	1,255,567,328	d59e8e8fa2e26d942c8a912846e4fc4f0ecf3ed8de2dc44e6e1ef44cfc51aae5
q8_0	908,687,328	d3fedf4d4fea36405bfce7ec0e8c132cb2c89b393ea2f4a61f97ba19352a5c46

Q8 now retains decoder.*, encoder_projector.* and joint.head.* in their exact source BF16 values. Conversion rejects non-BF16-representable staging values for those tensors. It still quantizes the eligible encoder matrices; BatchNorm compensation remains F32. The protected package is 7,227,056 bytes smaller than the initial 915,914,384-byte Q8 because the generic conversion previously left some LSTM tensors F32. This is mixed Q8 storage, not a lossless whole-model claim. Reconvert old Q8 packages to obtain the new policy/spec.

Reference and whole-recording results

Original references use the unmodified checkpoint through Transformers 5.19.0, PyTorch 2.8.0 CPU/F32, including its 4000-sample prefix. The publisher's unmodified CUDA/BF16 runner was also executed; see the initial comparison. Six unique mono 16 kHz fixtures were checked: LibriSpeech 2086-149220-0033.wav (7.435 s), four local English video excerpts (7.9925, 12, 12 and 30 s), and 1 s silence. The video excerpts start at 0, 20, 240 and 90 seconds in We need to talk about 5min.mp4 and are not shipped with source. Each native session also repeats short/long/short without reloading.

Backend	F32 exact text / timestamp bounds	BF16	Protected Q8
CPU	9/9	9/9	9/9
CUDA	9/9	9/9	9/9
Vulkan	9/9	9/9	9/9

All 81/81 checks pass with the final full-context default. Silence returns empty text. The initial F32 CPU intermediate comparison had maximum absolute error 1.684e-6 over 96 valid encoder frames; mel error was 1.689e-4. The padded 97th encoder row is not valid output. These fixtures do not establish corpus WER or forced-alignment accuracy.

The complete 299.99375 s recording now matches the original F32 transcript exactly, including punctuation/case, in every native result below:

Backend	GGUF	Processing time	Peak extra GPU memory incl. model	Peak process RSS
CPU	f32	48.986 s	0.00 GiB	5.17 GiB
CPU	q8_0	62.621 s	0.00 GiB	4.45 GiB
CUDA	f32	0.859 s	5.19 GiB	2.97 GiB
CUDA	bf16	0.820 s	5.03 GiB	2.17 GiB
CUDA	q8_0	0.801 s	4.51 GiB	2.21 GiB
VULKAN	f32	0.899 s	5.33 GiB	2.91 GiB
VULKAN	bf16	0.908 s	5.04 GiB	2.12 GiB
VULKAN	q8_0	0.892 s	4.61 GiB	2.11 GiB

GPU timing is the median of the last two of three full requests after a short warmup; CPU timing is one full request and includes its full-graph build. Both exclude model load/file I/O and include preprocessing/decoding. CPU times varied between runs; this is a memory/quality fix, not a CPU speedup claim. Original PyTorch CUDA timings were 2.436 s F32 and 2.049 s BF16 on the same recording. NVML device-total memory was sampled every 100 ms minus a pre-process baseline; RSS is sampled process working set. Memory includes model and temporary buffers.

Buffered streaming frontend reuse

Bark streams now cache one window of unnormalized log-mel features. Only frames with their full FFT/preemphasis support inside both windows are reused. Window edges are recomputed, and F64 normalization is repeated over the entire current window in the original reduction order. An unaligned hop grid or absent overlap falls back to fresh extraction; starting/resetting a stream clears the cache. Default raw-cache data is about 0.68 MiB of host RAM (1401 x 128 floats), plus frontend metadata, with no additional GPU tensors. This is Bark-only; stock Parakeet/Phonon, offline inference, attention and shared kernels retain their existing paths. No GGUF reconversion is required.

The model-local unit test compares every normalized float bit-for-bit against fresh extraction for growing/sliding windows, boundary impulses, silence, unaligned windows/tails, absent overlap and zero/one-valid-frame cases. Actual streaming before/after comparisons pass 81/81 checks: the six fixtures above plus repeated short/long/short, for CPU/CUDA/Vulkan x F32/BF16/Q8. Text and word timestamps match exactly within each backend/precision. The complete five-minute Q8 stream also retains exactly the same text/timestamps on all three backends. These comparisons use the prior buffered implementation at 889e3161, not the original model's different whole-recording context. Both warm-bench binaries include the start_stream() lifecycle correction; the previous benchmark only called reset(), which cannot activate the Bark stream wrapper.

The 66 HTTP/SSE checks below were repeated successfully with this path, including hotwords and stream reuse. The actual stock Parakeet streaming/reset regression also passes. Default 2 s centers, 10 s left context and 2 s right context are unchanged. Processing-time improvements do not reduce the amount of future audio required for an output, and this remains buffered bidirectional inference.

Five-minute Q8 buffered stream	Before	After	Less processing time
CUDA	3.154 s	2.735 s	13.3%
Vulkan	4.451 s	3.911 s	12.1%

These are medians of five streams after one warmup, with identical default contexts, 3200-sample input chunks, eight inference threads and OMP_NUM_THREADS=8, OMP_WAIT_POLICY=PASSIVE for both binaries. Input is fed without real-time delays. Timing includes preprocessing/decoding and excludes model load/file I/O. Logged frontend time falls by about 73-76%. Use the controlled-before/ and controlled-after/ artifacts for these measurements; unrestricted-thread exploratory trials were sensitive to background VM load. A separate single-stream CPU trial without warmup, using the original automatic frontend thread settings, measured 71.377 -> 69.696 s. That small CPU difference is indicative only; encoder computation dominates its runtime.

Reproduce a before/after sequence with the same model and audio on each binary:

python tests/parakeet_tdt/compare_bark3_streaming.py \
  --before-bin build-before/bin/parakeet_warm_bench \
  --after-bin build/bark3/bin/parakeet_warm_bench \
  --model models/underdog-bark-3.0-pro-q8_0.gguf --backend cuda \
  --audio short-16k-mono.wav long-16k-mono.wav short-16k-mono.wav \
  --output-dir build/bark3/streaming-comparison

VAD, hotwords and lifecycle

There are 66 passing server checks, 22 per backend on CPU/CUDA/Vulkan:

• The six reference recordings in one long-lived Q8 session.
• Zero-score and explicit-empty hotwords preserve the original transcript; enabled word lists and word/spoken variants run. Raw-PCM HTTP/SSE streams accept a vocabulary prompt and then an unbiased request in the same server.
• Two speech clips separated by silence return both reference transcripts through offline Silero VAD, with word offsets inside the original recording. Each independent span receives its own model prefix. Silence-only VAD returns empty text, and max-token limits are respected across spans.
• Malformed hotwords, excessive scores, zero/non-finite VAD duration and unknown chunk mode return HTTP 400; a following valid request succeeds.

A separate default-path Silero CLI check reproduces LibriSpeech exactly. A bounded high-strength positive control forces a requested new token sequence, verifying hotword selection is active; it is not a recommended quality setting. VAD/context changes and intentional biasing need not match the unbiased whole recording. These are native features, not features of the publisher's runner.

Relevant CTests pass in CPU/GPU builds: model-local prefix/streaming/resampling, allocation-lifetime/value checks, variant config, hotword policy, GGUF resource source and model spec. Five Bark converter checks pass. Stock Parakeet golden offline/long-form/stereo/fixed-window and streaming/reset regressions pass; Phonon's unchanged converter checks and loader/catalog sync are also checked.

Limits and artifacts

English only is the checkpoint's declared training/support scope. Its unlimited bidirectional attention and whole-utterance normalization prevent standard native KV caching from preserving old encoder activations as audio grows; streaming remains buffered. Full-context attention still grows with duration and retains the existing encoder frame limit. VAD is offline-only. Quantization can differ on other recordings despite exact matches here. Metal/HIP and large-corpus accuracy are untested.

Current local evidence is under outputs/ub3p/: memory-fix/final/ (whole-file runs and graph-buffer traces), memory-fix/matrix/ (81 reference checks), features-test/ (66 HTTP/SSE checks), models-features.json, and q8-profiles/ (profile comparisons and old Q8 backup). original-comparison/ retains the original reference outputs and baseline timing. Changes are model-local and were kept separate from shared backend/kernel work. streaming-optimization/ contains the streaming baseline/updated benchmarks, 81 comparison checks in matrix/, and the repeated HTTP/SSE checks in features-test/.