# Kitten TTS 2 native validation record Powered by Stellon Labs. ## Updated BF16, Q8 and hybrid Q4 (2026-10-16) Base: audio.cpp `624cf918`. Upstream weights: `f2eee74f4467855ad58fe16f1fb7b7df473263fd`. All exports use the default S3 MeanFlow decoder. Shared framework, S3, ggml and CMake files are unchanged. Windows x64, Ryzen AI 9 HX 470, RTX 4060 Ti 16 GB, CUDA 13.3, eight CPU threads. ### Conversion and runtime validation - Seven preparation/packing unit tests passed. - Small end-to-end exports verified `--type bf16`, `--type q8_0`, `--type q4_0` and the Q8 default. Tensor bytes matched the established conversion recipes; unsupported options were rejected. - The regenerated GGUFs omit custom LM export metadata. Every tensor's shape, type and payload matches its previous export, and all 48 prepared voice entries retain identical conditioning. The Q8 filename and package ID retain `v2`. - All 196 body matrices preserve full BF16 values exactly in Q8_0/Q4_0 storage. The Q4 embedding's relative L2 error against BF16 is 0.00442545 overall and 0.00405134 on speech-token rows. These are weight errors, not quality scores. - CPU and CUDA session checks passed for all three storage combinations: preset synthesis, seeded repeat, voice switching, long-form chunks, cloning, cached clone repeat, changed reference, and invalid-input handling. Outputs were finite, non-silent mono 24 kHz. - CPU and CUDA builds passed for the CLI and session probe; the CUDA server was rebuilt with the updated WebUI. Q4 CUDA down projections load as BF16 to avoid non-finite quantized execution, adding about 483 MiB of resident weights. CPU retains Q4. No student decoder or upstream TL2/emb4 format is used. ### Model contract and WebUI defaults The contract defaults projection storage to `native`. Q8 is the sole default package and UI recommendation. All three packages share `models/kitten-tts2-v2/` and keep distinct GGUF filenames. - Model-manager inspection selected `kitten_tts2_q8_v2`. - Rechecked on 2026-10-17 with an isolated package-manager fixture containing an existing `models/kitten-tts2/` installation: all three updated packages installed under `models/kitten-tts2-v2/` without overwrite. Installing and removing them preserved the old folder, sibling GGUFs, shared sidecars and unrelated files. Downloads were mocked for this check. - The existing frontend catalog resolved all three new paths and retained Q8 as the default without WebUI source changes. - Frontend catalog and Studio function checks exposed all three packages with distinct labels and submitted no precision override by default. - Svelte checks passed with zero errors/warnings; the WebUI build passed. - Final GGUFs produced identical seeded PCM with the weight-type option omitted versus explicit `native`, for all three files on both CPU and CUDA. ### English quality spot-check Earlier Voxtral Realtime ASR checks recovered the requested words in short preset and clone samples for Q8 and hybrid Q4 on both backends. In the 40-word chunked passage, Q4 CPU and Q8 CUDA omitted "A small bird sings outside the window" (17.5% normalized WER); Q8 CPU and Q4 CUDA recovered the whole text. This demonstrates output sensitivity to quantized/backend arithmetic and the need for broader listening review. It does not establish perceptual parity or superiority of either quantization. An additional Q4 Bella sample omitted "sings outside the window"; Q8 recovered the complete text. Reproduction: [updated-profile validation](README.md#updated-bf16-q8-and-hybrid-q4). ## Post-merge timing and model-local cleanup (2026-10-04) Baseline: upstream `d3ab9df288131aeb83a3c361f0eb1e464b41dea0`, which includes Kitten PR #776 and Chatterbox correctness PR #778. The follow-up starts the `session.wall_ms` timer before reference processing, uses existing framework modules for speaker normalization/activation/statistics/projection and the speech-head slice, and removes production prompt/logit/code/mel/speaker dumps. It changes only Kitten implementation, tests, and documentation. No shared framework, Chatterbox, or ggml code is changed. Speaker `abs`, temporal max pooling, and centered-variance subtraction/square/ scalar scale retain their existing ggml operations because there are no direct framework modules for them. The host speaker projection retains its BF16 rounding points and accumulation order. Low-level graph allocation, buffer I/O, weight-view initialization, and exact Q4 packing also remain model-local. The hardware, upstream assets and published Q8 GGUF are the same as listed in the environment section below. Before/after session runs use eight threads, seed 1234 and the default backend precision policy, with custom tracing unset. Only controlled CUDA component comparisons use `NVIDIA_TF32_OVERRIDE=0`. ### Output and component comparisons - CUDA and CPU-only Release builds passed for CLI, server, both Kitten probes and the model-spec system test. The model-spec test passed in both builds. - Both CPU and CUDA passed the seven-request session probe: four preset/chunked requests and three cloning requests, with cached repeats, voice/reference changes and invalid-input checks. - All **14 before/after float32 WAVs are exactly equal**, compared within the same backend (maximum sample error 0). CPU/CUDA equality is not claimed. - All six controlled component outputs are also exactly equal before/after within each backend, including speaker identity and BF16 projection. - All 13 CUDA multilingual server cases passed, advertising 48 voices and exercising English, nine other language presets, Chinese chunking and German/Chinese cloning. Prepared reference tokens/transcripts match upstream assets, and output is finite, non-silent, mono 24 kHz. The temporary server was stopped after testing. - Loader/catalog synchronization, all four preparation tests and the model-spec system test passed. Component errors against the upstream Python reference, using Bruno's reference WAV for the speaker and `component_codes.json` for the decoder: | Component | CPU maximum error | CUDA maximum error | |---|---:|---:| | XVectorSincNet normalized identity | 0.000001563 | 0.000001386 | | BF16 speaker projection, same identity input | 0 | 0 | | S3 encoder hidden states | 0.000001907 | 0.000002146 | | Two-step meanflow mel, zero noise | 0.000592232 | 0.003253222 | | HiFT pitch, same native mel | 0.000000298 | 0.000000268 | | HiFT waveform, same mel and zero random draws | 0.000013851 | 0.000015877 | All pass the existing limits. The speaker probe now writes its input WAV and output JSON through test-only code using the public speaker API. It does not require runtime dump hooks. Decoder comparisons retain the same zero-noise and fade handling described below. ### Reproduction and evidence Both existing Visual Studio 2022 x64 build directories were reconfigured and rebuilt with the following commands (PowerShell). Both caches select only `kitten_tts2`, disable llamafile, enable tests, and suppress regeneration. The CUDA cache enables CUDA with architecture 89 and disables the native model manager; the CPU-only cache disables CUDA and enables the native model manager. ```powershell cmake -S . -B build-kitten2-cuda cmake --build build-kitten2-cuda --config Release --parallel 8 ` --target audiocpp_cli audiocpp_server kitten_tts2_session_probe ` kitten_tts2_components_probe model_spec_system_test cmake -S . -B build-kitten2-pr cmake --build build-kitten2-pr --config Release --parallel 8 ` --target audiocpp_cli audiocpp_server kitten_tts2_session_probe ` kitten_tts2_components_probe model_spec_system_test ``` CPU and CUDA inference checks used `build-kitten2-cuda/bin/Release/` with the corresponding backend argument. The separate CPU-only build was compile/link and model-spec tested; it was not used for another full synthesis run. The [test README](README.md) contains the current probe/comparison commands and assets. Run the session probe once on the baseline and once on this follow-up, using separate output directories and the same model, backend, threads and seed. Compare corresponding WAVs within each backend. Local evidence is under the ignored `build-kitten2-cuda/followup-validation/`: `before.json`, `after.json`, `before-`/`after-` session logs and WAV directories, component outputs/reference-comparison logs, and `after-multilingual/report.json`. No weights or generated audio are committed. Fresh LM prompt/logit traces are no longer exposed by production inference. `check_parity.py` is retained for archived traces only. The updated multilingual test checks public API execution and prepared assets, without claiming internal BPE/token-limit checks. No new listening study, other-GPU coverage or peak-memory measurement was performed for this cleanup. ## Historical pre-merge validation (2026-10-03) The remaining sections preserve the original port's validation context and measurements, including its now-merged correctness prerequisite and old trace workflow. Use the updated test README for current commands. ### Reviewed scope and prerequisite Validated locally on 2026-10-03 after removing the shared precision changes. The Kitten implementation depends on [Chatterbox correctness PR #778](https://github.com/0xShug0/audio.cpp/pull/778), commit `f96a8e78a923aabd4874f06eef60b170358fdc80`. That separate change corrects relative-shift padding and the attention/feed-forward LayerNorm epsilon. The numerical and synthesis results below use the cleaned Kitten runtime together with that prerequisite. The starting Kitten checkout was `c41cc827` on base `2ba9fa80`; the shared CUDA, S3 and HiFT precision edits and Kitten's speaker precision override were removed before testing. The three correctness hunks from #778 were applied for the builds and then removed from the Kitten working tree. These results do not claim that the Kitten branch on its original base passes encoder parity without #778. The final Kitten diff contains no changes to `src/framework/`, `include/engine/framework/`, `external/ggml/`, or `src/models/chatterbox/`. Its implementation and headers live in `src/community_models/kitten_tts2/` and `include/engine/community_models/kitten_tts2/`. Model-specific tests, documentation, specification and preparation tools accompany it. Changes outside those model-specific files are limited to: - `CMakeLists.txt`: model registration and Kitten preparer/probe targets. - Shared documentation indexes: entries linking to Kitten's model guide. - `webui/configs/models_catalog.json`: the Kitten model-manager entry. - `webui/native/dist/index.html`: the generated interface containing that catalog entry; the WebUI imports the catalog at build time. ## Environment and assets - Windows x64, Visual Studio 2022 Build Tools, MSVC 19.44.35228.0, Release. - AMD Ryzen AI 9 HX 470, 12 physical cores, approximately 52 GiB RAM. - NVIDIA RTX 4060 Ti, 16 GiB VRAM, compute capability 8.9, driver 610.88, CUDA Toolkit 13.3.73. Eight host inference threads. - Python 3.12.10 and PyTorch 2.13.0 for offline reference comparisons. - KittenTTS source `226cb35`; Kitten model snapshot `baa41e5d2c5f64be0095365672a7858542261271`. - S3 checkpoint from ResembleAI/chatterbox-turbo revision `749d1c1a46eb10492095d68fbcf55691ccf137cd`, SHA-256 `d65cb687a2ed581ee6cc297e919ffefa63386944f42364ae13b78a594945514f`. The full inference path is native C++/GGML. Python, PyTorch, TorchScript and ONNX Runtime are not runtime dependencies. The default `decoder.pt` export is used only by the Python comparison script. ## Build and test setup The existing CUDA build was reconfigured/rebuilt with these settings (PowerShell line continuations), with #778 applied in the validation checkout: ```powershell cmake -S . -B build-kitten2-cuda -G "Visual Studio 17 2022" -A x64 ` -DAUDIOCPP_MODEL_SET=custom -DAUDIOCPP_MODELS=kitten_tts2 ` -DENGINE_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89 ` -DENGINE_ENABLE_LLAMAFILE=OFF -DENGINE_BUILD_TESTS=ON ` -DCMAKE_SUPPRESS_REGENERATION=ON cmake --build build-kitten2-cuda --config Release --parallel 8 ` --target audiocpp_cli audiocpp_server audiocpp_gguf ` audiocpp_kitten_tts2_prepare_voices kitten_tts2_session_probe ` kitten_tts2_components_probe model_spec_system_test ``` Binaries are in `build-kitten2-cuda/bin/Release/`. The same build supports CPU and CUDA selection. Both backends were exercised by the native session probe; the server language checks used CUDA. Commands and reference assets are documented in [README.md](README.md). After moving all Kitten headers from `src/` into `include/`, both the CUDA build above and the existing CPU-only `build-kitten2-pr/` build were reconfigured and rebuilt successfully for the same seven targets. Both model-spec tests passed. These builds also included #778 temporarily; its source changes were then removed from the Kitten working tree. The move changes include paths and removes the preparer's private `src/` include directory without changing the inference logic covered by the numerical runs. For strict CUDA component parity, `NVIDIA_TF32_OVERRIDE=0` was set before launching both the session probe that captured speaker intermediates and the component probe. All ordinary CPU/CUDA synthesis runs and the multilingual server run used an environment with this override removed. The production implementation leaves shared precision and dispatch policy unchanged. ## Passed checks after precision cleanup - Release build of CLI, server, converter, voice preparer, both native probes and model-spec test. - Model-spec test, loader/catalog synchronization and four preparation tests. - On both CPU and CUDA: packaged tensor types, exact seeded repeats, preset switching, three-chunk synthesis, native cloning, cached clones and changed references; rejection of unknown voices, missing transcripts and short clips. - LM parity: 301 exact prompt tokens; maximum logit error 0.253082, mean 0.052854, cosine 0.99999946; matching top speech token 2957. The controlled first preset produced 72 codes and 71,520 waveform samples. - All six controlled CUDA component comparisons, and CPU S3/HiFT comparisons. - All 13 CUDA multilingual server cases, with 48 registered voices: English Bruno, nine additional language presets, Chinese chunking, and German/Chinese cloning. Reference codec tokens and reference/target BPE matched the upstream assets. Audio was finite, non-silent mono 24 kHz, and generation ended before the token limit. The temporary server was stopped. ## Controlled component results The CPU and CUDA decoder comparisons below use the same speech codes and weights. Random draws are fixed to zero for the meanflow solver; HiFT receives identical native mel within each backend comparison, with zero phase/noise draws. CUDA comparisons use the test-only TF32 setting above. | Component | CPU maximum error | CUDA maximum error | |---|---:|---:| | XVectorSincNet normalized identity | Not repeated in this pass | 0.000003476 | | BF16 speaker projection, same identity input | Not repeated in this pass | 0 | | S3 encoder hidden states | 0.000001907 | 0.000002146 | | Two-step meanflow mel, zero noise | 0.000592232 | 0.003253222 | | HiFT pitch, same native mel | 0.000000298 | 0.000000268 | | HiFT waveform, same mel and zero random draws | 0.000013851 | 0.000015877 | These pass the unchanged tolerances documented in the test README. The waveform comparison excludes the first 960 samples where the upstream wrapper applies its fade. Speaker parity uses the exact native-resampled input, so it does not measure equality with upstream librosa/soxr preprocessing. A CUDA ablation with identical inputs measured encoder maximum errors of 0.318668097 without the correctness fixes, 0.000009358 with the relative-shift fix only, and 0.000002146 with both fixes. The original code fails the 0.0001 encoder tolerance; the shift-only variant passes, with the matching epsilon further reducing error. This evidence is also recorded in #778. ## Session timings without a TF32 override These historical measurements used eight threads and seed 1234. Model/session loading is excluded; the first clone includes lazy reference-encoder loading. Tracing was enabled. These are individual desktop measurements with other activity, not isolated benchmark averages or a performance comparison. | Request | CPU wall s | CPU audio s | CPU RTF | CUDA wall s | CUDA audio s | CUDA RTF | |---|---:|---:|---:|---:|---:|---:| | Bruno, first request | 8.992 | 3.10 | 2.901 | 1.199 | 2.98 | 0.402 | | Bruno, cached repeat | 5.531 | 3.10 | 1.784 | 0.963 | 2.98 | 0.323 | | Switch to Bella | 9.168 | 3.52 | 2.604 | 1.323 | 3.52 | 0.376 | | Three-chunk passage | 25.500 | 13.11 | 1.945 | 4.802 | 13.70 | 0.351 | | Clone Bruno reference, first | 10.710 | 3.62 | 2.959 | 2.535 | 3.50 | 0.724 | | Clone, cached repeat | 6.636 | 3.62 | 1.833 | 1.196 | 3.50 | 0.342 | | Change clone reference to Bella | 10.783 | 4.20 | 2.567 | 1.541 | 3.80 | 0.406 | RTF is wall time divided by generated audio duration. CPU and CUDA can generate different speech codes and durations. These checks establish execution and repeatability within a backend; they do not measure perceptual quality. ## Language coverage and earlier packaging checks The package provides Arabic, Chinese, French, German, Hindi, Italian, Portuguese, Russian and Spanish presets, plus 38 English voices and `PreparedBruno`. The 39 original prepared entries are preserved as parsed JSON; the nine additions retain upstream transcripts, NPZ identities and codec tokens. Earlier packaging checks compared all nine NPZ assets with NumPy and the BF16 speaker projections with PyTorch. Eight projections matched exactly; Spanish's maximum error was 0.00048828125, below the 0.032 tolerance. Earlier CPU and CUDA server runs also passed 13 multilingual cases. The current cleanup repeats the CUDA server cases; it does not claim a new CPU multilingual server run. Earlier Svelte checks and interface builds passed. The header/precision cleanup does not change the UI catalog or its generated bundle. Existing CLI/server import checks found no Torch, Python or ONNX Runtime DLL dependency. ## VRAM estimate (earlier measurement) The model spec and WebUI retain the 7 GB estimate for the native Q8 package. This estimate came from the earlier implementation with shared precision edits, before the cleanup. It has not been remeasured after those edits were removed. A server was sampled every 200 ms during a 238-character preset request and a clone request with the same text, retaining both in one session: | Measurement | Total GPU memory (MiB) | |---|---:| | Desktop baseline before starting the server | 3,497 | | Peak during preset speech | 8,316 | | Peak during cloning | 9,530 | | After stopping the server | 3,523 | The baseline-subtracted deltas were approximately 4.71 GiB for preset speech and 5.89 GiB for cloning. These are sampled total-device deltas, not isolated per-process allocations. Seven GB provides guidance with headroom, not a guaranteed maximum or verified minimum for all inputs and weight types. ## Evidence and limits Current logs, reports, WAVs and traces remain local under the ignored `build-kitten2-cuda/review-split/` directory: `results.json`, `combined-build.log`, `cpu-session.log`, `cuda-session.log`, `controlled-cuda-reference.log`, `both-fixes-cpu-reference.log`, `lm-parity.json`, and `multilingual/report.json`. Baseline comparison failures in this directory are intentional ablations. No weights or generated audio are committed. Earlier logs remain in `build-kitten2-pr/` and `build-kitten2-cuda/`. The header-move rebuild results are in `header-rebuild.json` and the `headers-{cpu,cuda}-{configure,build,model-spec}.log` files in the same directory. - These historical runs required #778 separately; current upstream includes it. - CUDA was tested on one RTX 4060 Ti; other GPU backends, NVIDIA generations, sustained-load stability and worst-case memory remain unvalidated. - Only the full default decoder is ported; student decoders are unsupported. - Cloning requires a transcript and a 1–30 second clip. Automatic ASR, streaming and upstream written-to-spoken normalization are not implemented. - Language checks establish execution and text preservation. Pronunciation, intelligibility, speaker similarity, every English voice, and the ten other upstream-advertised languages have not been comprehensively evaluated. - The component harness exercises reused Chatterbox S3 arithmetic. Separate end-to-end Chatterbox speech/voice-conversion quality tests were not performed.