voice_audio_multi/docs/reports/outetts_validation.md

# OuteTTS validation

This procedure exercises OuteTTS 1.0 1B in one long-lived audio.cpp session.
It covers normal TTS, framework long-form text chunking, voice cloning, repeated
reference-profile cache hits, cached-step graph reuse, stage timings, and
default-versus-`mem_saver` memory behavior.

## Model setup

Install the safetensors model, IBM DAC, and Qwen3 Forced Aligner resources:

```bash
python tools/model_manager_deprecated.py install outetts_1_0_1b --models-dir models
```

The model-manager package id is `outetts_1_0_1b`. It creates these model
roots:

- `models/Llama-OuteTTS-1.0-1B`
- `models/DAC.speech.v1.0`
- `models/Qwen3-ForcedAligner-0.6B`

The standalone GGUF command is documented in [TTS](tts.md#outetts). The packed
file used below contains the OuteTTS language model, IBM DAC, Qwen aligner,
tokenizers, configuration, and package specification.

## Python reference setup

The parity run used the official OuteTTS repository at commit
`f5eac6e70d792844c6a6959d900a47af2c061a5b` (`outetts` 0.4.4), Python
3.10, PyTorch 2.5.1+cu121, and Transformers 4.52.3:

```powershell
git clone https://github.com/edwko/OuteTTS build\reference\OuteTTS
git -C build\reference\OuteTTS checkout f5eac6e70d792844c6a6959d900a47af2c061a5b
python -m venv --system-site-packages build\reference\venv
build\reference\venv\Scripts\python.exe -m pip install -e build\reference\OuteTTS --no-deps
build\reference\venv\Scripts\python.exe -m pip install transformers==4.52.3 llama-cpp-python==0.3.9 polars natsort mecab-python3 unidic-lite uroman openai-whisper ftfy pyloudnorm
```

The reference loaded `models/Llama-OuteTTS-1.0-1B` through
`outetts.Backend.HF` with FP32 CUDA weights and loaded the official IBM DAC
checkpoint from
`models/DAC.speech.v1.0/weights_24khz_1.5kbps_v1.0.pth`. Both implementations
used temperature `0.4`, repetition penalty `1.1` over the latest 64 tokens,
top-k `40`, top-p `0.9`, min-p `0.05`, the request-file seeds, and the same
maximum number of new tokens.

Temperature zero was deliberately not used. OuteTTS 1.0's official
`generation_config.json`, README example, and Python API all select sampled
generation with temperature `0.4`; changing to greedy generation would test a
different inference contract. OuteTTS has no diffusion or latent-noise input.
Its only request-time randomness is language-model token sampling.

For a controlled clone comparison, Qwen word timings were generated once and
passed to the official Python `AudioProcessor.create_speaker_from_dict` path:

```powershell
build\windows-cuda-release\bin\audiocpp_cli.exe `
  --task align --family qwen3_forced_aligner `
  --model ..\models\Qwen3-ForcedAligner-0.6B_Q8\Qwen3-ForcedAligner-0.6B_Q8.gguf `
  --backend cuda --audio assets\resources\b.wav `
  --text "Some call me nature. Others call me Mother Nature. I've been here for over 4.5 billion years. 22,500 times longer than you." `
  --language en --words-out build\reference\b_words.json
```

Python still performed the official DAC encoding and feature extraction. This
keeps different alignment algorithms from contaminating the TTS comparison.
The audio.cpp cold-clone timing includes its Qwen alignment, while the Python
number starts from the supplied word timings.

Run the committed official-reference driver after creating the alignment JSON:

```powershell
build\reference\venv\Scripts\python.exe `
  tools\community_models\outetts_reference.py `
  --model ..\models\Llama-OuteTTS-1.0-1B `
  --dac ..\models\DAC.speech.v1.0\weights_24khz_1.5kbps_v1.0.pth `
  --alignment-json build\reference\b_words.json `
  --request-file tests\outetts\warm_bench_requests.json `
  --device cuda --dtype fp32 `
  --out-dir build\reference\official_script_fp32
```

The driver resets both `torch.manual_seed` and
`torch.cuda.manual_seed_all` immediately before every `model.generate` call,
after all prompt, alignment, and DAC-profile work. It writes native 24 kHz
WAVs plus `boundaries.json`, `memory.json`, `stdout.log`, and `command.json`.

## Exact build commands

The Windows validation used the project build script to establish the compiler
and backend configuration, then explicitly enabled the warm-benchmark targets.

Windows CPU:

```powershell
.\scripts\build_windows.ps1 -Preset windows-cpu-release -ConfigureOnly
cmake -S . -B build\windows-cpu-release -DENGINE_BUILD_WARMBENCH=ON
cmake --build build\windows-cpu-release --parallel 4 --target audiocpp_cli audiocpp_server outetts_warm_bench
```

Windows CUDA 12.4:

```powershell
$env:CUDA_PATH = "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4"
.\scripts\build_windows.ps1 -Preset windows-cuda-release -ConfigureOnly
cmake -S . -B build\windows-cuda-release -DENGINE_BUILD_WARMBENCH=ON
cmake --build build\windows-cuda-release --parallel 4 --target audiocpp_cli audiocpp_server outetts_warm_bench
```

## Exact standalone-GGUF path tests

The tested GGUF was deliberately outside the PR worktree and its directory
contained only one file:

```text
..\models\Llama-OuteTTS-1.0-1B_Q8\Llama-OuteTTS-1.0-1B_Q8.gguf
```

Normal TTS:

```powershell
build\windows-cuda-release\bin\audiocpp_cli.exe `
  --task tts --family outetts `
  --model ..\models\Llama-OuteTTS-1.0-1B_Q8\Llama-OuteTTS-1.0-1B_Q8.gguf `
  --backend cuda `
  --text "This is the standalone GGUF path test." `
  --max-tokens 256 --request-option seed=1234 `
  --out ..\outputs\outetts_path_test_tts.wav
```

Voice cloning using the aligner and DAC embedded in the same GGUF:

```powershell
build\windows-cuda-release\bin\audiocpp_cli.exe `
  --task clon --family outetts `
  --model ..\models\Llama-OuteTTS-1.0-1B_Q8\Llama-OuteTTS-1.0-1B_Q8.gguf `
  --backend cuda `
  --voice-ref assets\resources\b.wav `
  --reference-text "Some call me nature. Others call me Mother Nature. I've been here for over 4.5 billion years. 22,500 times longer than you." `
  --request-option reference_language=en `
  --text "This is the standalone GGUF cloning path test." `
  --max-tokens 256 --request-option seed=42 `
  --out ..\outputs\outetts_path_test_clone.wav
```

Both commands loaded successfully without an external package spec, tokenizer,
DAC directory, aligner directory, or other sidecar. The generated 24 kHz mono
WAVs passed ffmpeg decoding:

| Artifact | Duration | Bytes | SHA-256 |
|---|---:|---:|---|
| `..\outputs\outetts_path_test_tts.wav` | 1.399667s | 67228 | `92DA5D8438D4E6A79FBDDBD8EDA77D1AFD2A9B47F61085ECD3AD45C913102904` |
| `..\outputs\outetts_path_test_clone.wav` | 1.453000s | 69788 | `E5B46E62C2DB89A654F7EC21A7BE31F575D6F667F8578224386E7C552492B05E` |

## Exact long-lived-session runs

CUDA default:

```powershell
build\windows-cuda-release\bin\outetts_warm_bench.exe `
  --model ..\models\Llama-OuteTTS-1.0-1B_Q8\Llama-OuteTTS-1.0-1B_Q8.gguf `
  --backend cuda --threads 8 `
  --request-file tests\outetts\warm_bench_requests.json `
  --hold-seconds 5 `
  --audio-out-dir ..\outputs\outetts_review_cuda_default `
  --log-file build\logs\warmbench\outetts-cuda-default.log
```

Expected trace evidence:

- `outetts.text_chunk_count` is greater than one for `tts_longform`.
- `outetts.llama.step.graph_reused=1` appears after the first compatible
  generation request or chunk.
- the second identical reference reports `outetts.reference_cache.hit=1`.
- `tts_cold` and `tts_repeat` are byte-identical, as are `clone_cold`
  and `clone_repeat`; this verifies that warm graph/profile reuse does not
  change deterministic output.
- `outetts.aligner.runtime_reused=1` is observable for uncached references
  while the default session retains the aligner.
- only one active OuteTTS Llama runtime is retained; switching between native
  TTS weights and the CUDA F32 cloning fallback replaces the previous runtime.

Run the same request sequence in a fresh process with memory saver enabled:

```powershell
build\windows-cuda-release\bin\outetts_warm_bench.exe `
  --model ..\models\Llama-OuteTTS-1.0-1B_Q8\Llama-OuteTTS-1.0-1B_Q8.gguf `
  --backend cuda --threads 8 `
  --request-file tests\outetts\warm_bench_requests.json `
  --session-option outetts.mem_saver=true `
  --hold-seconds 5 `
  --audio-out-dir ..\outputs\outetts_review_cuda_mem_saver `
  --log-file build\logs\warmbench\outetts-cuda-mem_saver.log
```

CPU default:

```powershell
build\windows-cpu-release\bin\outetts_warm_bench.exe `
  --model ..\models\Llama-OuteTTS-1.0-1B_Q8\Llama-OuteTTS-1.0-1B_Q8.gguf `
  --backend cpu --threads 8 `
  --request-file tests\outetts\warm_bench_requests.json `
  --hold-seconds 5 `
  --audio-out-dir ..\outputs\outetts_review_cpu_default `
  --log-file build\logs\warmbench\outetts-cpu-default-final.log
```

Safetensors CUDA parity run (DAC is resolved by the package spec; the aligner
path is supplied because it is not embedded in the safetensors directory):

```powershell
build\windows-cuda-release\bin\outetts_warm_bench.exe `
  --model ..\models\Llama-OuteTTS-1.0-1B `
  --backend cuda --threads 8 `
  --request-file tests\outetts\warm_bench_requests.json `
  --session-option outetts.weight_type=f32 `
  --session-option outetts.aligner_path=..\models\Qwen3-ForcedAligner-0.6B `
  --audio-out-dir build\reference\cpp_f32_final_explicit `
  --log-file build\reference\cpp_f32_final_explicit.log
```

The memory-saver trace reports a positive
`outetts.llama.step.released_cache_capacity` after generation and
`outetts.aligner.runtime_released=1` after an uncached reference.

Sample total-device VRAM every 250 ms while each fresh benchmark runs. Ensure
that no unrelated CUDA workload is active:

```powershell
nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits -lms 250
```

Compare peak and final resident VRAM, request wall time, output duration, and RTF
between the two runs. Keep model, backend, device, seed, requests, and
quantization identical.

## Generated artifacts

Each benchmark directory contains:

```text
tts_cold_1.wav
tts_repeat_1.wav
tts_longform_1.wav
clone_cold_1.wav
clone_repeat_1.wav
```

The exact generated directories and logs were:

- `..\outputs\outetts_review_cuda_default`
- `..\outputs\outetts_review_cuda_mem_saver`
- `..\outputs\outetts_review_cpu_default`
- `build\logs\warmbench\outetts-cuda-default.log`
- `build\logs\warmbench\outetts-cuda-mem_saver.log`
- `build\logs\warmbench\outetts-cpu-default-final.log`
- `build\logs\warmbench\outetts-ab-torch-sampler\default-{1,2,3}-audio`
- `build\logs\warmbench\outetts-ab-torch-sampler\mem_saver-{1,2,3}-audio`
- `build\reference\cpp_f32_final_explicit`
- `build\reference\cpp_q8_torch_sampler`
- `build\reference\official_script_fp32\outputs`
- `build\reference\official_script_fp32\boundaries.json`

These validation artifacts are reproducible local outputs and are not committed
to the repository.

## Measured validation

The committed five-request sequence was measured with the packed Q8 GGUF on an
NVIDIA GeForce RTX 3090 (CUDA 12.4). Each CUDA mode was run in three fresh
processes in alternating order. Values below are mean +/- sample standard
deviation. The Python HF and audio.cpp safetensors columns are single loaded-
session runs; model load is excluded from every per-request measurement.

| Request | Python HF FP32 wall / RTF | C++ FP32 wall / RTF | C++ Q8 wall / RTF |
|---|---:|---:|---:|
| `tts_cold` | 4599.12 ms / 3.485 | 5366.24 ms / 4.066 | 3211.84 +/- 17.30 ms / 2.434 |
| `tts_repeat` | 3777.69 ms / 2.863 | 3864.98 ms / 2.929 | 3030.79 +/- 18.60 ms / 2.297 |
| `tts_longform` | 15129.91 ms / 2.690 | 15547.80 ms / 2.764 | 13015.83 +/- 171.29 ms / 2.342 |
| `clone_cold` | 5762.51 ms / 3.792 | 5988.01 ms / 3.940 | 6432.68 +/- 65.11 ms / 4.233 |
| `clone_repeat` | 5522.67 ms / 3.634 | 4231.19 ms / 2.784 | 4220.23 +/- 44.07 ms / 2.777 |

These are per-request timings from a single loaded session; model loading is
excluded. Python and C++ FP32 generated 1.31967, 1.31967, 5.62533, 1.51967,
and 1.51967 seconds respectively. Q8 generated 5.55867 seconds for the
long-form request and matched the other four durations.

The memory-saver A/B is also reported per request rather than inferred from a
single aggregate run:

| Request | Default CUDA Q8 | Memory saver CUDA Q8 | Wall delta | Default RTF | Memory-saver RTF |
|---|---:|---:|---:|---:|---:|
| `tts_cold` | 3211.84 +/- 17.30 ms | 3223.24 +/- 12.89 ms | +0.35% | 2.434 | 2.443 |
| `tts_repeat` | 3030.79 +/- 18.60 ms | 3044.58 +/- 18.76 ms | +0.45% | 2.297 | 2.307 |
| `tts_longform` | 13015.83 +/- 171.29 ms | 13070.17 +/- 105.32 ms | +0.42% | 2.342 | 2.351 |
| `clone_cold` | 6432.68 +/- 65.11 ms | 6467.00 +/- 46.38 ms | +0.53% | 4.233 | 4.256 |
| `clone_repeat` | 4220.23 +/- 44.07 ms | 4238.27 +/- 17.07 ms | +0.43% | 2.777 | 2.789 |

Memory saver was slightly slower on every request in this controlled repeat:
0.35% to 0.53%, and 0.44% over the mean five-request wall total. These small
differences are near normal run-to-run variance, so there is no evidence that
memory saver is a speed optimization. It remains useful
because the independently sampled peak VRAM fell from 17653 MiB to 5780 MiB
(67.3%) with the same 294 MiB post-session resident VRAM. It is therefore a
memory-versus-speed option, not a speed optimization.

The Windows CPU Q8 run is likewise reported per request:

| Request | Wall | Audio | RTF |
|---|---:|---:|---:|
| `tts_cold` | 10517.2 ms | 1.31967s | 7.970 |
| `tts_repeat` | 10349.9 ms | 1.31967s | 7.843 |
| `tts_longform` | 41604.0 ms | 5.67867s | 7.326 |
| `clone_cold` | 22849.3 ms | 1.58633s | 14.404 |
| `clone_repeat` | 18205.6 ms | 1.58633s | 11.477 |

All generated WAV files passed an ffmpeg decode check. The traces confirmed
four framework chunks for the long-form request, compatible step-graph reuse in
default mode, explicit graph release in memory-saver mode, and a reference-
profile cache hit on the repeated clone. In the original instrumented default
CUDA run, cold reference preparation took about 890 ms (109 ms alignment,
139 ms DAC encoding, and 214 ms profile construction), while the repeated
cached reference took 0.29 ms.

## Path and parity results

- The standalone path test loaded the one-file GGUF from outside the worktree;
  no adjacent sidecars or `model_specs` directory were present.
- Within CUDA, cold and repeated TTS were byte-identical with SHA-256
  `9D1AF0D25212129B3C26880F7C928CFFA813A4B9EF22F12F390A4C372D51D406`.
- Within CUDA, cold and repeated cloning were byte-identical with SHA-256
  `EB7E6B5A1330F845F45D3DA21C8AC28490572F95218A6F0D3DA2A83CB797EFE6`.
- The CUDA default and memory-saver outputs had the same hashes.
- Within CPU, cold and repeated TTS were byte-identical with SHA-256
  `A315C0AD425C99910597FCC4BAB9D80CB5F4C893176EBD9099BA6C6AA7B66BFD`.
- Within CPU, cold and repeated cloning were byte-identical with SHA-256
  `1A2009C0D512773A4B5082BF37EDEDD6A52A37CC6D556772C62EA26550A83B4D`.
- CPU and CUDA WAV bytes are not identical because backend floating-point
  execution differs; deterministic reuse parity was therefore checked within
  each backend rather than asserted across backends.

Python parity uses the repository's
`tools/audiocpp_cli/compare_audiocpp_cli_path_results.py` implementation: PCM
WAV cosine plus an 80-band `log1p` mel-spectrogram cosine (`n_fft=1024`,
`hop=256`). Both sides write the DAC-native 24 kHz signal directly. Raw WAV
cosine is highly phase-sensitive; log-mel cosine is the more informative value
when the DAC encoders are numerically close but not bit-identical.

The CUDA sampler now uses the framework's PyTorch-compatible exponential-race
random stream, equivalent to the `torch.multinomial` path used by Transformers.
Temporary trace-level instrumentation compared prompt strings, prompt token
IDs, generated token IDs, codec codes, and WAV frame counts at component
boundaries. It was removed after validation to avoid normal-log spam. The
permanent Python driver records the Python side in `boundaries.json`.

Boundary results under the controlled FP32 run:

| Request | Chunks | Python/C++ prompt IDs | Generated IDs per chunk | Python/C++ frames |
|---|---:|---:|---:|---:|
| `tts_cold` | 1 | 22 / 22 | 256 | 31672 / 31672 |
| `tts_repeat` | 1 | 22 / 22 | 256 | 31672 / 31672 |
| `tts_longform` | 4 | 20,22,14,21 / same | 256 each | 135008 / 135008 |
| `clone_cold` | 1 | 2367 / 2367 | 256 | 36472 / 36472 |
| `clone_repeat` | 1 | 2367 / 2367 | 256 | 36472 / 36472 |

The sampled prompt/generated boundary values for all six ordinary-TTS chunks
had zero mismatches. Clone prompt construction also matches the official word
labels and token count. The C++ DAC reference encoder still differs slightly
from Python: among 1046 reference-profile frames, C1 differs on 48 and C2 on
118. That numerical boundary explains why cloning is not waveform-identical.

Primary native/FP32 parity, measured only after the controlled random stream
and frame counts matched:

| Request | WAV cosine | Log-mel cosine | C++ / Python frames |
|---|---:|---:|---:|
| `tts_cold` | 0.999812247 | 0.999997481 | 31672 / 31672 |
| `tts_repeat` | 0.999812238 | 0.999997483 | 31672 / 31672 |
| `tts_longform` | 0.997324530 | 0.999294328 | 135008 / 135008 |
| `clone_cold` | -0.007753064 | 0.889403845 | 36472 / 36472 |
| `clone_repeat` | -0.007753064 | 0.889403845 | 36472 / 36472 |

Only after that path was clean was the packed Q8 model compared with the same
FP32 Python reference. Quantization is expected to perturb autoregressive
sampling, so this is a secondary characterization rather than the correctness
gate:

| Request | WAV cosine | Log-mel cosine | C++ Q8 / Python frames |
|---|---:|---:|---:|
| `tts_cold` | -0.016326171 | 0.696795792 | 31672 / 31672 |
| `tts_repeat` | -0.016326591 | 0.696792751 | 31672 / 31672 |
| `tts_longform` | 0.577927575 | 0.920741352 | 133408 / 135008 |
| `clone_cold` | -0.033990775 | 0.882715855 | 36472 / 36472 |
| `clone_repeat` | -0.033990775 | 0.882715855 | 36472 / 36472 |

## Backend coverage

| Backend | Coverage |
|---|---|
| Windows CPU | CLI/server/warm-benchmark build; normal TTS, long-form TTS, cloning, cache reuse, graph reuse, WAV decode, timing, and RSS run |
| Windows CUDA 12.4 | CLI/server/warm-benchmark build; normal TTS, long-form TTS, cloning, standalone-path loading, cache/graph reuse, three-run default/memory-saver A/B, Python HF parity, WAV decode, timing, RTF, and VRAM run |
| Linux CPU | GitHub Actions compile check passed for CLI, server, and GGUF converter; no model runtime measurement claimed |
| Linux Vulkan | GitHub Actions compile check passed for CLI, server, and GGUF converter; no model runtime measurement claimed |
| macOS CPU | GitHub Actions compile check passed for CLI, server, and GGUF converter; no model runtime measurement claimed |
| Metal | Not enabled or runtime-tested in this validation |

## Known limitations

- OuteTTS is offline-only in this implementation.
- Voice cloning requires an accurate transcript of the reference WAV and rejects
  references longer than 20 seconds.
- Quantized CUDA cloning expands the OuteTTS language-model weights to F32 for
  generation correctness. Default mode can therefore have a high transient
  peak while the aligner is retained; `outetts.mem_saver=true` releases the
  aligner and cached-step graph between phases.
- Long-form output concatenates independently generated chunks. `max_tokens`
  applies to each chunk; the runtime splits text further to fit an explicit cap
  and retries a chunk that reaches the cap without EOS/audio-end as smaller
  chunks rather than returning silently truncated speech. It reports an error
  only when the remaining text cannot be split any further.
- CPU and CUDA outputs are deterministic within each tested backend but are not
  expected to be byte-identical across backends.
- The controlled CUDA sampling stream is PyTorch-compatible. CPU sampling is
  deterministic for a fixed seed but still uses the C++ standard-library
  distribution and is not claimed to reproduce PyTorch token-for-token.
- Quantized weights can change sampled tokens and output length even when the
  seed and sampler implementation match, as shown by the Q8 long-form frame
  drift above.