voice_audio_multi/tests/mira_tts/README.md

MiraTTS validation

This directory follows the long-lived validation pattern used for the OuteTTS port in PR #63. The model is loaded once and exercised by a sequence containing cold, repeat, changed-prompt, long-form, and post-long-form repeat requests. Streaming has a separate long-lived sequence so that offline and streaming sessions do not duplicate model weights in VRAM.

The benchmark reports one JSON object per request with wall time, generated duration, real-time factor, sample/frame counts, a deterministic audio hash, and output path. Streaming runs additionally report first-event latency and the number of progressively emitted audio events.

Build

cmake -S . -B build/windows-cuda-release `
  -DGGML_CUDA=ON -DENGINE_BUILD_WARMBENCH=ON `
  -DCMAKE_BUILD_TYPE=Release
cmake --build build/windows-cuda-release --config Release `
  --target mira_tts_warm_bench -j 8

Native long-lived runs

Set paths once:

$bench = "build/windows-cuda-release/bin/mira_tts_warm_bench.exe"
$model = "../models_v3_test/MiraTTS-GGUF/mira-tts-bf16.gguf"
$voice = "../models_v3_test/MiraTTS-comparison/00-reference-voice.wav"
$spec = "model_specs/mira_tts.json"
$out = "build/logs/warmbench/mira_tts"

Run the five-case offline sequence:

& $bench --model $model --model-spec-override $spec --backend cuda `
  --voice-ref $voice --request-file tests/mira_tts/offline_requests.json `
  --audio-out-dir "$out/native-offline" `
  --summary-file "$out/native-offline.json" `
  --log-file "$out/native-offline.log"

Run the repeated streaming sequence:

& $bench --model $model --model-spec-override $spec --backend cuda `
  --run-mode streaming --voice-ref $voice `
  --request-file tests/mira_tts/streaming_requests.json `
  --audio-out-dir "$out/native-streaming" `
  --summary-file "$out/native-streaming.json" `
  --log-file "$out/native-streaming.log"

Validate deterministic repeats, streaming events, durations, hashes, and WAV readability (with FFprobe when available):

python tests/mira_tts/validate_bench.py `
  --offline-summary "$out/native-offline.json" `
  --streaming-summary "$out/native-streaming.json"

For a memory measurement, start the benchmark with --hold-seconds 30 and sample the process plus the selected GPU while it is holding the loaded session. Record both peak host RSS and peak device memory; do not report only the final idle value.

Trusted Python sequence

Run the same offline cases through a local, revision-pinned upstream snapshot:

python tools/community_models/mira_tts_reference_bench.py `
  --model-dir <verified-upstream-snapshot> --reference $voice `
  --request-file tests/mira_tts/offline_requests.json `
  --output-dir "$out/python-offline" `
  --summary-file "$out/python-offline.json"

Compare every matched request. Exact frame count is required for the strongest deterministic parity run:

python tools/community_models/compare_mira_tts_outputs.py `
  --cpp-summary "$out/native-offline.json" `
  --python-summary "$out/python-offline.json" `
  --output "$out/parity.json" --require-exact-frames

The default gates are waveform cosine >= 0.95 and log-mel cosine >= 0.95. Threshold changes must be justified by a saved artifact and must not hide a frame-count, token-boundary, or sampling mismatch.

Acceptance matrix

Check	Required evidence
Cold/repeat determinism	clone_cold, clone_repeat, and clone_repeat_after_longform have identical hashes
Long-lived lifecycle	All offline requests finish in one process without reloading the model
Long form	longform produces non-empty audio within its token budget
Streaming	More than one event, finite first-event latency, merged non-empty WAV
Streaming determinism	stream_cold and stream_repeat have identical hashes and event counts
Python/native parity	Per-case WAV cosine, log-mel cosine, and frame counts are recorded
Audio validity	Every WAV is readable by FFmpeg and has the reported sample rate/channels
Resource use	Peak RSS, peak GPU memory, wall time, duration, and RTF are recorded
Backends	CUDA is required; CPU/Vulkan results or explicit limitations are documented

Run deterministic checks before experimenting with unseeded sampling. If a repeat hash changes, inspect generated speech-token boundaries, prompt/context tokens, and component traces before comparing subjective audio quality.

Exact-output optimization comparisons

For performance-only changes, run the same fixtures with the same model, reference, seed, thread count, device, and request options before and after the change. Then compare each offline/streaming summary pair:

python tests/mira_tts/compare_optimization.py --before <baseline.json> --after <candidate.json>
python -m unittest discover -s tests/mira_tts -p test_compare_optimization.py

The comparison rejects changed request sets, audio hashes, WAV bytes, sample metadata, or streaming event counts. It reports timing changes without treating a single noisy measurement as a performance failure. Run validate_bench.py as well to check repeat determinism and streaming lifecycle behavior. Relative WAV paths are resolved from the current directory, normally the repository root.