# Drama 3 regression tests

## Offload Regression Group

The [case group](offload_regression_cases.json) covers three noise seeds, the
standard five-request sequence, three long-short-long sequences, and resident
controls. Each case starts a fresh session; its requests share that session.

```bash
RUN_DIR="build/logs/drama3-regression/$(date +%Y%m%d-%H%M%S)"
mkdir -p "$RUN_DIR"
conda run --no-capture-output -n qwen3-tts python tools/audiocpp_cli/run_audiocpp_cli_path_tests.py \
  --cases tests/community_models/fish_drama3/offload_regression_cases.json \
  --models-root /path/to/models \
  --audiocpp-cli-bin build/debug/bin/audiocpp_cli \
  --backend cuda --threads 8 --measure-resources --resource-sample-ms 25 --log \
  --out-root "$RUN_DIR" > "$RUN_DIR/runner.log" 2>&1
```

Run the whole group sequentially and preserve the baseline. Check repeat hashes,
noise and transcript quality, baseline parity, peak VRAM, and end-to-end RTF.
Exclude only the standard sequence's warmup when averaging warm RTF.

## Unit and HTTP Tests

`fish_drama3_backbone_allocation_test` injects failures during persistent-buffer
and graph allocation, checks that repeated failures retain no buffers, and then
runs a decode step. It covers both allocator paths with a small CPU fixture and
requires no model files. The test compiles the model constructor with local
allocation wrappers; the production build has no fault-injection hooks.

`fish_drama3_codec_source_test` compares packed codec weights with independent
split checkpoints in F32, F16 and BF16. It also checks existing split weights,
partial layouts and malformed shapes. `fish_drama3_expert_quantization_test`
checks explicit expert-bank overrides in Q2_K, Q3_K, Q4_K, Q5_K, Q6_K and Q8_0,
while preserving the default rank-two conversion policy and rejecting ordinary
rank-three convolution overrides. Neither test needs model files or a GPU.

These checks compare the native Qwen3.5 tokenizer and NFC normalization with
32 vectors generated by the official Drama 3 tokenizer. They also check five
malformed UTF-8 inputs and recovery using the same tokenizer instance.

Only `tokenizer.json` and `tokenizer_config.json` from
[fishaudio/drama-3](https://huggingface.co/fishaudio/drama-3) are required.
Place them in `models/Drama-3-source`. Model weights and a GPU are unnecessary.

```powershell
uv run --no-project --python 3.12 --with huggingface_hub hf download fishaudio/drama-3 tokenizer.json tokenizer_config.json --revision dd64a85cd82e9521f7202610ede97b367fc83946 --local-dir models/Drama-3-source
```

From a Visual Studio Developer PowerShell with CMake and Ninja available:

```powershell
cmake -S . -B build/drama3-tokenizer-tests -G Ninja -DCMAKE_BUILD_TYPE=Release -DAUDIOCPP_MODEL_SET=custom -DAUDIOCPP_MODELS=fish_drama3 -DENGINE_ENABLE_CUDA=OFF -DENGINE_ENABLE_VULKAN=OFF -DENGINE_BUILD_TESTS=ON -DAUDIOCPP_DRAMA3_TOKENIZER_DIR="$PWD/models/Drama-3-source"
cmake --build build/drama3-tokenizer-tests --target fish_drama3_backbone_allocation_test fish_drama3_codec_source_test fish_drama3_expert_quantization_test fish_drama3_tokenizer_test model_spec_system_test gguf_tensor_source_test quantize_chunking_test text_chunking_test streaming_audio_test --parallel 4
ctest --test-dir build/drama3-tokenizer-tests -R '^(fish_drama3_(backbone_allocation|codec_source|expert_quantization|tokenizer)_test|model_spec_system_test|gguf_tensor_source_test|quantize_chunking_test|text_chunking_test|streaming_audio_test)$' --output-on-failure
```

The same CMake options work on Linux. Use an absolute tokenizer directory and
a shell-appropriate path in place of the PowerShell `$PWD` expression.
This selection runs the four Drama 3 tests and five existing framework tests.

The vectors cover combining marks, several scripts, numbers, delivery and
speaker tags, whitespace, empty input, and an embedded NUL supplied as bytes.
To regenerate expected IDs directly with the official tokenizer, use
[uv](https://docs.astral.sh/uv/):

```powershell
uv run --no-project tests/community_models/fish_drama3/tokenizer_vectors.py --tokenizer models/Drama-3-source/tokenizer.json --output local/drama3-tokenizer-vectors.json
```

Without `AUDIOCPP_DRAMA3_TOKENIZER_DIR`, the target can still be built and run
directly with the sidecar directory and vector JSON as its two arguments.
Normal builds do not download tokenizer files or model weights.

To check a downloaded GGUF's embedded resource specification and actual codec
weights, pass its path to the codec test. This compares every projection with
the corresponding rows of the original packed weight. It needs no GPU and
does not load backbone weights:

```powershell
.\build\drama3-tokenizer-tests\bin\fish_drama3_codec_source_test.exe models\Drama-3-GGUF\drama-3-backbone-8bit-experts-Q4_K_S.gguf
```

Multiple GGUF paths can be checked in one invocation.

For inference checks, start the server from the
[model guide](../../../docs/models/fish_drama3.md) with model id
`drama-3`, then run the portable HTTP test from the repository root:

```powershell
uv run --no-project tests/community_models/fish_drama3/server_smoke.py --base-url http://127.0.0.1:8035 --output-directory build/drama3-http-smoke
```

This requires a loaded Drama 3 GGUF. It checks WAV/SSE sample parity, streamed
take artifacts, continuation, fresh reference encoding, voice cloning from
audio versus prepared codes, long-form text chunking, and a repeated seeded
request after those mixed tasks. It uses its own generated audio as the
reference. It also checks HTTP 400 responses for invalid tags, sampling limits,
VQ grids, reference/history JSON, descriptor fields, and acoustic-token artifacts.
Run the test on the server's machine because reference audio paths
must be readable by the server. WAVs and a JSON report are saved in the output
directory; no model or reference dataset is downloaded by the test.
The default per-request timeout is 600 seconds to allow a cold load from slower
storage. Use `--timeout` to adjust it.

## Warm first-audio timing

With the same server running, use the standard-library benchmark:

```powershell
uv run --no-project --python 3.12 tests/community_models/fish_drama3/benchmark.py --base-url http://127.0.0.1:8035 --output-directory build/drama3-benchmark --count 100
```

It generates and prepares a reference before timing, checks file-path and Base64
reference uploads, excludes ten warmup requests, then measures five short English
prompts in rotation. TTFA runs from the HTTP request write to the first nonempty
raw PCM chunk. Every stream is fully consumed, and repeated seeded PCM must match.
The report includes median and nearest-rank p95 TTFA and median RTF. It measures
receipt of audio bytes, not browser playback or cold model loading.

The output directory contains `summary.json`, `measured.csv`, the exact requests,
and generated WAVs. Use separate output directories for builds being compared.
Keep model, expert placement, context length, thread count, and other workloads
consistent. The documented comparison used Q4_K experts with a Q8_0 backbone,
zero CPU expert layers, context length 2048, eight threads, and concurrency one.