voice_audio_multi/docs/models/fish_drama3.md

Fish Audio Drama 3

fish_drama3 is a community model port of fishaudio/drama-3. Inference runs in native audio.cpp with GGML; Python is used to prepare the checkpoint.

Built with Fish Audio.

The base model includes the 40-layer hybrid attention/MoE backbone, 256 experts with eight selected per token, four shared enrichment layers, nine residual codebook layers with persistent timeline caches, and the Fish DAC encoder/decoder. It supports offline and streaming TTS, voice references, speaker and event tags in text, long-form chunking, and continuation/edit history.

Variants

Both variants use --family fish_drama3 and the same CLI and server routes. The loader reads layer-specific expert counts and the output-head layout from the checkpoint, so Drama3-small does not need a separate family.

Variant	GGUF packages	Source
Drama3 base	Q2_K, Q3_K, and Q4_K expert packages	Jmica/Drama-3-GGUF
Drama3-small	fish_drama3_small_iq3_xxs: Drama3-small-GGUF/drama3-small-IQ3_XXS.gguf	pentacoxian-dev/drama3-small-GGUF

Install the small variant with:

python tools/model_manager_v2.py install fish_drama3_small_iq3_xxs --models-root models

Use --model models/Drama3-small-GGUF/drama3-small-IQ3_XXS.gguf with the generation examples below. Base-model weight sizes and memory measurements in this guide do not describe the small checkpoint.

Windows build

Install Visual Studio 2022 C++ build tools, a CUDA toolkit that supports your GPU, and Git. Use the Drama 3 branch and initialize its public frontend submodule:

git clone --branch drama-3-native https://github.com/JarodMica/audio.cpp.git
Set-Location audio.cpp
git -c submodule.external/audio.cpp-server-frontends.url=https://github.com/0xShug0/audio.cpp-server-frontends.git submodule update --init --recursive

The command-local HTTPS override lets the public frontend submodule download without GitHub SSH credentials. It does not modify .gitmodules or global Git configuration.

Run the build helper from the repository root. It selects the required model dependencies and detects the CUDA architecture:

.\scripts\build_windows.ps1 -Preset windows-cuda-release -ModelSet custom -Models fish_drama3 -Target audiocpp_cli -Jobs 4
.\scripts\build_windows.ps1 -Preset windows-cuda-release -ModelSet custom -Models fish_drama3 -Target audiocpp_server -Jobs 4

The executables are in build\windows-cuda-release\bin. A CPU runtime uses the windows-cpu-release preset with the same model selection. To override automatic CUDA detection, add -CudaArchitectures with the architecture for your GPU.

Download a runnable GGUF

Install uv, then download one package from Jmica/Drama-3-GGUF. For the Q2 expert package:

uv run --no-project --python 3.12 --with huggingface_hub hf download `
  Jmica/Drama-3-GGUF drama-3-backbone-8bit-experts-Q2_K.gguf `
  --local-dir models\Drama-3-GGUF

The model manager lists the base and small packages:

uv run --no-project --python 3.12 tools\model_manager_v2.py list | Select-String 'fish_drama3'
uv run --no-project --python 3.12 tools\model_manager_v2.py install fish_drama3_q2_experts_q8_backbone --models-root models

File	Size	Expert banks	Other backbone matrices
drama-3-backbone-8bit-experts-Q4_K_S.gguf	22.91 GiB	Q4_K	Q8_0
drama-3-backbone-8bit-experts-Q3_K_S.gguf	18.93 GiB	Q3_K	Q8_0
drama-3-backbone-8bit-experts-Q2_K.gguf	15.88 GiB	Q2_K	Q8_0
drama-3-backbone-4bit-experts-Q4_K_S.gguf	21.78 GiB	Q4_K	Q4_K, with Q4_0 text embedding

These packages use standard GGML K-quants. The Q3_K_S and Q4_K_S file profiles use pure expert banks: all 120 routed expert projection banks have uniform Q3_K or Q4_K storage. Their storage-type selection corresponds to a standard --pure K profile with explicit precision overrides for the dense backbone, text embedding and audio components. Architecture-specific higher-bit promotion heuristics are not applied; an unmodified default llama.cpp preset can choose different types for some tensors. See the standard converter's pure and per-tensor options. The native converter takes --expert-type q3_k or --expert-type q4_k. backbone-8bit means Q8_0 backbone matrices; backbone-4bit means Q4_K matrices with a Q4_0 text embedding. Q2 experts use uniform Q2_K storage.

All four packages retain FP16 audio-tower matrices and waveform codec weights. Backbone and audio-tower normalization vectors stay FP32; codec normalization vectors use FP16 with the rest of the codec. The packages embed their tokenizer, configuration and model contract. Stock audio.cpp releases without this port cannot run Drama 3.

Prepare weights with uv

Conversion is optional when using a downloaded package. Build the native converter if you want to prepare a different precision:

.\scripts\build_windows.ps1 -Preset windows-cpu-release -ModelSet custom -Models fish_drama3 -Target audiocpp_gguf -Jobs 4

Download the complete official checkpoint into models\drama-3-original. The revision below is the checkpoint used for validation:

uv run --no-project --python 3.12 --with huggingface_hub hf download fishaudio/drama-3 --revision dd64a85cd82e9521f7202610ede97b367fc83946 --local-dir models\drama-3-original

Its original per-expert FP8 tensors have 128-by-128 inverse scales. The preparation script applies those scales and packs each projection into an expert bank.

uv run --python 3.12 tools\community_models\convert_fish_drama3.py `
  --checkpoint models\drama-3-original `
  --output models\drama3-f16-local.gguf `
  --quantized-output models\drama-3-backbone-8bit-experts-Q4_K_S.gguf `
  --gguf-tool build\windows-cpu-release\bin\audiocpp_gguf.exe

The intermediate tensor archive is approximately 68.2 GiB. The runnable package is approximately 22.9 GiB, with Q4_K routed experts, Q8_0 backbone matrices, FP16 audio tower and codec weights, and embedded configuration, tokenizer and license files. Codec weight normalization is folded during packaging. Allow space for both files during conversion. The intermediate archive is retained.

An existing intermediate can be reused with --input-gguf in place of --output. --expert-type selects the routed expert format; --backbone-type selects other backbone matrices; --embedding-type selects the text embedding format. Keep the embedding at a CUDA-supported lookup type such as Q8_0 or Q4_0. Conversion uses CPU PyTorch dependencies managed by uv and does not load weights onto a GPU.

Generate and stream

Host sampling is used by default. Set --session-option fish_drama3.device_sampling=true to enable tower decode and codebook sampling in one cached backend graph, or select the cuda_perf preset. Graph probability arithmetic is F32, so sampled outputs may differ from host sampling.

fish_drama3.perf_mode defaults to off. The opt-in cuda_perf preset sets the following defaults; explicit session options override the preset:

Session option	off	cuda_perf	Effect
fish_drama3.fused_moe	false	true	Cached-decode routing and SwiGLU fusion.
fish_drama3.kv_cache_type	f16	f16	KV storage precision, not weight precision. Set f32 to use F32 KV storage.
fish_drama3.kv_cache_alignment	1	256	Cached attention-prefix buckets; all history retained.
fish_drama3.device_sampling	false	true	Cached tower decode and device sampling.
fish_drama3.codec_batch_frames	4	4	Streaming codec batch ceiling; existing default retained.
fish_drama3.codec_async_decode	true	true	Streaming AR/codec overlap; existing default retained.
fish_drama3.backbone_prefill_cache_mb	256	256	Backbone prefill graph-cache budget in MiB; existing default retained.
fish_drama3.tower_prefill_cache_mb	256	256	Tower prefill graph-cache budget in MiB; existing default retained.

off retains the defaults listed above. Explicit session options override either preset, including fish_drama3.device_sampling=false with cuda_perf.

Fused arithmetic and attention shapes can change sampled audio. To enable the combined CUDA path, add:

--session-option fish_drama3.perf_mode=cuda_perf

Streaming defaults to a four-frame codec batch ceiling with AR/codec overlap. To use one-frame synchronous decoding:

--session-option fish_drama3.codec_batch_frames=1
--session-option fish_drama3.codec_async_decode=false

Batching starts with one frame, grows to the configured ceiling, and flushes the final partial batch. Larger batches retain more codec workspace and can introduce small waveform rounding differences without changing AR sampling. Overlap uses the dedicated codec backend and completes all audio before the request returns. Offline decoding is unchanged.

.\build\windows-cuda-release\bin\audiocpp_cli.exe `
  --task tts --family fish_drama3 --backend cuda --mode streaming `
  --model models\Drama-3-GGUF\drama-3-backbone-8bit-experts-Q2_K.gguf `
  --text 'A quick test of Drama Three on Windows.' `
  --max-tokens 500 --seed 123 --threads 8 `
  --session-option fish_drama3.cpu_expert_layers=8 `
  --session-option fish_drama3.context_length=2048 `
  --out speech.wav

For the tested 16 GB offload configurations, use 8 CPU expert layers with Q2, 12 with Q3, 24 with Q4-experts, or 20 with Q4-backbone. For a 24 GB configuration, use 4 with Q4-experts or 0 with the other packages. These are starting settings, not enforced memory limits. They were measured on a 96 GB GPU; physical 16 GB and 24 GB cards have not been tested. These counts control where expert weights and matmuls run. The backbone's attention/dense weights and the audio tower use the selected backend. fish_drama3.codec_backend=cpu moves codec computation to CPU as well.

If Vulkan voice cloning produces incomplete speech or reaches the token limit, try --session-option fish_drama3.codec_encoder_f32_accumulation=true.

This opt-in control uses F32 accumulation in the Vulkan reference encoder while keeping F16 storage. It defaults to false and does not change codec decoding or other backends. It also works with existing GGUFs without replacing their specs.

On CUDA, BF16 reference-encoder arithmetic is enabled by default with F16/BF16 codec weights. Set --session-option fish_drama3.codec_encoder_bf16_compute=false to restore the previous encoder arithmetic. This changes reference-conditioned generation but does not change codec decoding or backbone inference.

To reduce reference-encoder convolution memory, set --session-option fish_drama3.codec_encoder_chunk_samples=65536. The value is measured in samples at the codec sample rate and must be zero (the default, full encoding) or a positive multiple of 512. Chunking starts when the padded reference exceeds the chunk size plus 16,384 context samples; shorter references use the full graph. Overlapping convolution windows reuse one cached graph; the transformer and quantizer still process the full feature sequence, so total memory is not constant. Chunking can cost additional time and produce small backend-dependent numerical differences. Existing GGUFs accept the option without a spec override.

Sampling defaults to top_k=4097, which keeps every candidate for the semantic token and each tower codebook, so only top_p=0.9 filters. This matches Fish's official Drama 3 Space, which dropped the top-k cap because top_k=30 left takes stuck in silence. With --request-option top_k=30, long chunked requests can stall for about 9-21 s. The repetition-avoiding resample keeps its own top_k=30, top_p=0.95.

Host memory is required for offloaded weights. Context capacity includes text, reference audio frames, prior takes, and the requested generation budget.

Disk-backed experts

Set fish_drama3.disk_experts=true to keep the full routed expert banks in the GGUF file and read only selected experts into a bounded native-format cache. fish_drama3.expert_cache_size sets resident slots per layer (8 to 256; default 32). Dense backbone, tower, and codec weights remain resident. The existing cpu_expert_layers setting still selects CPU execution for those layers, but their expert caches are bounded too. No GGUF conversion is required.

fish_drama3.disk_expert_read_threads selects 1 to 64 host workers for copying selected experts from the memory-mapped GGUF into staging. The default is 1 (serial); try 4 to enable parallel copies using OpenMP or Apple dispatch. This applies only with disk_experts=true and pin_experts=false. Backend uploads and inference are unchanged. The host reader is backend-independent; Apple builds do not require OpenMP for parallel reads.

This is a memory/speed trade-off, not a faster MoE kernel. With cpu_expert_layers=0, prompt prefill groups tokens by expert within each layer, so one upload serves multiple tokens. Attention and shared/router projections retain the resident batched-prefill path; cached experts retain their vector projection path. Session-owned graphs share one activation workspace and are reused for the same exact prompt length while retained. A different length replaces that graph set rather than reusing a rounded capacity for stateful attention. Offline disk mode releases backbone and tower prefill graphs before waveform decoding, without clearing continuation state. Later full prefills rebuild those workspaces; resident execution keeps its existing cache policy. Mixed CPU/backend expert execution retains token-wise prefill. Storage speed still matters, and resident mode remains the default. For the minimum cache:

--session-option fish_drama3.disk_experts=true
--session-option fish_drama3.expert_cache_size=8
--session-option fish_drama3.cpu_expert_layers=0

When sufficient host RAM is available, add fish_drama3.pin_experts=true and set fish_drama3.cpu_expert_layers=0. This loads full expert banks into pinned host memory once, then uploads cache misses directly without an intermediate host copy. Expert matmuls still run on the selected backend, and GPU cache size is unchanged. This option is off by default and requires pinned host-buffer support; allocation failure is an error, not a fallback to a different memory policy.

Logs report disk_experts.bytes_read, disk_experts.expert_loads, disk_experts.cache_hits, disk_experts.buffer_mb, and disk_experts.host_buffer_mb when the session is destroyed. These describe expert I/O and cache allocation, not total process peak VRAM. With pinned experts, bytes read include the initial host-bank load instead of per-request cache-miss reads.

Serve over HTTP

Save the following as drama3-server.json in the repository root:

{
  "host": "127.0.0.1",
  "port": 8035,
  "backend": "cuda",
  "threads": 8,
  "lazy_load": true,
  "max_loaded_models": 1,
  "idle_unload_ms": 300000,
  "ui": true,
  "models": [{
    "id": "drama-3",
    "family": "fish_drama3",
    "path": "models/Drama-3-GGUF/drama-3-backbone-8bit-experts-Q2_K.gguf",
    "task": "tts",
    "mode": "streaming",
    "session_options": {
      "fish_drama3.cpu_expert_layers": 8,
      "fish_drama3.context_length": 2048
    },
    "default_request_options": {"max_tokens": 500}
  }]
}

.\build\windows-cuda-release\bin\audiocpp_server.exe --config drama3-server.json

The server exposes POST /v1/audio/speech and the task APIs, including streaming events and continuation artifacts. See the server guide.

Voice references

For cloning, add --voice-ref reference.wav and --request-option 'reference_text=What the reference says.'. Use --task clon to require a reference. Ordered references can also be supplied through multi_reference_cond or references as a JSON string:

[{"text":"<|speaker:0|> First reference.","audio_path":"first.wav"},
 {"text":"<|speaker:1|> Second reference.","audio_path":"second.wav"}]

Descriptors accept vq_codes instead of audio_path: a nonempty integer grid with ten rows, one per codebook. Text accepts <|speaker:0|> through <|speaker:15|> and event tags such as [sighs] and [laughs].

Continuation and editing

previous_takes is a JSON string containing ordered earlier takes. Each object has text and either vq_codes or audio_path. Earlier text strings can also be given; the runtime generates their codes before the requested take. previous_takes_file accepts the same data from a JSON file, including a single take object. Use it for VQ grids that exceed the Windows command-line limit.

--request-option 'previous_takes=["The report is due on Wednesday."]'
--text 'The report is due on Thursday.'

The library returns an AcousticTokens artifact named drama3_take, with family=fish_drama3 and schema=drama3_take_v1. Its UTF-8 JSON payload contains the text and VQ grid. Keep this artifact and pass it in the next request's input_artifacts to reuse the original codes for an edit. Multiple artifacts carry chained history. The current take's audio is returned separately.

Streaming sessions use IStreamingVoiceTaskSession. start_stream() runs synchronously and calls the event sink while audio is generated; finalize() returns the complete audio and the reusable take artifact. An application can invoke generation on its worker thread to keep its UI responsive.

Long-form text uses the framework chunker, with text_chunk_size=128 and text_chunk_mode=tag_aware by default. Each new chunk includes the previous chunk as context. Caller-supplied history remains ordered in every chunk.

Validation limits

Reproducible codec, tokenizer, quantization and HTTP smoke commands are in the Drama 3 test guide.

The port was exercised on native Windows with an RTX PRO 6000 Blackwell 96 GB, CUDA 12.8 and eight CPU worker threads. English speech, voice references, streaming audio and continuation/edit artifacts were checked. Streaming emits 44.1 kHz audio, with streamed chunks matching the complete output in the tests.

Independent checks cover hybrid backbone prefill and cached decode, actual Q2_K, Q3_K and Q4_K expert bank types and layouts against dequantized PyTorch weights, CPU/GPU expert splits, audio tower prefill/cached decode, and exact prompt rows against the official builder. Repeated seeded generation and offline/streaming generation produced identical codes within each profile. English transcript checks recovered the requested speech, including Wednesday-to-Thursday and noon-to-five edits.

This is an experimental port. Some emotion and nonverbal prompts produce prolonged reactions, omit dialogue or reach the generation cap. This occurs with both the original Q4-expert package and smaller formats. A heavier Q2 CPU offload trial also reached the long-form cap; the documented eight-layer configuration completed that case. These checks do not establish full perceptual parity with the original checkpoint. Multilingual pronunciation, speaker similarity and event delivery have not received a broad perceptual evaluation. A native seed reproduces a take within the same execution configuration; changing offload placement can change floating-point results and the sampled take.

The weights are subject to the Fish Audio Community License. See the model repository's LICENSE and NOTICE.