# Fish Audio Drama 3 `fish_drama3` is a community model port of [fishaudio/drama-3](https://huggingface.co/fishaudio/drama-3). Inference runs in native audio.cpp with GGML; Python is used to prepare the checkpoint. Built with Fish Audio. The base model includes the 40-layer hybrid attention/MoE backbone, 256 experts with eight selected per token, four shared enrichment layers, nine residual codebook layers with persistent timeline caches, and the Fish DAC encoder/decoder. It supports offline and streaming TTS, voice references, speaker and event tags in text, long-form chunking, and continuation/edit history. ## Variants Both variants use `--family fish_drama3` and the same CLI and server routes. The loader reads layer-specific expert counts and the output-head layout from the checkpoint, so Drama3-small does not need a separate family. | Variant | GGUF packages | Source | |---|---|---| | Drama3 base | Q2_K, Q3_K, and Q4_K expert packages | [Jmica/Drama-3-GGUF](https://huggingface.co/Jmica/Drama-3-GGUF) | | Drama3-small | `fish_drama3_small_iq3_xxs`: `Drama3-small-GGUF/drama3-small-IQ3_XXS.gguf` | [pentacoxian-dev/drama3-small-GGUF](https://huggingface.co/pentacoxian-dev/drama3-small-GGUF) | Install the small variant with: ```bash python tools/model_manager_v2.py install fish_drama3_small_iq3_xxs --models-root models ``` Use `--model models/Drama3-small-GGUF/drama3-small-IQ3_XXS.gguf` with the generation examples below. Base-model weight sizes and memory measurements in this guide do not describe the small checkpoint. ## Windows build Install Visual Studio 2022 C++ build tools, a CUDA toolkit that supports your GPU, and Git. Use the Drama 3 branch and initialize its public frontend submodule: ```powershell git clone --branch drama-3-native https://github.com/JarodMica/audio.cpp.git Set-Location audio.cpp git -c submodule.external/audio.cpp-server-frontends.url=https://github.com/0xShug0/audio.cpp-server-frontends.git submodule update --init --recursive ``` The command-local HTTPS override lets the public frontend submodule download without GitHub SSH credentials. It does not modify `.gitmodules` or global Git configuration. Run the build helper from the repository root. It selects the required model dependencies and detects the CUDA architecture: ```powershell .\scripts\build_windows.ps1 -Preset windows-cuda-release -ModelSet custom -Models fish_drama3 -Target audiocpp_cli -Jobs 4 .\scripts\build_windows.ps1 -Preset windows-cuda-release -ModelSet custom -Models fish_drama3 -Target audiocpp_server -Jobs 4 ``` The executables are in `build\windows-cuda-release\bin`. A CPU runtime uses the `windows-cpu-release` preset with the same model selection. To override automatic CUDA detection, add `-CudaArchitectures` with the architecture for your GPU. ## Download a runnable GGUF Install [uv](https://docs.astral.sh/uv/getting-started/installation/), then download one package from [Jmica/Drama-3-GGUF](https://huggingface.co/Jmica/Drama-3-GGUF). For the Q2 expert package: ```powershell uv run --no-project --python 3.12 --with huggingface_hub hf download ` Jmica/Drama-3-GGUF drama-3-backbone-8bit-experts-Q2_K.gguf ` --local-dir models\Drama-3-GGUF ``` The model manager lists the base and small packages: ```powershell uv run --no-project --python 3.12 tools\model_manager_v2.py list | Select-String 'fish_drama3' uv run --no-project --python 3.12 tools\model_manager_v2.py install fish_drama3_q2_experts_q8_backbone --models-root models ``` | File | Size | Expert banks | Other backbone matrices | |---|---:|---|---| | `drama-3-backbone-8bit-experts-Q4_K_S.gguf` | 22.91 GiB | Q4_K | Q8_0 | | `drama-3-backbone-8bit-experts-Q3_K_S.gguf` | 18.93 GiB | Q3_K | Q8_0 | | `drama-3-backbone-8bit-experts-Q2_K.gguf` | 15.88 GiB | Q2_K | Q8_0 | | `drama-3-backbone-4bit-experts-Q4_K_S.gguf` | 21.78 GiB | Q4_K | Q4_K, with Q4_0 text embedding | These packages use standard GGML K-quants. The Q3_K_S and Q4_K_S file profiles use pure expert banks: all 120 routed expert projection banks have uniform Q3_K or Q4_K storage. Their storage-type selection corresponds to a standard `--pure` K profile with explicit precision overrides for the dense backbone, text embedding and audio components. Architecture-specific higher-bit promotion heuristics are not applied; an unmodified default llama.cpp preset can choose different types for some tensors. See the standard converter's [pure and per-tensor options](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/quantize.cpp). The native converter takes `--expert-type q3_k` or `--expert-type q4_k`. `backbone-8bit` means Q8_0 backbone matrices; `backbone-4bit` means Q4_K matrices with a Q4_0 text embedding. Q2 experts use uniform Q2_K storage. All four packages retain FP16 audio-tower matrices and waveform codec weights. Backbone and audio-tower normalization vectors stay FP32; codec normalization vectors use FP16 with the rest of the codec. The packages embed their tokenizer, configuration and model contract. Stock audio.cpp releases without this port cannot run Drama 3. ## Prepare weights with uv Conversion is optional when using a downloaded package. Build the native converter if you want to prepare a different precision: ```powershell .\scripts\build_windows.ps1 -Preset windows-cpu-release -ModelSet custom -Models fish_drama3 -Target audiocpp_gguf -Jobs 4 ``` Download the complete official checkpoint into `models\drama-3-original`. The revision below is the checkpoint used for validation: ```powershell uv run --no-project --python 3.12 --with huggingface_hub hf download fishaudio/drama-3 --revision dd64a85cd82e9521f7202610ede97b367fc83946 --local-dir models\drama-3-original ``` Its original per-expert FP8 tensors have 128-by-128 inverse scales. The preparation script applies those scales and packs each projection into an expert bank. ```powershell uv run --python 3.12 tools\community_models\convert_fish_drama3.py ` --checkpoint models\drama-3-original ` --output models\drama3-f16-local.gguf ` --quantized-output models\drama-3-backbone-8bit-experts-Q4_K_S.gguf ` --gguf-tool build\windows-cpu-release\bin\audiocpp_gguf.exe ``` The intermediate tensor archive is approximately 68.2 GiB. The runnable package is approximately 22.9 GiB, with Q4_K routed experts, Q8_0 backbone matrices, FP16 audio tower and codec weights, and embedded configuration, tokenizer and license files. Codec weight normalization is folded during packaging. Allow space for both files during conversion. The intermediate archive is retained. An existing intermediate can be reused with `--input-gguf` in place of `--output`. `--expert-type` selects the routed expert format; `--backbone-type` selects other backbone matrices; `--embedding-type` selects the text embedding format. Keep the embedding at a CUDA-supported lookup type such as Q8_0 or Q4_0. Conversion uses CPU PyTorch dependencies managed by uv and does not load weights onto a GPU. ## Generate and stream Host sampling is used by default. Set `--session-option fish_drama3.device_sampling=true` to enable tower decode and codebook sampling in one cached backend graph, or select the `cuda_perf` preset. Graph probability arithmetic is F32, so sampled outputs may differ from host sampling. `fish_drama3.perf_mode` defaults to `off`. The opt-in `cuda_perf` preset sets the following defaults; explicit session options override the preset: | Session option | `off` | `cuda_perf` | Effect | |---|---|---|---| | `fish_drama3.fused_moe` | `false` | `true` | Cached-decode routing and SwiGLU fusion. | | `fish_drama3.kv_cache_type` | `f16` | `f16` | KV storage precision, not weight precision. Set `f32` to use F32 KV storage. | | `fish_drama3.kv_cache_alignment` | `1` | `256` | Cached attention-prefix buckets; all history retained. | | `fish_drama3.device_sampling` | `false` | `true` | Cached tower decode and device sampling. | | `fish_drama3.codec_batch_frames` | `4` | `4` | Streaming codec batch ceiling; existing default retained. | | `fish_drama3.codec_async_decode` | `true` | `true` | Streaming AR/codec overlap; existing default retained. | | `fish_drama3.backbone_prefill_cache_mb` | `256` | `256` | Backbone prefill graph-cache budget in MiB; existing default retained. | | `fish_drama3.tower_prefill_cache_mb` | `256` | `256` | Tower prefill graph-cache budget in MiB; existing default retained. | `off` retains the defaults listed above. Explicit session options override either preset, including `fish_drama3.device_sampling=false` with `cuda_perf`. Fused arithmetic and attention shapes can change sampled audio. To enable the combined CUDA path, add: ```text --session-option fish_drama3.perf_mode=cuda_perf ``` Streaming defaults to a four-frame codec batch ceiling with AR/codec overlap. To use one-frame synchronous decoding: ```text --session-option fish_drama3.codec_batch_frames=1 --session-option fish_drama3.codec_async_decode=false ``` Batching starts with one frame, grows to the configured ceiling, and flushes the final partial batch. Larger batches retain more codec workspace and can introduce small waveform rounding differences without changing AR sampling. Overlap uses the dedicated codec backend and completes all audio before the request returns. Offline decoding is unchanged. ```powershell .\build\windows-cuda-release\bin\audiocpp_cli.exe ` --task tts --family fish_drama3 --backend cuda --mode streaming ` --model models\Drama-3-GGUF\drama-3-backbone-8bit-experts-Q2_K.gguf ` --text 'A quick test of Drama Three on Windows.' ` --max-tokens 500 --seed 123 --threads 8 ` --session-option fish_drama3.cpu_expert_layers=8 ` --session-option fish_drama3.context_length=2048 ` --out speech.wav ``` For the tested 16 GB offload configurations, use 8 CPU expert layers with Q2, 12 with Q3, 24 with Q4-experts, or 20 with Q4-backbone. For a 24 GB configuration, use 4 with Q4-experts or 0 with the other packages. These are starting settings, not enforced memory limits. They were measured on a 96 GB GPU; physical 16 GB and 24 GB cards have not been tested. These counts control where expert weights and matmuls run. The backbone's attention/dense weights and the audio tower use the selected backend. `fish_drama3.codec_backend=cpu` moves codec computation to CPU as well. If Vulkan voice cloning produces incomplete speech or reaches the token limit, try `--session-option fish_drama3.codec_encoder_f32_accumulation=true`. This opt-in control uses F32 accumulation in the Vulkan reference encoder while keeping F16 storage. It defaults to false and does not change codec decoding or other backends. It also works with existing GGUFs without replacing their specs. On CUDA, BF16 reference-encoder arithmetic is enabled by default with F16/BF16 codec weights. Set `--session-option fish_drama3.codec_encoder_bf16_compute=false` to restore the previous encoder arithmetic. This changes reference-conditioned generation but does not change codec decoding or backbone inference. To reduce reference-encoder convolution memory, set `--session-option fish_drama3.codec_encoder_chunk_samples=65536`. The value is measured in samples at the codec sample rate and must be zero (the default, full encoding) or a positive multiple of 512. Chunking starts when the padded reference exceeds the chunk size plus 16,384 context samples; shorter references use the full graph. Overlapping convolution windows reuse one cached graph; the transformer and quantizer still process the full feature sequence, so total memory is not constant. Chunking can cost additional time and produce small backend-dependent numerical differences. Existing GGUFs accept the option without a spec override. Sampling defaults to `top_k=4097`, which keeps every candidate for the semantic token and each tower codebook, so only `top_p=0.9` filters. This matches Fish's official Drama 3 Space, which dropped the top-k cap because `top_k=30` left takes stuck in silence. With `--request-option top_k=30`, long chunked requests can stall for about 9-21 s. The repetition-avoiding resample keeps its own `top_k=30`, `top_p=0.95`. Host memory is required for offloaded weights. Context capacity includes text, reference audio frames, prior takes, and the requested generation budget. ### Disk-backed experts Set `fish_drama3.disk_experts=true` to keep the full routed expert banks in the GGUF file and read only selected experts into a bounded native-format cache. `fish_drama3.expert_cache_size` sets resident slots per layer (8 to 256; default 32). Dense backbone, tower, and codec weights remain resident. The existing `cpu_expert_layers` setting still selects CPU execution for those layers, but their expert caches are bounded too. No GGUF conversion is required. `fish_drama3.disk_expert_read_threads` selects 1 to 64 host workers for copying selected experts from the memory-mapped GGUF into staging. The default is 1 (serial); try 4 to enable parallel copies using OpenMP or Apple dispatch. This applies only with `disk_experts=true` and `pin_experts=false`. Backend uploads and inference are unchanged. The host reader is backend-independent; Apple builds do not require OpenMP for parallel reads. This is a memory/speed trade-off, not a faster MoE kernel. With `cpu_expert_layers=0`, prompt prefill groups tokens by expert within each layer, so one upload serves multiple tokens. Attention and shared/router projections retain the resident batched-prefill path; cached experts retain their vector projection path. Session-owned graphs share one activation workspace and are reused for the same exact prompt length while retained. A different length replaces that graph set rather than reusing a rounded capacity for stateful attention. Offline disk mode releases backbone and tower prefill graphs before waveform decoding, without clearing continuation state. Later full prefills rebuild those workspaces; resident execution keeps its existing cache policy. Mixed CPU/backend expert execution retains token-wise prefill. Storage speed still matters, and resident mode remains the default. For the minimum cache: ```text --session-option fish_drama3.disk_experts=true --session-option fish_drama3.expert_cache_size=8 --session-option fish_drama3.cpu_expert_layers=0 ``` When sufficient host RAM is available, add `fish_drama3.pin_experts=true` and set `fish_drama3.cpu_expert_layers=0`. This loads full expert banks into pinned host memory once, then uploads cache misses directly without an intermediate host copy. Expert matmuls still run on the selected backend, and GPU cache size is unchanged. This option is off by default and requires pinned host-buffer support; allocation failure is an error, not a fallback to a different memory policy. Logs report `disk_experts.bytes_read`, `disk_experts.expert_loads`, `disk_experts.cache_hits`, `disk_experts.buffer_mb`, and `disk_experts.host_buffer_mb` when the session is destroyed. These describe expert I/O and cache allocation, not total process peak VRAM. With pinned experts, bytes read include the initial host-bank load instead of per-request cache-miss reads. ## Serve over HTTP Save the following as `drama3-server.json` in the repository root: ```json { "host": "127.0.0.1", "port": 8035, "backend": "cuda", "threads": 8, "lazy_load": true, "max_loaded_models": 1, "idle_unload_ms": 300000, "ui": true, "models": [{ "id": "drama-3", "family": "fish_drama3", "path": "models/Drama-3-GGUF/drama-3-backbone-8bit-experts-Q2_K.gguf", "task": "tts", "mode": "streaming", "session_options": { "fish_drama3.cpu_expert_layers": 8, "fish_drama3.context_length": 2048 }, "default_request_options": {"max_tokens": 500} }] } ``` ```powershell .\build\windows-cuda-release\bin\audiocpp_server.exe --config drama3-server.json ``` The server exposes `POST /v1/audio/speech` and the task APIs, including streaming events and continuation artifacts. See the [server guide](../../app/server/README.md). ## Voice references For cloning, add `--voice-ref reference.wav` and `--request-option 'reference_text=What the reference says.'`. Use `--task clon` to require a reference. Ordered references can also be supplied through `multi_reference_cond` or `references` as a JSON string: ```json [{"text":"<|speaker:0|> First reference.","audio_path":"first.wav"}, {"text":"<|speaker:1|> Second reference.","audio_path":"second.wav"}] ``` Descriptors accept `vq_codes` instead of `audio_path`: a nonempty integer grid with ten rows, one per codebook. Text accepts `<|speaker:0|>` through `<|speaker:15|>` and event tags such as `[sighs]` and `[laughs]`. ## Continuation and editing `previous_takes` is a JSON string containing ordered earlier takes. Each object has `text` and either `vq_codes` or `audio_path`. Earlier text strings can also be given; the runtime generates their codes before the requested take. `previous_takes_file` accepts the same data from a JSON file, including a single take object. Use it for VQ grids that exceed the Windows command-line limit. ```powershell --request-option 'previous_takes=["The report is due on Wednesday."]' --text 'The report is due on Thursday.' ``` The library returns an `AcousticTokens` artifact named `drama3_take`, with `family=fish_drama3` and `schema=drama3_take_v1`. Its UTF-8 JSON payload contains the text and VQ grid. Keep this artifact and pass it in the next request's `input_artifacts` to reuse the original codes for an edit. Multiple artifacts carry chained history. The current take's audio is returned separately. Streaming sessions use `IStreamingVoiceTaskSession`. `start_stream()` runs synchronously and calls the event sink while audio is generated; `finalize()` returns the complete audio and the reusable take artifact. An application can invoke generation on its worker thread to keep its UI responsive. Long-form text uses the framework chunker, with `text_chunk_size=128` and `text_chunk_mode=tag_aware` by default. Each new chunk includes the previous chunk as context. Caller-supplied history remains ordered in every chunk. ## Validation limits Reproducible codec, tokenizer, quantization and HTTP smoke commands are in the [Drama 3 test guide](../../tests/community_models/fish_drama3/README.md). The port was exercised on native Windows with an RTX PRO 6000 Blackwell 96 GB, CUDA 12.8 and eight CPU worker threads. English speech, voice references, streaming audio and continuation/edit artifacts were checked. Streaming emits 44.1 kHz audio, with streamed chunks matching the complete output in the tests. Independent checks cover hybrid backbone prefill and cached decode, actual Q2_K, Q3_K and Q4_K expert bank types and layouts against dequantized PyTorch weights, CPU/GPU expert splits, audio tower prefill/cached decode, and exact prompt rows against the official builder. Repeated seeded generation and offline/streaming generation produced identical codes within each profile. English transcript checks recovered the requested speech, including Wednesday-to-Thursday and noon-to-five edits. This is an experimental port. Some emotion and nonverbal prompts produce prolonged reactions, omit dialogue or reach the generation cap. This occurs with both the original Q4-expert package and smaller formats. A heavier Q2 CPU offload trial also reached the long-form cap; the documented eight-layer configuration completed that case. These checks do not establish full perceptual parity with the original checkpoint. Multilingual pronunciation, speaker similarity and event delivery have not received a broad perceptual evaluation. A native seed reproduces a take within the same execution configuration; changing offload placement can change floating-point results and the sampled take. The weights are subject to the Fish Audio Community License. See the model repository's [LICENSE](https://huggingface.co/Jmica/Drama-3-GGUF/blob/main/LICENSE) and [NOTICE](https://huggingface.co/Jmica/Drama-3-GGUF/blob/main/NOTICE).