voice_audio_multi/docs/models/fish_audio.md

# Fish Audio S2 Pro

Fish Audio S2 Pro is wired as `--family fish_audio` for offline and streaming
text to speech (`--task tts`) and reference voice cloning (`--task clon`). The
integration supports standalone GGUF packages, session-level reference caching,
framework text chunking for long-form text, and Fish-style multi-reference
conditioning.

## Install

```bash
python3 tools/model_manager_v2.py install fish_audio_s2_pro_q8_0
```

The default package installs:

```text
models/Fish-Audio-S2-Pro-GGUF/fish-audio-s2-pro-q8_0.gguf
```

> [!TIP]
> **Quantization:** Need to run on 6 GB or 8 GB consumer GPUs? See the [Fish Audio S2 Pro GGUF Quantization Recipe (Q6_K & Q4_K)](../proposals/fish_audio_quantization_recipe.md) to save up to **1.82 GB VRAM** while maintaining crystal clear speech synthesis.

## Quick Start

Text-to-speech:

```bash
audiocpp_cli --task tts --family fish_audio \
  --model models/Fish-Audio-S2-Pro-GGUF/fish-audio-s2-pro-q8_0.gguf \
  --backend cuda \
  --text "Hello from Fish Audio." \
  --out out.wav
```

Reference voice clone:

```bash
audiocpp_cli --task clon --family fish_audio \
  --model models/Fish-Audio-S2-Pro-GGUF/fish-audio-s2-pro-q8_0.gguf \
  --backend cuda \
  --text "The final render is ready for review." \
  --voice-ref assets/resources/b.wav \
  --reference-text "Some call me nature. Others call me Mother Nature." \
  --out out_ref.wav
```

## Streaming

Add `--mode streaming` to either CLI example to emit audio while the AR model
generates codes. The codec retains its causal attention and convolution state
within each text chunk and resets that state at chunk and request boundaries.
The streaming graph is reused between requests; offline decoding is unchanged.

For HTTP streaming, configure the model with `"mode": "streaming"` and send
`"stream_format": "sse"` to `/v1/audio/speech`. Each `speech.audio.delta` contains
only new PCM audio, not a growing prefix. See the [server API](../../app/server/README.md)
for the streaming response format. The complete text is supplied up front;
this is output streaming, not incremental text input.

## Model

| Field | Value |
|---|---|
| Family | `fish_audio` |
| Tasks | `tts`, `clon` |
| Modes | `offline`, `streaming` |
| Model path | `models/Fish-Audio-S2-Pro-GGUF/fish-audio-s2-pro-q8_0.gguf` |
| Languages | Model auto-handles language; tested paths cover English and Chinese-style prompts |
| Voice input | Optional reference WAV through `--voice-ref`; transcript through `--reference-text` when known |
| Built-in voices | Not exposed |

## Common Options (use directly)

| Option | Values | Default | Meaning |
|---|---|---:|---|
| `--voice-ref` | WAV path | not set | Reference speaker audio for voice cloning. |
| `--reference-text` | text | empty string | Reference transcript; required with inline reference audio. |
| `--max-tokens` | integer | `1024` | Maximum generated semantic tokens per chunk. `0` uses the default. |
| `--text-chunk-size` | integer chars | `200` | Long-form chunk size. |
| `--text-chunk-mode` | `default`, `tag_aware`, `japanese`, `endline` | `default` | Framework text chunking mode. |
| `--temperature` | `(0, 2)` | `0.8` | Sampling temperature. |
| `--top-k` | integer | `30` | Top-k sampling limit. |
| `--top-p` | `(0, 1]` | `0.8` | Nucleus sampling limit. |
| `--seed` | integer | random when omitted | Sampling seed for reproducible output. |

## Request Options (use with `--request-option`)

| Option | Values | Default | Meaning |
|---|---|---:|---|
| `multi_reference_cond` | JSON array | not set | Ordered Fish Audio reference conditioning pairs. Each entry requires `audio` and `text`. |

## Session Options (use with `--session-option`)

| Option | Values | Default | Meaning |
|---|---|---:|---|
| `fish_audio.mem_saver` | bool | `false` | Release cached AR runtime graphs after each request. |
| `fish_audio.lora` | checkpoint file | not set | Prepared Fish Speech adapter. Slow/fast decoder projections and embeddings are supported; codec weights are unchanged. |
| `fish_audio.lora_scale` | finite float | `1` | Multiplier for the adapter's alpha/r scaling. |
| `fish_audio.reference_cache_slots` | integer | `1` | Prepared reference-audio cache slots. |
| `fish_audio.weight_type` | `native`, `f32`, `f16`, `bf16`, `q8_0` | `native` | AR matmul weight storage type. |
| `fish_audio.codec_weight_type` | `native`, `f32`, `f16`, `q8_0` | `native` | Codec conv/matmul weight storage type. |
| `fish_audio.codec_encoder_bf16_compute` | bool | `false` | Use BF16 reference-encoder arithmetic on CUDA with F16/BF16 codec weights. Does not change codec decoding. |
| `fish_audio.ar_graph_arena_mb` | integer MiB >= 1 | `512` | AR graph arena size. |
| `fish_audio.ar_weight_context_mb` | integer MiB >= 1 | `32` | AR weight context size. |
| `fish_audio.codec_graph_arena_mb` | integer MiB >= 1 | `512` | Codec graph arena size. |
| `fish_audio.codec_weight_context_mb` | integer MiB >= 1 | `32` | Codec weight context size. |

To prepare a Fish Speech Lightning checkpoint, use its training alpha (not alpha/r):

```bash
python tools/lora/prepare_fish_audio_adapter.py --input lora.ckpt --alpha 16 --output adapter.safetensors
```

The exporter transposes embedding factors to the runtime's matrix convention and
stores alpha with every target. Use an adapter trained for the exact base checkpoint.

## Multi-Reference Conditioning

`multi_reference_cond` is a Fish Audio request option. It takes an ordered JSON
array of reference pairs:

```json
[
  {
    "audio": "assets/resources/a.wav",
    "text": "First reference transcript."
  },
  {
    "audio": "assets/resources/b.wav",
    "text": "Second reference transcript."
  }
]
```

Each entry must include:

| Field | Meaning |
|---|---|
| `audio` | Reference WAV path. |
| `text` | Transcript for that reference audio. |

The option has two useful cases.

### Case 1: One Generated Voice

Use multiple references without speaker tags when you want them to condition one
generated voice/style.

```bash
audiocpp_cli --task clon --family fish_audio \
  --model models/Fish-Audio-S2-Pro-GGUF/fish-audio-s2-pro-q8_0.gguf \
  --backend cuda \
  --text "The review is ready, and I will check the final numbers." \
  --request-option 'multi_reference_cond=[{"audio":"assets/resources/a.wav","text":"First reference transcript."},{"audio":"assets/resources/b.wav","text":"Second reference transcript."}]' \
  --out out_multi_ref.wav
```

In this mode the references are packed into the prompt context. The target text
does not assign individual lines to reference speakers.

### Case 2: Speaker-Tagged Turns

Use Fish speaker tags in the target text when you want the generated output to
follow a speaker order. Reference array index `0` maps to `<|speaker:0|>`, index
`1` maps to `<|speaker:1|>`, and so on. Speaker ids should stay within the
provided reference indexes.

```bash
audiocpp_cli --task clon --family fish_audio \
  --model models/Fish-Audio-S2-Pro-GGUF/fish-audio-s2-pro-q8_0.gguf \
  --backend cuda \
  --text "<|speaker:0|>The review is ready. <|speaker:1|>I will check the final numbers. <|speaker:0|>Please send me the final summary." \
  --request-option 'multi_reference_cond=[{"audio":"assets/resources/a.wav","text":"First reference transcript."},{"audio":"assets/resources/b.wav","text":"Second reference transcript."}]' \
  --request-option max_new_tokens=220 \
  --out out_speakers.wav
```

The reference transcripts may already contain speaker tags. If they do not, the
runtime tags them by reference order before packing the prompt.