voice_audio_multi/docs/models/fish_audio.md

Fish Audio S2 Pro

Fish Audio S2 Pro is wired as --family fish_audio for offline and streaming text to speech (--task tts) and reference voice cloning (--task clon). The integration supports standalone GGUF packages, session-level reference caching, framework text chunking for long-form text, and Fish-style multi-reference conditioning.

Install

python3 tools/model_manager_v2.py install fish_audio_s2_pro_q8_0

The default package installs:

models/Fish-Audio-S2-Pro-GGUF/fish-audio-s2-pro-q8_0.gguf

Tip

Quantization: Need to run on 6 GB or 8 GB consumer GPUs? See the Fish Audio S2 Pro GGUF Quantization Recipe (Q6_K & Q4_K) to save up to 1.82 GB VRAM while maintaining crystal clear speech synthesis.

Quick Start

Text-to-speech:

audiocpp_cli --task tts --family fish_audio \
  --model models/Fish-Audio-S2-Pro-GGUF/fish-audio-s2-pro-q8_0.gguf \
  --backend cuda \
  --text "Hello from Fish Audio." \
  --out out.wav

Reference voice clone:

audiocpp_cli --task clon --family fish_audio \
  --model models/Fish-Audio-S2-Pro-GGUF/fish-audio-s2-pro-q8_0.gguf \
  --backend cuda \
  --text "The final render is ready for review." \
  --voice-ref assets/resources/b.wav \
  --reference-text "Some call me nature. Others call me Mother Nature." \
  --out out_ref.wav

Streaming

Add --mode streaming to either CLI example to emit audio while the AR model generates codes. The codec retains its causal attention and convolution state within each text chunk and resets that state at chunk and request boundaries. The streaming graph is reused between requests; offline decoding is unchanged.

For HTTP streaming, configure the model with "mode": "streaming" and send "stream_format": "sse" to /v1/audio/speech. Each speech.audio.delta contains only new PCM audio, not a growing prefix. See the server API for the streaming response format. The complete text is supplied up front; this is output streaming, not incremental text input.

Model

Field	Value
Family	fish_audio
Tasks	tts, clon
Modes	offline, streaming
Model path	models/Fish-Audio-S2-Pro-GGUF/fish-audio-s2-pro-q8_0.gguf
Languages	Model auto-handles language; tested paths cover English and Chinese-style prompts
Voice input	Optional reference WAV through --voice-ref; transcript through --reference-text when known
Built-in voices	Not exposed

Common Options (use directly)

Option	Values	Default	Meaning
--voice-ref	WAV path	not set	Reference speaker audio for voice cloning.
--reference-text	text	empty string	Reference transcript; required with inline reference audio.
--max-tokens	integer	1024	Maximum generated semantic tokens per chunk. 0 uses the default.
--text-chunk-size	integer chars	200	Long-form chunk size.
--text-chunk-mode	default, tag_aware, japanese, endline	default	Framework text chunking mode.
--temperature	(0, 2)	0.8	Sampling temperature.
--top-k	integer	30	Top-k sampling limit.
--top-p	(0, 1]	0.8	Nucleus sampling limit.
--seed	integer	random when omitted	Sampling seed for reproducible output.

Request Options (use with --request-option)

Option	Values	Default	Meaning
multi_reference_cond	JSON array	not set	Ordered Fish Audio reference conditioning pairs. Each entry requires audio and text.

Session Options (use with --session-option)

Option	Values	Default	Meaning
fish_audio.mem_saver	bool	false	Release cached AR runtime graphs after each request.
fish_audio.lora	checkpoint file	not set	Prepared Fish Speech adapter. Slow/fast decoder projections and embeddings are supported; codec weights are unchanged.
fish_audio.lora_scale	finite float	1	Multiplier for the adapter's alpha/r scaling.
fish_audio.reference_cache_slots	integer	1	Prepared reference-audio cache slots.
fish_audio.weight_type	native, f32, f16, bf16, q8_0	native	AR matmul weight storage type.
fish_audio.codec_weight_type	native, f32, f16, q8_0	native	Codec conv/matmul weight storage type.
fish_audio.codec_encoder_bf16_compute	bool	false	Use BF16 reference-encoder arithmetic on CUDA with F16/BF16 codec weights. Does not change codec decoding.
fish_audio.ar_graph_arena_mb	integer MiB >= 1	512	AR graph arena size.
fish_audio.ar_weight_context_mb	integer MiB >= 1	32	AR weight context size.
fish_audio.codec_graph_arena_mb	integer MiB >= 1	512	Codec graph arena size.
fish_audio.codec_weight_context_mb	integer MiB >= 1	32	Codec weight context size.

To prepare a Fish Speech Lightning checkpoint, use its training alpha (not alpha/r):

python tools/lora/prepare_fish_audio_adapter.py --input lora.ckpt --alpha 16 --output adapter.safetensors

The exporter transposes embedding factors to the runtime's matrix convention and stores alpha with every target. Use an adapter trained for the exact base checkpoint.

Multi-Reference Conditioning

multi_reference_cond is a Fish Audio request option. It takes an ordered JSON array of reference pairs:

[
  {
    "audio": "assets/resources/a.wav",
    "text": "First reference transcript."
  },
  {
    "audio": "assets/resources/b.wav",
    "text": "Second reference transcript."
  }
]

Each entry must include:

Field	Meaning
audio	Reference WAV path.
text	Transcript for that reference audio.

The option has two useful cases.

Case 1: One Generated Voice

Use multiple references without speaker tags when you want them to condition one generated voice/style.

audiocpp_cli --task clon --family fish_audio \
  --model models/Fish-Audio-S2-Pro-GGUF/fish-audio-s2-pro-q8_0.gguf \
  --backend cuda \
  --text "The review is ready, and I will check the final numbers." \
  --request-option 'multi_reference_cond=[{"audio":"assets/resources/a.wav","text":"First reference transcript."},{"audio":"assets/resources/b.wav","text":"Second reference transcript."}]' \
  --out out_multi_ref.wav

In this mode the references are packed into the prompt context. The target text does not assign individual lines to reference speakers.

Case 2: Speaker-Tagged Turns

Use Fish speaker tags in the target text when you want the generated output to follow a speaker order. Reference array index 0 maps to <|speaker:0|>, index 1 maps to <|speaker:1|>, and so on. Speaker ids should stay within the provided reference indexes.

audiocpp_cli --task clon --family fish_audio \
  --model models/Fish-Audio-S2-Pro-GGUF/fish-audio-s2-pro-q8_0.gguf \
  --backend cuda \
  --text "<|speaker:0|>The review is ready. <|speaker:1|>I will check the final numbers. <|speaker:0|>Please send me the final summary." \
  --request-option 'multi_reference_cond=[{"audio":"assets/resources/a.wav","text":"First reference transcript."},{"audio":"assets/resources/b.wav","text":"Second reference transcript."}]' \
  --request-option max_new_tokens=220 \
  --out out_speakers.wav

The reference transcripts may already contain speaker tags. If they do not, the runtime tags them by reference order before packing the prompt.