Fabio
c36510c21e
Some checks failed
Linux Build / Linux cpu (push) Has been cancelled
Linux Build / Linux cpu-clang (push) Has been cancelled
Linux Build / Linux vulkan (push) Has been cancelled
mac-build / macOS CPU (push) Has been cancelled
Nix Build / Nix (cpu) (push) Has been cancelled
Nix Build / Nix (vulkan) (push) Has been cancelled
windows-build / Windows CPU (push) Has been cancelled
120 lines
7.8 KiB
Markdown
120 lines
7.8 KiB
Markdown
# Higgs Audio v3 TTS
|
|
|
|
Higgs Audio v3 TTS is a voice-clone TTS model. The current integration uses
|
|
the framework chunker for long text and keeps the reference prompt state in the
|
|
model session. Use `clon` for voice-cloning requests; the existing `tts` route
|
|
remains an alias to the same model path for compatibility.
|
|
|
|
For chunked generation without reference audio, the first generated chunk's
|
|
audio codes and transcript condition the remaining chunks to preserve its voice.
|
|
This reference is local to the request; supplied reference audio takes precedence.
|
|
|
|
## Install
|
|
|
|
The model manager installs the Q8_0 standalone GGUF package by default:
|
|
|
|
```bash
|
|
python3 tools/model_manager_v2.py install --models-root models higgs_audio_tts_4b_q8_0
|
|
```
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
audiocpp_cli --task clon --family higgs_audio_tts --model models/Higgs-Audio-v3-TTS-4B-GGUF/higgs-audio-v3-tts-4b-q8_0.gguf --backend cuda --text "Hello from Higgs Audio." --voice-ref assets/resources/b.wav --reference-text "Some call me nature. Others call me Mother Nature. I've been here for over 4.5 billion years. 22,500 times longer than you." --out out.wav
|
|
```
|
|
|
|
## Model
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Family | `higgs_audio_tts` |
|
|
| Model path | `models/Higgs-Audio-v3-TTS-4B-GGUF/higgs-audio-v3-tts-4b-q8_0.gguf` when installed through the model manager |
|
|
| Tasks | `tts`, `clon` |
|
|
| Modes | `offline`, `streaming` |
|
|
| Languages | Model auto-handles supported languages |
|
|
| Voice input | Reference WAV through `--voice-ref`; transcript through `--reference-text` when known |
|
|
| Built-in voices | Not exposed |
|
|
|
|
## Common Options (use directly)
|
|
|
|
| Option | Values | Default | Meaning |
|
|
|---|---|---:|---|
|
|
| `--voice-ref` | WAV path | optional | Reference speaker audio for cloning; generation without a reference is also supported. |
|
|
| `--reference-text` | text | empty string | Transcript for reference audio. |
|
|
| `--text-chunk-size` | integer chars | `1024` | Long-form chunk size. |
|
|
| `--max-tokens` | integer | `2048` | Maximum generated AR tokens; zero uses the default. Must not exceed the configured context length. |
|
|
| `--temperature` | float | `0.8` | AR sampling temperature. Must be positive. |
|
|
| `--top-k` | integer | `30` | AR top-k sampling limit. The narrower default is less prone to premature EOC than the Python client's `50`. |
|
|
| `--top-p` | float | `0.8` | AR nucleus sampling probability. Must be positive. |
|
|
| `--repetition-penalty` | float | `1.1` | Accepted for Python API compatibility; audio sampling does not consume this value. Compatibility-only, but must be finite and positive. |
|
|
|
|
## Request Options (use with `--request-option`)
|
|
|
|
| Option | Values | Default | Meaning |
|
|
|---|---|---:|---|
|
|
| `max_tokens` | integer >= 0 | `2048` | Maximum generated AR tokens; zero uses the default. Must not exceed the configured context length. |
|
|
| `allow_incomplete_generation` | boolean | `false` | Return generated audio with a warning instead of failing when `max_tokens` is reached before EOC. Audio may be truncated or contain silence/noise. |
|
|
| `temperature` | float | `0.8` | AR sampling temperature. Must be positive. |
|
|
| `top_p` | float | `0.8` | AR nucleus sampling probability. Must be positive. |
|
|
| `top_k` | integer | `30` | AR top-k sampling limit. |
|
|
| `filter_nan_logits` | boolean | `false` | Filter NaN and infinity during greedy and stochastic top-k selection. All-invalid logits still fail. |
|
|
| `repetition_penalty` | float | `1.1` | Accepted for Python API compatibility; audio sampling does not consume this value. Compatibility-only, but must be finite and positive. |
|
|
| `seed` | integer >= 0 | random | Torch sampling seed; omitted chooses a random seed. |
|
|
| `reference_text` | text | empty string | Transcript accompanying reference audio. |
|
|
| `text_chunk_size` | integer | `1024` | Framework long-form text chunk size. |
|
|
| `text_chunk_mode` | `default`, `tag_aware`, `japanese`, `endline` | `default` | Framework text chunking mode. |
|
|
| `stream_frames_per_event` | integer >= 1 | `32` | Streaming audio frame cap; each codec frame contains 40 ms of audio. |
|
|
| `stream_chunk_policy` | `grow`, `fixed` | `grow` | Start with one frame and grow to the cap, or emit fixed-size chunks. Streaming only. |
|
|
|
|
If mixed finite and non-finite logits disrupt greedy or stochastic top-k sampling, try
|
|
`--request-option filter_nan_logits=true`, or `"filter_nan_logits": true` in the
|
|
server request's `options`. Greedy sampling (`top_k=1`) then selects the highest
|
|
finite logit. It does not repair invalid model output; all-invalid logits still fail.
|
|
|
|
For best-effort output when the model does not finish within the token budget, use
|
|
`--request-option allow_incomplete_generation=true`. On the server, set
|
|
`"allow_incomplete_generation": true` in the request's `options` object.
|
|
This applies to offline and streaming generation. Normal EOC stopping is unchanged;
|
|
without the option, reaching the limit before EOC still fails.
|
|
|
|
## Streaming
|
|
|
|
Set the session mode to `streaming`. Audio is decoded and emitted during AR
|
|
generation, once all eight delayed codebooks for each frame are available.
|
|
The default chunk schedule is 1, 2, 4, 8, 16, then 32 frames per event;
|
|
the final event may be shorter. The non-causal codec needs ten frames of
|
|
future context, so a one-frame first event still waits for that context.
|
|
This is not a one-AR-step latency guarantee.
|
|
|
|
Streaming keeps bounded codec windows and reuses their graphs within the
|
|
session. Text chunking and reference conditioning remain the same as offline.
|
|
The final WAV contains exactly the concatenated audio deltas. Floating-point
|
|
differences from offline decoding are possible because codec graph shapes differ.
|
|
|
|
For the server, configure `"mode": "streaming"` on the model entry and request
|
|
`"stream_format": "sse"` from `/v1/audio/speech`. Streaming controls belong in
|
|
the request's `options` object. They are not applied to offline generation.
|
|
|
|
## Session Options (use with `--session-option`)
|
|
|
|
Try `--session-option higgs_audio_tts.reference_prefill_mode=exact` for excessive
|
|
repetition or unexpectedly long, mostly silent output on repeated voice-cloning
|
|
requests. In server configuration, set
|
|
`"session_options": {"higgs_audio_tts.reference_prefill_mode": "exact"}` on the model entry.
|
|
This preserves reference-KV reuse and leaves full-prompt prefill and AR decoding
|
|
unchanged. It may increase prefill time and is not a general generation-quality fix.
|
|
The mode is fixed for the session and applies to offline and streaming generation.
|
|
|
|
| Option | Values | Default | Meaning |
|
|
|---|---|---:|---|
|
|
| `higgs_audio_tts.reference_prefill_mode` | `auto`, `exact` | `auto` | Cached-reference prefill attention; `auto` preserves existing behavior, `exact` enables the workaround described above. |
|
|
| `higgs_audio_tts.weight_type` | `native`, `f32`, `f16`, `bf16`, `q8_0` | `native` | Shared AR and codec weight storage type. |
|
|
| `higgs_audio_tts.ar_weight_type` | `native`, `f32`, `f16`, `bf16`, `q8_0` | shared setting | Autoregressive decoder weight storage override. |
|
|
| `higgs_audio_tts.codec_weight_type` | `native`, `f32`, `f16`, `bf16`, `q8_0` | shared setting | Audio codec weight storage override. |
|
|
| `higgs_audio_tts.ar_weight_context_mb` | integer MiB >= 1 | not set | AR weight context size. |
|
|
| `higgs_audio_tts.codec_weight_context_mb` | integer MiB >= 1 | `1536` | Codec weight context size. |
|
|
| `higgs_audio_tts.ar_decode_graph_arena_mb` | integer MiB >= 1 | `512` | AR decode graph arena size. |
|
|
| `higgs_audio_tts.codec_decode_graph_arena_mb` | integer MiB >= 1 | `128` | Codec decode graph arena size. |
|
|
| `higgs_audio_tts.codec_encode_graph_arena_mb` | integer MiB >= 1 | `256` | Reference-audio codec encode graph arena size. |
|
|
| `higgs_audio_tts.reference_cache_slots` | integer >= 0 | `1` | Encoded reference-audio cache slots; `0` disables reuse. |
|
|
| `higgs_audio_tts.attention` | `auto`, `flash`, `eager` | `auto` | Attention kernel. `auto` uses flash except on Volta/Turing GPUs (e.g. V100), where it falls back to eager to avoid missing MMA kernels. |
|