Fabio
c36510c21e
Some checks failed
Linux Build / Linux cpu (push) Has been cancelled
Linux Build / Linux cpu-clang (push) Has been cancelled
Linux Build / Linux vulkan (push) Has been cancelled
mac-build / macOS CPU (push) Has been cancelled
Nix Build / Nix (cpu) (push) Has been cancelled
Nix Build / Nix (vulkan) (push) Has been cancelled
windows-build / Windows CPU (push) Has been cancelled
122 lines
6.7 KiB
Markdown
122 lines
6.7 KiB
Markdown
# F5-TTS (community model)
|
||
|
||
[F5-TTS](https://github.com/SWivid/F5-TTS) is an open-source zero-shot voice-cloning TTS built on a
|
||
flow-matching diffusion transformer (DiT) with a ConvNeXt text conditioner and a Vocos vocoder.
|
||
This community port also targets the [Habibi-TTS](https://github.com/SWivid/Habibi-TTS) finetune —
|
||
a multi-dialect Arabic checkpoint suite (MSA, SAU, UAE, ALG, IRQ, EGY, MAR, OMN, TUN, LEV, SDN, LBY)
|
||
from the same authors — which uses the identical architecture, giving Arabic support through the
|
||
same family (`habibi` / `habibi_tts` are registered as aliases).
|
||
|
||
**Status: M4 — inference wired end to end.** The session (`src/community_models/f5_tts/session.cpp`)
|
||
runs full synthesis via `f5_synthesize` (text pipeline with Habibi dialect tokens → batched-CFG DiT
|
||
Euler sampler → Vocos vocoder) and is served by `audiocpp_server`. Parity vs the reference PyTorch
|
||
implementation is covered by golden harnesses (DiT stage taps, batched CFG incl. the null branch,
|
||
tokenizer ids) plus a whisper.cpp ASR pronunciation check; see `/mnt/ai/f5-parity/run_all.sh`.
|
||
|
||
## Milestones
|
||
|
||
Each milestone is gated on parity against the reference PyTorch implementation (cosine similarity
|
||
≥ 0.999 on fixed inputs) plus a listening check, matching the evidence bar described in #54 and
|
||
PR #180.
|
||
|
||
| Milestone | Scope | Status |
|
||
|---|---|---|
|
||
| M0 | Family registration, model spec, stub session, this doc | done |
|
||
| M1 | Weight loading + mel-Vocos decode path (ConvNeXt + iSTFT) | done (mel-corr 0.9963) |
|
||
| M2 | DiT forward (RoPE, adaLN) + ConvNeXt text conditioner | done (all stages cosine 1.000000) |
|
||
| M3 | CFM sampler (Euler, sway + EPSS, CFG null-branch parity), inference wiring, En/Ar samples | done |
|
||
| M4 | Long-form chunking, RTF/VRAM evidence, server wiring | done (0.51x RTF on RTX 3090; GGUF packages hosted at trklou/audio.cpp) |
|
||
|
||
## Server usage
|
||
|
||
`audiocpp_server.json` entry: family `f5_tts`, model path = checkpoint directory (a `*.gguf`
|
||
package, or a safetensors checkpoint + `vocab.txt` for development). Session options:
|
||
`f5_tts.vocos_path` (only needed with safetensors checkpoints lacking a bundled vocoder),
|
||
`f5_tts.dialect` (default UNK), `f5_tts.frame_budget` (256-8192 mel frames per CFM pass;
|
||
omitted uses 2048, explicit zero is rejected).
|
||
Requests take `reference_text` (required), `dialect`, `speed`, `seed`, `num_inference_steps`,
|
||
`guidance_scale` (legacy alias `cfg_strength`), `sway_sampling_coef`, `strip_diacritics`.
|
||
|
||
Omitting `seed` selects fresh randomness; explicitly setting `seed=0` is deterministic.
|
||
`speed` defaults to 1.0 for duration estimation, but generated text chunks shorter
|
||
than 10 characters use 0.3 instead.
|
||
|
||
**Diacritics (harakat):** Habibi was trained on ASR transcripts, which are undiacritized —
|
||
the harakat/tanwin/shadda tokens exist in the vocab but are severely undertrained, and raw
|
||
diacritized input degrades to garbled speech with character repetitions (identical in the
|
||
Python reference; not a port issue). By default the frontend strips combining marks
|
||
(U+0640, U+064B–U+065F, U+0670) before synthesis, so `أَيْنَ اللَّوْنُ الأَحْمَر؟` reads as
|
||
`أين اللون الأحمر؟`. Disable per request with `strip_diacritics=false`.
|
||
|
||
## Quickstart (from a fresh clone)
|
||
|
||
Everything needed is installable from this repository — no Python inference stack required:
|
||
|
||
```bash
|
||
# 1. Build (f5_tts is included in AUDIOCPP_MODEL_SET=full, or select it explicitly)
|
||
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DAUDIOCPP_MODEL_SET=custom \
|
||
-DAUDIOCPP_MODELS=f5_tts -DENGINE_ENABLE_CUDA=ON
|
||
cmake --build build --parallel --target audiocpp_cli audiocpp_server
|
||
|
||
# 2. Download the model package (GGUF: DiT + Vocos vocoder in one file + vocab.txt)
|
||
python3 tools/model_manager_v2.py install habibi_unified
|
||
# Per-dialect specialized checkpoints (stronger accent): habibi_alg, habibi_egy,
|
||
# habibi_irq, habibi_mar, habibi_msa, habibi_sau, habibi_uae
|
||
# Standalone vocoder for the original safetensors checkpoints: vocos_mel_24khz
|
||
|
||
# 3. Synthesize (zero-shot: any short reference WAV + its transcript)
|
||
build/bin/audiocpp_cli --task tts --family habibi \
|
||
--model <models>/Habibi-TTS/Unified \
|
||
--voice-ref /path/to/reference.wav \
|
||
--reference-text 'transcript of the reference audio' \
|
||
--text 'أهلا، هذا نص عربي تجريبي.' \
|
||
--request-option dialect=UNK --out out.wav
|
||
```
|
||
|
||
The GGUF package is self-contained: the DiT lives under the `transformer` namespace and the
|
||
Vocos vocoder under `vocos`, so no separate vocoder download or `f5_tts.vocos_path` option is
|
||
needed (the option and safetensors fallbacks still work for the original HF checkpoints).
|
||
Dialects: `UNK MSA SAU UAE ALG IRQ EGY MAR OMN TUN LEV SDN LBY`.
|
||
Reference audio longer than half the frame budget is truncated (with a warning): ~10.9s at the
|
||
default 2048-frame budget on CUDA, ~5.1s on CPU. There the DiT graph pool is allocated up front
|
||
(9 MiB per mel frame, hard-capped at 12 GiB for the batched-CFG graph and 6 GiB for the plain
|
||
one), so a pass above 1024 frames cannot build a graph at all and the budget is clamped to 960
|
||
frames. Keep refs under that and make sure the transcript matches, otherwise the transcript tail
|
||
leaks into the output.
|
||
|
||
### Converting checkpoints to GGUF
|
||
|
||
For LoRA support, reconvert the matching original checkpoint with the current
|
||
converter; do not use the currently published F5-TTS GGUF packages as the base.
|
||
PEFT DiT adapters can then be loaded with `--session-option f5_tts.lora=<path>`.
|
||
Keep `adapter_config.json` beside the adapter weights. The optional
|
||
`f5_tts.lora_scale` multiplies alpha/r and defaults to `1`. Adapted weights and
|
||
DiT graph caches are session-owned; the Vocos vocoder is not adapted.
|
||
See the [LoRA routing table](../lora.md).
|
||
|
||
`tools/community_models/convert_f5_tts.py` wraps `audiocpp_gguf` and produces one self-contained GGUF per
|
||
checkpoint (`transformer` + `vocos` namespaces) plus the standalone vocoder package:
|
||
|
||
```bash
|
||
python3 tools/community_models/convert_f5_tts.py --checkpoint-root /models/Habibi-TTS \
|
||
--vocos /models/vocos-mel-24khz/vocos.safetensors \
|
||
--converter build/bin/audiocpp_gguf --output-dir gguf-packages
|
||
```
|
||
|
||
The safetensors source layout stays supported for development; the packaged/default format is GGUF.
|
||
|
||
## Relevant building blocks already in-tree
|
||
|
||
- Vocos vocoder: `src/models/vevo2/components.cpp`, `src/models/index_tts2/`
|
||
- iSTFT: `src/models/miocodec/`, `src/models/seed_vc/`
|
||
- Flow matching: `src/models/vevo2/fm.cpp`
|
||
- RoPE DiT / adaLN: `src/models/stable_audio/foundation/rf_dit.cpp`
|
||
|
||
## Checkpoints
|
||
|
||
| Model | Source | License |
|
||
|---|---|---|
|
||
| F5-TTS Base (en/zh) | `SWivid/F5-TTS` | cc-by-nc-4.0 |
|
||
| Habibi Unified (ar) | `SWivid/Habibi-TTS` | cc-by-nc-sa-4.0 |
|
||
|
||
Both are non-commercial licenses; keep that in mind before shipping anything built on them.
|