voice_audio_multi/docs/models/f5_tts.md

F5-TTS (community model)

F5-TTS is an open-source zero-shot voice-cloning TTS built on a flow-matching diffusion transformer (DiT) with a ConvNeXt text conditioner and a Vocos vocoder. This community port also targets the Habibi-TTS finetune — a multi-dialect Arabic checkpoint suite (MSA, SAU, UAE, ALG, IRQ, EGY, MAR, OMN, TUN, LEV, SDN, LBY) from the same authors — which uses the identical architecture, giving Arabic support through the same family (habibi / habibi_tts are registered as aliases).

Status: M4 — inference wired end to end. The session (src/community_models/f5_tts/session.cpp) runs full synthesis via f5_synthesize (text pipeline with Habibi dialect tokens → batched-CFG DiT Euler sampler → Vocos vocoder) and is served by audiocpp_server. Parity vs the reference PyTorch implementation is covered by golden harnesses (DiT stage taps, batched CFG incl. the null branch, tokenizer ids) plus a whisper.cpp ASR pronunciation check; see /mnt/ai/f5-parity/run_all.sh.

Milestones

Each milestone is gated on parity against the reference PyTorch implementation (cosine similarity ≥ 0.999 on fixed inputs) plus a listening check, matching the evidence bar described in #54 and PR #180.

Milestone	Scope	Status
M0	Family registration, model spec, stub session, this doc	done
M1	Weight loading + mel-Vocos decode path (ConvNeXt + iSTFT)	done (mel-corr 0.9963)
M2	DiT forward (RoPE, adaLN) + ConvNeXt text conditioner	done (all stages cosine 1.000000)
M3	CFM sampler (Euler, sway + EPSS, CFG null-branch parity), inference wiring, En/Ar samples	done
M4	Long-form chunking, RTF/VRAM evidence, server wiring	done (0.51x RTF on RTX 3090; GGUF packages hosted at trklou/audio.cpp)

Server usage

audiocpp_server.json entry: family f5_tts, model path = checkpoint directory (a *.gguf package, or a safetensors checkpoint + vocab.txt for development). Session options: f5_tts.vocos_path (only needed with safetensors checkpoints lacking a bundled vocoder), f5_tts.dialect (default UNK), f5_tts.frame_budget (256-8192 mel frames per CFM pass; omitted uses 2048, explicit zero is rejected). Requests take reference_text (required), dialect, speed, seed, num_inference_steps, guidance_scale (legacy alias cfg_strength), sway_sampling_coef, strip_diacritics.

Omitting seed selects fresh randomness; explicitly setting seed=0 is deterministic. speed defaults to 1.0 for duration estimation, but generated text chunks shorter than 10 characters use 0.3 instead.

Diacritics (harakat): Habibi was trained on ASR transcripts, which are undiacritized — the harakat/tanwin/shadda tokens exist in the vocab but are severely undertrained, and raw diacritized input degrades to garbled speech with character repetitions (identical in the Python reference; not a port issue). By default the frontend strips combining marks (U+0640, U+064B–U+065F, U+0670) before synthesis, so أَيْنَ اللَّوْنُ الأَحْمَر؟ reads as أين اللون الأحمر؟. Disable per request with strip_diacritics=false.

Quickstart (from a fresh clone)

Everything needed is installable from this repository — no Python inference stack required:

# 1. Build (f5_tts is included in AUDIOCPP_MODEL_SET=full, or select it explicitly)
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DAUDIOCPP_MODEL_SET=custom \
      -DAUDIOCPP_MODELS=f5_tts -DENGINE_ENABLE_CUDA=ON
cmake --build build --parallel --target audiocpp_cli audiocpp_server

# 2. Download the model package (GGUF: DiT + Vocos vocoder in one file + vocab.txt)
python3 tools/model_manager_v2.py install habibi_unified
# Per-dialect specialized checkpoints (stronger accent): habibi_alg, habibi_egy,
# habibi_irq, habibi_mar, habibi_msa, habibi_sau, habibi_uae
# Standalone vocoder for the original safetensors checkpoints: vocos_mel_24khz

# 3. Synthesize (zero-shot: any short reference WAV + its transcript)
build/bin/audiocpp_cli --task tts --family habibi \
    --model <models>/Habibi-TTS/Unified \
    --voice-ref /path/to/reference.wav \
    --reference-text 'transcript of the reference audio' \
    --text 'أهلا، هذا نص عربي تجريبي.' \
    --request-option dialect=UNK --out out.wav

The GGUF package is self-contained: the DiT lives under the transformer namespace and the Vocos vocoder under vocos, so no separate vocoder download or f5_tts.vocos_path option is needed (the option and safetensors fallbacks still work for the original HF checkpoints). Dialects: UNK MSA SAU UAE ALG IRQ EGY MAR OMN TUN LEV SDN LBY. Reference audio longer than half the frame budget is truncated (with a warning): ~10.9s at the default 2048-frame budget on CUDA, ~5.1s on CPU. There the DiT graph pool is allocated up front (9 MiB per mel frame, hard-capped at 12 GiB for the batched-CFG graph and 6 GiB for the plain one), so a pass above 1024 frames cannot build a graph at all and the budget is clamped to 960 frames. Keep refs under that and make sure the transcript matches, otherwise the transcript tail leaks into the output.

Converting checkpoints to GGUF

For LoRA support, reconvert the matching original checkpoint with the current converter; do not use the currently published F5-TTS GGUF packages as the base. PEFT DiT adapters can then be loaded with --session-option f5_tts.lora=<path>. Keep adapter_config.json beside the adapter weights. The optional f5_tts.lora_scale multiplies alpha/r and defaults to 1. Adapted weights and DiT graph caches are session-owned; the Vocos vocoder is not adapted. See the LoRA routing table.

tools/community_models/convert_f5_tts.py wraps audiocpp_gguf and produces one self-contained GGUF per checkpoint (transformer + vocos namespaces) plus the standalone vocoder package:

python3 tools/community_models/convert_f5_tts.py --checkpoint-root /models/Habibi-TTS \
    --vocos /models/vocos-mel-24khz/vocos.safetensors \
    --converter build/bin/audiocpp_gguf --output-dir gguf-packages

The safetensors source layout stays supported for development; the packaged/default format is GGUF.

Relevant building blocks already in-tree

• Vocos vocoder: src/models/vevo2/components.cpp, src/models/index_tts2/
• iSTFT: src/models/miocodec/, src/models/seed_vc/
• Flow matching: src/models/vevo2/fm.cpp
• RoPE DiT / adaLN: src/models/stable_audio/foundation/rf_dit.cpp

Checkpoints

Model	Source	License
F5-TTS Base (en/zh)	SWivid/F5-TTS	cc-by-nc-4.0
Habibi Unified (ar)	SWivid/Habibi-TTS	cc-by-nc-sa-4.0

Both are non-commercial licenses; keep that in mind before shipping anything built on them.