# Audio8 TTS

Audio8 TTS Preview 0.6B (Qwen backbone) and 0.1B (Falcon-H1 hybrid Mamba2+attention) are compact multilingual text-to-speech models with zero-shot voice cloning, ported natively into audio.cpp as the community family `audio8_tts`. They use a DualAR architecture derived from Fish Audio
S2 Pro: a slow semantic transformer generates speech semantics, a fast codebook transformer expands each semantic step into a full codec frame, and a neural codec renders 44.1 kHz audio. The native path executes all three
stages directly on ggml with no Python dependency.

> **Status:** `0.6B` Qwen is fully native. The `0.1B` Falcon-H1 path uses a
> stateful native Mamba2 + hybrid-GQA slow-AR forward and has been validated
> against the Transformers reference. The resolved implementation diary is
> retained in the [archive](../archive/audio8_tts_falcon_h1_status.md).

| Field | Value |
|---|---|
| Family | `audio8_tts` |
| Model directory | any directory holding the standalone GGUF (or a safetensors snapshot layout) |
| Task | `tts`, `clon` |
| Modes | `offline`, `streaming` |
| Languages | auto, yue, zh, nl, en, fr, de, it, ja, ko, pl, es |
| Voice input | optional reference WAV plus its exact transcript (clone) |
| Output | mono 44.1 kHz WAV |
| Backbones | `0.6B`: Qwen `slow_backbone=qwen` (24L, dim 896, 14H/2KV, RoPE 1e6), `0.1B`: Falcon-H1 `slow_backbone=falcon_h1` (dim 512, `d_inner 768=32×24`, `d_state 64`, `d_conv 4`, `dt_rank 24`, `GQA 8/2`, RoPE 1e11) |
| GGUF examples | `Audio8-TTS-Preview-0.6b-GGUF/audio8-tts-preview-0.6b-q8_0.gguf` (1.4G), `Audio8-TTS-Preview-0.1B-GGUF/audio8-tts-preview-0.1b-q8_0.gguf` (812M) |
| External ggml SSM | `external/ggml` already provides `ggml_ssm_conv/scan` for `cpu/cuda/metal/vulkan/opencl/sycl/cann` — no fork needed |

## Source

Upstream project and checkpoints:

- Project: <https://github.com/Audio8-AI/Audio8_TTS>
- Hugging Face organization: <https://huggingface.co/Audio8> (an official
  `Audio8/Audio8-TTS-Preview-0.6B-ONNX-INT4` runtime export is published;
  the PyTorch preview checkpoint ships the files below)
- License: Apache-2.0

The port targets the HF preview checkpoint snapshot containing:

```
config.json                     flat arktts configuration
model.safetensors               226 DualAR tensors (BF16)
codec.pth                       neural codec weights (PyTorch pickle)
tokenizer.json                  Qwen-style BPE tokenizer
tokenizer_config.json           tokenizer metadata
modeling_arktts.py              slow/fast AR reference implementation
modeling_arktts_codec.py        codec reference implementation
processing_arktts.py            prompt-format reference implementation
```

## What it does

Two tasks are exposed:

- **TTS** (`--task tts`): synthesize speech from text with automatic language
  handling across the eleven advertised languages.
- **Clone** (`--task clon`, or `tts` with a reference): condition generation
  on a speaker-reference WAV and its transcript to reproduce that voice.

Generation applies repetition-aware sampling (RAS): semantic logits are
restricted to the valid semantic range plus EOS, sampled at the request
temperature, and re-sampled with a high-stability fallback distribution when
the drawn token repeats within a sliding window, which suppresses the
looping artifacts typical of small AR speech models. Long inputs are split
into word-budget chunks (default 200 characters) and concatenated.

## Usage

Build with the family enabled and run against a converted GGUF:

```bash
cmake -S . -B build/linux-cpu-release -DCMAKE_BUILD_TYPE=Release \
      -DENGINE_ENABLE_CUDA=OFF -DENGINE_ENABLE_METAL=OFF \
      -DENGINE_ENABLE_VULKAN=OFF -DAUDIOCPP_MODEL_SET=custom \
      -DAUDIOCPP_MODELS=audio8_tts
cmake --build build/linux-cpu-release --target audiocpp_cli -j "$(nproc)"
```

Plain synthesis:

```bash
audiocpp_cli --task tts --family audio8_tts \
  --model models/Audio8-TTS-Preview-0.6B-GGUF/audio8-tts-preview-0.6b-q8_0.gguf \
  --text "The quick brown fox jumps over the lazy dog." \
  --seed 42 --metrics \
  --out tts.wav
```

Zero-shot cloning — provide a clean reference WAV and the exact words spoken
in it:

```bash
audiocpp_cli --task clon --family audio8_tts \
  --model models/Audio8-TTS-Preview-0.6B-GGUF/audio8-tts-preview-0.6b-q8_0.gguf \
  --voice-ref reference.wav \
  --reference-text "The exact words spoken in reference.wav." \
  --text "Hello from my cloned voice." \
  --seed 42 --out clone.wav
```

Multiple ordered references can be conditioned through one request option:

```bash
--request-option 'multi_reference_cond=[{"audio":"ref1.wav","text":"..."},{"audio":"ref2.wav","text":"..."}]'
```

### Controls

| Option | Default | Meaning |
|---|---:|---|
| `--voice-ref <wav>` | none | Speaker-reference audio for cloning. |
| `--reference-text <text>` | none | Exact transcript of the reference audio. |
| `--request-option temperature=<f>` | `0.8` | Semantic-token sampling temperature in `(0, 2)`; both endpoints excluded. |
| `--request-option top_p=<f>` | `0.8` | Nucleus threshold in `(0, 1]`; zero excluded, one included. |
| `--request-option top_k=<n>` | `30` | Top-k limit. |
| `--request-option seed=<n>` | random | Sampling seed for reproducible output. |
| `--request-option max_tokens=<n>` | `1024` | Maximum semantic steps per chunk; zero selects the default. |
| `--request-option text_chunk_size=<n>` | `200` | Word-budget chunk size in characters. |
| `--session-option audio8_tts.weight_type=<mode>` | `native` | AR matmul weight storage type. |
| `--session-option audio8_tts.codec_weight_type=<mode>` | `native` | Codec weight storage type. |
| `--session-option audio8_tts.mem_saver=<bool>` | `false` | Release cached AR graphs after each request. |
| `--session-option audio8_tts.reference_cache_slots=<n>` | `1` | Prepared reference cache slots. |
| `--session-option audio8_tts.ar_graph_arena_mb=<n>` | `512` | AR graph arena size; integer MiB >= 1. |
| `--session-option audio8_tts.ar_weight_context_mb=<n>` | `512` | AR weight context size; integer MiB >= 1. |
| `--session-option audio8_tts.codec_graph_arena_mb=<n>` | `512` | Codec graph arena size; integer MiB >= 1. |
| `--session-option audio8_tts.codec_weight_context_mb=<n>` | `512` | Codec weight context size; integer MiB >= 1. |

## Architecture

Three ggml graphs mirror the Python reference exactly (0.6B Qwen path):

1. **Slow semantic AR** — 24-layer Llama-style decoder (dim 896, 14 heads +
   2 KV heads, head_dim 64, FFN 4864, RoPE base 1e6, packed QKV *with*
   bias). A single prefill graph ingests the whole prompt
   (`[1, steps, 896]`) and a step graph decodes one column per step against
   a static KV cache. Vocabulary is 155776 Qwen-style tokens; valid speech
   semantics span `[semantic_begin_id, semantic_end_id]` =
   `[151678, 155773]`. **0.1B Falcon-H1 variant:** dim 512, hybrid per-layer
   `input_layernorm → parallel mamba2 (in_proj 1688=768+896+24, conv1d 896×1×4,
   dt_bias/A_log/D, out_proj) + GQA attention (q 512/512, k/v 128/512,
   RoPE 1e11) → pre_ff_layernorm → FFN (768)**, plus `embedding_multiplier
   0.1088`/`lm_head 0.0781`/`ssm/attn` multipliers from `types.h`. Current
   `falcon_forward_stateless:861` is a **stub** (RMSNorm + in_proj split +
   conv bias SiLU + gated out_proj + FFN, `attn_out=0`, no
   `ggml_ssm_conv/B/C/dt/A/D/ggml_ssm_scan` nor recurrent `conv[3,896]/ssm[64,32,24]`
   state, full recompute `O(N²)`); full Mamba2 tracked in
   `docs/FALCON_H1_0.1B_PORT_PLAN.md` M2/M3 (reuses `external/ggml` SSM backends).
2. **Fast codebook AR** — 4-layer decoder (same width, no attention biases,
   untied output head over 4096 codes). Conditioned on the slow hidden
   state, it autoregressively expands one semantic token into a frame of
   10 codebook indices (10 × 4096-entry books). Shared by both backbones
   (0.1B uses compact vocab 1024+EOS → expanded 4097 with `semantic 65537…`).
3. **Neural codec** — window-transformer encoder/decoder (8 layers, 16
   heads, FFN 1216) with Snake1d residual units, ConvNeXt blocks, causal
   transposed upsampling, and a downsample quantizer holding one semantic +
   nine acoustic codebooks. Decoding turns each 10-code frame into 2048
   samples at 44.1 kHz (≈21.5 frames/s).

Prompting follows `ArkttsProcessor._prompt_segments`: a chat-template system
turn ("convert the provided text to speech", optionally with reference text
and reference codes placed at the semantic span) followed by the user turn
and an `<|voice|>` assistant anchor. On semantic begin/end positions the
input embedding is the **plain sum** of the token embedding and all ten
codebook embeddings — deliberately without fish_audio's `1/sqrt(n)`
scaling, which arktts does not use.

Sampling is Gumbel-max (`argmax(softmax(logits/T) − log u)`) with legacy
top-k/top-p filtering, RAS window 10, and EOS `151645`.

## Porting procedure

The port reused the proven `fish_audio` family instead of starting from
scratch, since arktts shares its DualAR design:

1. **Copy + rename**: duplicate `src/models/fish_audio/*` into
   `src/community_models/audio8_tts/*` (headers likewise), rename
   `FishAudio*` → `Audio8Tts*`, rewrite namespaces and include paths, add
   the `audiocpp_add_model(audio8_tts …)` CMake block. The renamed skeleton
   had to compile before any semantic change.
2. **Contract first**: `types.h` (flat `arktts` config parsing, RAS
   parameters, uniform 10×4096 codebooks), `assets.h/.cpp`, the spec-backed
   `make_audio8_tts_loader()` in `session.cpp`, and schema-v1
   `model_specs/audio8_tts.json`.
3. **Parallel adaptation**, each citing the mirrored Python line:
   prompt format (`prompt_builder.cpp`), embedding lookup + sampling
   (`ar.cpp`), codec graphs (`codec.cpp`), option plumbing
   (`session.cpp`, `generator.cpp`).
4. **Codec conversion tooling** (below), then GGUF packaging.
5. **End-to-end validation** by SenseVoice ASR round-trip: every generated
   clip must transcribe verbatim. This caught one real bug — a leftover
   fish_audio `semantic_scale` multiply in the embedding lookup produced
   stationary noise until removed, after which seed-fixed clips transcribe
   verbatim (e.g. *"The quick brown fox jumps over the lazy dog."*, 3.44 s,
   RMS 0.13).

## Created files

| File | Purpose |
|---|---|
| `include/engine/community_models/audio8_tts/types.h` | Flat `arktts` config structs (slow/fast AR, codec, RAS params). |
| `include/.../audio8_tts/assets.h` + `src/.../assets.cpp` | Checkpoint loading: config, tokenizer, safetensors/GGUF tensor binding into host/device weights. |
| `include/.../audio8_tts/session.h` + `src/.../session.cpp` | `Audio8TtsSession`, request/session options, reference handling, and the spec-backed `make_audio8_tts_loader()`. |
| `include/.../audio8_tts/prompt_builder.h` + `src/.../prompt_builder.cpp` | Chat-prompt assembly and reference-code placement (`_prompt_segments`). |
| `include/.../audio8_tts/tokenizer_text.h` + `src/.../tokenizer_text.h/.cpp` | Text normalization matching Python `clean()` (`" ".join(split())`). |
| `include/.../audio8_tts/ar.h` + `src/.../ar.cpp` | Slow AR prefill/step graphs, fast codebook AR, embeddings, RAS/top-k/top-p/Gumbel sampling. |
| `include/.../audio8_tts/generator.h` + `src/.../generator.cpp` | Generation loop orchestration across both AR runtimes. |
| `include/.../audio8_tts/codec.h` + `src/.../codec.cpp` | Codec decode graphs (codes → 44.1 kHz waveform) and reference-audio encode for cloning. |
| `model_specs/audio8_tts.json` | Schema-v1 contract: metadata, languages, options, packages, GGUF/safetensors sources. |
| `CMakeLists.txt` | `audiocpp_add_model(audio8_tts …)` registration block. |
| `tools/community_models/convert_audio8_tts_codec.py` | Torch-free `codec.pth` → `codec.safetensors` converter. |
| `tools/community_models/convert_audio8_tts.py` | Safetensors snapshot → standalone GGUF packager. |
| `docs/models/audio8_tts.md` | This document. |

## Converting to GGUF

Two offline steps turn a fresh HF snapshot into standalone GGUFs. Neither tool downloads anything or requires a Python torch install.

**Step 1 — convert the codec** (`codec.pth` is a PyTorch pickle, unreadable by the C++ loader). The converter reads it with a zipfile/pickle parser, fuses new-style parametrization and legacy weight-norm pairs into the plain `*.conv.weight` / `*.in_proj.weight` keys the C++ graphs bind, and asserts shape anchors against the checkpoint:

```bash
python3 tools/community_models/convert_audio8_tts_codec.py \
    /path/to/Audio8-TTS-Preview-0.6b/codec.pth \
    /path/to/Audio8-TTS-Preview-0.6b/codec.safetensors
```

**Step 2 — package one self-contained GGUF** (AR + codec tensors, embedded config/tokenizer/spec):

```bash
# 16-bit reference package (AR BF16, codec F32 -> BF16)
python3 tools/community_models/convert_audio8_tts.py \
    --model-dir /path/to/Audio8-TTS-Preview-0.6b \
    --converter build/bin/audiocpp_gguf --type bf16

# default Q8_0 package, like fish_audio ships
python3 tools/community_models/convert_audio8_tts.py \
    --model-dir /path/to/Audio8-TTS-Preview-0.6b \
    --converter build/bin/audiocpp_gguf --type q8_0
```

Each output embeds 681 tensors (226 AR + 455 codec) in two namespaces `model_weights.*`, `codec_weights.*`). If a Q8_0 package audibly drifts, keep the codec namespace at 16 bit via the tool's mixed-storage options —codec conv stacks are quantization-sensitive.

**Where to get the inputs**: the upstream checkpoint from the
[Audio8 Hugging Face organization](https://huggingface.co/Audio8) / [Audio8-AI GitHub](https://github.com/Audio8-AI/Audio8_TTS) (file list under *Source* above). 

## Limitations and TODO

Current limitations (2026-08-29):

- **0.6B**: validated on CPU via SenseVoice ASR (`The quick brown fox…` 3.02s RMS 0.15, `你好，欢迎使用audio8。` 2.32s, `Artificial intelligence…` 2.97s) with `--family audio8_tts` GGUF `q8_0`/`bf16`; CUDA/Vulkan/Metal SSM backends are built but not yet exercised for this family (ggml kernels already present).
- **0.1B**: weight-complete and `audiocpp_cli` builds, but slow AR is a documented stub (`ar.cpp:861 TODO(Falcon-H1)`, `attn_out=0`, no `ggml_ssm_scan`/state, full recompute) — STT currently `like.` vs target; no `/tmp` writes or Python dependency. Full hybrid Mamba2 port is planned, not blocked on ggml (see `docs/FALCON_H1_0.1B_PORT_PLAN.md`).
- Streaming is supported for TTS and voice cloning through the standard pull-event session path.
- Cloning uses the same DualAR loop (reference WAV + `reference_text` → prompt builder) — 0.6B cloning path is structurally identical to TTS but has not yet been ASR-evaluated against real reference voices beyond the TTS evidence above.
- No conversation-turn continuation (Python supports multi-turn prompting; the C++ v1 path is single-request).
- ASR round-trip verifies intelligibility, not speaker similarity; formal parity runs against the Python/ONNX reference are still outstanding.

TODO:

- [ ] Complete 0.1B Falcon-H1 Mamba2 port (`ggml_ssm_conv/B/C/dt/A/D/scan` + recurrent `conv/ssm` ring + hybrid attention `K=1`) — `docs/FALCON_H1_0.1B_PORT_PLAN.md` M2/M3 (3–3.5d), then SenseVoice verification of `out/*0.1b.wav`
- [ ] Clone-task validation with real reference voices (0.6B first, then 0.1B after Mamba2)
- [ ] Exercise CUDA/Metal/Vulkan backends for both models (ggml SSM kernels already vendored)