voice_audio_multi/docs/models/echo_tts.md

# Echo-TTS

Echo-TTS is an English zero-shot voice-cloning TTS model. A 2.8B diffusion transformer (EchoDiT)
generates 80-dimensional latents in PCA space, which the Fish S1-DAC autoencoder decodes to 44.1 kHz
audio. Cloning takes a reference wav with **no transcript required**.

Upstream: [jordand/echo-tts-base](https://huggingface.co/jordand/echo-tts-base) ·
autoencoder: [jordand/fish-s1-dac-min](https://huggingface.co/jordand/fish-s1-dac-min)

| Family | `echo_tts` |
|---|---|
| Languages | en |
| Model directory | `models/echo-tts` |
| Modes | offline |
| Sample rate | 44 100 Hz |
| Tasks | `tts`, `clon` |

## Text Chunk Size

Use `--text-chunk-size 300` or `--request-option text_chunk_size=300` to set the
maximum Unicode codepoints per text chunk. The default is 300; explicit values
must be positive. Each chunk is generated separately within the model's fixed
audio window.

## Status

**Work in progress.** Every stage is *implemented*; the open question is how much of it is
*verified*. Those are tracked separately on purpose, because a clean build and plausible audio
prove neither.

| Milestone | Scope | Implemented | Numerically verified |
|---|---|---|---|
| M0 | Family registration, model spec v1 | yes | n/a |
| M1 | GGUF conversion, DiT, PCA inverse, Fish decode | yes | **yes** — denoiser and 40-step trajectory both above gate |
| M2 | Native speaker encoding (Fish encoder + RVQ) | yes | folded into the denoiser probe below |
| M3 | Long-form via the framework text chunker | yes | **no** |
| M4 | Q8_0 conversion, RTF and memory evidence | partial | **no** |

Cloning is self-contained — `session.cpp` calls `codec_->encode_latents` directly, so no pre-computed
speaker latent is required.

### Numerical parity against PyTorch

Measured on an RTX 3090 (sm_86), CUDA, F16 GGUF, by `tests/echo_tts/echo_tts_dit_parity.cpp`
against a dump from `tools/community_models/echo_tts/echo_tts_reference.py`. Gates are cosine over the
flattened tensors **and** max-absolute-error, never equality: cosine alone cannot see a uniform
scale error, and the host Philox stream matches CUDA to ~2 ULP rather than bit-exactly.

**The reference's own weight dtype dominates every number here**, and getting it wrong is what made
the sampler look broken for most of this port's life. ggml accumulates in F32 regardless of the
stored weight type, so the like-for-like comparison against an F16 GGUF is a **float32** reference,
not an F16 one. Run that way, every check passes:

| Check | vs float32 reference | Gate | Verdict |
|---|---|---|---|
| Denoiser, one conditional forward at t = 0.7 | cosine 0.999999899, max-abs 0.0126 | ≥ 0.999, ≤ 0.25 | **PASS** |
| 40-step sampler, reference initial noise injected | cosine 0.999516, max-abs 0.318 | ≥ 0.999, ≤ 4.0 | **PASS** |
| 40-step sampler, our own seeded draw | cosine 0.999542, max-abs 0.319 | ≥ 0.999, ≤ 4.0 | **PASS** |

For contrast, the same GGUF against lower-precision references — this is the whole of the residual
that earlier revisions of this document reported as unexplained:

| Check | vs bfloat16 reference | vs float16 reference |
|---|---|---|
| Denoiser | cosine 0.999977 | cosine 0.999999188 |
| Sampler, injected noise | cosine 0.913082 | cosine 0.988459 |
| Sampler, seeded draw | cosine 0.905481 | cosine 0.976972 |

Generate the F32 dump with `--force-dtype float32`, which also disables TF32 — TF32 has a 10-bit
mantissa, no better than F16, and Ampere would otherwise use it for matmuls and defeat the point.

**The denoiser probe passes and is the number that carries the port.** It is what settles the four
details that fail *silently* rather than loudly — half-head RoPE, the interleaved (not NEOX) rotary
pairing, the speaker patchify reshape, and the adaLN `shift/scale/gate` order. A 0 % WER cannot do
that job: it scores the words, not the speaker identity, so a wrong patchify reshape in particular
could yield fluent, correctly-worded speech in the wrong voice and still read as 0 %.

Note what the probe does **not** isolate. The reference text ids and speaker latents are injected,
but `prepare_conditioning()` then runs this port's own text encoder, speaker encoder and KV
projections, so the number covers the combined conditioning-plus-denoiser path. That is enough to
catch a wrong block; it is not enough to localise one. Per-block activation dumps
(`echo_tts_pack_reference.py --blocks`) exist for that and have not been run.

**The 40-step trajectory now passes, and the earlier shortfall was the reference, not this port.**
The evidence that pointed there before it was confirmed:

- **It was not the RNG.** Injecting the reference's own initial noise scored no better than our
  seeded draw (0.988 vs 0.977), so the Philox difference was never the mechanism.
- **It tracked dtype.** Each step up in reference precision moved the seeded trajectory: 0.905 at
  bfloat16, 0.977 at float16, 0.9995 at float32.
- **It compounded with step count.** At 4 steps the bfloat16 comparison scored 0.9965; at 40 it
  scored 0.9055. Monotonic degradation with step count is the signature of accumulating rounding in
  one of the two implementations, amplified every step by dual CFG at 3.0 and 8.0.
- **A line-by-line review of the sampler found no defect** — schedule, inclusive CFG bounds, single
  application of `truncation_factor`, three-lane CFG combination, the Euler update and the
  speaker-KV boundary all agree with `inference.py`.

The rounding was PyTorch's, not ours. Credit to @dignome for running the float32 reference that
settled it; the numbers above were then reproduced independently on an RTX 3090 with the gates
enforced.

### Host-side checks

`tests/echo_tts/echo_tts_host_units.cpp` is registered with `add_test` and needs neither a GPU nor
the checkpoint. It covers WhisperD normalisation (including the asymmetric quote rewrite and the
bare-`S1` tag suppression), full byte-token id vectors, truncation and padding, PCA projection and
inversion pinned independently against hand-computed values on a rectangular non-symmetric basis,
and the flattening-point crop including its standard-deviation *and* mean thresholds.

Separately, and **by hand rather than in CI**: PCA project/unproject at 5.7e-06 against numpy on
the real (80, 1024) basis, the Euler dual-CFG update at 6.6e-07 against a numpy transcription of
`inference.py`, the timestep embedding at 0.0 diff, and the tokenizer at 140/140 ids on the parity
prompt. `combine_cfg_lanes` and `euler_timestep_schedule` have no registered coverage.

### End-to-end run, RTX 3090 (sm_86), CUDA, F16 GGUF

| Check | Result |
|---|---|
| Conversion | `manifest OK`; 1117 DiT tensors written, 219 blockwise tensors dropped, 495 codec tensors |
| GGUF verifier | pass — 1614 tensors (`dit_weights/` 1117, `ae/` 495, `pca/` 2), F16 1043 / F32 571 |
| `latent_scale` | 0.0555555559694767 (= 1/18), matching the reference |
| Generation | exit 0, 44 100 Hz mono, no NaNs, peak 0.80 (below the normalisation threshold) |
| ASR round-trip, 15 words | WER 0 % — the only diffs are Whisper writing spoken "dot" as punctuation |
| ASR round-trip, 32 words | WER 0.0 %, 0 edits |
| Throughput | 9.195 s of audio in 7.89 s wall — **RTF 0.86 cold**, including the 5.5 GB model load |

Transcription used `faster-whisper-large-v3-turbo`. Generation cost is essentially constant across
those two runs (7.75 s vs 7.89 s) because the window is fixed at 640 frames, so longer text inside
one chunk is close to free.

WER on 32 words is a small sample and scores intelligibility only. It shows the pipeline runs end to
end and produces the right words; the denoiser cosine above is what shows the graph is right.

### What is still missing

- No per-block DiT activation dump, so the passing denoiser cosine proves correctness without
  localising where any future regression lives.
- No A/B of the flash-attention path against `AUDIOCPP_ECHO_TTS_NO_FLASH=1` on a fixed seed.
- No listening comparison of F16 against Q8_0.
- Warm RTF and VRAM-stability-across-requests numbers.

None of these block the port from working; they bound how much the numbers above prove.

## Known limitations

### Fixed 29.72-second generation window

Echo is trained to generate at most **640 latents**, and 640 × 2048 ÷ 44100 = **29.7215 s**. This is
a property of the model, not of this port.

Behaviour outside that window:

- Text corresponding to more than ~30 s is **spoken faster** to fit, rather than truncated. This is
  learned behaviour arising from global attention over the text, not an explicit compression step.
- The upstream tokenizer hard-truncates text past **768 UTF-8 bytes**.
- Requesting a shorter window does *not* compress the whole utterance into it — upstream documents
  that the model generates a **prefix** of the utterance instead.

`long_form` is therefore **not** claimed in `capabilities` at this stage.

### Blockwise generation does not extend the window

Upstream ships a blockwise sampler that generates in connected blocks and supports continuing from
existing audio. It **subdivides** the ≤30 s window rather than extending it: upstream requires
`sum(block_sizes) + continuation_length < 640` "to be in-distribution with training data", and
documents prefix plus continuation as "up to 30 seconds combined". Upstream also notes blockwise
"hasn't been thoroughly tested".

## Licence — read before using output commercially

Echo-TTS is **CC-BY-NC-SA-4.0**, and the restriction covers **generated audio, not only the
weights**. The output constraint is inherited from the Fish S1-DAC autoencoder — the same mechanism
that makes Fish Speech's own outputs non-commercial.

Practically: **audio produced by this model may not be used commercially**, regardless of how the
rest of your stack is licensed. audio.cpp itself is Apache 2.0 and is unaffected; model weights are
a separate download.

There is existing precedent in-tree — `fish_audio` (Fish Audio S2 Pro) carries the identical
output restriction from the identical dependency.

## Why this model

Selected by comparing every model tracked in [tts-bench](https://github.com/5uck1ess/tts-bench) — a
public benchmark covering **62 local TTS models** across speed, objective scores, and blind human
preference — against audio.cpp's existing support table.

| Measure | Echo-TTS | Field |
|---|---|---|
| Blind cloning Elo | **1162** | #3 of 40 (35 games; 738 cloning votes total) |
| Speaker similarity (SIM) | **0.836** | 2nd of 41 scored models |
| UTMOS (naturalness) | 4.21 | — |
| WER (intelligibility) | 7.45 % | — |
| Frozen pairwise study | **21-1-6** | near-tied 1st of 28 |

Two honest caveats: the cloning arena averages ~30 games per model, so gaps under ~100 Elo are
noise, and the ranking uses a single reference clip. Echo's standing is robust to both — it is
top-3 on human votes *and* 2nd on objective SIM, which are independent measurements.

Compute profile suits a GGUF port: ~2.8 B parameters at 1.35× RTFx and 9.4 GB VRAM in PyTorch on an
RTX 3090, so there is real work to amortise.

## Architecture

| Component | Params | Role |
|---|---:|---|
| EchoDiT trunk, 24 blocks | 1.75 B | Joint attention + SwiGLU MLP, adaLN timestep modulation |
| Text encoder | 294 M | UTF-8 **byte** tokens (256 vocab) — no phonemizer or G2P |
| Speaker encoder | 294 M | Reference PCA latents → speaker states |
| Latent encoder | 294 M | Blockwise only; omitted in M1 |
| PCA state | 83 K | Fish 1024-D ↔ DiT 80-D, `latent_scale` = 1/18 |
| Fish S1-DAC | 391 M weights | Reference encoding and waveform decoding |

Sampling is 40 Euler steps with **two independent CFG scales** — text (default 3.0) and speaker
(default 8.0) — gated to `t ∈ [0.5, 1.0]` (this is the default - matching upstream - sampling behavior). There is also optional **single CFG scaling**  (`echo_tts.cfg_mode=joint`) with combined text and speaker guidance scale (default 8.0), as well as optional `t_schedule_mode=sway` timestep scheduling, with default 12 Euler steps.

Note the Fish checkpoint stores an additional 303.6 M elements of `freqs_cis` and `causal_mask`
buffers. These are regenerated at runtime rather than shipped in the GGUF.

## Options

| Option | Type | Default | Description |
|---|---|---|---|
| `text_guidance_scale` | float | 3.0 | Classifier-free guidance scale on the text condition (Independent-CFG mode). Advertised range is recommended, not enforced by the parser. |
| `speaker_guidance_scale` | float | 8.0 | Classifier-free guidance scale on the speaker condition (Independent CFG mode). Advertised range is recommended, not enforced by the parser. |
| `guidance_scale` | float | 8.0 | Classifier-free guidance scale on the text+speaker condition (Joint-CFG mode). Advertised range is recommended, not enforced by the parser. |
| `cfg_mode` | string | independent | What type of classifier-free guidance to use. `independent` is the default and upstream implementation, using two guidance scales (one for text, one for the speaker). `joint` uses a single guidance scale on both the text and the speaker. |
| `num_inference_steps` | int | 40/12 | Euler sampler steps. Default 40 for `t_schedule_mode=linear`, default 12 for `t_schedule_mode=sway`. |
| `t_schedule_mode` | string | linear | `linear` time scheduling is the default upstream; `sway` scheduling modifies the grid to place more timesteps where the noise is higher. Sway time schedules can reduce the number of steps needed to produce similar quality. |
| `sway_coeff` | float | -1.0 | Sway coefficient for producing timesteps. Values must be finite and produce decreasing timesteps (validated at request) |
| `truncation_factor` | float | 0.8 | Initial-noise truncation factor. Advertised range is recommended, not enforced by the parser. |
| `speaker_kv_scale` | float | 1.0 | Force-speaker KV scaling. 1.0 disables; 1.5 is the upstream default when enabled. Advertised range is recommended, not parser-enforced; zero is invalid for reciprocal scaling. |
| `seed` | int | 0 | RNG seed for the initial latent. Negative signed-integer values are treated as zero. |
| `reference_duration_sec` | float | 15.0 | Trim the speaker reference to at most this many seconds before encoding. Shorter references are cheaper and often clone better; upstream's guidance favours around 10 s. Defaults to 15 s. Values above the trained maximum of 297.1 s are clamped. Set per request, or as a session default from CLI or server config. Must be positive. |
| `max_duration_sec` | float | — | Cap the generation window, in seconds (up to 29.7215 s, the trained maximum; larger values are clamped to it). Quantised down to a whole latent frame of 46.44 ms. Left unset, the window is estimated per chunk from text length and widened automatically if the utterance does not finish inside it, which is substantially cheaper for short text. Must be positive; rounded to at least one latent frame. |
| `mem_saver` | bool | false | Release Fish S1-DAC decode graph state when prior to and encode graph state after encoding reference speaker audio to reduce peak VRAM usage and steady-state VRAM usage when speakers are already cached.
| `guidance_interval` | int | 1 | Refresh the two unconditional CFG lanes only every Nth guided step. 1 reproduces upstream exactly. See [performance notes](../community_models/echo_tts_performance.md). |

Session options are `echo_tts.reference_duration_sec`,
`echo_tts.reference_cache_slots`, `echo_tts.mem_saver`, and `echo_tts.cfg_mode`.

## Text format

Prompts follow the [WhisperD](https://huggingface.co/jordand/whisper-d-v1a) transcription style:

- `[S1] ` is prepended automatically when neither `[S1]` nor `[S2]` is present.
- Colons, semicolons, and em dashes are normalised to commas.
- Commas generally function as pauses.
- Exclamation points and other emphatic punctuation increase expressiveness but can reduce quality.

Multi-speaker dialogue is expressed with `[S1]` / `[S2]` tags.

## Reference audio

Up to 5 minutes is accepted; 10 seconds or less works well. Audio is mixed to mono, resampled to
44.1 kHz, and peak-limited before encoding.

## Running it

```
audiocpp_cli \
    --family echo_tts \
    --model /path/to/Echo-TTS-GGUF \
    --task clon \
    --voice-ref reference.wav \
    --text "[S1] Alright, I'm going to demo this new model." \
    --out out.wav
```

`tts` is an alias to the same synthesis path. Route selection does not change
the model's existing input requirements.

The speaker reference is `--voice-ref`, not `--target-voice`; the latter is for
path-based voice conversion. No transcript of the reference is needed. Useful
request options: `num_inference_steps` (default 40), `text_guidance_scale`
(3.0), `speaker_guidance_scale` (8.0), `truncation_factor` (0.8), and `seed`.

## Quantisation

`--precision q8_0` produces a roughly half-size GGUF:

| | F16 | Q8_0 |
| --- | ---: | ---: |
| DiT | 4.76 GB | 2.53 GB |
| codec | 0.78 GB | 0.50 GB |

Q8_0 packs 32 weights per block behind one shared scale, so a tensor qualifies
only when its last logical dimension is a multiple of 32. The converter routes
each tensor accordingly rather than quantising blindly:

* **Q8_0** -- every 2-D matmul weight with a conforming row length. That is all
  but one DiT tensor, and 78% of codec weights.
* **F16** -- convolution kernels (`ggml_conv_1d` has no quantised path, which is
  why `codec.cpp` takes matmul and conv storage types separately) and the one
  non-conforming matmul, `in_proj.weight` at (2048, 80).
* **F32** -- norm weights, biases, snake alphas, LayerScale/ConvNeXt gammas and
  the codebooks, exactly as at other precisions.

Round-trip error is around 6e-05 RMSE with cosine similarity above 0.9999 on
weight-like distributions. Because the scale is per 32-weight block, the large
outliers this model carries in its late blocks and in `k_norm` degrade only
their own block rather than a whole row -- and `k_norm` is F32 regardless.

Quality has not been compared against F16 on real audio. Start with `orig` and
treat Q8_0 as an experiment until someone listens to both.

## Limiting the reference length

`reference_duration_sec` trims the speaker reference before encoding. Shorter
references cost less and often clone better -- upstream's guidance favours
around 10 s, and a long clip averages timbre over more prosodic variation.

Per request (bare name):

```
--request-option reference_duration_sec=30
```

As a default for a CLI run or a server, in the session scope (family-prefixed,
which is how the framework namespaces session and load options):

```
--session-option echo_tts.reference_duration_sec=30
```

In a server config file the same key goes under `session_options`, with a string
value. A request value overrides the session default. Values above the trained
maximum of 297.1 s are clamped rather than rejected. Trimming happens before
chunked encoding, so a cap also bounds encode time and VRAM.

## Reference encoding cost

Encoding the speaker reference is linear in its length: one Fish encode pass per
~29.7 s chunk, so a 4m29s clip is ten passes against one for a 28 s clip. At the
trained maximum that is roughly 22% of a request's arithmetic, before per-graph
launch overhead.

The result depends only on the audio and the trim length, so it is cached across
requests. A server rotating a few voices pays the cost once per voice instead of
once per request:

```
--session-option echo_tts.reference_cache_slots=8
```

Default 4; `0` disables it. Each slot holds only the projected latent, at most
2 MB. The cache lives with the session, so it helps a running server and not a
one-shot CLI invocation. Beyond caching, the levers are `reference_duration_sec`
and shorter references generally -- around 10 s is one chunk, the floor.

## The Fish S1-DAC autoencoder

Echo decodes its 80-D PCA latents through the Fish S1 DAC and encodes speaker
references with the same model. audio.cpp ships that codec as a framework
runtime, `engine::codecs::FishDacCodecRuntime`, so Echo and `fish_audio` share
one implementation -- but **not** the weights. `fish_audio` ships Fish Audio S2
Pro; Echo is trained against the S1 DAC (`jordand/fish-s1-dac-min`), and
`pca_state.safetensors` is fitted to that
codec's latent space specifically. Pointing Echo at S2 Pro would produce
plausible-looking latents and wrong audio, with no error anywhere.

The S1 weights are therefore packaged inside Echo's own GGUF, in the `ae`
namespace:

```
python3 tools/community_models/echo_tts/convert_echo_tts.py \
    --model-dir /path/to/echo-tts-base \
    --fish-dir  /path/to/fish-s1-dac-min \
    --outfile   Echo-TTS-GGUF/model.gguf
```

No companion model and no extra options are needed at run time. Two details the
converter handles:

* **Weight normalisation is folded.** The checkpoint stores it in two forms --
  `conv.parametrizations.weight.original0/original1` on the convolutions and
  legacy `weight_g`/`weight_v` on the quantiser projections -- and `codec.cpp`
  expects plain `conv.weight`. Both reduce to `w = g * v / ||v||` with the norm
  taken over every axis but the first. Note that for `ConvTranspose1d` axis 0 is
  the *input* channel count, so `g` is sized by input channels there; the
  decoder's four transposed convolutions are the only place this bites.
* **Fused qkv projections are split.** `autoencoder.py` keeps one `wqkv`
  linear and splits its output into three equal blocks; `codec.cpp` loads
  `attention.q_proj` / `k_proj` / `v_proj` separately, so the converter
  partitions the weight rows in the same order.
* **Exact tensor shapes are carried in metadata.** `ggml_n_dims()` ignores
  trailing dimensions of size 1, so a `(1, C, 1)` snake alpha would read back as
  `(C, 1)` and fail `codec.cpp`'s `{1, C, 1}` shape check. The converter emits
  `audiocpp.tensor_ranks` (INT32) and `audiocpp.tensor_shapes` (INT64) in tensor
  order, which audio.cpp uses in preference to the lossy inference.
* **The GGUF embeds its own model spec.** `package.cpp` refuses to load a
  published GGUF that does not, so the converter copies
  `model_specs/echo_tts.json` into the `audiocpp.model_spec.*` metadata keys.
  A distributed file is therefore self-describing and does not depend on the
  reader having a matching `model_specs/` checkout. Use `--model-spec` to embed
  a spec from elsewhere.
* **Namespaces are separated by `/`, not `.`** -- `dit_weights/...`, `pca/...`,
  `ae/...`. `PrefixedTensorSourceView` matches on `prefix + "/"`, so a
  dot-separated name is never routed and the loader reports the namespace as
  non-existent rather than the tensor as missing.
* **The codec namespace is `ae`, not `codec_weights`.** ggml caps tensor
  names at 64 characters (`GGML_MAX_NAME`) and rejects the whole file at load
  time if any name reaches it. The longest name `codec.cpp` loads is already 60
  characters, so only a three-character prefix fits; `codec_weights.` would push
  157 of the 455 codec tensors over. The converter refuses to write a GGUF that
  would trip this, and `verify_echo_gguf.py` re-checks it.
* **Registered buffers are dropped.** Two causal masks and three RoPE tables
  account for 305 MB of the 1.87 GB checkpoint and are rebuilt at graph
  construction, so they are not stored.

That leaves roughly 1.57 GB of codec weights on top of the 4.76 GB DiT.
`docs/archive/echo_tts_autoencoder_reuse.md` covers how the two
families share the codec implementation and where the seam sits in
`src/framework/codecs/fish_dac_codec_runtime.cpp`.