Fabio
c36510c21e
Some checks failed
Linux Build / Linux cpu (push) Has been cancelled
Linux Build / Linux cpu-clang (push) Has been cancelled
Linux Build / Linux vulkan (push) Has been cancelled
mac-build / macOS CPU (push) Has been cancelled
Nix Build / Nix (cpu) (push) Has been cancelled
Nix Build / Nix (vulkan) (push) Has been cancelled
windows-build / Windows CPU (push) Has been cancelled
134 lines
6.7 KiB
Markdown
134 lines
6.7 KiB
Markdown
# GLM-TTS
|
|
|
|
GLM-TTS is a zero-shot Chinese and English speech-synthesis model. The native
|
|
audio.cpp path executes its Llama speech-token generator, Whisper-VQ reference
|
|
encoder, Flow/DiT mel generator, CAMPPlus speaker encoder, and HiFT vocoder.
|
|
Both advertised routes are reference-conditioned: provide a clean WAV and the
|
|
exact words spoken in it.
|
|
|
|
GLM-TTS is v1-native. `model_specs/glm_tts.json` is the single source of truth
|
|
for metadata, packages, normalized options, and GGUF/safetensors resources.
|
|
The runtime uses the generic spec-backed loader; the legacy request aliases
|
|
listed below remain an internal compatibility layer for existing users.
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Family | `glm_tts` |
|
|
| Model directory | `models/GLM-TTS` |
|
|
| Task | `tts`, `clon` |
|
|
| Modes | `offline` |
|
|
| Languages | Chinese, English |
|
|
| Voice input | Required reference WAV plus its exact transcript |
|
|
| Output | mono 24 kHz WAV |
|
|
|
|
Install the default package:
|
|
|
|
```bash
|
|
python tools/model_manager_v2.py install glm_tts --models-root models
|
|
```
|
|
|
|
The default install is the standalone Q8 GGUF package.
|
|
|
|
The original safetensors preparation path remains available through the
|
|
deprecated manager:
|
|
|
|
```bash
|
|
python tools/model_manager_deprecated.py install glm_tts --models-dir models
|
|
```
|
|
|
|
That installer downloads `zai-org/GLM-TTS`, converts the official Flow and HiFT
|
|
PyTorch checkpoints to safetensors, prepares the ChatGLM tokenizer resources,
|
|
and installs the matching CAMPPlus safetensors weights. The latter are sourced
|
|
from `mlx-community/index-tts2-mlx` because the GLM-TTS repository publishes
|
|
CAMPPlus as ONNX only; the native output was checked directly against that
|
|
official ONNX graph.
|
|
|
|
Run the prepared safetensors package:
|
|
|
|
```bash
|
|
audiocpp_cli --task clon --family glm_tts \
|
|
--model models/GLM-TTS --backend cuda \
|
|
--voice-ref reference.wav \
|
|
--reference-text "The exact words spoken in reference.wav." \
|
|
--text "Hello from GLM-TTS." \
|
|
--top-k 25 --top-p 0.8 --temperature 1.0 \
|
|
--seed 0 --max-tokens 256 --out glm_tts.wav
|
|
```
|
|
|
|
The `tts` task accepts the same reference arguments. It is an alias for the
|
|
same zero-shot synthesis path rather than an unconditioned preset voice.
|
|
|
|
## Standalone GGUF
|
|
|
|
After installing the package, create a standalone mixed GGUF. The
|
|
autoregressive Llama group remains F16 for speech-token quality; the speech
|
|
tokenizer, Flow, HiFT, and CAMPPlus groups use Q8_0:
|
|
|
|
```bash
|
|
audiocpp_gguf \
|
|
--input llama_weights=models/GLM-TTS/llm/model.safetensors.index.json \
|
|
--input speech_tokenizer_weights=models/GLM-TTS/speech_tokenizer/model.safetensors \
|
|
--input flow_weights=models/GLM-TTS/flow/model.safetensors \
|
|
--input hift_weights=models/GLM-TTS/hift/model.safetensors \
|
|
--input campplus_weights=models/GLM-TTS/frontend/campplus.safetensors \
|
|
--root models/GLM-TTS \
|
|
--family glm_tts \
|
|
--model-spec model_specs/glm_tts.json \
|
|
--type q8_0 \
|
|
--keep-type "llama_weights/*=f16" \
|
|
--overwrite \
|
|
--output models/GLM-TTS-Q8/GLM-TTS_Q8.gguf
|
|
```
|
|
|
|
The resulting file embeds all five tensor groups, the package specification,
|
|
configs, and tokenizer sidecars. It runs from a directory containing only the
|
|
GGUF:
|
|
|
|
```bash
|
|
audiocpp_cli --task clon --family glm_tts \
|
|
--model models/GLM-TTS-Q8/GLM-TTS_Q8.gguf --backend cuda \
|
|
--voice-ref reference.wav \
|
|
--reference-text "The exact words spoken in reference.wav." \
|
|
--text "Hello from GLM-TTS." \
|
|
--seed 0 --max-tokens 256 --out glm_tts_q8.wav
|
|
```
|
|
|
|
## Controls
|
|
|
|
| Option | Values | Default | Meaning |
|
|
|---|---|---:|---|
|
|
| `--reference-text` | text | required | Exact transcript of `--voice-ref`. |
|
|
| `--max-tokens` | integer | automatic | Maximum generated speech tokens; otherwise the official 2x-to-20x text-token bounds are used. Nonpositive values select the automatic text-derived budget. |
|
|
| `--temperature` | float | `1.0` | Speech-token temperature; official default 1.0. Nonpositive values select greedy decoding. |
|
|
| `--top-k` | integer | `25` | Speech-token top-k; official default 25. Nonpositive values disable top-k filtering. |
|
|
| `--top-p` | float | `0.8` | Speech-token nucleus threshold; official default 0.8. Out-of-range values are not rejected by the parser. |
|
|
| `--seed` | integer | `0` | Seed used by token sampling, Flow noise, and HiFT. |
|
|
| `--request-option num_inference_steps=<n>` | integer > 0 | checkpoint config | Flow Euler integration steps; commonly 10. |
|
|
| `--request-option flow_guidance_scale=<float>` | float | checkpoint config | Flow classifier-free guidance rate; commonly 0.7. |
|
|
| `--request-option flow_noise_file=<path>` | raw F32 file | none | Optional exact initial Flow noise for parity tests. Legacy `flow_noise_path` remains accepted. |
|
|
| `--request-option hift_source_random_file=<path>` | raw F32 file | none | Optional exact HiFT phase-uniform and Gaussian values for parity tests. Legacy `hift_source_random_path` remains accepted. |
|
|
| `--request-option hift_prior_noise_count=<n>` | integer | `0` | Torch RNG offset before normal HiFT source generation. |
|
|
|
|
The legacy GLM-TTS request keys `flow_steps`, `cfg_rate`, `flow_noise_file`,
|
|
`hift_source_random_file`, and `hift_prior_noise_values` remain accepted for
|
|
backward compatibility.
|
|
| `--session-option glm_tts.weight_type=native\|f32\|f16\|bf16\|q8_0` | enum | `native` | Requested component weight storage type. |
|
|
| `--session-option glm_tts.mem_saver=true\|false` | bool | `false` | Release the reference-only Whisper-VQ and CAMPPlus runtimes after caching the voice, while keeping Llama, Flow, and HiFT warm. |
|
|
| `--session-option glm_tts.aggressive_mem_saver=true\|false` | bool | `false` | Also release Llama, Flow, and HiFT after each stage. This minimizes VRAM but reloads the generation path on every request. |
|
|
| `--session-option glm_tts.reference_cache_slots=<n>` | integer | `1` | Prepared reference-audio cache slots. Reusing a reference skips Whisper-VQ, mel, fbank, and CAMPPlus preparation; `0` disables it. |
|
|
|
|
Balanced mem-saver is intended for a server repeatedly using a cached
|
|
reference voice. On a reference-cache miss it first releases the warm
|
|
generation path, prepares and caches the new voice, then reconstructs the
|
|
generation path. This prevents the reference and generation weight groups
|
|
from overlapping in VRAM. Set `glm_tts.aggressive_mem_saver=true` only when
|
|
the lowest possible peak VRAM is more important than request latency; it
|
|
implies balanced mem-saver even when `glm_tts.mem_saver` is omitted.
|
|
|
|
The reference transcript must match the audio. A mismatched transcript changes
|
|
both semantic and speaker conditioning and can substantially reduce quality.
|
|
Q8 generation can select a slightly different speech-token sequence from the
|
|
native checkpoint, so waveform identity is not expected.
|
|
|
|
See [GLM-TTS validation](../reports/glm_tts_validation.md) for exact component
|
|
parity, path-test, timing, and output details.
|