Fabio
c36510c21e
Some checks failed
Linux Build / Linux cpu (push) Has been cancelled
Linux Build / Linux cpu-clang (push) Has been cancelled
Linux Build / Linux vulkan (push) Has been cancelled
mac-build / macOS CPU (push) Has been cancelled
Nix Build / Nix (cpu) (push) Has been cancelled
Nix Build / Nix (vulkan) (push) Has been cancelled
windows-build / Windows CPU (push) Has been cancelled
114 lines
4.9 KiB
Markdown
114 lines
4.9 KiB
Markdown
# FireRedAudio
|
|
|
|
FireRedAudio is a multimodal speech and audio model. The native audio.cpp path
|
|
covers Chinese ASR, audio understanding, TTS cloning, voice design, semantic
|
|
editing, and acoustic editing from a single GGUF package.
|
|
|
|
## Quick Start
|
|
|
|
Chinese ASR:
|
|
|
|
```bash
|
|
audiocpp_cli \
|
|
--task asr \
|
|
--family firered_audio \
|
|
--model models/FireRedAudio-GGUF/firered-audio-orig.gguf \
|
|
--backend cuda \
|
|
--audio input.wav \
|
|
--text "Transcribe speech to text." \
|
|
--request-option template_name=asr \
|
|
--text-out transcript.txt
|
|
```
|
|
|
|
Voice cloning:
|
|
|
|
```bash
|
|
audiocpp_cli \
|
|
--task clon \
|
|
--family firered_audio \
|
|
--model models/FireRedAudio-GGUF/firered-audio-orig.gguf \
|
|
--backend cuda \
|
|
--language zh \
|
|
--text "安徽淮南秦师傅发现，停在小区的爱车右前驾驶窗玻璃被砸。" \
|
|
--voice-ref reference/FireRedAudio/assets/examples/tts_zh_prompt.wav \
|
|
--reference-text "同时，他强调微调要科学有序。" \
|
|
--request-option template_name=tts_clone \
|
|
--out firered_audio_clone.wav
|
|
```
|
|
|
|
Audio understanding:
|
|
|
|
```bash
|
|
audiocpp_cli \
|
|
--task asr \
|
|
--family firered_audio \
|
|
--model models/FireRedAudio-GGUF/firered-audio-orig.gguf \
|
|
--backend cuda \
|
|
--audio input.wav \
|
|
--text "What is happening in this audio?" \
|
|
--request-option template_name=understand \
|
|
--request-option enable_thinking=true \
|
|
--request-option max_tokens=1024 \
|
|
--text-out answer.txt
|
|
```
|
|
|
|
## Model
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Family | `firered_audio` |
|
|
| Tasks | `asr`, `tts`, `clon`, `vdes`, `gen` |
|
|
| Mode | `offline` |
|
|
| Default package | `models/FireRedAudio-GGUF/firered-audio-q8_0.gguf` |
|
|
| Generation paths | `tts_clone`, `voice_design`, `semantic_edit`, `acoustic_edit` |
|
|
| Text paths | `asr`, `understand` |
|
|
|
|
## Templates
|
|
|
|
| `template_name` | Task | Inputs |
|
|
|---|---|---|
|
|
| `asr` | `asr` | `--audio`, optional prompt through `--text` |
|
|
| `understand` | `asr` | `--audio`, question through `--text` |
|
|
| `tts_clone` | `clon` | `--text`, `--voice-ref`, `--reference-text` |
|
|
| `voice_design` | `vdes` | `--text`, `--request-option instruct=<text>` |
|
|
| `semantic_edit` | `gen` | `--audio`, `--request-option instruct=<text>` |
|
|
| `acoustic_edit` | `gen` | `--audio`, `--request-option instruct=<text>` |
|
|
|
|
## Common Options (use directly)
|
|
|
|
| Option | Values | Default | Meaning |
|
|
|---|---|---:|---|
|
|
| `--language` | language tag | `zh` | Generation or text-path language tag. |
|
|
| `--voice-ref` | WAV path | required for clone | Prompt/reference voice. |
|
|
| `--reference-text` | text | empty | Transcript for the prompt audio. |
|
|
|
|
## Request Options (use with `--request-option`)
|
|
|
|
| Option | Values | Default | Meaning |
|
|
|---|---|---:|---|
|
|
| `template_name` | `asr`, `understand`, `tts_clone`, `voice_design`, `semantic_edit`, `acoustic_edit` | see description | Defaults to voice_design for design, acoustic_edit for generation with input audio, tts_clone for other generation, and asr for ASR. |
|
|
| `language` | language tag | `zh` | Generation or text-path language tag. |
|
|
| `reference_text` | text | empty | Transcript for the prompt audio. |
|
|
| `instruct` | text | required for design/edit | Generation voice/delivery instruction. Legacy `instruction` remains accepted. ASR and understanding instead use the text input as their task prompt. |
|
|
| `num_inference_steps` | integer > 0 | `10` | FireRedAudio DiT flow steps per latent patch. |
|
|
| `guidance_scale` | float > 0 | `2.0` | FireRedAudio DiT CFG scale. |
|
|
| `max_audio_steps` | integer > 0 | `750` | Maximum generated RedAE latent patches. Legacy `max_new_audio_steps` remains accepted. |
|
|
| `min_audio_steps` | integer >= 0 | `6` | Minimum audio patches before stop may be accepted. Legacy `min_new_audio_steps` remains accepted. |
|
|
| `max_text_tokens` | integer > 0 | `512` | Text-mode token budget before audio mode. Legacy `max_new_text_tokens` remains accepted. |
|
|
| `max_tokens` | integer > 0 | see description | Output token budget; defaults to 300, or 1024 for understand with enable_thinking=true. |
|
|
| `enable_thinking` | bool | `false` | Enable open thinking block for `understand`; invalid for plain `asr`. |
|
|
| `top_k` | integer >= 0 | `20` | Understanding top-k sampling. |
|
|
| `top_p` | `0..1` | `0.8` | Understanding nucleus sampling. |
|
|
| `temperature` | float >= 0 | `0.7` | Text sampling temperature for understanding. Understanding-mode sampling requires a positive value; zero does not select greedy decoding. |
|
|
| `seed` | integer >= 0 | `1234` | Generation seed. |
|
|
|
|
## Session Options (use with `--session-option`)
|
|
|
|
| Option | Values | Default | Meaning |
|
|
|---|---|---:|---|
|
|
| `firered_audio.reference_cache_slots` | integer >= 0 | `2` | Prepared reference-audio cache slots. |
|
|
| `firered_audio.mem_saver` | bool | `false` | Release runtime graphs after request phases. |
|
|
| `firered_audio.weight_type` | `native`, `f32`, `f16`, `bf16`, `q8_0` | `native` | Weight storage override for experiments. |
|
|
|
|
Legacy `firered_audio.helper_graph_arena_mb` remains accepted but has no effect
|
|
on allocation; it is not a memory control.
|