voice_audio_multi/docs/models/omnivoice.md

OmniVoice

OmniVoice supports multilingual TTS, voice cloning, voice design, non-verbal tag tokens, long-form chunking, and chunked pseudo-streaming output.

Route	Task	Mode	Inputs
Auto voice	tts	offline, streaming	--text
Voice clone	tts, clon	offline, streaming	--text, --voice-ref, optional --reference-text
Voice design	tts, vdes	offline, streaming	--text, --instruct

Model Layout

Use a local OmniVoice model package:

models/OmniVoice/

The package should contain the OmniVoice model weights and the audio-tokenizer files expected by model_specs/omnivoice.json. GGUF packages can also be used when they embed the package spec.

Offline CLI

Voice clone:

audiocpp_cli --task tts --family omnivoice --model /path/to/OmniVoice --backend cuda --text "Hello from OmniVoice." --voice-ref assets/resources/b.wav --reference-text "Some call me nature. Others call me Mother Nature. I've been here for over 4.5 billion years. 22,500 times longer than you." --out out.wav

Voice design:

audiocpp_cli --task tts --family omnivoice --model /path/to/OmniVoice --backend cuda --text "Hello from OmniVoice." --instruct "female, young adult, moderate pitch" --out out.wav

Auto voice:

audiocpp_cli --task tts --family omnivoice --model /path/to/OmniVoice --backend cuda --text "Hello from OmniVoice." --out out.wav

clon and vdes are aliases to the same synthesis implementation. Route selection does not add or remove input requirements.

Streaming CLI

OmniVoice upstream Python does not expose model-native streaming. audio.cpp exposes chunked pseudo streaming: the session emits one audio event per generated text chunk, then returns a merged final WAV.

audiocpp_cli --task tts --mode streaming --family omnivoice --model /path/to/OmniVoice --backend cuda --text "Hello from OmniVoice. This text can be split into chunk events." --voice-ref assets/resources/b.wav --reference-text "Some call me nature. Others call me Mother Nature. I've been here for over 4.5 billion years. 22,500 times longer than you." --text-chunk-size 160 --out stream.wav --out-dir stream_chunks

--out writes the final merged WAV. --out-dir writes emitted chunk WAVs such as chunk_0.wav, chunk_1.wav, and so on.

Server Streaming

Configure OmniVoice with mode: "streaming":

{
  "host": "127.0.0.1",
  "port": 8080,
  "backend": "cuda",
  "device": 0,
  "threads": 8,
  "models": [
    {
      "id": "omnivoice-stream",
      "family": "omnivoice",
      "path": "/path/to/OmniVoice",
      "task": "tts",
      "mode": "streaming"
    }
  ]
}

SSE request:

curl -N http://127.0.0.1:8080/v1/audio/speech \
  -H "Content-Type: application/json" \
  -H "Accept: text/event-stream" \
  -d '{
    "model": "omnivoice-stream",
    "input": "Hello from OmniVoice streaming.",
    "response_format": "pcm",
    "stream_format": "sse",
    "voice": "assets/resources/b.wav",
    "reference_text": "Some call me nature. Others call me Mother Nature. I have been here for over four billion years.",
    "options": {
      "text_chunk_size": 160,
      "text_chunk_mode": "tag_aware"
    }
  }'

The SSE stream emits speech.audio.delta events followed by speech.audio.done.

Common Options (use directly)

Option	Values	Default	Meaning
--voice-ref	WAV path	not set	Reference speaker audio for voice cloning.
--reference-text	text	empty string	Transcript for the reference audio.
--instruct	text	empty string	Voice-design instruction.
--language	language hint	auto	Optional language hint.
--text-chunk-size	integer chars	disabled	Enables framework text chunking and controls pseudo-streaming chunk size.
--text-chunk-mode	default, tag_aware, japanese, endline	tag_aware	Framework text chunking mode when --text-chunk-size is set.
--num-inference-steps	integer	32	Decoder diffusion steps.
--guidance-scale	float	2.0	Decoder CFG strength.
--seed	unsigned 32-bit integer	random session RNG	Generation seed; omitted requests continue the session RNG state.

The corresponding request-option names are reference_text, instruct, language, text_chunk_size, text_chunk_mode, num_inference_steps, guidance_scale, and seed.

Request Options (use with --request-option)

Option	Values	Default	Meaning
speed	float	1.0	Speech speed multiplier.
duration_sec	seconds	estimated	Optional target output duration.
shift	float	0.1	Diffusion timestep schedule shift.
denoise	bool	true	Add the denoise control token when reference audio is present.
preprocess_prompt	bool	true	Preprocess reference audio before tokenization.
postprocess_output	bool	true	Apply output postprocessing and trimming.
layer_penalty_factor	float	5.0	Layer-order penalty for masked audio-codebook position selection.
position_temperature	float	5.0	Gumbel temperature for masked-position selection.
class_temperature	nonnegative float	0.0	Audio-token class sampling temperature; 0 selects classes greedily.
audio_chunk_duration_sec	seconds	15.0	Model-side automatic chunk duration when framework chunking is not explicitly enabled.
audio_chunk_threshold_sec	seconds	30.0	Estimated audio length threshold before model-side chunking is used.

Legacy request names duration, t_shift, audio_chunk_duration, and audio_chunk_threshold remain accepted, respectively. Use one spelling per option. Both spellings work with existing GGUFs; no reconversion is required.

Session Options (use with --session-option)

Arena and weight-context sizes are in MiB.

Option	Values	Default	Meaning
omnivoice.mem_saver	bool	false	Release staged runtime graphs after request phases; default false. Cannot be combined with perf_mode=flash_attention.
omnivoice.perf_mode	off, flash_attention	off	Generator performance mode; default off keeps the exact-safe attention path. flash_attention requires mem_saver=false.
omnivoice.weight_type	native, f32, f16, bf16, q8_0	native	Generator weight storage fallback.
omnivoice.generator_weight_type	native, f32, f16, bf16, q8_0	inherits weight_type	Explicit generator weight storage override.
omnivoice.audio_tokenizer_weight_type	native, f32, f16, bf16, q8_0	native	Runtime storage type for audio tokenizer weights.
omnivoice.audio_tokenizer_codebook_weight_type	native, f32, f16, bf16	f32	Storage type for the audio tokenizer codebooks, codebook score matrix, semantic feature projection, and fc2. These ignore audio_tokenizer_weight_type. native keeps the packaged type and is rejected when that type is quantized, as in the Q8_0 GGUF.
omnivoice.audio_tokenizer_graph_arena_mb	positive integer	128	Audio tokenizer graph arena.
omnivoice.generator_prefill_graph_arena_mb	positive integer	256	Generator prefill graph arena.
omnivoice.generator_decode_graph_arena_mb	positive integer	256	Generator decode graph arena.
omnivoice.audio_tokenizer_weight_context_mb	positive integer	128	Audio tokenizer weight tensor context.
omnivoice.generator_weight_context_mb	positive integer	256	Generator weight tensor context.

omnivoice.perf_mode=flash_attention is only available on the normal graph path and cannot be combined with omnivoice.mem_saver=true.

The two weight_type options quantize SafeTensors weights at load time, which is not the same as loading a prebuilt Q8 GGUF package. On CUDA, omnivoice.generator_weight_type=f16 measured 1.26x faster than native, while runtime omnivoice.audio_tokenizer_weight_type=q8_0 produced unusable audio without being faster. See the OmniVoice weight-type benchmark.

VoiceTut-TTS (Egyptian Arabic)

VoiceTut-TTS is an OmniVoice fine-tune for Egyptian Arabic and Arabic-English code-switching. It has the same architecture and tensor layout as OmniVoice and reuses the OmniVoice audio tokenizer unchanged, so it runs through the omnivoice family with every route and option above. It was trained with the arz (Egyptian Arabic) language id, so pass --language arz. Model and integration by @Mohammedaly22.

Package	Format	Notes
voicetut_tts_q8_0	GGUF Q8_0	Standalone GGUF with the audio tokenizer embedded.
voicetut_tts_f16	GGUF F16	Closest to the original weights.

Voice clone with Egyptian Arabic and English code-switching:

audiocpp_model_manager install voicetut_tts_q8_0 --models-dir models
audiocpp_cli --task tts --family omnivoice --model models/VoiceTut-TTS-GGUF --backend cuda --language arz --text "عندي meeting بكرة الصبح، فهحاول أخلص الشغل بدري النهارده." --voice-ref speaker.wav --reference-text "<transcript of speaker.wav>" --out out.wav

The upstream repository ships 17 reference speakers in reference_speakers/ with transcripts in references.json. Copy them into a server voice_dir (one <Name>.wav per speaker plus a prompt_text file with <Name>|<transcript> lines) to use them as named voices, for example "voice": "Mohamed". See the server voice library.

To build the GGUF packages, place the VoiceTut weights and the OmniVoice audio_tokenizer/ directory in one model root, then convert:

audiocpp_gguf --input weights=models/VoiceTut-TTS/model.safetensors   --input audio_tokenizer_weights=models/VoiceTut-TTS/audio_tokenizer/model.safetensors   --root models/VoiceTut-TTS --output models/VoiceTut-TTS-GGUF/voicetut-tts-q8_0.gguf   --family omnivoice --type q8_0 --overwrite

Tags

Non-verbal tags are written directly in --text. Supported spellings include:

[laughter] [sigh] [confirmation-en] [question-en] [question-ah] [question-oh]
[question-ei] [question-yi] [surprise-ah] [surprise-oh] [surprise-wa]
[surprise-yo] [dissatisfaction-hnn]