voice_audio_multi/docs/models/sopro_tts.md

Sopro V2 Turbo (sopro_tts)

samuel-vitorino/sopro-v2-turbo is a 120M-parameter zero-shot voice-cloning TTS covering English, European Portuguese, French and German, released under Apache-2.0. It clones from 5–20 s of reference audio and outputs 24 kHz mono.

Not the same model as soprano_tts. audio.cpp's existing soprano_tts family is ekwek/Soprano-1.1-80M, an unrelated project with a Qwen3 backbone and a different decoder. Sopro V2 Turbo shares nothing with it beyond a similar name, so it ships as its own family. The --family hints sopro, sopro_v2 and sopro_v2_turbo all resolve to sopro_tts.

Installation

Both packages come from the model repo. The f16 GGUF is a single self-contained file and the one the WebUI installs; the upstream safetensors checkpoint runs directly in full precision:

python3 tools/model_manager_v2.py install sopro_v2_turbo_f16
# -> models/sopro-v2-turbo-GGUF/sopro-v2-turbo-f16.gguf
python3 tools/model_manager_v2.py install sopro_v2_turbo_safetensors
# -> models/sopro-v2-turbo/{config.json,tokenizer.model,*.safetensors}

The GGUF was packed from the safetensors with the command below. audiocpp_gguf takes one namespaced input per stage, and --root makes it embed config.json and tokenizer.model as sidecars. Keep the mel front ends' filterbanks and windows in f32 when packing your own: --type f16 would otherwise round them too, and the reference mel the acoustic head copies from drifts by several dB.

build/bin/audiocpp_gguf \
    --input model=models/sopro-v2-turbo/model.safetensors \
    --input semantic_encoder=models/sopro-v2-turbo/semantic_encoder.safetensors \
    --input speaker_encoder=models/sopro-v2-turbo/speaker_encoder.safetensors \
    --input vocoder=models/sopro-v2-turbo/vocoder.safetensors \
    --family sopro_tts --root models/sopro-v2-turbo \
    --keep-type 'vocoder/feature_extractor.*=f32' --keep-type 'vocoder/head.istft.window*=f32' \
    --keep-type 'speaker_encoder/frontend.*=f32' --keep-type 'semantic_encoder/frontend.*=f32' \
    --output models/sopro-v2-turbo-GGUF/sopro-v2-turbo-f16.gguf --type f16

Build

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release \
    -DAUDIOCPP_MODEL_SET=custom -DAUDIOCPP_MODELS=sopro_tts
cmake --build build --target audiocpp_cli -j"$(nproc)"

Run

Zero-shot cloning always needs a reference clip:

build/bin/audiocpp_cli \
    --task tts --family sopro_tts \
    --model models/sopro-v2-turbo \
    --backend cpu --threads 8 \
    --text "Sopro is a lightweight text-to-speech model that runs on device." \
    --voice-ref ref.wav \
    --request-option language=en \
    --out out.wav --metrics

--task clon works the same way. The reference is resampled to 24 kHz, cropped at a pause near reference_duration_sec, and level-normalised before the speaker and semantic encoders see it. That normalisation is boost-only and peak-guarded (sopro 2.2): a reference already at or above the −19.8 dB prompt level is passed through untouched, and a boost is never large enough to push the peak past 0.95. The output gain is fixed by the level the reference ends up at: it maps that level to −23 dB (output_gain), the same for every segment and in both run modes.

Streaming

--mode streaming emits audio as the semantic LM runs, like SoproTTS.stream upstream:

build/bin/audiocpp_cli \
    --task tts --family sopro_tts \
    --model models/sopro-v2-turbo \
    --backend cpu --threads 8 --mode streaming \
    --text "$(cat article.txt)" \
    --voice-ref ref.wav --language en \
    --out stream.wav --out-dir chunks/

Each event carries a chunk_<n> named audio buffer that is already levelled, gated and faded, so a consumer can play events back to back; --out writes the whole utterance, which is exactly the concatenation of the events.

How it works (streaming.cpp, a port of sopro/streaming.py):

• The reference prompt is trimmed to a whole number of semantic tokens and its complete stream_chunk_frames (64-frame) chunks are solved once, before the first text segment.
• Every 16 semantic tokens, the acoustic head solves the frames that can no longer change (a 3-token lookahead margin behind the last token, rounded down to whole chunks). Each frame attends to everything solved before it within its chunk boundary, and the solver keeps every DiT layer's keys and values per Euler step, so a chunk costs only its own frames.
• The vocoder decodes each chunk once its receptive field (27 frames each side) is complete and overlap-adds it into a streaming ISTFT.
• The opening audio of each text segment is held until speech starts, then the lead-in is cut the same way the offline path trims it.

Streaming is not a re-cut of the offline output: it solves with chunk-limited attention and its own noise draws, as the Python package does, so the two modes differ for the same seed. On an 8-core Apple M-series CPU at 6 threads the first chunk arrives about 1.6–1.8 s after the request starts (reference encoding ~0.3–0.45 s, prompt solve ~0.8–0.95 s, first chunk ~0.4–0.5 s for a 10 s reference), and each later 0.68 s chunk takes about 0.2 s. reference_duration_sec scales the prompt solve.

Voice cache

sopro_tts.voice_cache_slots=<n> (off by default) keeps the last n prepared voices, keyed by the clip and reference_duration_sec, together with each streamed voice's solved prompt, like reusing a prepare_reference result upstream. A repeated voice skips the encoders and the prompt solve, which brought the first chunk down from ~1.8 s to ~0.4–0.5 s in the setup above. A streamed 10 s voice holds about 60 MB.

To prepare a voice without generating anything, pass audiocpp_session_prepare a request that carries only the voice audio (plus reference_duration_sec or num_inference_steps if the requests that follow set them).

The cache never changes the audio: the draws a voice needs for itself (room tone for a clip without a usable pause, the prompt noise) are seeded from the clip rather than the request.

Options

Request option	Type	Default	Meaning
language	string	see description	Language tag (en, pt, fr, de); omitted inherits transcript language, then the session default.
temperature	float	see description	Semantic sampling temperature; nonpositive selects greedy decoding. Omitted uses checkpoint generation.temperature.
top_p	float	see description	Semantic nucleus cutoff; values outside [0,1] are disabled/clamped by the sampler. Omitted uses checkpoint generation.top_p.
top_k	int	see description	Semantic top-k; nonpositive disables filtering. Omitted uses checkpoint generation.top_k.
num_inference_steps	int	see description	Positive acoustic sampling steps; omitted uses checkpoint generation.steps.
max_duration_sec	float	see description	Positive maximum segment duration; must be >= min_duration_sec. Omitted uses checkpoint generation.max_seconds.
min_duration_sec	float	see description	Nonnegative minimum duration; must be <= max_duration_sec. Omitted uses checkpoint generation.min_seconds.
reference_duration_sec	float	see description	Positive reference window in seconds; omitted uses checkpoint generation.ref_seconds.
text_chunk_size	int	see description	Positive codepoint budget per segment; omitted uses checkpoint generation.max_segment_chars.
seed	int	(random)	Seeds semantic sampling and the acoustic noise prior

Session option	Default	Meaning
sopro_tts.language	(empty)	Default language tag for requests that do not set one
sopro_tts.voice_cache_slots	0	Prepared voices kept between requests; see Voice cache
sopro_tts.matmul_weight_type	f32	Storage type for matmul weights (native, f32, f16, bf16, q8_0)
sopro_tts.conv_weight_type	f32	Storage type for convolution weights (native, f32, f16)

Numeric generation defaults come from the checkpoint's config.json generation block, so a retrained variant picks up its own values without a code change. Language inherits the transcript language, then the session default; an omitted seed is random.

Architecture notes

Five stages run per request, mirroring sopro/ upstream:

1. Text (text_tokenizer.cpp) — SentencePiece unigram, 8192 pieces, plus the reference's punctuation clean-up and sentence/clause/word segmentation. No phonemiser.
2. Speaker encoder (speaker_encoder.cpp, ~11M) — 16 kHz log-mel into a three-stage gated depthwise ResNet with squeeze-excite, then attentive-statistics pooling for identity and multi-scale mean/std pooling for style. The convolution trunk runs on the backend; the two pooling heads and their MLPs run on the host, where they cost a few hundred kFLOP.
3. Semantic encoder (semantic_encoder.cpp, ~82M) — a Whisper-style front end and six non-causal transformer layers, resampled to one frame per 1024 output samples and quantised by an FSQ head with levels [7,5,5,5,5] (4375 codes).
4. Semantic LM (semantic_lm.cpp) — 12 pre-norm blocks, dim 512, QK RMS-norm, SwiGLU, half-rotation RoPE. The prompt is [style prefix | text | prompt tokens | BOS], where the prompt tokens are the first prompt_tokens reference tokens for the first segment and the tail of the previous segment's tokens after that. Because the only structural difference from a Qwen3 decoder is a LayerScale vector on each residual branch, and those branches end in a bias-free projection, the scale is folded into that projection's rows at load time and the shared QwenCausalDecodeRuntime runs the stack unmodified. The eight-query style prefix cross-attention runs on the host.
5. Acoustic head + vocoder (acoustic.cpp, vocoder.cpp) — an 8-block adaptive-layer-norm DiT solving a rectified flow in two Euler steps on a sway-sampled time grid, with the prompt mel re-pinned after every step; then an 8-block Vocos ConvNeXt backbone and one centred ISTFT over the full band. The vocoder config is causal with a lookahead of 3 frames on every 7-tap conv, which is the stock centred Vocos layout; other lookaheads are padded to match. mu (the upsampled semantic conditioning) is built in its own graph because it is constant across solver steps.

Implementation details worth knowing:

• Front-end buffers come from the checkpoint. torchaudio stores its analysis window and mel filterbank as persistent buffers, and all three front ends load those rather than rebuilding the filterbank, which removes the usual mel-parity risk. A checkpoint exported without them fails at load with a message naming the missing tensor.
• Grouped convolutions are split. The DiT's causal positional embedding uses Conv1d(512, 512, k=31, groups=16); ggml has no grouped conv1d, so the weight is split into 16 independent convolutions at load time.
• The velocity graph re-uploads every leaf per Euler step. ggml_gallocr exempts only GGML_TENSOR_FLAG_OUTPUT tensors from being freed and reused (ggml_gallocr_free_node in ggml-alloc.c); an input leaf's arena space is handed to a later intermediate once its last consumer has run. That is correct for a one-shot graph, but the solver replays the velocity graph once per step, so staging mu, cond_mel, cond_mask, spk and the RoPE positions once would leave the second and later steps reading whatever overwrote them. SoproVelocityGraph::run uploads all of them before every compute; it costs a few hundred kB per step against a multi-GFLOP DiT pass.

Known limitations

• Without the voice cache, every request prepares its voice again. The reference arrives with each request, so a stream pays the encoders and the prompt solve unless sopro_tts.voice_cache_slots is set.
• Sampling RNG is not torch-bit-exact. sample_next_token reproduces the reference's masking, temperature, top-k and top-p arithmetic exactly, but draws from a seeded std::mt19937_64 rather than torch's generator, so a given seed will not reproduce the Python output sample-for-sample. The same seed is reproducible within audio.cpp.
• No int8 AR path. The upstream --int8 CPU option quantises only the semantic LM; the closest here, sopro_tts.matmul_weight_type=q8_0, quantises the matmuls of every stage.
• The text front end is deliberately minimal upstream: prefer words to symbols (one plus two, not 1 + 2), and avoid mixing languages inside one sentence.

Validation status

Verified against the real checkpoint on a 16-core x86-64 CPU build, 8 threads. Every stage was reimplemented independently in numpy, driven from the checkpoint's own weights, and diffed against the C++.

Stage	Check	Result
Vocoder mel front end	vs. numpy STFT + checkpoint filterbank	max diff 1.9e-3
Semantic encoder mel	vs. numpy	max diff 1.9e-5
Semantic encoder transformer	vs. numpy, all 6 layers	max diff 6.0e-5
FSQ token ids	vs. numpy	188/188 identical
Speaker encoder mel / trunk / heads	vs. numpy	max diff 3.7e-5
Acoustic mu, spk, time embedding	vs. numpy	max diff 1.3e-5
Acoustic velocity field, every Euler step	vs. numpy	max diff 5.8e-3
Acoustic self-reconstruction	NMSE vs. the reference's own mel	0.38
Fixed seed reproducibility	byte-identical WAV across runs	pass
Streaming chunk sum	chunks vs. --out	exact
Voice cache on vs. off	server, 7 requests per mode (repeats, a new seed, an eviction)	byte-identical
Voice prepared ahead vs. not	C API, audiocpp_session_prepare with only the voice, both modes	byte-identical
Long-form, 6026 chars, text_chunk_size=200	~371 s of audio, Apple M3, 6 threads	offline 118 s, streaming 136 s; peak RSS 1.57 / 1.95 GB
matmul_weight_type f16 / bf16 / q8_0	vs. f32, same seed, both modes	f16 0.12–0.16 dB, bf16 0.5–2.1 dB spectrum; q8_0 samples different tokens
Published f16 GGUF	vs. safetensors, 12 cases, same seed, both modes	0.03–0.08 dB spectrum on 8 of 12, both greedy cases included; the f16 LM samples different tokens on the 4 long sampled cases
orig GGUF	vs. safetensors, same seed, both modes	byte-identical

The numpy references above are independent reimplementations from the same source, which catches implementation bugs but not a shared misreading of the architecture.

Parity with the Python package (sopro 2.2.0)

Measured end to end against SoproTTS.synthesize and SoproTTS.stream from the published 2.2.0 wheel and the published checkpoint, CPU backend: EN and PT, single- and multi-segment text, 16- and 32-bit references, one reference short enough to take the room-tone crop branch. For these checks the C++ run was given the Python run's random draws (AR tokens, solver noise, room tone), so every other stage is computed independently on each side.

Check	Result
Reference crop, normalisation, 16 kHz resample	max sample diff 5e-7
Reference semantic tokens	identical, all cases
Reference analysis mel	max diff 6e-5 to 1.3e-3 on 16-bit references
Acoustic solve (2 Euler steps), PT case	max diff 6e-4, mean 6.5e-5 (normalised mel units)
Vocoder + gain + trims + fades, PT case	77.5 dB waveform SNR when fed the same solved mel
Greedy AR (temperature 0, C++ samples its own tokens)	identical tokens and length, 2 cases
Output length	identical, all cases
Output level	within 0.02 dB
Output spectrum (STFT magnitude, cells within 60 dB of peak)	mean diff 0.03–0.05 dB on 16-bit references
Streaming vs SoproTTS.stream, 5 cases (up to 47 chunks, 2 segments)	identical length and chunk boundaries
Streaming output spectrum	mean diff 0.03 dB on 16-bit references

Two things to read the numbers right:

• Compare spectra, not samples. The vocoder predicts phase, so the waveform is very sensitive to its input: adding 1e-5 of noise to the normalised mel moves the Python output itself to ~52 dB waveform SNR, and the 6e-4 float32 difference of the solve gives ~31 dB. The magnitude spectra stay within a few hundredths of a dB.
• Near-silent reference bands are float32 noise. On two 32-bit studio references the analysis mel differs (up to 0.3) in cells ~30 dB below the frame, mostly near 11–12 kHz. Those cells sit below float32 resolution: adding 1e-8 of noise to the Python reference moves them by up to 0.45. Those outputs differ by 0.16–0.26 dB mean; with the C++ reference mel fed to Python the gap closes to 0.01–0.02 dB.

Debugging

tests/sopro_tts/sopro_probe.cpp (built with -DENGINE_BUILD_WARMBENCH=ON) exercises each stage in isolation against a reference clip:

build/bin/sopro_probe models/sopro-v2-turbo reference.wav /tmp/soproprobe

It reports the crop_on_pause decision, a mel round trip through the vocoder (which is phase-invariant and so the meaningful vocoder check), the FSQ token histogram, the speaker embedding statistics, and an acoustic self-reconstruction NMSE. It also writes probe_vocoder_roundtrip.wav — the reference passed through mel then the vocoder; if that sounds like the speaker, the whole back half of the pipeline is fine.

Setting SOPRO_DUMP_DIR=<dir> additionally dumps the encoder and solver intermediates as raw f32 for diffing against a reference implementation.

References

• Model card: https://huggingface.co/samuel-vitorino/sopro-v2-turbo
• Reference implementation: https://github.com/samuel-vitorino/sopro
• Blog post: https://research.haloneuro.ai/posts/sopro-v2