voice_audio_multi/docs/models/soulx_flashhead.md

SoulX-FlashHead

Important

cuDNN is strongly recommended for Lite on CUDA: about 2× faster warm inference with roughly half the peak VRAM in the RTX 5090 Q8 measurement (9,152 → 5,168 MiB; 44% less VRAM). Prebuilt binaries do not include cuDNN support. Build from source with cuDNN and -DGGML_CUDA_CUDNN_CONV3D=ON, then select --session-option 'soulx_flashhead.exec_plan={"vae":{"impl":"cudnn"}}'.

SoulX-FlashHead generates portrait video from a reference image and speech audio. The family uses a shared audio-conditioned DiT with variant-specific video VAEs: Lite uses LTX and Pro uses Wan. Only Lite GGUF weights are provided.

Upstream: Soul-AILab/SoulX-FlashHead.

Download Lite

GGUF packages: audio-cpp/SoulX-FlashHead-GGUF. Q8 is the default package; BF16 is also available. Both include all required weights and configuration.

hf download audio-cpp/SoulX-FlashHead-GGUF SoulX-FlashHead-Lite-Q8_0.gguf \
  --local-dir models/SoulX-FlashHead-GGUF

Convert Lite

Download Model_Lite/ and VAE_LTX/ from Soul-AILab/SoulX-FlashHead-1_3B, and the model and configuration files from facebook/wav2vec2-base-960h.

python tools/community_models/soulx_flashhead/convert_gguf.py \
  --source /path/to/SoulX-FlashHead-1_3B \
  --audio-encoder /path/to/wav2vec2-base-960h \
  --type bf16 \
  --output /path/to/SoulX-FlashHead-GGUF/SoulX-FlashHead-Lite-BF16.gguf

The GGUF contains the DiT, video VAE, speech encoder, configuration and model contract. No Python runtime or original checkpoint files are needed for C++ inference.

Generate

build/debug/bin/audiocpp_cli \
  --family soulx_flashhead --task gen \
  --model /path/to/SoulX-FlashHead-Lite-Q8_0.gguf \
  --backend cuda --threads 8 \
  --audio /path/to/speech.wav \
  --request-option reference_image_file=/path/to/portrait.png \
  --out-dir outputs/flashhead \
  --log --log-file outputs/flashhead/run.log

Use --mode streaming to supply audio incrementally. Streaming input must be mono 16 kHz. Lite emits 24 frames per audio block at 25 FPS, carrying nine motion frames between blocks. The audio encoder uses an eight-second sliding window. A partial final audio block is padded with silence, matching upstream.

Output artifacts contain RGB24 frames with width, height, frame count, frame rate and chunk index metadata. Streaming consumers can display each chunk as it arrives.

Request option	Default	Meaning
reference_image_file	required	Reference portrait; resized and center-cropped.
height, width	512	Frame size; divisible by the variant's VAE spatial stride.
num_inference_steps	4	Denoising steps for distilled checkpoints.
seed	42	Generation seed.
color_correction_strength	1	Match the portrait's color statistics; zero disables correction.
audio_guidance_scale	3	Audio guidance for non-distilled checkpoints; unused by Lite/Pro.

Encode the artifacts as an MP4, optionally muxing the source speech:

python tools/community_models/soulx_flashhead/render_video.py \
  --input outputs/flashhead --audio /path/to/speech.wav \
  --output outputs/flashhead/video.mp4

Variant boundaries

The DiT reads channel counts, transformer dimensions, patch size and temporal/spatial VAE strides from the embedded configuration. Lite selects the LTX VAE; Pro and the non-distilled architecture select the shared Wan VAE modules. Non-distilled weights use Euler updates with audio_guidance_scale (default 3); distilled checkpoints use the upstream re-noising update. Only Lite GGUFs are provided. Pro has also been executed from normalized source weights. Its Wan VAE passes the forced-latent RGB comparison against upstream. The unreleased non-distilled checkpoint has not been validated and is not an advertised package. The converter accepts --variant lite|pro|pretrained; its default is Lite. Use --prepare-only with an output directory to normalize a source checkpoint without creating a GGUF. The directory can be loaded directly by this family:

python tools/community_models/soulx_flashhead/convert_gguf.py \
  --source /path/to/SoulX-FlashHead-1_3B \
  --audio-encoder /path/to/wav2vec2-base-960h \
  --variant pro --prepare-only --output /path/to/FlashHead-Pro-normalized

Use that directory as --model in the generation command above.

Lite uses the session-owned LTX video VAE runtime, which caches its encode/decode graphs and shares their scratch arena. Video stays on the backend through color correction and motion encoding; RGB frames are copied to the host for delivery. CUDA attention uses SageAttention2 when the build and device support its head dimensions, otherwise Flash Attention.

Enable the optional native cuDNN Conv3D path for Lite CUDA builds:

cmake -S . -B build/debug -DCMAKE_BUILD_TYPE=Debug \
  -DENGINE_ENABLE_CUDA=ON -DGGML_CUDA_CUDNN_CONV3D=ON \
  -DCUDNN_ROOT=/path/to/cudnn
cmake --build build/debug --parallel 8

When explicitly selected through the execution plan below, this requires cuDNN headers and libraries. Native Lite convolutions use F16 operands, F32 accumulation, and F16 output storage; channel bias is added in F32 to the F16 convolution result before the final F16 cast. Residual storage is also F16. Normalization accumulates in F32 with F16 output on the native path. DiT intermediates, color processing, and public VAE inputs/outputs use F32. These paths avoid reproducing every Python BF16 intermediate rounding; small numerical differences are expected.

Server streaming

Configure a streaming entry and start audiocpp_server with that config:

{
  "host": "127.0.0.1",
  "port": 8080,
  "backend": "cuda",
  "threads": 8,
  "lazy_load": true,
  "models": [{
    "id": "flashhead",
    "family": "soulx_flashhead",
    "path": "/path/to/SoulX-FlashHead-Lite-Q8_0.gguf",
    "task": "gen",
    "mode": "streaming"
  }]
}

curl -N http://127.0.0.1:8080/v1/tasks/stream \
  -H 'Content-Type: application/json' \
  -d '{"model":"flashhead","stream_format":"sse","include_artifacts":true,"request":{"audio":"/path/to/speech-16k-mono.wav","options":{"reference_image_file":"/path/to/portrait.png"}}}'

Each task.stream.event carries video chunks in event.artifacts. Decode the artifact's Base64 payload as RGB24 using its width, height, frames and FPS metadata. chunk_index gives the order. Chunks arrive during generation; task.stream.done ends the request and does not repeat their payloads. For a complete non-streaming result, configure an offline entry and use /v1/tasks/run instead. A streaming entry delivers video through stream events; /v1/tasks/run does not collect those events into its response.

Select the generic VAE with an omitted execution plan or explicitly with --session-option 'soulx_flashhead.exec_plan={"vae":{"impl":"ggml"}}'. For Lite on CUDA, select the optional cuDNN implementation with --session-option 'soulx_flashhead.exec_plan={"vae":{"impl":"cudnn"}}'. This requires the cuDNN Conv3D build option above; an unavailable implementation is rejected rather than silently substituted.