voice_audio_multi/docs/models/controlfoley.md

ControlFoley

ControlFoley is a multimodal Foley generator. It can generate sound from text alone, from video alone, from text plus video, or from reference audio plus video.

Quick Start

Text-to-audio:

audiocpp_cli \
  --task gen \
  --family controlfoley \
  --model models/ControlFoley-GGUF/controlfoley-large-44k-f32.gguf \
  --backend cuda \
  --text "A bird sings melodically in a forest." \
  --request-option duration_sec=10 \
  --request-option num_inference_steps=25 \
  --request-option guidance_scale=4.5 \
  --request-option seed=123 \
  --out foley.wav

Video-conditioned generation:

audiocpp_cli \
  --task gen \
  --family controlfoley \
  --model models/ControlFoley-GGUF/controlfoley-large-44k-f32.gguf \
  --backend cuda \
  --text "skateboarding" \
  --request-option video_file=/path/to/video.mp4 \
  --request-option duration_sec=8 \
  --request-option num_inference_steps=25 \
  --request-option guidance_scale=4.5 \
  --out foley_video.wav

Model

Field	Value
Family	controlfoley
Task	gen
Mode	offline
Packages	controlfoley-large-44k-f32.gguf, controlfoley-large-44k-q8_0.gguf
Text input	Optional prompt through --text
Video input	Optional video path through --request-option video_file=<path>
Audio input	Optional reference WAV through --audio

Paths

Path	Required inputs
T2A	--text
TV2A	--text, --request-option video_file=<path>
TC-V2A	--text, --request-option video_file=<path>, --request-option mask_away_clip=true
AC-V2A	--audio, --request-option video_file=<path>
V2A	--request-option video_file=<path>

Common Options (use directly)

Option	Values	Default	Meaning
--text	prompt text	optional	Text conditioning prompt.
--audio	WAV path	optional	Reference audio for audio-conditioned video generation.

Request Options (use with --request-option)

Option	Values	Default	Meaning
video_file	video path	optional	Video conditioning source. Legacy video remains accepted.
duration_sec	float	8	Target temporal budget.
num_inference_steps	integer > 0	25	Euler flow inference steps.
guidance_scale	float > 0	4.5	Classifier-free guidance strength. Must be > 0. Values strictly between 0 and 1 bypass CFG and negative-prompt conditioning.
negative_prompt	text	empty	Negative text branch for CFG.
mask_away_clip	bool	false	Disable OpenCLIP video conditioning while keeping visual and sync conditioning.
seed	integer >= 0	42	Sampler seed.

Session Options (use with --session-option)

Option	Values	Default	Meaning
controlfoley.weight_type	native, f32, f16, bf16, q8_0	native	Weight storage override for experiments.