Fabio
c36510c21e
Some checks failed
Linux Build / Linux cpu (push) Has been cancelled
Linux Build / Linux cpu-clang (push) Has been cancelled
Linux Build / Linux vulkan (push) Has been cancelled
mac-build / macOS CPU (push) Has been cancelled
Nix Build / Nix (cpu) (push) Has been cancelled
Nix Build / Nix (vulkan) (push) Has been cancelled
windows-build / Windows CPU (push) Has been cancelled
86 lines
2.9 KiB
Markdown
86 lines
2.9 KiB
Markdown
# ControlFoley
|
|
|
|
ControlFoley is a multimodal Foley generator. It can generate sound from text
|
|
alone, from video alone, from text plus video, or from reference audio plus
|
|
video.
|
|
|
|
## Quick Start
|
|
|
|
Text-to-audio:
|
|
|
|
```bash
|
|
audiocpp_cli \
|
|
--task gen \
|
|
--family controlfoley \
|
|
--model models/ControlFoley-GGUF/controlfoley-large-44k-f32.gguf \
|
|
--backend cuda \
|
|
--text "A bird sings melodically in a forest." \
|
|
--request-option duration_sec=10 \
|
|
--request-option num_inference_steps=25 \
|
|
--request-option guidance_scale=4.5 \
|
|
--request-option seed=123 \
|
|
--out foley.wav
|
|
```
|
|
|
|
Video-conditioned generation:
|
|
|
|
```bash
|
|
audiocpp_cli \
|
|
--task gen \
|
|
--family controlfoley \
|
|
--model models/ControlFoley-GGUF/controlfoley-large-44k-f32.gguf \
|
|
--backend cuda \
|
|
--text "skateboarding" \
|
|
--request-option video_file=/path/to/video.mp4 \
|
|
--request-option duration_sec=8 \
|
|
--request-option num_inference_steps=25 \
|
|
--request-option guidance_scale=4.5 \
|
|
--out foley_video.wav
|
|
```
|
|
|
|
## Model
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Family | `controlfoley` |
|
|
| Task | `gen` |
|
|
| Mode | `offline` |
|
|
| Packages | `controlfoley-large-44k-f32.gguf`, `controlfoley-large-44k-q8_0.gguf` |
|
|
| Text input | Optional prompt through `--text` |
|
|
| Video input | Optional video path through `--request-option video_file=<path>` |
|
|
| Audio input | Optional reference WAV through `--audio` |
|
|
|
|
## Paths
|
|
|
|
| Path | Required inputs |
|
|
|---|---|
|
|
| T2A | `--text` |
|
|
| TV2A | `--text`, `--request-option video_file=<path>` |
|
|
| TC-V2A | `--text`, `--request-option video_file=<path>`, `--request-option mask_away_clip=true` |
|
|
| AC-V2A | `--audio`, `--request-option video_file=<path>` |
|
|
| V2A | `--request-option video_file=<path>` |
|
|
|
|
## Common Options (use directly)
|
|
|
|
| Option | Values | Default | Meaning |
|
|
|---|---|---:|---|
|
|
| `--text` | prompt text | optional | Text conditioning prompt. |
|
|
| `--audio` | WAV path | optional | Reference audio for audio-conditioned video generation. |
|
|
|
|
## Request Options (use with `--request-option`)
|
|
|
|
| Option | Values | Default | Meaning |
|
|
|---|---|---:|---|
|
|
| `video_file` | video path | optional | Video conditioning source. Legacy `video` remains accepted. |
|
|
| `duration_sec` | float | `8` | Target temporal budget. |
|
|
| `num_inference_steps` | integer > 0 | `25` | Euler flow inference steps. |
|
|
| `guidance_scale` | float > 0 | `4.5` | Classifier-free guidance strength. Must be > 0. Values strictly between 0 and 1 bypass CFG and negative-prompt conditioning. |
|
|
| `negative_prompt` | text | empty | Negative text branch for CFG. |
|
|
| `mask_away_clip` | bool | `false` | Disable OpenCLIP video conditioning while keeping visual and sync conditioning. |
|
|
| `seed` | integer >= 0 | `42` | Sampler seed. |
|
|
|
|
## Session Options (use with `--session-option`)
|
|
|
|
| Option | Values | Default | Meaning |
|
|
|---|---|---:|---|
|
|
| `controlfoley.weight_type` | `native`, `f32`, `f16`, `bf16`, `q8_0` | `native` | Weight storage override for experiments. |
|