Fabio
c36510c21e
Some checks failed
Linux Build / Linux cpu (push) Has been cancelled
Linux Build / Linux cpu-clang (push) Has been cancelled
Linux Build / Linux vulkan (push) Has been cancelled
mac-build / macOS CPU (push) Has been cancelled
Nix Build / Nix (cpu) (push) Has been cancelled
Nix Build / Nix (vulkan) (push) Has been cancelled
windows-build / Windows CPU (push) Has been cancelled
106 lines
4.5 KiB
Markdown
106 lines
4.5 KiB
Markdown
# SAM Audio
|
|
|
|
SAM Audio separates a described sound from a recording. The native port accepts
|
|
text descriptions, masked reference images/videos, and positive/negative time
|
|
anchors in offline mode.
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
audiocpp_cli \
|
|
--task s2s \
|
|
--family sam_audio \
|
|
--model /path/to/SAM-Audio-GGUF/sam-audio-small-f32.gguf \
|
|
--backend cuda \
|
|
--audio input.wav \
|
|
--text "man speaking" \
|
|
--seed 42 \
|
|
--out-dir outputs/separation \
|
|
--log
|
|
```
|
|
|
|
The output directory contains `target.wav` (the described sound) and
|
|
`residual.wav` (the remaining sounds). Both are mono at 48 kHz. The output may
|
|
include padding to the next 40 ms codec frame.
|
|
|
|
## Common Options (use directly)
|
|
|
|
| Option | Values | Default | Meaning |
|
|
|---|---|---|---|
|
|
| `--audio` | audio path | required | Recording to separate; converted to mono at 48 kHz. |
|
|
| `--text` | text | required | Description of the target sound. |
|
|
| `--out-dir` | directory | not set | Save both separated waveforms. |
|
|
|
|
## Request Options (use with `--request-option`)
|
|
|
|
| Option | Values | Default | Meaning |
|
|
|---|---|---|---|
|
|
| `reference_image_file` | image file | not set | Masked RGB image of the target sound source, held constant throughout the recording. Legacy `reference_image_path` remains accepted. |
|
|
| `reference_video_file` | video file | not set | Masked video of the target sound source, aligned to audio by frame timestamps. Legacy `reference_video_path` remains accepted. |
|
|
| `seed` | integer, `-1` for random | `42` | Random seed. Also accepts the common `--seed` flag. |
|
|
| `num_inference_steps` | positive integer | `16` | Midpoint solver steps; each step evaluates the denoiser twice. |
|
|
| `anchors` | JSON array | not set | Positive (`+`) or negative (`-`) time spans in seconds. |
|
|
|
|
For example, mark a target occurrence from 0.5 to 2 seconds and an unwanted sound
|
|
from 3 to 4 seconds:
|
|
|
|
```bash
|
|
--request-option 'anchors=[["+",0.5,2.0],["-",3.0,4.0]]'
|
|
```
|
|
|
|
Spans require `0 <= start < end`. For overlapping spans, the later entry takes
|
|
precedence. The recording is processed as a whole; memory use grows with its length.
|
|
|
|
For a static visual prompt, add `--request-option reference_image_file=masked.png`.
|
|
Mask unwanted regions in the image before supplying it. The model does not create
|
|
the mask. Vision weights load on the first visual request and remain available
|
|
for later requests in the same session.
|
|
|
|
For a moving visual prompt, use `--request-option reference_video_file=masked.mp4`
|
|
instead. Video decoding uses FFmpeg/libav runtime libraries; no FFmpeg command-line
|
|
process is launched. Audio still comes from `--audio`, and should share the video's
|
|
time origin. Use either the image or video option, not both. Frames are selected by
|
|
nearest timestamp and encoded in bounded batches with a reused graph.
|
|
|
|
## Session Options (use with `--session-option`)
|
|
|
|
| Option | Values | Default | Meaning |
|
|
|---|---|---|---|
|
|
| `sam_audio.memory_bounded` | boolean | `false` | Experimental bounded-workspace execution for long recordings; the option name is provisional. |
|
|
|
|
This mode keeps full-recording diffusion and evaluates codec convolutions in
|
|
tiles with complete receptive-field context. Watermark recurrent state is
|
|
carried between tiles. It does not split the recording into independent
|
|
separation requests or crossfade the results. Different computation shapes can
|
|
produce small floating-point differences.
|
|
|
|
GPU workspace is bounded by fixed codec tiles and the model's configured
|
|
diffusion capacity (400 seconds for Small). Intermediate recordings are held
|
|
in host RAM, whose use grows with duration. Extra transfers and overlap
|
|
computation may be slower. This does not extend the model's context limit.
|
|
|
|
Until the option name is finalized, existing GGUFs are unchanged. To try it
|
|
with the current checkout, add these flags to the command above:
|
|
|
|
```bash
|
|
--model-spec-override model_specs/sam_audio.json \
|
|
--session-option sam_audio.memory_bounded=true
|
|
```
|
|
|
|
## Convert Weights
|
|
|
|
With the official checkpoint and configuration in a local directory:
|
|
|
|
```bash
|
|
python tests/sam_audio/convert_gguf.py \
|
|
--model /path/to/sam-audio-small \
|
|
--output /path/to/prepared-sam-audio \
|
|
--gguf-output /path/to/SAM-Audio-GGUF/sam-audio-small-f32.gguf
|
|
```
|
|
|
|
The converter includes the T5 text encoder, tokenizer, configuration, and model
|
|
spec in the GGUF, and checks every tensor against its source. It uses
|
|
`build/debug/bin/audiocpp_gguf` by default; `--converter` selects another binary.
|
|
|
|
Model weights are subject to Meta's SAM License supplied with the original
|
|
checkpoint. The bundled T5 encoder retains its upstream license.
|