voice_audio_multi/docs/models/ace_step.md

# ACE-Step

ACE-Step is wired as `--family ace_step --task gen`. It generates and edits music from text, lyrics, and optional source audio. The route controls whether source audio is ignored, optional, or required. Supplied `audio_codes` can also satisfy the source-input requirement for routes that require audio.

Common CLI shape:

```bash
audiocpp_cli --task gen --family ace_step --model models/Ace-Step1.5 --backend cuda --task-route <route> ...
```

## Model

| Field | Value |
|---|---|
| Family | `ace_step` |
| Model directory | `models/Ace-Step1.5` |
| Task | `gen` |
| Default DiT | Inferred from model filename (`base`, `sft`, `turbo`, and `xl`); otherwise `acestep-v15-turbo` |
| Optional DiT | `acestep-v15-base`, `acestep-v15-xl-turbo`, `acestep-v15-xl-sft` |
| Default LM | `acestep-5Hz-lm-1.7B` |
| Prompt input | `--text` |
| Lyrics input | `--lyrics` |
| Source audio | Route-dependent through `--audio` |

## Text To Music

Generate a full song or music clip from prompt text and optional lyrics. Source audio is ignored by this route.

| Field | Value |
|---|---|
| Route | `text2music` |
| Source audio | Ignored |
| Planner | Used unless `audio_codes` are supplied |
| Duration | `--duration-seconds`; `-1` lets the planner/model choose |
| Metadata controls | Optional `bpm`, `keyscale`, `timesignature`, and `language` |

```bash
audiocpp_cli --task gen --family ace_step --model models/Ace-Step1.5 --backend cuda --task-route text2music --text "cinematic synth pop with clear vocals" --lyrics "We rise with the morning light" --duration-seconds 60 --out song.wav
```

Use the base DiT instead of turbo:

```bash
audiocpp_cli --task gen --family ace_step --model models/Ace-Step1.5 --backend cuda --task-route text2music --text "acoustic folk ballad" --lyrics "The river remembers our names" --duration-seconds 60 --load-option ace_step.dit_model_path=acestep-v15-base --out song.wav
```

## Complete

Complete or continue a source audio track. Source audio is optional: if provided, it conditions the continuation; otherwise the route behaves like a completion prompt.

| Field | Value |
|---|---|
| Route | `complete` |
| Source audio | Optional |
| Planner | Used |
| Duration | Source duration is not locked unless requested metadata or planner output controls it |
| Track classes | Optional `complete_track_classes` list changes the completion instruction |

```bash
audiocpp_cli --task gen --family ace_step --model models/Ace-Step1.5 --backend cuda --task-route complete --audio input.wav --text "finish this as a cinematic rock track" --out complete.wav
```

## Lego

Compose or transform a track from source audio plus a prompt. Source audio is required and preserved as the repaint context.

| Field | Value |
|---|---|
| Route | `lego` |
| Source audio | Required |
| Planner | Used |
| Duration | Locked to source audio |
| Repaint window | Used internally over the source context |
| Track name | Optional `track_name` changes the generated instruction |

```bash
audiocpp_cli --task gen --family ace_step --model models/Ace-Step1.5 --backend cuda --task-route lego --audio input.wav --text "generate a brighter guitar layer" --request-option track_name=guitar --out lego.wav
```

## Extract

Extract a target track from source audio. Source audio is required, and the route uses extraction-specific defaults for guidance and shift.

| Field | Value |
|---|---|
| Route | `extract` |
| Source audio | Required |
| Planner | Not used |
| Duration | Locked to source audio |
| Track name | Optional `track_name`; when omitted, the default extract instruction is used |
| Route defaults | `guidance_scale=7.0`, `shift=3.0`, planner chain-of-thought metadata disabled |

```bash
audiocpp_cli --task gen --family ace_step --model models/Ace-Step1.5 --backend cuda --task-route extract --audio song.wav --text "extract vocals" --request-option track_name=vocals --out vocals.wav
```

## Cover

Generate a cover from source audio using cover-tokenizer conditioning. Source audio is required and duration is locked to the source.

| Field | Value |
|---|---|
| Route | `cover` |
| Source audio | Required |
| Planner | Not used |
| Cover conditioning | Uses the FSQ cover tokenizer |
| Duration | Locked to source audio |

```bash
audiocpp_cli --task gen --family ace_step --model models/Ace-Step1.5 --backend cuda --task-route cover --audio source.wav --text "turn this into energetic pop vocals" --lyrics "We keep moving through the night" --out cover.wav
```

## Cover Without FSQ

Generate a cover from source audio without the FSQ cover-tokenizer conditioning path.

| Field | Value |
|---|---|
| Route | `cover-nofsq` |
| Source audio | Required |
| Planner | Not used |
| Cover conditioning | Does not use the FSQ cover tokenizer |
| Duration | Locked to source audio |

```bash
audiocpp_cli --task gen --family ace_step --model models/Ace-Step1.5 --backend cuda --task-route cover-nofsq --audio source.wav --text "make a softer acoustic cover" --lyrics "We keep moving through the night" --out cover_nofsq.wav
```

## Repaint

Replace a time span inside source audio. Source audio and a repaint window are required.

| Field | Value |
|---|---|
| Route | `repaint` |
| Source audio | Required |
| Planner | Not used |
| Duration | Locked to source audio |
| Required window | `--repaint-start`, `--repaint-end` |
| Repaint policy | `repaint_mode`, `repaint_strength`, or direct repaint injection/crossfade options |

```bash
audiocpp_cli --task gen --family ace_step --model models/Ace-Step1.5 --backend cuda --task-route repaint --audio song.wav --text "replace the middle with a brighter chorus" --repaint-start 20 --repaint-end 35 --out repaint.wav
```

## Common Options (use directly)

| Option | Values | Default | Meaning |
|---|---|---:|---|
| `--task-route` | `text2music`, `complete`, `lego`, `extract`, `cover`, `cover-nofsq`, `repaint` | `text2music` | ACE-Step operation. |
| `--text` | text | required | Music prompt or edit instruction. |
| `--lyrics` | text | empty string | Vocal lyrics. |
| `--audio` | WAV path | route-dependent | Source audio for complete/edit/extract/cover routes. |
| `--duration-seconds` | float, `-1` for auto | `-1` | Target duration. Source-locked routes use source duration. |
| `--language` | language code | text input language, then `en` | Vocal language for lyrics; request-map `language` takes precedence. |
| `--track-name` | text | empty string | Track name used by `lego` and `extract` instructions. |
| `--repaint-start` | seconds | required for `repaint` | Start time for repaint. |
| `--repaint-end` | seconds | required for `repaint` | End time for repaint. |
| `--repaint-mode` | `balanced`, `conservative`, `aggressive` | unset | Preset repaint blending policy; supplying strength alone selects `balanced`. |
| `--repaint-strength` | `0..1` | `0.5` when a preset is supplied | Strength for balanced mode; values are clamped to this range. |
| `--num-inference-steps` | integer | `8` | Denoising steps; turbo checkpoints cap this at 8. |
| `--guidance-scale` | float | `1.0`; `7.0` for `extract` unless overridden | Diffusion CFG; ignored by turbo checkpoints. |
| `--seed` | integer | random if omitted | Generation seed. |

## Request Options (use with `--request-option`)

| Option | Values | Default | Meaning |
|---|---|---:|---|
| `complete_track_classes` | track class or JSON array | empty list | Track classes for `complete`, e.g. `'["vocals","guitar"]'`. A plain string is one class, not a comma-separated list. |
| `bpm` | integer | not set | Force BPM metadata; otherwise the planner chooses it when used. |
| `keyscale` | text | not set | Force key metadata; otherwise the planner chooses it when used. |
| `timesignature` | text | not set | Force time signature metadata; otherwise the planner chooses it when used. |
| `negative_prompt` | text | `NO USER INPUT` | Negative prompt. |
| `audio_codes` | ACE semantic code text | not set | Skip planner token generation and use supplied audio codes. |
| `audio_cover_strength` | float | `1.0` | Cover strength for cover/edit-style conditioning. |
| `cover_noise_strength` | float | `0.0` | Noise strength for cover conditioning. |
| `lm_temperature` | float | `0.85` | Planner sampling temperature. |
| `lm_guidance_scale` | float | `2.0` | Planner CFG scale. Legacy `lm_cfg_scale` remains accepted. |
| `duration_sec` | float | `-1.0` | Target duration; non-positive leaves selection to the planner. Legacy `duration_seconds` remains accepted. |
| `enable_thinking` | bool | `true` | Enable planner thinking when supported by the checkpoint. Legacy `thinking` remains accepted. |
| `lm_top_k` | integer | `0` | Planner top-k; `0` disables top-k. |
| `lm_top_p` | float | `0.9` | Planner top-p. |
| `lm_repetition_penalty` | float | `1.0` | Planner repetition penalty. |
| `sampler_mode` | `euler`, `heun` | `euler` | Diffusion sampler mode. |
| `shift` | float | `1.0`; `3.0` for `extract` unless overridden | Diffusion timestep schedule shift. |
| `retake_seed` | integer, `-1` to clear | not set | Optional retake noise seed. |
| `retake_variance` | float | `0.0` | Retake noise mixing strength. |
| `flow_edit_morph` | bool | `false` | Status: parsed for text2music, but not usable because the flow-edit diffusion overlay is not implemented. |
| `dcw_enabled` | bool | `false` | Status: experimental dynamic-cfg wavelet path. Keep disabled unless validating that path. |

## Session Options (use with `--session-option`)

| Option | Values | Default | Meaning |
|---|---|---:|---|
| `ace_step.lora` | adapter path | none | DiT decoder adapter file or PEFT directory; merged during weight loading. |
| `ace_step.lora_scale` | finite float | `1.0` | Adapter strength multiplied by metadata scaling; zero disables it. |
| `ace_step.weight_type` | `native`, `f32`, `f16`, `bf16`, `q8_0` | `native` | Shared weight storage type. |
| `ace_step.dit_weight_type` | same as above | inherits `ace_step.weight_type` | DiT override; `f16` is promoted to `bf16`. |
| `ace_step.dit_attention` | `auto`, `eager`, `flash` | `auto` | DiT attention implementation. Auto selects flash on supported GPUs and eager on CPU. |
| `ace_step.planner_weight_type` | same as above | inherits `ace_step.weight_type` | Planner LM override. |
| `ace_step.planner_prefix_bucketing` | bool | `true` | Bucket planner CFG attention to the valid prefix. Set `false` for full-capacity attention; other optimizations remain enabled. |
| `ace_step.planner_fused_projections` | bool | `false`; `true` in `cuda_perf` | CUDA packed QKV and gate/up projections with fused SwiGLU. Can change generated codes. |
| `ace_step.planner_f16_decode` | bool | `false`; `true` in `cuda_perf` | CUDA F16 KV and prefix-bucketed decoding for metadata and CFG-off codes. CFG-on code decoding already uses F16 KV. Can change generated codes. |
| `ace_step.planner_independent_cfg_prefill` | bool | `false`; `true` in `cuda_perf` | Prefill CUDA CFG branches at their actual lengths instead of left-padding them to a shared length. Can change generated codes. |
| `ace_step.text_encoder_weight_type` | same as above | inherits `ace_step.weight_type` | Text encoder override. |
| `ace_step.vae_weight_type` | same as above | inherits `ace_step.weight_type` | VAE override. |
| `ace_step.perf_mode` | `off`, `cuda_perf` | `off` | `cuda_perf` enables the three CUDA planner options above and selects VAE tiles of 1024 latent frames with 64-frame overlap per side. Explicit options override the preset. |
| `ace_step.vae_tile_frames` | positive integer | backend policy | Maximum latent frames evaluated per VAE tile. `cuda_perf` sets 1024. |
| `ace_step.vae_tile_overlap_frames` | non-negative integer | backend policy | Context per side, in latent frames; must be less than half the tile capacity. `cuda_perf` sets 64. |
| `ace_step.mem_saver` | bool | `false` | Release staged graph/cache state after request phases to reduce resident VRAM. Later requests may rebuild released graphs. |

## DiT Performance

Static conditioning, attention masks and cross-attention K/V stay resident
during a diffusion pass and refresh when conditioning changes. They are not
rebuilt and uploaded at every solver step.

`ace_step.dit_attention=auto` uses flash attention on supported GPUs and eager
attention on CPU. Set it to `eager` to retain the previous attention lowering.
Flash preserves the model's full/sliding masks and F32 accumulation, but can
change floating-point results. This option does not convert model weights.

## Planner Performance

Planner decoding reads K/V cache views directly. CFG decoding stores K/V in
F16 and reuses cached graphs sized to the valid prefix, rounded up to 256
steps. The complete history remains visible; this is not sliding-window
attention. Model weights are not converted by these optimizations. Different
cache precision or attention shapes can change sampled output at a fixed seed.
Disable prefix bucketing with
`--session-option ace_step.planner_prefix_bucketing=false`.

The default leaves projection packing, F16 metadata/CFG-off decoding and
independent CFG prefill disabled. Enable them individually with the planner
options above, or together with `ace_step.perf_mode=cuda_perf`. These change
floating-point execution and can produce different generated codes and WAVs,
even at the same seed. Explicit `false` values override the preset. The preset
does not change planner weight precision; `ace_step.planner_weight_type`
remains a separate explicit choice.

Each planner control works independently; enabling one does not enable the
others. Independent CFG prefill is ignored when planner CFG is off. Requests
can switch CFG on/off within one session; their prefill precision and decode
batch-size graphs are cached separately.

Prefill/decode graphs are session-owned and reused when `mem_saver=false`.
This avoids rebuilding them on later requests but retains their GPU memory.

## VAE Performance

The Snake activation uses a broadcast graph that allows existing CUDA fusion,
without changing weight precision. This fix is independent of `perf_mode`.

With `perf_mode=off` and no tile overrides, tiling is unchanged: long GPU
requests use 128-frame tiles with 32-frame overlap per side (stride 64),
and long CPU requests use 256/64. Requests up to 512 latent frames on GPU
or 256 on CPU retain the existing single-graph decode path.

For larger tiles, add:

```bash
--session-option ace_step.perf_mode=cuda_perf
```

This selects 1024/64 and enables the CUDA planner optimizations described
above, without changing weight precision or DiT attention selection.
Larger tiles reduce redundant overlap computation but
use more peak VRAM and can change floating-point results. The preset does
not select a backend; use `--backend` separately.

Explicit values override the preset, for example to retain the smaller tiles:

```bash
--session-option ace_step.perf_mode=cuda_perf \
--session-option ace_step.vae_tile_frames=128 \
--session-option ace_step.vae_tile_overlap_frames=32
```

An explicit tile capacity also applies to short requests. If only the tile
capacity is set in `off` mode, overlap is derived from the existing 64-frame
policy, reduced to fit the tile. Explicit overlap is never silently reduced:
`2 * overlap < tile_frames` must hold. When reducing the preset's tile size,
override its overlap too if necessary.

## Weight Adapters

DiT LoRA/LoKr adapters can be selected with
`--session-option ace_step.lora=/path/to/adapter`. PEFT directories use
`adapter_model.safetensors` and `adapter_config.json`; SafeTensors files may use
native A/B, ComfyUI up/down, or ordinary matrix LoKr factors. Only decoder
targets are adapted; the planner, conditioning encoders, and VAE remain unchanged.
Paths relative to the model root are accepted. Adapter changes require a new
session. Merging a quantized base dequantizes and requantizes the adapted weights;
it is not equivalent to adapting the original full-precision checkpoint first.

## Load Options (use with `--load-option`)

| Option | Values | Default | Meaning |
|---|---|---:|---|
| `ace_step.dit_model_path` | `acestep-v15-turbo`, `acestep-v15-base`, `acestep-v15-xl-turbo`, `acestep-v15-xl-sft` | filename-derived; otherwise `acestep-v15-turbo` | Override DiT variant selection inside the model root. |

### Spec Scope And Compatibility

The spec lists generation, musical metadata, planner, editing, retake, and
semantic-code controls, plus weight storage, memory saving, and DiT selection.
Request names are unprefixed; session and load names use `ace_step.` in CLI/server
configuration. Prompt text and source audio remain structured inputs.

The following legacy names remain accepted:

| Scope | Canonical | Legacy |
|---|---|---|
| Request | `duration_sec` | `duration_seconds` |
| Request | `lm_guidance_scale` | `lm_cfg_scale` |
| Request | `enable_thinking` | `thinking` |
| Request | `audio_codes` | `audio_code_string` |
| Session | `ace_step.mem_saver` | `mem_saver` |

The shared parser rejects conflicting values supplied under both spellings.
`audio_codes` is the existing exception: it takes precedence over
`audio_code_string`.

Older standalone GGUFs do not need re-emission or a local spec override to use
these controls: ACE-Step parses them directly, without an embedded-spec option
whitelist. The spec expansion does not introduce stricter runtime validation.

To keep the embedded spec compact, it omits debug noise inputs, graph arenas,
individual planner CoT switches, low-level velocity/repaint tuning, experimental
`dcw_*`, and the unimplemented `flow_edit_morph` path. Existing runtime support
for omitted controls is unchanged. Repaint preset fields have no spec default
because supplying a preset changes behavior compared with leaving both unset.

ACE-Step GGUF packages are variant-specific. The model filename selects the DiT
variant by default. Use `--load-option ace_step.dit_model_path=<variant>` when
the filename does not identify the packaged variant.

### XL variants

`acestep-v15-xl-turbo` and `acestep-v15-xl-sft` are the larger DiT: 32 layers of
2560 against turbo's 24 of 2048, with 32 attention heads of 128 (so the attention
width is 4096, wider than the model). The condition encoder, audio tokenizer and
detokenizer stay at 2048 — the `encoder_hidden_size` group in the XL config — and
the DiT's condition embedder bridges the two. The XL timbre encoder also prepends
a CLS token to the reference frames and reads that position back, where earlier
variants read the first audio frame.

Both are **optional package resources**: they are only loadable when their
weights are present, and a package without them loads and runs exactly as
before. Selecting one that is not installed reports which directory is missing.
The upstream snapshots ship four safetensors shards plus a
`model.safetensors.index.json`, which the package spec points at directly.

The two differ only in `is_turbo`: XL Turbo is guidance-distilled and ignores
`guidance_scale`, XL SFT takes the CFG path the way `acestep-v15-base` does.
Their dimensions, encoder group and head configuration are identical.

`ace_step_xl_turbo_bf16` and `ace_step_xl_sft_bf16` install them as GGUFs
(14.2 GiB each), self-contained the way the Turbo and Base GGUFs are — XL DiT,
planner LM, text encoder and VAE in one file. `ace_step_xl_turbo_q8dit` and
`ace_step_xl_sft_q8dit` are the same packages with the DiT at q8_0 and the
planner LM, text encoder and VAE left at bf16 (9.97 GiB); see the measurements
at the end of this section for what that costs:

```bash
audiocpp_cli --task gen --family ace_step --model models/ACE-Step1.5-GGUF/xl-turbo --backend cuda --task-route text2music --text "warm lo-fi hip hop with a soft rhodes piano" --duration-seconds 60 --load-option ace_step.dit_model_path=acestep-v15-xl-turbo --out song.wav
```

Running one from a safetensors tree instead is worth a `dit_weight_type=bf16`,
because the XL snapshots are stored in float32 and `native` puts 19.9 GB of
weights on the card:

```bash
audiocpp_cli --task gen --family ace_step --model models/Ace-Step1.5 --backend cuda --task-route text2music --text "warm lo-fi hip hop with a soft rhodes piano" --duration-seconds 60 --load-option ace_step.dit_model_path=acestep-v15-xl-sft --session-option ace_step.dit_weight_type=bf16 --out song.wav
```

Measured on an RTX 5090, 20 s of audio, weight loading included and the weights
warm in the page cache: 87 s from safetensors at `native`, 25 s from safetensors
at `bf16`, 15 s from the bf16 GGUF, both variants alike (turbo, for reference:
11 s). Reading the weights off disk adds roughly 10 s either way.

Building an XL GGUF yourself still names every namespace the spec requires,
because `audiocpp_gguf` validates the conversion against all of them — but the
turbo and base entries only have to *exist*. `--exclude-prefix` drops their
tensors before any data is read, so a 76-byte placeholder stands in for the 9 GB
of weights that would otherwise be downloaded and thrown away:

```bash
python -c "import json,struct; h=json.dumps({'x':{'dtype':'F32','shape':[1,1],'data_offsets':[0,4]}}).encode(); h+=b' '*((8-len(h)%8)%8); open('placeholder.safetensors','wb').write(struct.pack('<Q',len(h))+h+bytes(4))"
```

```bash
audiocpp_gguf --root models/Ace-Step1.5 --family ace_step \
  --input dit_turbo_weights=placeholder.safetensors \
  --input dit_turbo_silence_latent=placeholder.safetensors \
  --input dit_base_weights=placeholder.safetensors \
  --input dit_base_silence_latent=placeholder.safetensors \
  --input dit_xl_turbo_weights=models/Ace-Step1.5/acestep-v15-xl-turbo/model.safetensors.index.json \
  --input dit_xl_turbo_silence_latent=models/Ace-Step1.5/acestep-v15-xl-turbo/silence_latent.safetensors \
  --input lm_weights=models/Ace-Step1.5/acestep-5Hz-lm-1.7B/model.safetensors \
  --input text_encoder_weights=models/Ace-Step1.5/Qwen3-Embedding-0.6B/model.safetensors \
  --input vae_weights=models/Ace-Step1.5/vae/diffusion_pytorch_model.safetensors \
  --exclude-prefix dit_turbo_ --exclude-prefix dit_base_ \
  --type bf16 --output ace-step-1.5-xl-turbo-bf16.gguf
```

Swap `dit_xl_turbo_*` for `dit_xl_sft_*` to build the SFT one. Upstream ships
`silence_latent.pt` where the spec wants safetensors;
`tests/ace_step/convert_silence_latent.py --input <variant>/silence_latent.pt`
converts it. Of the turbo and base snapshots only `config.json` is genuinely
needed — those are required sidecars, a few KB each.

`ace_step` is graded `No (planner sampling can fail)` for q8_0 in
[gguf.md](../gguf.md), and that grade is about the planner LM, not the DiT: at a
fixed seed a fully quantised build samples a different token path and returns an
unrelated song. Quantising the DiT alone keeps the planner exact, which
`--keep-type` expresses:

```bash
  --keep-type "lm_weights*=bf16" \
  --keep-type "text_encoder_weights*=bf16" \
  --keep-type "vae_weights*=bf16" \
  --keep-type "dit_xl_turbo_silence_latent*=bf16" \
  --type q8_0 --output ace-step-1.5-xl-turbo-q8dit.gguf
```

Measured on an RTX 5090 against the bf16 build at the same seed and prompt,
20 s of audio: 9.97 GiB against 14.2 GiB and 9.3 s against 14.2 s, with a 0.989
waveform correlation against the bf16 output — 0.997 on a sung 40 s take, 0.999
for XL SFT. A fully quantised build of the same weights correlates 0.09, and its
ASR transcript is a different lyric line.

Re-quantising a finished GGUF works too, since a GGUF input carries its own
namespaces, package spec and sidecars:

```bash
audiocpp_gguf --input ace-step-1.5-xl-turbo-bf16.gguf --type q8_0 \
  --keep-type "lm_weights*=bf16" --output ace-step-1.5-xl-turbo-q8dit.gguf
```