voice_audio_multi/docs/models/ace_step.md

ACE-Step

ACE-Step is wired as --family ace_step --task gen. It generates and edits music from text, lyrics, and optional source audio. The route controls whether source audio is ignored, optional, or required. Supplied audio_codes can also satisfy the source-input requirement for routes that require audio.

Common CLI shape:

audiocpp_cli --task gen --family ace_step --model models/Ace-Step1.5 --backend cuda --task-route <route> ...

Model

Field	Value
Family	ace_step
Model directory	models/Ace-Step1.5
Task	gen
Default DiT	Inferred from model filename (base, sft, turbo, and xl); otherwise acestep-v15-turbo
Optional DiT	acestep-v15-base, acestep-v15-xl-turbo, acestep-v15-xl-sft
Default LM	acestep-5Hz-lm-1.7B
Prompt input	--text
Lyrics input	--lyrics
Source audio	Route-dependent through --audio

Text To Music

Generate a full song or music clip from prompt text and optional lyrics. Source audio is ignored by this route.

Field	Value
Route	text2music
Source audio	Ignored
Planner	Used unless audio_codes are supplied
Duration	--duration-seconds; -1 lets the planner/model choose
Metadata controls	Optional bpm, keyscale, timesignature, and language

audiocpp_cli --task gen --family ace_step --model models/Ace-Step1.5 --backend cuda --task-route text2music --text "cinematic synth pop with clear vocals" --lyrics "We rise with the morning light" --duration-seconds 60 --out song.wav

Use the base DiT instead of turbo:

audiocpp_cli --task gen --family ace_step --model models/Ace-Step1.5 --backend cuda --task-route text2music --text "acoustic folk ballad" --lyrics "The river remembers our names" --duration-seconds 60 --load-option ace_step.dit_model_path=acestep-v15-base --out song.wav

Complete

Complete or continue a source audio track. Source audio is optional: if provided, it conditions the continuation; otherwise the route behaves like a completion prompt.

Field	Value
Route	complete
Source audio	Optional
Planner	Used
Duration	Source duration is not locked unless requested metadata or planner output controls it
Track classes	Optional complete_track_classes list changes the completion instruction

audiocpp_cli --task gen --family ace_step --model models/Ace-Step1.5 --backend cuda --task-route complete --audio input.wav --text "finish this as a cinematic rock track" --out complete.wav

Lego

Compose or transform a track from source audio plus a prompt. Source audio is required and preserved as the repaint context.

Field	Value
Route	lego
Source audio	Required
Planner	Used
Duration	Locked to source audio
Repaint window	Used internally over the source context
Track name	Optional track_name changes the generated instruction

audiocpp_cli --task gen --family ace_step --model models/Ace-Step1.5 --backend cuda --task-route lego --audio input.wav --text "generate a brighter guitar layer" --request-option track_name=guitar --out lego.wav

Extract

Extract a target track from source audio. Source audio is required, and the route uses extraction-specific defaults for guidance and shift.

Field	Value
Route	extract
Source audio	Required
Planner	Not used
Duration	Locked to source audio
Track name	Optional track_name; when omitted, the default extract instruction is used
Route defaults	guidance_scale=7.0, shift=3.0, planner chain-of-thought metadata disabled

audiocpp_cli --task gen --family ace_step --model models/Ace-Step1.5 --backend cuda --task-route extract --audio song.wav --text "extract vocals" --request-option track_name=vocals --out vocals.wav

Cover

Generate a cover from source audio using cover-tokenizer conditioning. Source audio is required and duration is locked to the source.

Field	Value
Route	cover
Source audio	Required
Planner	Not used
Cover conditioning	Uses the FSQ cover tokenizer
Duration	Locked to source audio

audiocpp_cli --task gen --family ace_step --model models/Ace-Step1.5 --backend cuda --task-route cover --audio source.wav --text "turn this into energetic pop vocals" --lyrics "We keep moving through the night" --out cover.wav

Cover Without FSQ

Generate a cover from source audio without the FSQ cover-tokenizer conditioning path.

Field	Value
Route	cover-nofsq
Source audio	Required
Planner	Not used
Cover conditioning	Does not use the FSQ cover tokenizer
Duration	Locked to source audio

audiocpp_cli --task gen --family ace_step --model models/Ace-Step1.5 --backend cuda --task-route cover-nofsq --audio source.wav --text "make a softer acoustic cover" --lyrics "We keep moving through the night" --out cover_nofsq.wav

Repaint

Replace a time span inside source audio. Source audio and a repaint window are required.

Field	Value
Route	repaint
Source audio	Required
Planner	Not used
Duration	Locked to source audio
Required window	--repaint-start, --repaint-end
Repaint policy	repaint_mode, repaint_strength, or direct repaint injection/crossfade options

audiocpp_cli --task gen --family ace_step --model models/Ace-Step1.5 --backend cuda --task-route repaint --audio song.wav --text "replace the middle with a brighter chorus" --repaint-start 20 --repaint-end 35 --out repaint.wav

Common Options (use directly)

Option	Values	Default	Meaning
--task-route	text2music, complete, lego, extract, cover, cover-nofsq, repaint	text2music	ACE-Step operation.
--text	text	required	Music prompt or edit instruction.
--lyrics	text	empty string	Vocal lyrics.
--audio	WAV path	route-dependent	Source audio for complete/edit/extract/cover routes.
--duration-seconds	float, -1 for auto	-1	Target duration. Source-locked routes use source duration.
--language	language code	text input language, then en	Vocal language for lyrics; request-map language takes precedence.
--track-name	text	empty string	Track name used by lego and extract instructions.
--repaint-start	seconds	required for repaint	Start time for repaint.
--repaint-end	seconds	required for repaint	End time for repaint.
--repaint-mode	balanced, conservative, aggressive	unset	Preset repaint blending policy; supplying strength alone selects balanced.
--repaint-strength	0..1	0.5 when a preset is supplied	Strength for balanced mode; values are clamped to this range.
--num-inference-steps	integer	8	Denoising steps; turbo checkpoints cap this at 8.
--guidance-scale	float	1.0; 7.0 for extract unless overridden	Diffusion CFG; ignored by turbo checkpoints.
--seed	integer	random if omitted	Generation seed.

Request Options (use with --request-option)

Option	Values	Default	Meaning
complete_track_classes	track class or JSON array	empty list	Track classes for complete, e.g. '["vocals","guitar"]'. A plain string is one class, not a comma-separated list.
bpm	integer	not set	Force BPM metadata; otherwise the planner chooses it when used.
keyscale	text	not set	Force key metadata; otherwise the planner chooses it when used.
timesignature	text	not set	Force time signature metadata; otherwise the planner chooses it when used.
negative_prompt	text	NO USER INPUT	Negative prompt.
audio_codes	ACE semantic code text	not set	Skip planner token generation and use supplied audio codes.
audio_cover_strength	float	1.0	Cover strength for cover/edit-style conditioning.
cover_noise_strength	float	0.0	Noise strength for cover conditioning.
lm_temperature	float	0.85	Planner sampling temperature.
lm_guidance_scale	float	2.0	Planner CFG scale. Legacy lm_cfg_scale remains accepted.
duration_sec	float	-1.0	Target duration; non-positive leaves selection to the planner. Legacy duration_seconds remains accepted.
enable_thinking	bool	true	Enable planner thinking when supported by the checkpoint. Legacy thinking remains accepted.
lm_top_k	integer	0	Planner top-k; 0 disables top-k.
lm_top_p	float	0.9	Planner top-p.
lm_repetition_penalty	float	1.0	Planner repetition penalty.
sampler_mode	euler, heun	euler	Diffusion sampler mode.
shift	float	1.0; 3.0 for extract unless overridden	Diffusion timestep schedule shift.
retake_seed	integer, -1 to clear	not set	Optional retake noise seed.
retake_variance	float	0.0	Retake noise mixing strength.
flow_edit_morph	bool	false	Status: parsed for text2music, but not usable because the flow-edit diffusion overlay is not implemented.
dcw_enabled	bool	false	Status: experimental dynamic-cfg wavelet path. Keep disabled unless validating that path.

Session Options (use with --session-option)

Option	Values	Default	Meaning
ace_step.lora	adapter path	none	DiT decoder adapter file or PEFT directory; merged during weight loading.
ace_step.lora_scale	finite float	1.0	Adapter strength multiplied by metadata scaling; zero disables it.
ace_step.weight_type	native, f32, f16, bf16, q8_0	native	Shared weight storage type.
ace_step.dit_weight_type	same as above	inherits ace_step.weight_type	DiT override; f16 is promoted to bf16.
ace_step.dit_attention	auto, eager, flash	auto	DiT attention implementation. Auto selects flash on supported GPUs and eager on CPU.
ace_step.planner_weight_type	same as above	inherits ace_step.weight_type	Planner LM override.
ace_step.planner_prefix_bucketing	bool	true	Bucket planner CFG attention to the valid prefix. Set false for full-capacity attention; other optimizations remain enabled.
ace_step.planner_fused_projections	bool	false; true in cuda_perf	CUDA packed QKV and gate/up projections with fused SwiGLU. Can change generated codes.
ace_step.planner_f16_decode	bool	false; true in cuda_perf	CUDA F16 KV and prefix-bucketed decoding for metadata and CFG-off codes. CFG-on code decoding already uses F16 KV. Can change generated codes.
ace_step.planner_independent_cfg_prefill	bool	false; true in cuda_perf	Prefill CUDA CFG branches at their actual lengths instead of left-padding them to a shared length. Can change generated codes.
ace_step.text_encoder_weight_type	same as above	inherits ace_step.weight_type	Text encoder override.
ace_step.vae_weight_type	same as above	inherits ace_step.weight_type	VAE override.
ace_step.perf_mode	off, cuda_perf	off	cuda_perf enables the three CUDA planner options above and selects VAE tiles of 1024 latent frames with 64-frame overlap per side. Explicit options override the preset.
ace_step.vae_tile_frames	positive integer	backend policy	Maximum latent frames evaluated per VAE tile. cuda_perf sets 1024.
ace_step.vae_tile_overlap_frames	non-negative integer	backend policy	Context per side, in latent frames; must be less than half the tile capacity. cuda_perf sets 64.
ace_step.mem_saver	bool	false	Release staged graph/cache state after request phases to reduce resident VRAM. Later requests may rebuild released graphs.

DiT Performance

Static conditioning, attention masks and cross-attention K/V stay resident during a diffusion pass and refresh when conditioning changes. They are not rebuilt and uploaded at every solver step.

ace_step.dit_attention=auto uses flash attention on supported GPUs and eager attention on CPU. Set it to eager to retain the previous attention lowering. Flash preserves the model's full/sliding masks and F32 accumulation, but can change floating-point results. This option does not convert model weights.

Planner Performance

Planner decoding reads K/V cache views directly. CFG decoding stores K/V in F16 and reuses cached graphs sized to the valid prefix, rounded up to 256 steps. The complete history remains visible; this is not sliding-window attention. Model weights are not converted by these optimizations. Different cache precision or attention shapes can change sampled output at a fixed seed. Disable prefix bucketing with --session-option ace_step.planner_prefix_bucketing=false.

The default leaves projection packing, F16 metadata/CFG-off decoding and independent CFG prefill disabled. Enable them individually with the planner options above, or together with ace_step.perf_mode=cuda_perf. These change floating-point execution and can produce different generated codes and WAVs, even at the same seed. Explicit false values override the preset. The preset does not change planner weight precision; ace_step.planner_weight_type remains a separate explicit choice.

Each planner control works independently; enabling one does not enable the others. Independent CFG prefill is ignored when planner CFG is off. Requests can switch CFG on/off within one session; their prefill precision and decode batch-size graphs are cached separately.

Prefill/decode graphs are session-owned and reused when mem_saver=false. This avoids rebuilding them on later requests but retains their GPU memory.

VAE Performance

The Snake activation uses a broadcast graph that allows existing CUDA fusion, without changing weight precision. This fix is independent of perf_mode.

With perf_mode=off and no tile overrides, tiling is unchanged: long GPU requests use 128-frame tiles with 32-frame overlap per side (stride 64), and long CPU requests use 256/64. Requests up to 512 latent frames on GPU or 256 on CPU retain the existing single-graph decode path.

For larger tiles, add:

--session-option ace_step.perf_mode=cuda_perf

This selects 1024/64 and enables the CUDA planner optimizations described above, without changing weight precision or DiT attention selection. Larger tiles reduce redundant overlap computation but use more peak VRAM and can change floating-point results. The preset does not select a backend; use --backend separately.

Explicit values override the preset, for example to retain the smaller tiles:

--session-option ace_step.perf_mode=cuda_perf \
--session-option ace_step.vae_tile_frames=128 \
--session-option ace_step.vae_tile_overlap_frames=32

An explicit tile capacity also applies to short requests. If only the tile capacity is set in off mode, overlap is derived from the existing 64-frame policy, reduced to fit the tile. Explicit overlap is never silently reduced: 2 * overlap < tile_frames must hold. When reducing the preset's tile size, override its overlap too if necessary.

Weight Adapters

DiT LoRA/LoKr adapters can be selected with --session-option ace_step.lora=/path/to/adapter. PEFT directories use adapter_model.safetensors and adapter_config.json; SafeTensors files may use native A/B, ComfyUI up/down, or ordinary matrix LoKr factors. Only decoder targets are adapted; the planner, conditioning encoders, and VAE remain unchanged. Paths relative to the model root are accepted. Adapter changes require a new session. Merging a quantized base dequantizes and requantizes the adapted weights; it is not equivalent to adapting the original full-precision checkpoint first.

Load Options (use with --load-option)

Option	Values	Default	Meaning
ace_step.dit_model_path	acestep-v15-turbo, acestep-v15-base, acestep-v15-xl-turbo, acestep-v15-xl-sft	filename-derived; otherwise acestep-v15-turbo	Override DiT variant selection inside the model root.

Spec Scope And Compatibility

The spec lists generation, musical metadata, planner, editing, retake, and semantic-code controls, plus weight storage, memory saving, and DiT selection. Request names are unprefixed; session and load names use ace_step. in CLI/server configuration. Prompt text and source audio remain structured inputs.

The following legacy names remain accepted:

Scope	Canonical	Legacy
Request	duration_sec	duration_seconds
Request	lm_guidance_scale	lm_cfg_scale
Request	enable_thinking	thinking
Request	audio_codes	audio_code_string
Session	ace_step.mem_saver	mem_saver

The shared parser rejects conflicting values supplied under both spellings. audio_codes is the existing exception: it takes precedence over audio_code_string.

Older standalone GGUFs do not need re-emission or a local spec override to use these controls: ACE-Step parses them directly, without an embedded-spec option whitelist. The spec expansion does not introduce stricter runtime validation.

To keep the embedded spec compact, it omits debug noise inputs, graph arenas, individual planner CoT switches, low-level velocity/repaint tuning, experimental dcw_*, and the unimplemented flow_edit_morph path. Existing runtime support for omitted controls is unchanged. Repaint preset fields have no spec default because supplying a preset changes behavior compared with leaving both unset.

ACE-Step GGUF packages are variant-specific. The model filename selects the DiT variant by default. Use --load-option ace_step.dit_model_path=<variant> when the filename does not identify the packaged variant.

XL variants

acestep-v15-xl-turbo and acestep-v15-xl-sft are the larger DiT: 32 layers of 2560 against turbo's 24 of 2048, with 32 attention heads of 128 (so the attention width is 4096, wider than the model). The condition encoder, audio tokenizer and detokenizer stay at 2048 — the encoder_hidden_size group in the XL config — and the DiT's condition embedder bridges the two. The XL timbre encoder also prepends a CLS token to the reference frames and reads that position back, where earlier variants read the first audio frame.

Both are optional package resources: they are only loadable when their weights are present, and a package without them loads and runs exactly as before. Selecting one that is not installed reports which directory is missing. The upstream snapshots ship four safetensors shards plus a model.safetensors.index.json, which the package spec points at directly.

The two differ only in is_turbo: XL Turbo is guidance-distilled and ignores guidance_scale, XL SFT takes the CFG path the way acestep-v15-base does. Their dimensions, encoder group and head configuration are identical.

ace_step_xl_turbo_bf16 and ace_step_xl_sft_bf16 install them as GGUFs (14.2 GiB each), self-contained the way the Turbo and Base GGUFs are — XL DiT, planner LM, text encoder and VAE in one file. ace_step_xl_turbo_q8dit and ace_step_xl_sft_q8dit are the same packages with the DiT at q8_0 and the planner LM, text encoder and VAE left at bf16 (9.97 GiB); see the measurements at the end of this section for what that costs:

audiocpp_cli --task gen --family ace_step --model models/ACE-Step1.5-GGUF/xl-turbo --backend cuda --task-route text2music --text "warm lo-fi hip hop with a soft rhodes piano" --duration-seconds 60 --load-option ace_step.dit_model_path=acestep-v15-xl-turbo --out song.wav

Running one from a safetensors tree instead is worth a dit_weight_type=bf16, because the XL snapshots are stored in float32 and native puts 19.9 GB of weights on the card:

audiocpp_cli --task gen --family ace_step --model models/Ace-Step1.5 --backend cuda --task-route text2music --text "warm lo-fi hip hop with a soft rhodes piano" --duration-seconds 60 --load-option ace_step.dit_model_path=acestep-v15-xl-sft --session-option ace_step.dit_weight_type=bf16 --out song.wav

Measured on an RTX 5090, 20 s of audio, weight loading included and the weights warm in the page cache: 87 s from safetensors at native, 25 s from safetensors at bf16, 15 s from the bf16 GGUF, both variants alike (turbo, for reference: 11 s). Reading the weights off disk adds roughly 10 s either way.

Building an XL GGUF yourself still names every namespace the spec requires, because audiocpp_gguf validates the conversion against all of them — but the turbo and base entries only have to exist. --exclude-prefix drops their tensors before any data is read, so a 76-byte placeholder stands in for the 9 GB of weights that would otherwise be downloaded and thrown away:

python -c "import json,struct; h=json.dumps({'x':{'dtype':'F32','shape':[1,1],'data_offsets':[0,4]}}).encode(); h+=b' '*((8-len(h)%8)%8); open('placeholder.safetensors','wb').write(struct.pack('<Q',len(h))+h+bytes(4))"

audiocpp_gguf --root models/Ace-Step1.5 --family ace_step \
  --input dit_turbo_weights=placeholder.safetensors \
  --input dit_turbo_silence_latent=placeholder.safetensors \
  --input dit_base_weights=placeholder.safetensors \
  --input dit_base_silence_latent=placeholder.safetensors \
  --input dit_xl_turbo_weights=models/Ace-Step1.5/acestep-v15-xl-turbo/model.safetensors.index.json \
  --input dit_xl_turbo_silence_latent=models/Ace-Step1.5/acestep-v15-xl-turbo/silence_latent.safetensors \
  --input lm_weights=models/Ace-Step1.5/acestep-5Hz-lm-1.7B/model.safetensors \
  --input text_encoder_weights=models/Ace-Step1.5/Qwen3-Embedding-0.6B/model.safetensors \
  --input vae_weights=models/Ace-Step1.5/vae/diffusion_pytorch_model.safetensors \
  --exclude-prefix dit_turbo_ --exclude-prefix dit_base_ \
  --type bf16 --output ace-step-1.5-xl-turbo-bf16.gguf

Swap dit_xl_turbo_* for dit_xl_sft_* to build the SFT one. Upstream ships silence_latent.pt where the spec wants safetensors; tests/ace_step/convert_silence_latent.py --input <variant>/silence_latent.pt converts it. Of the turbo and base snapshots only config.json is genuinely needed — those are required sidecars, a few KB each.

ace_step is graded No (planner sampling can fail) for q8_0 in gguf.md, and that grade is about the planner LM, not the DiT: at a fixed seed a fully quantised build samples a different token path and returns an unrelated song. Quantising the DiT alone keeps the planner exact, which --keep-type expresses:

  --keep-type "lm_weights*=bf16" \
  --keep-type "text_encoder_weights*=bf16" \
  --keep-type "vae_weights*=bf16" \
  --keep-type "dit_xl_turbo_silence_latent*=bf16" \
  --type q8_0 --output ace-step-1.5-xl-turbo-q8dit.gguf

Measured on an RTX 5090 against the bf16 build at the same seed and prompt, 20 s of audio: 9.97 GiB against 14.2 GiB and 9.3 s against 14.2 s, with a 0.989 waveform correlation against the bf16 output — 0.997 on a sung 40 s take, 0.999 for XL SFT. A fully quantised build of the same weights correlates 0.09, and its ASR transcript is a different lyric line.

Re-quantising a finished GGUF works too, since a GGUF input carries its own namespaces, package spec and sidecars:

audiocpp_gguf --input ace-step-1.5-xl-turbo-bf16.gguf --type q8_0 \
  --keep-type "lm_weights*=bf16" --output ace-step-1.5-xl-turbo-q8dit.gguf