voice_audio_multi/docs/models/yue2.md

YuE2

YuE2 is wired as --family yue2 --task gen. It generates music from lyrics and a style prompt, with optional symbolic ABC conditioning.

Quick Start

Default packaged GGUF layout:

./build/debug/bin/audiocpp_cli \
  --task gen \
  --family yue2 \
  --model models/Yue2-3B-GGUF \
  --backend cuda \
  --threads 8 \
  --lyrics "[Verse]
Soft morning light is touching the window.
I hear the city waking below.
[Chorus]
Stay with the rhythm, let it carry us home.
Sing with the sunrise, we are never alone." \
  --request-option style="English, indie pop, bright acoustic guitar, soft drums, warm lead vocal, polished demo mix" \
  --request-option cot=off \
  --seed 1234 \
  --out yue2.wav \
  --log

The default session loads yue2-3b-q8_0.gguf for the main AR/NAR model and yue2-vae-f16.gguf for the VAE from the model root.

Model

Field	Value
Family	yue2
Model directory	models/Yue2-3B-GGUF
Task	gen
Main GGUF default	yue2-3b-q8_0.gguf
VAE GGUF default	yue2-vae-f16.gguf
Required sidecars	sidecars/yue2-model-config.json, sidecars/yue2-generation-config.json, sidecars/yue2-qwen.tiktoken, sidecars/yue2-vae-config.json
Lyrics input	--lyrics; --text is accepted as a fallback
Style input	--request-option style=<prompt>

Component Selection

Select the BF16 main model:

./build/debug/bin/audiocpp_cli \
  --task gen \
  --family yue2 \
  --model models/Yue2-3B-GGUF \
  --backend cuda \
  --threads 8 \
  --session-option yue2.model_gguf=yue2-3b-bf16.gguf \
  --session-option yue2.vae_gguf=yue2-vae-f16.gguf \
  --lyrics "[Verse]
Soft morning light is touching the window.
[Chorus]
Stay with the rhythm, let it carry us home." \
  --request-option style="English, pop rock, bright guitars, clean drums, warm vocal" \
  --request-option cot=off \
  --seed 1234 \
  --out yue2-bf16.wav \
  --log

Select the F32 VAE:

--session-option yue2.vae_gguf=yue2-vae-f32.gguf

The component paths are relative to --model; absolute paths are rejected.

Performance Controls

Defaults preserve the existing generation path. To enable the performance preset:

--session-option yue2.perf_mode=cuda_perf

Session option	Default (off)	cuda_perf
yue2.packed_projections	false	true
yue2.kv_cache_alignment	1	256
yue2.mem_saver	true	false
yue2.vae_weight_type	Shared weight_type, otherwise native	Unchanged
yue2.vae_tile_frames	Model configuration (normally 1024)	Model configuration capped at 512
yue2.attention	auto	auto (unchanged)

Packing combines AR QKV and gate/up projections without changing their weight dtype. KV alignment pads AR cache capacity without changing the refill boundary. With mem_saver=false, AR/NAR runtimes and cached graphs remain session-owned between requests instead of being released after their stages. This can substantially increase peak VRAM. Packing, aligned attention and different VAE tile boundaries can change numerical results. The preset does not change weight dtype.

Each control can be set independently of the preset. Explicit session options override preset values, including false and 1. For example, retain the other preset settings but release stages between requests:

--session-option yue2.perf_mode=cuda_perf \
--session-option yue2.mem_saver=true

vae_tile_frames is the latent core length; the model's halo is unchanged. attention controls NAR attention only; AR already uses flash attention. The cuda_perf preset is tuned for CUDA but is not backend-gated.

AR and NAR LoRA

Use yue2.ar_lora for score/semantic planning and yue2.nar_lora for acoustic rendering. Either adapter can be used alone or both together. Native A/B, PEFT A/B, and ComfyUI up/down SafeTensors layouts are supported, including the instrumental adapter's fused QKV and gate/up projections. Changing adapters requires a new session; merged weights are cached within that session, not merged per request.

For example, add the acoustic adapter alongside the AR adapter:

--session-option yue2.nar_lora=/path/to/nar_lora_joint_v4.safetensors \
--session-option yue2.nar_lora_scale=1.0

The NAR adapter includes full vae2llm and llm2vae projection replacements in addition to LoRA deltas. Its scale applies only to deltas; replacements are loaded at full strength. A scale of 0 disables the entire adapter, including replacements. No separate audio-tokenizer head is needed for text-to-music generation.

Load a YuE2 AR adapter with session options:

--session-option yue2.ar_lora=/path/to/ar_lora_inst_v3abc.safetensors \
--session-option yue2.ar_lora_scale=1.0 \
--request-option cot=full

Both FP32 and BF16 adapter files are supported. Use the standard A/B files from the instrumental adapter repository, including its ComfyUI fused file. Use yue2.nar_lora for NAR adapters. Relative adapter paths resolve under the model root. Reload the session after changing the adapter or scale; 0 disables it. With no adapter, weight loading and generation are unchanged.

The adapter modifies AR attention and MLP projections at load time, using BF16(W + BF16(scale * (B @ A))). Native YuE2 adapters retain their direct scale; other layouts apply their alpha/rank metadata when present. There is no per-token adapter computation. AR adapters leave NAR and VAE weights unchanged. Merged AR tensors are cached in CPU memory in their upload dtype for the session, so later requests skip merging. Only one dtype per adapted tensor is retained; unloading the session frees the cache. By default (mem_saver=true), AR weights are still released from VRAM before VAE decoding. Use the BF16 main GGUF for the closest match to the original weights. Loading onto Q8/Q4 bases merges into dequantized weights and requantizes the result; it is not equivalent to merging into the original BF16 model first.

For this instrumental adapter, use cot=full, instrument/style tags in style, and [instrumental] or section tags in lyrics. Its license is CC BY-NC 4.0.

ABC Conditioning

Use cot=melody or cot=full to run the symbolic route. External ABC requires one of those modes:

./build/debug/bin/audiocpp_cli \
  --task gen \
  --family yue2 \
  --model models/Yue2-3B-GGUF \
  --backend cuda \
  --threads 8 \
  --lyrics "[Verse]
Write the melody over this score." \
  --request-option style="English, folk pop, acoustic guitar, steady drums" \
  --request-option cot=melody \
  --request-option abc_file=/path/to/score.abc \
  --seed 1234 \
  --out yue2-abc.wav \
  --log

Inline ABC can be passed with --request-option abc=<abc text>.

Generated ABC Export

When the model generates its own plan (cot=melody or cot=full with no external abc / abc_file), a nonempty decoded ABC score is attached to the result as a score artifact (text/vnd.abc). The WebUI shows a Save ABC download in the result panel, and the CLI writes score.abc when --out-dir is set:

./build/debug/bin/audiocpp_cli \
  --task gen \
  --family yue2 \
  --model models/Yue2-3B-GGUF \
  --backend cuda \
  --threads 8 \
  --lyrics "..." \
  --request-option style="English, folk pop" \
  --request-option cot=full \
  --seed 1234 \
  --out yue2.wav \
  --out-dir yue2_out \
  --log
# -> yue2_out/score.abc

Semantic Token Export

Set export_semantic=true to attach the semantic stage output as a semantic artifact (application/vnd.yue2.semantic+json). The payload is a flat JSON array of codec indices, one integer per semantic frame (25 frames per second) in [0, 32768), without the stop token. The CLI writes semantic.json when --out-dir is set:

./build/debug/bin/audiocpp_cli \
  --task gen \
  --family yue2 \
  --model models/Yue2-3B-GGUF \
  --backend cuda \
  --threads 8 \
  --lyrics "..." \
  --request-option style="English, folk pop" \
  --request-option export_semantic=true \
  --seed 1234 \
  --out yue2.wav \
  --out-dir yue2_out \
  --log
# -> yue2_out/semantic.json

The artifact meta carries frames (the number of indices) and truncated (true when the stage stopped on semantic_max_tokens instead of the stop token).

The indices are the same values the reference YuE2 Python pipeline stores in its semantic.npy (1-D int32). To convert:

python3 -c "import json, numpy; numpy.save('semantic.npy', numpy.array(json.load(open('semantic.json')), dtype=numpy.int32))"

Stopping Early

stop_after ends the run before the remaining stages:

Value	Runs	Produces
abc	ABC planner	score artifact
semantic	ABC planner, semantic AR	semantic artifact; score only for generated nonempty ABC
audio	everything	audio; score only for generated nonempty ABC; semantic with export_semantic=true

stop_after=abc requires cot=melody or cot=full and no external abc / abc_file, because there would be nothing to generate. stop_after=semantic implies export_semantic=true, since the token stream is all that stage produces. No score is emitted with cot=off or externally supplied abc / abc_file.

A result from abc or semantic carries no audio. Use --out-dir on the CLI to collect the artifacts (--out writes nothing), and /v1/tasks/run on the server — /v1/audio/speech requires an audio output.

./build/debug/bin/audiocpp_cli \
  --task gen \
  --family yue2 \
  --model models/Yue2-3B-GGUF \
  --backend cuda \
  --threads 8 \
  --lyrics "..." \
  --request-option style="English, folk pop" \
  --request-option stop_after=semantic \
  --seed 1234 \
  --out-dir yue2_out \
  --log
# -> yue2_out/score.abc, yue2_out/semantic.json

Continuing From Semantic Tokens

semantic_prefix takes a JSON array of semantic codec indices, one per frame at 25 frames per second, each in [0,32768). The frames become forced history: the AR stage prefills them behind the prompt and samples the rest of the song from frame N. semantic_prefix_file reads the same text from a file and is ignored when semantic_prefix is set.

The returned stream, and therefore the NAR stage and the rendered audio, includes the forced frames. semantic_min_tokens, semantic_max_tokens and the repetition penalty window all count the total stream, so semantic_max_tokens must be at least N.

./build/debug/bin/audiocpp_cli \
  --task gen \
  --family yue2 \
  --model models/Yue2-3B-GGUF \
  --backend cuda \
  --threads 8 \
  --lyrics "..." \
  --request-option style="English, folk pop" \
  --request-option cot=off \
  --request-option semantic_prefix_file=/path/to/semantic.json \
  --request-option semantic_max_tokens=1200 \
  --seed 1234 \
  --out yue2-continue.wav \
  --log

With cot=melody or cot=full a prefix also requires abc or abc_file: without a score the run would plan a new one that the forced frames do not belong to.

To render exactly the given frames and sample nothing, set both token bounds to N:

  --request-option semantic_min_tokens=640 \
  --request-option semantic_max_tokens=640

That run skips the AR stage: the given frames are the stream, and with stop_after=semantic they are returned as they came. stop_after=audio still prefills them once for the NAR conditioning.

Results are deterministic for a given request, but a prefix does not reproduce the draws of an uninterrupted run: the sampler RNG starts at the first sampled frame, and prefilled K/V differ numerically from step-decoded K/V.

With a guidance scale other than 1.0 the forced frames are appended to both the positive and the negative prefix, and that path prefills through host K/V, so memory grows with N. The default cot=full route uses scale 1.0, which runs a single stream on the device.

A prefix does not have to come from the run it continues. Splicing the tokens of two renders of one score changes a song's style part-way through; examples/yue2_style_change is a complete script for it.

Common Options (use directly)

Option	Values	Default	Meaning
--lyrics	text	empty	Song lyrics; optional for instrumental generation.
--text	text	empty	Fallback lyrics source when --lyrics is not supplied.
--seed	integer in [0, 2^63)	1234	Generation seed. Must be nonnegative and less than 2^63.

Request Options (use with --request-option)

Option	Values	Default	Meaning
style	text	required	Music style prompt.
cot	off, melody, full	full	Symbolic planning route.
stop_after	abc, semantic, audio	audio	Last stage to run. See "Stopping Early".
abc	ABC text	empty	Inline ABC score; requires cot=melody or cot=full.
abc_file	path	empty	ABC score file; requires cot=melody or cot=full.
nar_noise_file	raw float32 file	empty	Provide a noise file for NAR generation, shaped [frames,64].
export_semantic	true, false	false	Attach the semantic token stream as a semantic artifact.
semantic_prefix	JSON array of codec indices	empty	Inline semantic frames to force at the start of the music stream.
semantic_prefix_file	path	empty	File holding the same JSON array; ignored when semantic_prefix is set.
guidance_scale	0..20	see description	Semantic classifier-free guidance scale. Default follows the upstream route defaults. Omitted defaults to 1.01 with cot=off, otherwise 1.0.
num_inference_steps	integer > 0	8	NAR midpoint ODE steps.
seed	integer in [0, 2^63)	1234	Generation seed. Must be nonnegative and less than 2^63.

Sampling Options (use with --request-option)

Defaults are checkpoint-configurable. The table shows fallback values when the checkpoint does not override them.

Option	Values	Default	Meaning
abc_temperature	0..5	0.7	ABC planner sampling temperature.
abc_top_p	0..1	0.9	ABC planner nucleus sampling probability. Must be in (0, 1].
abc_top_k	integer >= 1	30	ABC planner top-k limit.
abc_repetition_penalty	float > 0	1.005	ABC planner repetition penalty. Must be positive.
abc_penalty_window	integer >= 1	100	ABC planner repetition penalty window.
abc_min_tokens	integer >= 0	32	Minimum ABC planner tokens before EOS is accepted.
abc_max_tokens	integer >= abc_min_tokens	4096	Maximum ABC planner tokens. Zero is allowed when min_tokens is also zero.
semantic_temperature	0..5	1.0	Semantic codec sampling temperature.
semantic_top_p	0..1	0.95	Semantic codec nucleus sampling probability. Must be in (0, 1].
semantic_top_k	integer >= 1	100	Semantic codec top-k limit.
semantic_repetition_penalty	float > 0	1.2	Semantic codec repetition penalty. Must be positive.
semantic_penalty_window	integer >= 1	50	Semantic codec repetition penalty window.
semantic_min_tokens	integer >= 0	200	Minimum semantic tokens before EOS is accepted.
semantic_max_tokens	integer >= semantic_min_tokens	9000	Maximum semantic codec tokens. Zero is allowed when min_tokens is also zero.

Session Options (use with --session-option)

Option	Values	Default	Meaning
yue2.model_gguf	relative GGUF path	yue2-3b-q8_0.gguf	Main AR/NAR component.
yue2.vae_gguf	relative GGUF path	yue2-vae-f16.gguf	VAE component.
yue2.ar_lora	safetensors path	none	AR adapter; native, PEFT, and ComfyUI layouts. Relative paths use the model root.
yue2.ar_lora_scale	finite float	1.0	Adapter delta scale; 0 disables it.
yue2.nar_lora	safetensors path	none	NAR adapter; relative paths use the model root.
yue2.nar_lora_scale	finite float	1.0	NAR delta scale; replacements stay at full strength. 0 disables the entire adapter.
yue2.weight_type	native, f32, f16, bf16, q8_0, q4_0, q4_k	native	Shared weight storage fallback for the main model and VAE.
yue2.model_weight_type	native, f32, f16, bf16, q8_0, q4_0, q4_k	see description	YuE2 MoT weight storage type. Omitted inherits weight_type.
yue2.vae_weight_type	native, f32, f16, bf16, q8_0, q4_0, q4_k	see description	YuE2 VAE weight storage type. Omitted inherits weight_type.
yue2.model_weight_context_mb	MiB integer >= 1	32	Main model weight context size.
yue2.vae_weight_context_mb	MiB integer >= 1	32	VAE weight context size.
yue2.ar_prefill_graph_arena_mb	MiB integer >= 1	32	AR prefill graph arena size.
yue2.ar_decode_graph_arena_mb	MiB integer >= 1	32	AR one-token decode graph arena size.
yue2.nar_graph_arena_mb	MiB integer >= 1	32	NAR acoustic flow graph arena size.
yue2.vae_graph_arena_mb	MiB integer >= 1	32	VAE decode graph arena size.
yue2.attention	auto, flash, eager	auto	NAR acoustic-flow attention kernel. auto uses flash, except on Volta/Turing CUDA GPUs (missing MMA kernels) and Intel Vulkan GPUs (eager measured 2.2x faster) where it uses eager; explicit flash / eager override the probe. The AR decode path always uses flash.
yue2.attention_tile_rows	integer	0	Query rows per tile in eager NAR attention. Nonpositive values select automatic tiling to keep each tile's score matrix under 3 GiB. A song whose scores already fit runs as one tile. Positive values set the query-row budget per tile. Ignored by the flash kernel.
yue2.perf_mode	off, cuda_perf	off	Performance preset; see Performance Controls. Explicit controls override preset values.
yue2.packed_projections	boolean	false; true with cuda_perf	Pack AR QKV and gate/up projections without changing weight dtype.
yue2.kv_cache_alignment	integer >= 1	1; 256 with cuda_perf	Align AR KV capacity; values above 1 also use cached CUDA CFG attention windows of this size. May use more memory and change numerics.
yue2.mem_saver	boolean	true; false with cuda_perf	Release AR/NAR runtimes and VAE graphs after generation.
yue2.vae_tile_frames	integer >= 1	Model configuration; capped at 512 with cuda_perf	VAE latent core frames per tile, with unchanged halo. Explicit values override the preset cap.

Parity Probes

The parity probes compare the port against tensors dumped from the Python reference without adding reference-only inputs to the request surface. Build them with -DENGINE_BUILD_TESTS=ON (or -DENGINE_BUILD_EXTENDED_TESTS=ON / -DENGINE_BUILD_MODEL_TESTS=ON).

Probe	Reference dump	Compares
yue2_vae_parity_probe	tests/yue2/yue2_vae_reference_dump.py	VAE encode/decode planar tensors.
yue2_nar_parity_probe	tests/yue2/yue2_nar_reference_dump.py	NAR acoustic-flow latents.

yue2_nar_reference_dump.py runs ABC and semantic sampling greedily, draws the same fp32 noise yue2.nar.song_chunks uses, and writes prefix.i32, codec.i32, noise.f32, latents_ref.f32 and metadata.json. yue2_nar_parity_probe feeds exactly those tensors through Yue2ArRuntime / Yue2NarRuntime and reports max_abs, rmse and cosine against latents_ref.f32:

python tests/yue2/yue2_nar_reference_dump.py --reference-root reference/YuE \
    --model models/YuE2-3B --vae models/YuE2-Vae --out-dir /tmp/yue2nar
build/bin/yue2_nar_parity_probe --model models/Yue2-3B-GGUF \
    --reference /tmp/yue2nar --backend cuda

Measured cosine is >= 0.9999 with the bf16 package and >= 0.995 with the shipped q8_0 package, which accumulates more error over long prefixes. Use --min-cosine / --max-rmse for a stricter gate, and --model-gguf / --weight-type to match the deployment under test.