Fabio
c36510c21e
3.6 KiB
Magenta RealTime 2
Magenta RealTime 2 generates 48 kHz stereo instrumental music from a text style
prompt or an audio example. Both Small (230M) and Base (2.4B) use the same
gen route in offline and streaming modes. Text describes the music; it is not
a lyric transcript.
Download one self-contained F16 or Q8_0 package from audio-cpp/Magenta-RealTime-2-GGUF. Each GGUF includes the generator, SpectroStream codec, MusicCoCa style runtime weights, configuration, vocabulary, and v1 model spec.
audiocpp_cli --model Magenta-RealTime-2-GGUF/magenta-rt2-small-f16.gguf \
--family magenta_realtime_2 --task gen --backend cuda \
--text "disco funk" --request-option duration_sec=10 \
--out music.wav --log
Use magenta-rt2-base-f16.gguf for Base, or --backend cpu / vulkan
for another backend. Use the -q8_0.gguf packages for quantized transformer
matrices. Add --mode streaming to emit audio incrementally;
the same request produces the same final waveform in either mode. Streaming
emits up to four codec frames (160 ms) per audio event. Each request starts a
fresh piece; it does not continue the previous request.
For audio style conditioning, replace --text with --audio style.wav.
MusicCoCa embeds the recording's style rather than continuing its waveform.
| Request option | Default | Meaning |
|---|---|---|
duration_sec |
10 |
Duration, rounded up to 40 ms frames; minimum 0.04 seconds. |
temperature |
1.3 |
Sampling temperature; 0 selects greedy decoding. |
top_k |
40 |
Candidate limit from 1–1024; 0 keeps all valid codes. |
seed |
0 |
Unsigned 32-bit generation seed. |
guidance_scale |
3 |
Style guidance from -1–7. |
note_guidance_scale |
1 |
Note guidance from -1–7. |
drum_guidance_scale |
1 |
Drum guidance from -1–7. |
use_mapper |
true |
Map text embeddings into audio style space. Applies to text input. |
drums |
-1 |
Unspecified (-1), no drums (0), or drums (1). |
pianoroll_file |
unset | JSON array with one row per 25 Hz frame and 128 integer pitch states per row. |
Set these through --request-option name=value. Guidance is discretized to
the model's trained control tokens. Piano states are -1 (unspecified), 0
(off), 1 (sustain), 2 (onset), and 3 (model chooses sustain/onset).
| Session option | Default | Meaning |
|---|---|---|
magenta_realtime_2.weight_type |
native |
Transformer weight storage type. Codec and MusicCoCa remain F32. |
All generation graphs and bounded attention/codec state are cached per session. MusicCoCa has its own framework runtime and reads its tensors from the same GGUF. Inference does not require Python, JAX, or TensorFlow Lite.
Conversion
Download the original Google checkpoint and resources,
then run the converter with NumPy, safetensors, and ai-edge-litert installed:
python tools/models/convert_magenta_realtime_2_gguf.py \
--source Magenta-RealTime-2 --output Magenta-RealTime-2-GGUF \
--component small --type f16
python tools/models/convert_magenta_realtime_2_gguf.py \
--source Magenta-RealTime-2 --output Magenta-RealTime-2-GGUF \
--component base --type f16
Use --type q8_0 instead for Q8_0. Each conversion embeds all MusicCoCa
resources in the selected generator GGUF. SpectroStream, MusicCoCa, and
sensitive normalization, bias, scale, and sink tensors remain F32.
Original model: Google DeepMind Magenta RealTime 2. Weights retain CC-BY-4.0 and the upstream usage terms.