Fabio
c36510c21e
Some checks failed
Linux Build / Linux cpu (push) Has been cancelled
Linux Build / Linux cpu-clang (push) Has been cancelled
Linux Build / Linux vulkan (push) Has been cancelled
mac-build / macOS CPU (push) Has been cancelled
Nix Build / Nix (cpu) (push) Has been cancelled
Nix Build / Nix (vulkan) (push) Has been cancelled
windows-build / Windows CPU (push) Has been cancelled
74 lines
3.6 KiB
Markdown
74 lines
3.6 KiB
Markdown
# Magenta RealTime 2
|
||
|
||
Magenta RealTime 2 generates 48 kHz stereo instrumental music from a text style
|
||
prompt or an audio example. Both Small (230M) and Base (2.4B) use the same
|
||
`gen` route in offline and streaming modes. Text describes the music; it is not
|
||
a lyric transcript.
|
||
|
||
Download one self-contained F16 or Q8_0 package from
|
||
[audio-cpp/Magenta-RealTime-2-GGUF](https://huggingface.co/audio-cpp/Magenta-RealTime-2-GGUF).
|
||
Each GGUF includes the generator, SpectroStream codec, MusicCoCa style runtime
|
||
weights, configuration, vocabulary, and v1 model spec.
|
||
|
||
```bash
|
||
audiocpp_cli --model Magenta-RealTime-2-GGUF/magenta-rt2-small-f16.gguf \
|
||
--family magenta_realtime_2 --task gen --backend cuda \
|
||
--text "disco funk" --request-option duration_sec=10 \
|
||
--out music.wav --log
|
||
```
|
||
|
||
Use `magenta-rt2-base-f16.gguf` for Base, or `--backend cpu` / `vulkan`
|
||
for another backend. Use the `-q8_0.gguf` packages for quantized transformer
|
||
matrices. Add `--mode streaming` to emit audio incrementally;
|
||
the same request produces the same final waveform in either mode. Streaming
|
||
emits up to four codec frames (160 ms) per audio event. Each request starts a
|
||
fresh piece; it does not continue the previous request.
|
||
|
||
For audio style conditioning, replace `--text` with `--audio style.wav`.
|
||
MusicCoCa embeds the recording's style rather than continuing its waveform.
|
||
|
||
| Request option | Default | Meaning |
|
||
|---|---:|---|
|
||
| `duration_sec` | `10` | Duration, rounded up to 40 ms frames; minimum 0.04 seconds. |
|
||
| `temperature` | `1.3` | Sampling temperature; `0` selects greedy decoding. |
|
||
| `top_k` | `40` | Candidate limit from 1–1024; `0` keeps all valid codes. |
|
||
| `seed` | `0` | Unsigned 32-bit generation seed. |
|
||
| `guidance_scale` | `3` | Style guidance from -1–7. |
|
||
| `note_guidance_scale` | `1` | Note guidance from -1–7. |
|
||
| `drum_guidance_scale` | `1` | Drum guidance from -1–7. |
|
||
| `use_mapper` | `true` | Map text embeddings into audio style space. Applies to text input. |
|
||
| `drums` | `-1` | Unspecified (`-1`), no drums (`0`), or drums (`1`). |
|
||
| `pianoroll_file` | unset | JSON array with one row per 25 Hz frame and 128 integer pitch states per row. |
|
||
|
||
Set these through `--request-option name=value`. Guidance is discretized to
|
||
the model's trained control tokens. Piano states are `-1` (unspecified), `0`
|
||
(off), `1` (sustain), `2` (onset), and `3` (model chooses sustain/onset).
|
||
|
||
| Session option | Default | Meaning |
|
||
|---|---|---|
|
||
| `magenta_realtime_2.weight_type` | `native` | Transformer weight storage type. Codec and MusicCoCa remain F32. |
|
||
|
||
All generation graphs and bounded attention/codec state are cached per session.
|
||
MusicCoCa has its own framework runtime and reads its tensors from the same
|
||
GGUF. Inference does not require Python, JAX, or TensorFlow Lite.
|
||
|
||
## Conversion
|
||
|
||
Download the original [Google checkpoint and resources](https://huggingface.co/google/magenta-realtime-2),
|
||
then run the converter with NumPy, safetensors, and `ai-edge-litert` installed:
|
||
|
||
```bash
|
||
python tools/models/convert_magenta_realtime_2_gguf.py \
|
||
--source Magenta-RealTime-2 --output Magenta-RealTime-2-GGUF \
|
||
--component small --type f16
|
||
python tools/models/convert_magenta_realtime_2_gguf.py \
|
||
--source Magenta-RealTime-2 --output Magenta-RealTime-2-GGUF \
|
||
--component base --type f16
|
||
```
|
||
|
||
Use `--type q8_0` instead for Q8_0. Each conversion embeds all MusicCoCa
|
||
resources in the selected generator GGUF. SpectroStream, MusicCoCa, and
|
||
sensitive normalization, bias, scale, and sink tensors remain F32.
|
||
|
||
Original model: [Google DeepMind Magenta RealTime 2](https://github.com/magenta/magenta-realtime).
|
||
Weights retain CC-BY-4.0 and the upstream usage terms.
|