Fabio
c36510c21e
Some checks failed
Linux Build / Linux cpu (push) Has been cancelled
Linux Build / Linux cpu-clang (push) Has been cancelled
Linux Build / Linux vulkan (push) Has been cancelled
mac-build / macOS CPU (push) Has been cancelled
Nix Build / Nix (cpu) (push) Has been cancelled
Nix Build / Nix (vulkan) (push) Has been cancelled
windows-build / Windows CPU (push) Has been cancelled
231 lines
10 KiB
Markdown
231 lines
10 KiB
Markdown
# MiniMax-H3
|
|
|
|
MiniMax-H3 is a text-to-audio/video generation model. The current audio.cpp port is
|
|
packaged as component GGUF files: text encoder, DiT, Audio VAE, Video VAE, config, and
|
|
processor sidecars live together in one model directory.
|
|
|
|
## Package Layout
|
|
|
|
The package directory must contain the files referenced by `model_specs/minimax_h3.json`:
|
|
|
|
```text
|
|
MiniMax-H3-Q4-GGUF/
|
|
configuration.json
|
|
text_encoder_q4_k.gguf
|
|
dit.gguf
|
|
dit_int8.gguf optional experimental ConvRot DiT
|
|
audio_vae.gguf
|
|
audio_vae_folded_f16.gguf
|
|
video_vae.gguf
|
|
FL2VA/processor/
|
|
merges.txt
|
|
preprocessor_config.json
|
|
tokenizer.json
|
|
tokenizer_config.json
|
|
vocab.json
|
|
...
|
|
```
|
|
|
|
The runtime uses `audio_vae_folded_f16.gguf`. Keep `audio_vae.gguf` as the original
|
|
Audio VAE GGUF so the folded runtime file can be regenerated.
|
|
|
|
## Run
|
|
|
|
For performance, memory, and quality tradeoffs across step count, DiT variant,
|
|
and acceleration options, see the [MiniMax-H3 performance notes](../reports/minimax_h3_performance.md).
|
|
|
|
MiniMax-H3 packages contain several component GGUF files. Pass the DiT entry file
|
|
explicitly: `dit.gguf` for the default path, or `dit_int8.gguf` for the experimental
|
|
ConvRot INT8 path. The runtime uses that file's parent directory as the package root and
|
|
resolves the other component files from there.
|
|
|
|
```bash
|
|
PROMPT='A lively four speaker comedy scene in a small radio studio. Speaker one says, "Welcome back, everyone, today we are testing a microphone that only records embarrassing truths." Speaker two says, "That explains why it kept calling me a sandwich with ambition." Speaker three says, "Please focus, the sponsor asked for professionalism and at least one normal sentence." Speaker four says, "Fine. This is a normal sentence, delivered by a person standing next to a haunted coffee machine." The speakers laugh, interrupt each other lightly, and continue with clear natural voices, quick timing, and no background music.'
|
|
|
|
build/debug/bin/audiocpp_cli \
|
|
--task gen \
|
|
--family minimax_h3 \
|
|
--model models/MiniMax-H3-Q4-GGUF/dit.gguf \
|
|
--model-spec-override model_specs/minimax_h3.json \
|
|
--backend cuda \
|
|
--threads 8 \
|
|
--text "$PROMPT" \
|
|
--seed 20260808 \
|
|
--num-inference-steps 20 \
|
|
--guidance-scale 1.0 \
|
|
--request-option height=32 \
|
|
--request-option width=32 \
|
|
--request-option num_frames=481 \
|
|
--request-option return_video=false \
|
|
--out output.wav \
|
|
--metrics
|
|
```
|
|
|
|
Set `return_video=true` when you also want RGB24 video output as an artifact. Use
|
|
`--out-dir <dir>` to write custom artifacts from the CLI.
|
|
|
|
## Options
|
|
|
|
Common request options:
|
|
|
|
| Option | Default | Notes |
|
|
|---|---:|---|
|
|
| `num_inference_steps` | `50` | Joint DiT denoising steps. Lower values are faster. |
|
|
| `seed` | `42` | Deterministic initial latent seed. |
|
|
| `height` | `768` | Target video canvas height. Use smaller values for short audio-focused generations. |
|
|
| `width` | `1344` | Target video canvas width. Use smaller values for short audio-focused generations. |
|
|
| `num_frames` | `124` | Target video frame count before H3 alignment. MiniMax-H3 video is 24 fps. |
|
|
| `guidance_scale` | `1.0` | Classifier-free guidance scale. |
|
|
| `negative_prompt` | `" "` | Negative prompt used when guidance is active. |
|
|
| `sampler_mode` | `euler` | One of `euler`, `res_multistep`, `dpmpp_2m`, or `unipc`. Legacy `sampler` and `minimax_h3.sampler` remain accepted. |
|
|
| `flow_shift` | `12.0` | Video FlowMatch scheduler shift. |
|
|
| `audio_flow_shift` | `3.0` | Audio FlowMatch scheduler shift. |
|
|
| `return_video` | `false` | Decode RGB24 video frames in addition to audio. |
|
|
|
|
Memory and staged-weight options:
|
|
|
|
| Option | Default | Notes |
|
|
|---|---:|---|
|
|
| `minimax_h3.weight_context_mb` | `512` | Staged weight context size in MiB. |
|
|
| `minimax_h3.mem_saver` | `true` | Release staged DiT, audio VAE, and video VAE weights after their request phase instead of keeping them resident for the whole session. Prompt encoder weights are always request-scoped. Set `false` to keep generation weights cached when VRAM allows. |
|
|
|
|
Advanced options:
|
|
|
|
| Option | Default | Notes |
|
|
|---|---:|---|
|
|
| `text_layerwise` | `false` | Load prompt encoder weights in scoped layer groups. |
|
|
| `text_layerwise_batch` | `1` | Prompt encoder layer-group size when `text_layerwise=true`. |
|
|
| `dit_layerwise` | `false` | Load DiT weights in scoped prelude/block/final groups. |
|
|
| `dit_layerwise_batch` | `1` | DiT block-group size when `dit_layerwise=true`. |
|
|
| `dit_mlp_chunk_tokens` | `0` | Token chunk size for high-resolution layerwise MLP. Zero disables chunking. |
|
|
| `dit_acceleration` | `none` | Optional acceleration mode: `none`, `first_block_cache`, or `spectrum`. |
|
|
| `first_block_cache_threshold` | `0.1` | First-block cache residual-difference threshold. |
|
|
| `first_block_cache_start_percent` | `0.1` | Start caching at 10% denoising progress; fractions must satisfy `0 <= start < end <= 1`. |
|
|
| `first_block_cache_end_percent` | `0.95` | Stop caching at 95% denoising progress; used with the schedule to derive the default sigma window. |
|
|
| `first_block_cache_start_sigma` | derived from progress/schedule | Optional largest video sigma for caching; must be supplied together with `first_block_cache_end_sigma`. |
|
|
| `first_block_cache_end_sigma` | derived from progress/schedule | Optional smallest video sigma for caching; must be supplied together with `first_block_cache_start_sigma`. Explicit sigma endpoints override the derived window. |
|
|
| `first_block_cache_max_consecutive` | `2` | Maximum consecutive cache hits before forcing a full DiT step. |
|
|
| `spectrum_warmup_steps` | `1` | Full DiT warmup steps before spectrum forecasting. |
|
|
| `spectrum_initial_window` | `2.0` | Initial spectrum forecast scheduling window. |
|
|
| `spectrum_flex_window` | `0.75` | Spectrum forecast window growth after actual DiT steps. |
|
|
| `spectrum_degree` | `1` | Chebyshev polynomial degree for spectrum forecasting. |
|
|
| `spectrum_history_size` | `8` | Maximum spectrum forecast history rows. |
|
|
| `spectrum_ridge_lambda` | `0.1` | Ridge regularization for spectrum forecasting. |
|
|
|
|
## Convert
|
|
|
|
Most MiniMax-H3 components use the normal `audiocpp_gguf` converter. The DiT component is
|
|
the exception: it needs MiniMax-H3-specific AdaLN curve-table folding, so use the dedicated
|
|
DiT script for that component.
|
|
|
|
### DiT
|
|
|
|
```bash
|
|
python tools/community_models/minimax_h3/convert_dit_gguf.py \
|
|
--input /path/to/MiniMax-H3-NF4/minimax-h3-fl2va-nf4.safetensors \
|
|
--output models/MiniMax-H3-Q4-GGUF/dit.gguf \
|
|
--overwrite \
|
|
--type q4_0 --bnb-nf4-type q4_0 \
|
|
--override 'blocks.*.adaln_proj.linear.weight=q4_0' \
|
|
--override 'blocks.*.attn.out_proj.weight=bf16' \
|
|
--override 'final_layer.adaln_proj.linear.weight=q4_0' \
|
|
--override 'condition_proj.weight=bf16' \
|
|
--override 'time_embedder.proj_in.weight=bf16' \
|
|
--override 'time_embedder.proj_out.weight=bf16' \
|
|
--override 'final_layer.audio_out.weight=bf16' \
|
|
--override 'final_layer.video_out.weight=bf16' \
|
|
--override 'blocks.39.mlp.fc2.weight=bf16' \
|
|
--override 'blocks.40.mlp.fc2.weight=bf16' \
|
|
--override 'blocks.41.mlp.fc2.weight=bf16' \
|
|
--override 'blocks.42.mlp.fc2.weight=bf16' \
|
|
--override 'blocks.43.mlp.fc2.weight=bf16' \
|
|
--override 'blocks.44.mlp.fc2.weight=bf16' \
|
|
--override 'blocks.45.mlp.fc2.weight=bf16' \
|
|
--override 'blocks.46.mlp.fc2.weight=bf16' \
|
|
--override 'blocks.47.mlp.fc2.weight=bf16' \
|
|
--override 'blocks.48.mlp.fc2.weight=bf16' \
|
|
--override 'blocks.49.mlp.fc2.weight=bf16' \
|
|
--quantize-scope weights-2d --ineligible native \
|
|
--quantizer python \
|
|
--helper-chunk-rows 128 \
|
|
--h3-adaln-curve-grid 1024 \
|
|
--h3-adaln-curve-rank 64
|
|
```
|
|
|
|
The DiT GGUF should contain 532 tensors: one F32 AdaLN table, BF16 tensors for the
|
|
precision-sensitive override list, and Q4_0 tensors for the remaining quantized 2D
|
|
weights.
|
|
|
|
### DiT INT8 ConvRot
|
|
|
|
The optional `dit_int8.gguf` variant uses ConvRot INT8 linear layers for selected DiT
|
|
projections. It is CUDA-only and currently experimental. Keep the normal `dit.gguf` as the
|
|
default package DiT unless you explicitly want to test the INT8 path.
|
|
|
|
```bash
|
|
python tools/community_models/minimax_h3/convert_dit_gguf.py \
|
|
--input /path/to/MiniMax-H3-NF4/minimax-h3-fl2va-nf4.safetensors \
|
|
--output models/MiniMax-H3-Q4-GGUF/dit_int8.gguf \
|
|
--overwrite \
|
|
--type q4_0 --bnb-nf4-type q4_0 \
|
|
--overlay-input /path/to/MiniMax_H3_FL2VA_pruned_int8_convrot.safetensors \
|
|
--overlay-include 'blocks.*.attn.out_proj.weight' \
|
|
--overlay-include 'blocks.*.mlp.fc1.weight' \
|
|
--overlay-include 'blocks.*.mlp.fc2.weight' \
|
|
--override 'blocks.*.adaln_proj.linear.weight=q4_0' \
|
|
--override 'final_layer.adaln_proj.linear.weight=q4_0' \
|
|
--override 'condition_proj.weight=bf16' \
|
|
--override 'time_embedder.proj_in.weight=bf16' \
|
|
--override 'time_embedder.proj_out.weight=bf16' \
|
|
--override 'final_layer.audio_out.weight=bf16' \
|
|
--override 'final_layer.video_out.weight=bf16' \
|
|
--override 'blocks.39.mlp.fc2.weight=bf16' \
|
|
--override 'blocks.40.mlp.fc2.weight=bf16' \
|
|
--override 'blocks.41.mlp.fc2.weight=bf16' \
|
|
--override 'blocks.42.mlp.fc2.weight=bf16' \
|
|
--override 'blocks.43.mlp.fc2.weight=bf16' \
|
|
--override 'blocks.44.mlp.fc2.weight=bf16' \
|
|
--override 'blocks.45.mlp.fc2.weight=bf16' \
|
|
--override 'blocks.46.mlp.fc2.weight=bf16' \
|
|
--override 'blocks.47.mlp.fc2.weight=bf16' \
|
|
--override 'blocks.48.mlp.fc2.weight=bf16' \
|
|
--override 'blocks.49.mlp.fc2.weight=bf16' \
|
|
--quantize-scope weights-2d --ineligible native \
|
|
--quantizer python \
|
|
--helper-chunk-rows 128 \
|
|
--h3-adaln-curve-grid 1024 \
|
|
--h3-adaln-curve-rank 64
|
|
```
|
|
|
|
The INT8 overlay is intentionally limited to `attn.out_proj`, `mlp.fc1`, and `mlp.fc2`.
|
|
Do not overlay `attn.qkv_proj`: the isolated projection can look numerically close, but the
|
|
end-to-end model output is not valid with QKV overlaid. ConvRot stores rotated INT8 weights
|
|
and scales in the GGUF; activation rotation and quantization happen in the CUDA runtime.
|
|
|
|
### Audio VAE
|
|
|
|
Create the original Audio VAE GGUF first:
|
|
|
|
```bash
|
|
build/debug/bin/audiocpp_gguf \
|
|
--input /path/to/MiniMax-H3-NF4/audio_vae_nf4.safetensors \
|
|
--output models/MiniMax-H3-Q4-GGUF/audio_vae.gguf \
|
|
--type orig \
|
|
--bnb-nf4-type q4_k \
|
|
--family minimax_h3 \
|
|
--root models/MiniMax-H3-Q4-GGUF \
|
|
--overwrite
|
|
```
|
|
|
|
Then create the folded runtime Audio VAE GGUF:
|
|
|
|
```bash
|
|
python tools/community_models/minimax_h3/convert_fold_audio_vae_gguf.py \
|
|
--input models/MiniMax-H3-Q4-GGUF/audio_vae.gguf \
|
|
--output models/MiniMax-H3-Q4-GGUF/audio_vae_folded_f16.gguf \
|
|
--folded-type f16 \
|
|
--overwrite
|
|
```
|
|
|
|
The folded file replaces MiniMax-H3 BigVGAN weight-norm pairs with direct convolution
|
|
weights stored as F16. Other tensors are copied from `audio_vae.gguf`.
|