voice_audio_multi/docs/build/rockchip-npu.md

Rockchip NPU (RKNN) Decoder Path

Opt-in, per-stage acceleration of one model stage on Rockchip RK3588/RK3576/RK356x NPUs through librknnrt. This is not a ggml backend: ggml has no RKNN backend, and a full backend port is a different project. Here a single model stage that already computes through its own graph and ExecutionContext is routed to a pre-compiled RKNN network instead of ggml.

Currently implemented for three families: the qwen3_tts speech tokenizer decoder (the vocoder, codes to PCM), the piper_tts VITS decoder (latent to PCM) and the kokoro_tts iSTFTNet generator (latent to spectrogram). Everything below is measured on an Orange Pi 5 Plus (RK3588, 4x A76 + 4x A55).

Conditions on the board numbers. Every RTF, ms and RSS figure here was taken with the vendor rknpu_ondemand governor, which ramps the NPU to its 1000 MHz OPP: check cat /sys/class/devfreq/fdab0000.npu/cur_freq (1000000000 under load) and cat /sys/class/devfreq/fdab0000.npu/governor. The OPP table on that overlay starts at 300 MHz, so a board running the mainline rocket driver's performance-less default can sit at a fraction of that clock and every figure here scales with it. The CPU side of the same runs is --threads 4 on the A76 cluster, and the native build (scripts/build_linux.sh --backend cpu, which turns on the host-ISA ggml kernels): a plain cmake -DCMAKE_BUILD_TYPE=Release build of the same tree measures the qwen3_tts CPU path at RTF 4.287 and 4510 MB against the 3.010 and 4200 MB below, while the kokoro_tts CPU path reproduces to 1.409 against 1.415. Absolute CPU-path figures are therefore build-dependent, ratios are not. The board's DRAM read ceiling is 26.4-29.4 GB/s, measured with a cache-bypassing STREAM triad (4 A76 threads, 804 MB working set) and sysbench memory with a 64 MB buffer; sysbench with its default 1 MB buffer reports 61 GB/s because it never leaves L3, and stress-ng --stream is cache-sized too, so neither is a DRAM measurement.

Requirements

Board side:

• librknnrt.so (RKNPU2 runtime, 2.x) on the loader path, and rknn_api.h for the headers. The Rockchip SDK's install.sh puts the library in /usr/lib and the headers in /usr/local/include.
• The RKNPU kernel driver. Check with cat /sys/kernel/debug/rknpu/version or dmesg | grep -i rknpu.

Host side (x86_64, for building the .rknn):

• rknn-toolkit2 in a dedicated venv. Its pins are strict (numpy<=1.26.4, onnx==1.16.1, upper-bounded torch, Python 3.6-3.12); do not resolve them against a system Python.
• On Python 3.12 the 2.3.2 converter needs two patches the SDK does not ship: setuptools<81 (rknn.api.base_utils imports pkg_resources, removed in setuptools 81), and an onnx/mapping.py shim inside the venv's onnx package (the module is gone in onnx 1.16, and onnx 1.15 has no cp312 wheel). RKNN does import onnx then onnx.mapping.X, so the shim must also be bound by appending from onnx import mapping to onnx/__init__.py.

Build

Native on the board (the same flags work in a cross build):

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release \
  -DENGINE_ENABLE_RKNN=ON \
  -DENGINE_RKNN_INCLUDE_DIR=/usr/local/include \
  -DENGINE_RKNN_LIBRARY=/usr/lib/librknnrt.so \
  -DAUDIOCPP_MODEL_SET=custom -DAUDIOCPP_MODELS=qwen3_tts
cmake --build build --parallel --target audiocpp_cli

ENGINE_ENABLE_RKNN is OFF by default and nothing in the default build links librknnrt. With the flag off, selecting RKNN in qwen3_tts.exec_plan fails with Qwen3 speech decoder RKNN: built without ENGINE_ENABLE_RKNN rather than falling back silently.

The same build ships as a Docker image: .devops/rockchip.Dockerfile (arm64-only; the librknnrt.so + headers are fetched at build time from the pinned SDK tag), and the container reaches the NPU through the driver's own device nodes — see docs/docker.md.

Model preparation

Prebuilt engines for RK3588 ship as rknn packages in the model specs, for the three families whose engines are validated on that board — install them like any other package and name the files by bare file name in the plan:

python3 tools/model_manager_v2.py install piper_lessac_medium_decoder_rk3588
python3 tools/model_manager_v2.py install kokoro_82m_generator_rk3588
python3 tools/model_manager_v2.py install qwen3_tts_0_6b_base_speech_decoder_rk3588

Each installs into the family's GGUF weight directory, so an exec_plan artifact can name it by file alone. The engines are compiled per SoC: these are RK3588 images, and the converter runs below are the path for any other target (and for rebuilding from the source checkpoints).

Outside those packages, a .rknn is not downloadable and nothing in the runtime produces it. Build it on the host from the Qwen/Qwen3-TTS-12Hz-0.6B-Base checkpoint (the speech_tokenizer subdirectory, the same weights the GGUF package was converted from):

python3 tools/rknn/prepare_qwen3_tts_rknn_vocoder.py --model-dir <snapshot-dir> --codes-length 64
python3 tools/rknn/prepare_qwen3_tts_rknn_vocoder.py --model-dir <snapshot-dir> --codes-length 16

Each run re-traces the decoder as one fixed-shape streaming step, checks the stepwise decode against a full decode (SNR must stay above 40 dB), exports ONNX, and compiles it for --target (default rk3588). The graph is stateful: the causal-conv tails and the sliding-window KV history travel through the graph I/O as one flat fp32 cache, so a step decodes from continuous state and there are no chunk seams. --codes-length is compiled in, so the two-tier first chunk needs one file per batch size; the cache layout does not depend on it, so a small-batch step's cache feeds the large-batch model directly.

The two stages want different toolchains (the export needs the model package, the conversion needs the toolkit's pinned torch/onnx), so run them in the two venvs and hand the ONNX over:

<export-venv>/bin/python tools/rknn/prepare_qwen3_tts_rknn_vocoder.py \
  --model-dir <snapshot-dir> --codes-length 64 --skip-rknn --output vocoder_stateful_64.onnx
<converter-venv>/bin/python tools/rknn/prepare_qwen3_tts_rknn_vocoder.py \
  --onnx vocoder_stateful_64.onnx --target rk3588

--onnx reads the step contract (batch size, cache size, head dim) back out of the graph, so the conversion stage restates nothing.

The export stage is version-sensitive: a recent torch/onnx decomposes LayerNorm and Softmax into patterns (Pad, Exp, Shape, ConstantOfShape) that rknn.build aborts on inside fold_constant, with a core dump and no Python traceback. The script counts those ops after exporting and says so there, where it is cheap to act on. Measured with this graph: the toolkit-era pair exports 1064 nodes and compiles; a current pair exports 2653 and aborts.

A .rknn is compiled per target SoC and is not portable between chips. The graph is built fp16 (w16a16) and deliberately not quantized: the Snake activation makes Q8/INT8 audibly noisy (9.5-13.4 dB SNR) and fp16 measures lossless (56.8 dB).

--target accepts the RKNPU2 platforms the toolkit knows. Only rk3588 has been built and measured on a board here; rk3566 conversions of all three families have since been built successfully on the host (the engines are unmeasured on a board), the others are untested, not unsupported.

Piper (VITS) decoder

tools/rknn/prepare_piper_tts_rknn_vocoder.py slices a Piper voice ONNX at the seam the runtime already has — the inverse-flow input z [1,192,N] and its frame mask y_mask [1,1,N] (plus g for a multi-speaker voice) — checks the fixed-window decode against a full-length decode, and compiles the slice:

<export-venv>/bin/python tools/rknn/prepare_piper_tts_rknn_vocoder.py \
  --voice-onnx en_US-lessac-medium.onnx --skip-rknn --output decoder_z.onnx
<converter-venv>/bin/python tools/rknn/prepare_piper_tts_rknn_vocoder.py \
  --onnx decoder_z.onnx --target rk3588

The conversion stage re-runs the self-check on white-noise latents (the voice's own encoder is not involved once the slice exists), and its global gate applies only to decodes that contain seams: a latent shorter than the window is one tail window with no seam, and its global SNR is the documented tail deficit, not a stitching failure.

The seam is the flow input and not the input of dec.conv_pre: the flow belongs to the decoder half, and the runtime's latent is the pre-flow one. Slicing at conv_pre compiles a decoder that expects post-flow samples and feeds it pre-flow ones, which is quiet mush (measured: output RMS 4x low, no intelligible speech).

RKNN has no dynamic shapes, so the engine is compiled for one window of 93 latent slices and the runtime walks the latent over it: 45 slices emitted per step, 24 slices of real context on each side, a short window zero-padded and mask-zeroed at the end that has no data, and the window that reaches the end of the latent extends its left context to fill the window. The 24 is measured, not inherited: at the 45+5+5 window paroli ships, interior windows score 8-60 dB against a full-length decode (93: 98+ dB, 77: 51+ dB). The mask is 1 over the real slices and 0 over the padding — the flow multiplies its activations by the mask after every convolution, so a padded frame marked 1 propagates its conv biases into the real frames and costs 10-20 dB there.

The last slices of the latent are the one thing a fixed-shape window cannot reproduce: a full-length decode has the end of the tensor there, where a window has padding, and a conv layer two levels deep reads the pad's own conv output (nonzero, from the biases) where the full decode reads the implicit zeros past the tensor end. No mask value in the padding fixes it, because the mask reaches only the flow: the VITS decoder (conv_pre, 18 resblock convs, 3 upsamples, conv_post) is unmasked, so the pad's conv output is what the last real slices read. Measured: the last ~7 slices of a short latent land 20-40 dB off, the interior windows are exact. paroli ships the same trade.

--dec-mask does fix it, by masking inside the decoder: Mul(y_mask) after every /dec/ Conv/ConvTranspose, with the mask upsampled by all-ones ConvTranspose1d at each dec.ups stride (8/8/4) so it is the right length at each level. Measured on the tail window: 28.6 → 117.8 dB (47-frame latent) and 43.8 → 118.6 dB (30-frame) against a full-length decode, interiors unchanged byte for byte, and 4.8 → 45.8 dB for a 47-frame tail window run through librknnrt directly. It costs +21% window time (RTF 0.0688 → 0.0832 on the 443-character utterance) and changes nothing there, because every chunk of that utterance exceeds 69 frames, so no tail window is short: the defect is a short-utterance property. It stays opt-in because the board does not hear it: over 8 realistic short utterances (one word to a phone-number string, three of them short enough that the tail window carries 28-35 pad slices) the two engines score the same against the CPU decode within 0.1 dB on every one (25.82/25.82, 24.62/24.62, 33.27/33.24 on the padded three), because the fp16 engine's own noise floor against the CPU decode (22.4-36.2 dB) sits at or below the fp32 tail deficit (28.6-43.8 dB) — the defect is buried under the engine's noise, so the +21% buys nothing audible. The two engines' outputs do differ by ~60 dB, so the masked engine is live and distinct. A Resize node for the mask upsample measures 50-53 dB in the PC simulator and -7.9 dB on the board — Resize does not lower to a target the board runtime accepts, so the upsampler is a ConvTranspose1d of all ones (exact nearest, and the same operator the decoder already runs on the NPU).

Piper duration encoder

The same script with --encoder compiles the other half — the text encoder plus the duration discriminator — for a fixed phoneme-token width: it slices at enc_p.proj (the 2x inter_channels prior stats) and dp.proj (the flow conditioning), the two arrays the runtime's flow graphs and length regulator consume, right-pads tokens to --frames and compiles one engine per width. The slice is deterministic (all three piper scales scalars feed sampling, the duration ceiling and the discriminator's flow noise, strictly downstream of the seam), and piper's exported masks are pad-safe: a padded run reproduces the exact-length run on the real frames at 117-130 dB fp32, checked by the self-check on the graph the engine is compiled from. scales is not an input.

<rknn-venv>/bin/python tools/rknn/prepare_piper_tts_rknn_vocoder.py \
  --voice-onnx en_US-lessac-medium.onnx --encoder --frames 128

With the frame axis pinned the graph constant-folds (992 to 463 nodes at the BASIC level), and what remains in every attention layer is a relative-shift chain: Pad + Reshape + Slice shifting the [N, 2N-1] relative-position slabs by one step per query row, unrolled N times. The RKNN kernels compute exactly those chains as garbage on the board — probes show the flatten-Pad tails filled with uninitialized memory (NaN, ±65504), one attention layer alone drags the engine to 0.4 dB end to end, and a chain probe segfaults the runtime — while the PC simulator computes the same graph at 56 dB. The simulator is therefore not a gate for this export; a board probe is. The converter rewrites the chains into exact constant-index gathers (rel-logit bias via GatherElements, rel-value sum via Gather + Mul + ReduceSum, verified 121-131 dB fp32 against the raw slice), the engine computes 40-48 dB on the NPU, and the fold self-checks run on the rewritten graph.

The gathers cost O(N^3) weight bytes: the index initializers are constants, so the toolkit folds them into per-row Toeplitz expansions — 137 MB at 128 tokens, 402 MB at 192, 2.9 GB at 384, and 512 exhausts 31 GB of converter RAM. Compile narrow bins only. The sidecar walks the sorted ladder to the smallest engine that fits the token count and falls back to the ggml encoder above the widest: the wide widths are where the fp16 duration drift hurts most (see the measurements below), so the fallback is a quality gate, not a convenience.

Inflect v2 (Micro/Nano) decoder

tools/rknn/prepare_inflect_v2_rknn_decoder.py slices the official decode.onnx at the same kind of seam — the inverse-flow input z [1,192,N] and its frame mask y_mask [1,1,N], the pre-flow latent the runtime's decoder graph (reverse_flow(latent) into dec) already has on hand — checks the fixed-window decode against a full-length one, and compiles the slice:

<export-venv>/bin/python tools/rknn/prepare_inflect_v2_rknn_decoder.py \
  --decode-onnx decode.onnx --duration-onnx duration.onnx --skip-rknn --output decoder_z.onnx
<converter-venv>/bin/python tools/rknn/prepare_inflect_v2_rknn_decoder.py \
  --onnx decoder_z.onnx --decode-onnx decode.onnx --duration-onnx duration.onnx --target rk3588

The head of decode.onnx (z = m_p + exp(logs) * zp_noise * noise_scale) stays outside the slice, so the engine has no noise input and no noise_scale: the runtime draws the noise with its own variation knob and hands the engine the finished latent. Because the slice is pinned to the compiled window, the self-check needs decode.onnx itself for a full-length reference decode; without it the check degrades to one window-length latent decoded against itself.

The slice is folded before it is compiled (onnxslim, with the input shapes pinned to the window): the export sizes its spline-flow tensors from the latent length through Shape → ConstantOfShape → Slice, and handing that to rknn.build aborts the build inside fold_constant with a core dump and no traceback — the same failure the qwen3 export stage warns about, arriving here from the vendor graph rather than from a too-new torch. Folding first takes 979 nodes with 24 Shape and 24 ConstantOfShape down to 409 plain conv nodes, and the build finishes (23.9 MB engine, 14.9 MB of it weights).

The window is 256 latent slices: 192 emitted per step, 32 slices of real context on each side, a short window zero-padded and mask-zeroed at the end that has no data, and the window that reaches the end of the latent extends its left context to fill the window. The 32 is measured on the model's own prior samples at 480 slices against a full-length decode: 5.1 dB with no context, 40.0 dB at 20, 82.1 dB at 24, 128.8 dB at 32.

Unlike VITS's dec, this decoder half is masked in the flow and unmasked in the decoder, so the slice carries the decoder-half muls by default (mask_decoder_graph, the same construction as piper's opt-in --dec-mask): without them a latent whose decode is dominated by the window that reaches the end of the latent scores 7.5-17.9 dB against a full-length decode (30, 47, 91 and 210 slices), with them 94.9-120.2 dB, and the interior windows are unchanged (they run with mask 1 everywhere, where every injected Mul is an exact fp32 identity). A 480-slice latent is unaffected either way, because its tail window is long enough that the implicit-zero end is buried. --no-dec-mask drops the muls.

Kokoro (iSTFTNet) generator

tools/rknn/prepare_kokoro_tts_rknn_generator.py re-traces the generator the GGUF weights describe (decoder.module.generator.* in kokoro-v1_0.pth) at one fixed window and compiles it. The seam is the generator alone — x [1,512,W], the harmonic conditioning har [1,22,60W+1], the style vector, and the frame mask at the two rates the graph normalizes at, in and out as spec_phase [1,22,60W+1]. Source-signal synthesis, the forward STFT and the iSTFT head stay on CPU:

<export-venv>/bin/python tools/rknn/prepare_kokoro_tts_rknn_generator.py \
  --checkpoint kokoro-v1_0.pth --config config.json \
  --output-dir artifacts --frames 128 --skip-rknn
<converter-venv>/bin/python tools/rknn/prepare_kokoro_tts_rknn_generator.py \
  --onnx artifacts/generator_w128.onnx --target rk3588

The window is capped by the NPU, not by the model: an op-encoding field on the time axis is 13 bits, so the longest axis in the graph, the fine rate 60W + 1, has to stay under 8192. At W=512 the compiler emits ~800k REGTASK: The bit width of field value exceeds the limit, ... limit: 0x1fff, value: 0x7800 lines and the engine voices clipped noise (rms 32764 of 32767, SNR -27 dB against the CPU decode). 128 is the widest power of two inside the limit, and it emits 128 - 2*28 = 72 latent frames per step, where 28 is the measured receptive field. The line itself is not the failure: the RK3566 builds of the three shipping engines log a few hundred to a few thousand of them (measured: 352 for the Kokoro w128 engine, 355 for the Piper w93 engine, 6570 for the Qwen3 B=64 vocoder) and measure clean on the board; the RK3588 build of the Piper w93 engine logs none. What kills the engine is the magnitude — value: 0x7800 is 30720, four times past the field, repeated ~800k times. tools/rknn/rknn_build_log.py is the gate that separates the two regimes: it captures fds 1/2 around rknn.build, because those lines come from the C++ compiler and not from Python logging, and aborts past 100 000 lines.

The call sequence that gate has to wrap — config, load_onnx, build, export_rknn, release — is the same in all five converters and lives in tools/rknn/rknn_build.py (build_engine). What stays per converter is the ONNX surgery, the input names and shapes it declares, and the simulator-side verification.

The field is per axis and per op, and the ceiling is escapable: slicing the fine-rate axis inside the graph takes W=256 from 403 587 REGTASK lines to 0, and the re-traced module keeps every gate — 60.5 dB against the runtime's generator, 135.6 dB padded-window exactness, 124.4 dB export against torch. The engine then voiced fp16-saturated output (117 682 non-finite values out of a 14.85 s utterance, contiguous in columns 8329-13872, peak 65440 = fp16 max), and the cause is the AdaIN global pooling: its statistics are raw sums over the fine axis, so the same site that peaks at 7681 at W=128 peaks at 1.1e6 at W=256 (per-node maxima from an onnxruntime fp32 run: first node past fp16 max a Mul at 612 487, largest node an Add at 1 106 990). The diagnosis is a probe, not a shipped change: recomputing those statistics as count fractions holds the same graph's fp32 range to 15 361 with first-over = None. The cure in the producer is --max-axis, which moves the runtime's window walk inside the graph — windows of (max_axis-1)//scale frames with --padding frames of context per side, each window through the full unsliced module, emitted columns concatenated the way decode_windowed overwrites the seam column. Then every in-graph reduction axis is at most max_axis, so the 13-bit field and the fp16 range are satisfied by construction: W=256 (3 windows of 136 at padding 28) builds with 1 REGTASK line and voices 0 non-finite values, peak 56.25, 46.8 dB against the chunked torch reference, and W=160 (2 windows) builds with 0 lines. The 9.5 dB against the monolithic reference is the per-window AdaIN gap, not the engine: the chunked-vs-monolithic gate measures 9.9 dB and the fp32 PC simulator of the same graph gives the identical 9.5 dB. It does not ship: the in-graph escape replicates the module inside the engine (142.7 MB for 3 windows against 74.7 MB for one W=128 window), so widening the window that way costs the weight copies the runtime walk exists to avoid, and the stitched result is a per-window synthesis, not the monolithic one.

The cheap wide window — the widest mono window the field allows, W=136 (emitted 80, 15 windows for the 14.85 s utterance, 76.0 MB) — was built and measured on the board, and it is not a win: RTF 1.057 against 1.114 at W=128, RSS 2345 against 2340 MB, 0 non-finite values, and the pumping spread 7.67 dB against 7.23 dB. Fewer seams and statistics over more frames do not help, the same verdict overlap-blending gave: the pumping is per-window loudness, not seam geometry. The same run settles where the normalized-statistics cure applies at all: at W=136 the raw sums peak at 8161 (nothing over 65504) and at W=256 at 1.1e6, so the band of widths where the sums saturate and the field passes is empty. Normalized statistics matter only in a graph whose span exceeds 8191, which is the sliced wide window the field rejects, so they are not implemented.

The per-window statistics are the audible artifact; feeding them is not the fix, and that path is not in the build. Listening compared the NPU render with the CPU render of the same text and seed found the loudness of successive 0.9 s windows swinging -2.8 to +4.5 dB against the CPU render (spread 7.2 dB) with no clicks at the seams - the per-window renormalization, heard as compression. A --global-stats variant of the engine - the per-channel AdaIN mean/variance of a full-length decode fed into the engine as graph I/O (stats_in/stats_out/use_global, 7 inputs / 2 outputs), computed on CPU from the full latent by one full-length generator pass - was built and measured for exactly that reason. In fp32 it is exact: the stitched per-window decode against the full-length decode goes from 9.7 dB to 135.4 dB on the 486-frame latent, with the stitch, the tail and the padding exact once the statistics are global. That confirms the diagnosis and rules out this engine as the cure: on the board the fed statistics scale the window activations past the engine's fp16 range and every window after the first renders as a constant-energy tone. Measured on a 14.85 s utterance: windows 1..15 carry rms flat at 51.5 dB (the CPU render swings 57.1 to 67.4), spectral flatness 1.7e-17, and all the energy at DC, 4800 Hz and 9600 Hz - Nyquist/2 and 0.8 * Nyquist, i.e. a saturated output, not speech. Only window 0 is intelligible, because that one is taken from the CPU pass. The render also costs 1.8x the wall clock (RTF 2.18 against 1.08) and +1.5 GB RSS. The producer, the runtime and the engine contract therefore carry no code for it, and the runtime rejects a 7-input engine at load; the findings stay here as the reason. Building it needed one runtime detail: ggml_build_forward_expand in the vendored ggml does not set GGML_TENSOR_FLAG_OUTPUT, so a captured statistics node with no other consumer is freed by the allocator and its buffer is handed to a later node - the statistics nodes must be marked as graph outputs.

Three rewrites are compiled in, all of them the reference Kokoro RKNN port's:

• Weight-norm folded at load. Left in the graph, vector_norm traces to 51 ReduceL2 nodes and rknn.build aborts.
• Masked instance norm through GlobalAveragePool. A ReduceSum over the time axis lowers to a register field capped at 8191 (the same limit as above); GAP is the pooling engine's own reduction. The mask has to zero the padding at every stage boundary — after the upsampling reflect-pad, on the resblock sum, on the noise branch output — not only inside AdaIN's statistics, because a conv after a masked AdaIN writes real-frame sums into the pad and the next conv_transpose reads them where a full decode reads implicit zeros. With that, a padded window reproduces the same frames decoded unpadded to 135 dB.
• Snake's sin to a range-reduced minimax polynomial, and its 1/alpha folded into the following convolution. librknnrt has no NPU Sin: the 48 sine nodes of a 128-frame window ran on CPU and were 41% of the window's time (350 ms of 846 ms, RKNN_LOG_LEVEL=5 operator ranking). The polynomial is the fold (below) plus 3 Mul + 2 Add at degree 5, and degree 5, 7 and 9 measure the same 50.0 dB against the exact sine once the argument is folded, so the cheap fit wins. The alpha-fold is exact algebra and removes one multiply and one divide per Snake; elementwise ops are 84% of this graph's NPU time and the convolutions 15%, so removing full-tensor passes is where the wall clock is.
• --no-sin-reduce clips the Snake sine argument at ±pi instead of folding it (the pre-fold export, kept for reproducibility). The clip is what sets a 30.4 dB floor: 1.6% of the 49M Snake arguments of a 200-frame decode are past pi (max 20.5), and the clip evaluates the polynomial on a clamped argument, so degrees 5, 7 and 9 all plateau at 17.6 dB sine-level SNR. The fold k = Floor(x/(2*pi) + 0.5); xr = x - k*2*pi (Mul/Add/Floor/Mul/Sub, all NPU-native) puts the polynomial on xr in [-pi, pi], where the fit is valid: 41.7 dB sine-level at degree 5 (70.5 at 7, 103.1 at 9) and 50.0 dB at the generator output against the exact sine (clipped form: 30.4). The fold is not free: measured on the board with both engines built from the same producer revision and interleaved three times per utterance, it costs +36% generator time (RU 33.6 s: 28400 ms median against 20849, RTF 1.074 against 0.851; EN 5.5 s: 4482 against 3239 ms, +38%), and the NPU load says why — the clipped engine holds Core0 at 86-94% during a run, the folded one at 70-72%, so the Floor chain does not stay on the NPU end to end. Both engines log 0 REGTASK lines on RK3588 (the 352-line count belongs to the RK3566 build, above). The two NPU renders differ by 17.3 dB (RU; 17.0 dB on EN, max|d| 8020 of 32767), i.e. the change is audible, and in a listening test the folded engine ranks between the CPU render and the clipped one. The default is the fold on quality, not on speed. The CPU-vs-NPU SNR of both (0.46/0.48 dB) is the documented per-window AdaIN renormalization, not the sine.

The exported graph is checked against the torch graph it was traced from, on the last window of the runtime's walk, with the torch reference evaluated in the same sine form (124.6 dB), so the gate measures the export and not the approximation.

Verified

• Export stage: stepwise-vs-full-decode self-check 111.4 dB at B=64 and 103.7 dB at B=16, cache_numel 1303424, head_dim 64, kv_len 71.
• Conversion stage: converting a reference ONNX reproduces the reference .rknn byte for byte (identical MD5, 259688011 bytes).
• Kokoro export stage: the re-traced generator reproduces the runtime's own generator at 60.5 dB interior SNR against the runtime's own generator, a padded 128-frame window reproduces the same decoded unpadded at 135.1 dB, and the exported ONNX reproduces the torch graph it was traced from at 124.6 dB.

Component execution plans

Each participating model exposes one JSON-valued session option. Its keys name the configurable components:

Component	Option	Implementations	Artifact roles
Piper decoder	piper_tts.exec_plan	ggml, rknn	RKNN: required main
Piper duration_encoder	piper_tts.exec_plan	ggml, rknn	RKNN: required main (comma-separated ladder of per-width engines)
Piper duration_flows	piper_tts.exec_plan	ggml	None (backend may name cpu)
Kokoro generator	kokoro_tts.exec_plan	ggml, rknn	RKNN: required main
Qwen3 speech_decoder	qwen3_tts.exec_plan	ggml, rknn	RKNN: required main, optional first, window
sanoTTS decoder	sanotts.exec_plan	default, ggml, neon	None
Inflect v2 decoder	inflect_v2.exec_plan	ggml, rknn	RKNN: required main
CosyVoice3 flow	cosyvoice3.exec_plan	ggml, rknn	RKNN: required main (comma-separated ladder of per-length engines)

These options also work with older GGUFs whose embedded specs do not declare exec_plan; the model validates the plan without requiring a spec override. The replaced RKNN/NEON option names are not retained as aliases.

Each component uses the same optional fields: impl, backend, artifacts (an object of named file paths), and params (an object of implementation-specific settings). Omit the option, use {}, omit a component, or give it an empty object or impl=default to retain that component's existing selection. Explicit RKNN selection requires its compiled files and an RKNN-enabled build; files alone do not select RKNN. Unknown fields, unsupported implementations/backends, missing required artifacts, unknown components/artifact roles, and unsupported params are errors. An artifact path is used as given, except that a name that exists under the model's own directory — the directory --model names, the same root a GGUF sidecar is found under — resolves to it. So an engine installed with the weights is nameable by file ("main":"vocoder_stateful_64b_fp16.rknn"), while a board path, or a path relative to the process working directory, keeps working as before. Kokoro's RKNN generator accepts params.overlap (nonnegative integer, default 0) and params.gain (boolean, default false). The other implementations currently accept no params. Parameter types and bounds are checked by the model, including the engine-dependent emitted-window limit for overlap.

The current GGML components retain their session placement: an explicit backend must match the session's backend selection. The Piper duration stages are the exception: they accept the session backend or cpu, which is how a device session runs the small duration path on the host. NEON requires an AArch64 CPU session. RKNN selects the NPU through its implementation; omit backend rather than naming a ggml backend. This is not a global backend switch or a generic pipeline system.

CLI session options carry JSON text. Server configs also accept a JSON object:

"session_options": {
  "piper_tts.exec_plan": {
    "decoder": {
      "impl": "rknn",
      "artifacts": {"main": "/models/decoder.rknn"}
    }
  }
}

Run

./build/bin/audiocpp_cli --task tts --family qwen3_tts --mode streaming \
  --model models/Qwen3-TTS-12Hz-0.6B-Base-GGUF --text "..." --voice-ref ref.wav \
  --out out.wav \
  --session-option qwen3_tts.conv_weight_type=f16 \
  --session-option 'qwen3_tts.exec_plan={"speech_decoder":{"impl":"rknn","artifacts":{"main":"/root/tts-rknn/vocoder/vocoder_stateful_64b_fp16.rknn","first":"/root/tts-rknn/vocoder/vocoder_stateful_16_fp16.rknn"}}}'

artifacts.first is optional and only affects streaming: the first token group runs on the small-batch model, which is what cuts time-to-first-audio. When the RKNN decoder is active the ggml decoder weights are not loaded at all, which is where most of the memory saving comes from.

artifacts.window switches streaming to the ggml path's window semantics: every event runs a full decode of its whole [left context + new] window from a zeroed cache (the stateful graph again, so cache_out is ignored) and the left context's audio is dropped. The stateful default voices each token once, which is why it beats the CPU streaming decoder on speed and misses it on waveform; the window role re-voices the context and reproduces the CPU streaming function per chunk. It costs one engine step of 25 + frames_per_event tokens per event. Naming the main engine ("window":"vocoder_stateful_64b_fp16.rknn") aliases it — the resident B=64 step covers the default 25 + 12 window at one resident context; a dedicated smaller export compiled with --codes-length 25+stream_frames_per_event is cheaper per event. Offline decoding is unaffected.

Piper takes one option and is offline only:

./build/bin/audiocpp_cli --task tts --family piper_tts \
  --model models/Piper-TTS-GGUF/piper-en-us-lessac-medium-orig.gguf \
  --text "..." --text-chunk-size 60 --out out.wav \
  --session-option 'piper_tts.exec_plan={"decoder":{"impl":"rknn","artifacts":{"main":"/root/tts-rknn/piper-lessac-w93.rknn"}}}'

The same option pins the duration stages to the host, with the decoder untouched:

  --session-option 'piper_tts.exec_plan={"duration_encoder":{"backend":"cpu"},"duration_flows":{"backend":"cpu"}}'

and it moves the duration encoder to the NPU, over a comma-separated ladder of per-width engines (the same producer as the decoder, --encoder; short chunks pick the narrow engine, chunks wider than the ladder stay on the ggml encoder):

  --session-option 'piper_tts.exec_plan={"decoder":{"impl":"rknn","artifacts":{"main":"/root/tts-rknn/piper-lessac-w93.rknn"}},"duration_encoder":{"impl":"rknn","artifacts":{"main":"/root/tts-rknn/enc/encoder_n128.rknn,/root/tts-rknn/enc/encoder_n192.rknn"}}}'

The engine is per voice: a .rknn compiled from lessac's latent is validated against the loaded voice's channel count at init and fails with a message naming the producer if it does not match.

Kokoro takes the engine plus two measured parameters on its window walk and is offline only (generator.params.overlap, generator.params.gain inside the execution plan):

./build/bin/audiocpp_cli --task tts --family kokoro_tts \
  --model models/Kokoro-82M-GGUF/kokoro-82m-q8_0.gguf \
  --text "..." --seed 1234 --out out.wav \
  --session-option 'kokoro_tts.exec_plan={"generator":{"impl":"rknn","artifacts":{"main":"/root/tts-rknn/gen_w128.rknn"},"params":{"overlap":0,"gain":false}}}'

The engine is per model, not per voice: the generator weights are in the GGUF, and the engine is validated against the loaded model's latent channel count, its spectrogram row count and its audio samples per latent frame at init.

..._overlap advances the walk by emitted - overlap frames and blends the shared columns with a raised cosine (1.54x wall, no change in the per-window loudness spread, so it is a seam knob, not a loudness one). ..._gain corrects each window's loudness toward a full-length CPU decode of the same latent and is also what enables the near-Nyquist cap in the Kokoro measurements below (1.9x wall, +601 MB RSS, because the ggml generator graph stays loaded and runs once per utterance). Both default to the shipping walk.

Inflect v2 takes one option and is offline only:

./build/bin/audiocpp_cli --task tts --family inflect_v2 \
  --model models/Inflect-Micro-v2-GGUF/inflect-micro-v2-q8_0.gguf \
  --text "..." --seed 1234 --out out.wav \
  --session-option 'inflect_v2.exec_plan={"decoder":{"impl":"rknn","artifacts":{"main":"/root/tts-rknn/inflect-v2-micro-decoder-rk3588.rknn"}}}'

The engine is per model (Micro and Nano differ in latent width), validated against the loaded model's latent channel count and its samples-per-frame hop at init.

Measured on RK3588

300 Russian characters, greedy decoding, conv_weight_type=f16, all other stages on CPU:

mode	RTF / wall	peak RSS	decoder ms per token
offline, ggml decoder	3.01	4200 MB	130.1
offline, RKNN B=64	2.17	2560 MB	64.1
streaming, ggml decoder	151.2 s	-	365
streaming, RKNN 16+64	57.7 s, first audio 0.88 s	-	62.4
streaming, RKNN 16+64 window (main aliased)	93.2 s, first chunk 3.07 s	-	47.9

Streaming gains more than offline because the ggml streaming decoder re-voices the left-context tokens in every window, while the stateful model voices each token once. The window role buys the quality back at that same price: the same run re-voicing every window costs the extra 48 s of decoder and lands at 47.4 dB against the ggml streaming render instead of 16.4 dB.

Quality, split by what varies, because the numbers below are not comparable to each other and one of the rows is not about the NPU at all:

pair	SNR	envelope corr
backend, offline, 185 chars	27 dB	-
backend, offline, 307 chars	25.1 dB	0.9999
backend, streaming, 307 chars	16.4 dB	0.9988
backend, streaming window, 307 chars	47.4 dB	-
backend, RKNN B=16 vs B=64	57 dB	-
mode, same backend (streaming vs offline), 307 chars	-2.2 dB	0.187

The last row is not a sidecar cost. It measures the same on the CPU graph path (-2.24 dB on CPU against -2.23 dB on RKNN, one binary, one voice reference, seed=42, conv_weight_type=f16, 307 Russian characters), so it is the streaming mode, not the NPU. Cross-correlation alignment moves it by 0.1 dB at a -120 ms lag, so it is not a time shift either: the 20 ms RMS envelope correlation of 0.187 says the prosody and timing diverge across the whole phrase, which is what two valid syntheses of one text look like. GigaAM v3 CTC (16 kHz) transcribes all five renders of that matrix to the input text exactly (CER 0.00%). The backend rows are the same signal: in the offline row the per-second dips are confined to near-silence (the fp16 noise floor) and there is no discontinuity at the group boundaries.

Piper, measured on the same board

piper-en-us-lessac-medium-orig.gguf, 443 characters at --text-chunk-size 60 (10 chunks, 25.18 s of audio), seed 1234:

path	RTF	peak RSS	decoder ms
ggml decoder	0.198	299 MB	2070
RKNN decoder (93-slice window)	0.0696	175 MB	1030

2.85x wall, 2.0x on the decoder itself (0.95 → 0.46 ms per emitted slice), -42% RSS. The decoder-side headroom is smaller than the projection from paroli's numbers because the window carries 24+24 slices of context for 45 emitted slices, and this board's ggml VITS decoder is already fast (RTF 0.20).

Quality: ggml-vs-RKNN output SNR 24.1 dB global, 40 dB median per 256-sample slice, no alignment drift (best lag 0 samples in every window), and the max sample-to-sample step is unchanged (13384 vs 13412), so no seam clicks. The dips are near-silence (LSB noise) and one loud plosive onset, i.e. the fp16 noise floor, not the stitching.

The duration encoder ladder is a small, width-limited win, and the width limit is the quality story. On a 557-character mixed RU/EN text (8 chunks, 109-459 tokens each), the narrow bins are faster than the ggml encoder (63-71 ms vs 84-89 ms at 109-115 tokens, 1.25-1.35x) and the wide bin is slower (688-773 ms vs 440-758 ms at 285-373 tokens: 2.9 GB of Toeplitz weights saturate DRAM bandwidth), so a {128,192} ladder with a ggml fallback above 192 is the shape worth shipping: it moves 3 of the 8 chunks and costs nothing on the rest. Per-bin engines score 40-48 dB against fp32 on the board (stats and conditioning, n in {17, N-1, N}). Quality: the fp16 engine shifts each chunk's duration sum by the sum of its per-token sampling errors, so the drift grows with width — ±7% latent frames on the 128-bin chunks (the scale of the flow's own seeded sampling noise) against a one-sided −21…−23% on the 384-bin chunks, a 25% tempo shift, audible. GigaAM/ Whisper transcribe the narrow-bin render and the ggml render to the same text. The wide bins are fp16-exact per frame and systematically bad per utterance: keep the ladder narrow.

Kokoro, measured on the same board

kokoro-82m-q8_0.gguf, 185 Russian characters, seed 1234, --threads 4 (14.85 s of audio, 1188 latent frames, 17 windows of 72 emitted frames). The Kokoro voice set has no Russian voice (frontend.cpp accepts a, b, e, f, h, i, j, p, z; --language russian throws unsupported Kokoro language: russian), so this utterance is Russian text voiced through --voice-id af_heart, i. e. the built-in eSpeak-ng G2P applied to Cyrillic — a language transfer, not a supported Russian path, and a reproduction has to use the same voice and text:

path	RTF	peak RSS	generator ms
ggml generator	1.415	2777 MB	17748
RKNN generator (128-frame window, clipped sine)	0.867	2334 MB	9609

1.63x wall, 1.85x on the generator itself (14.9 → 8.1 ms per latent frame processed), -16% RSS. The shipping default is the folded engine, and it is slower than this row: on a 224-character (33.6 s) Russian utterance, engines built from one producer revision and interleaved three times, the folded engine measures RTF 1.074 / 28400 ms against the clipped engine's 0.851 / 20849 ms (+36%, see the sine bullet above). Quote the clipped row as the clipped engine's figure, not as the default's. The generator-side gain is smaller than the 5.1x measured in the reference Kokoro engine because this window carries 28+28 frames of context for 72 emitted frames (1.78x of compute spent on context, and the 28 is the exact receptive field, so it cannot shrink), and because the window width is capped at 136 frames by the register-field limit, which is what forces the 17 windows.

Quality: ggml-vs-RKNN output SNR 0.49 dB. That number is not a defect measurement and has to be read against what it compares. The generator normalizes per window (AdaIN), so a multi-window decode is a different valid synthesis and not an approximation of the full-length one: dropping 2 of 486 latent frames from the statistics moves the generator output by 25% RMS. What is measured is exactness per window — a padded window reproduces the same frames decoded unpadded to 135.1 dB — and the module reproduces the runtime's own generator to 60.5 dB interior (the floor is the q8_0 style affine). The reference Kokoro engine, with the same per-window normalization at T_fix=50, measures 17.11 dB against its own CPU decode on a 7.1 s utterance (2 windows); the gap between its 17 dB and our 0.5 dB is window count, not stitching. Utterances of 72 latent frames or fewer (about 0.9 s) fit one window and are exact.

The NPU render also carries 2-3 content-locked near-Nyquist clicks per 26 s render: at the columns where the graph's activations peak, the top iSTFT bins run 10-24x the CPU render's 9-12 kHz energy there. They are fp16 rounding in the convolutional accumulation, not the sine and not the seams - the clip sine, the range-reduced sine, the exact Sin node, the unfolded Snake (--no-snake-fold, which bounds the convolution input) and the w128/w136 grids all leave the same columns at the same magnitude - and RKNN 2.3.2 has no per-layer dtype knob. params.gain=true in the generator plan caps the top five log-magnitude rows at the full-length CPU decode's level at every column: on a 25.9 s render the worst of the top eight columns goes 14.7x -> 2.8x (shipping, no gain: 23.3x), for 1.9x wall and +601 MB RSS, because the ggml generator graph stays loaded and runs once per utterance. The residual 1.4-2.8x is the iSTFT Hann-overlap smear into the bins the cap does not touch.

The talker is the only DRAM-bound candidate, and it is not at the ceiling

The sidecars and the code predictor read 0.08-0.72 GB/s of weights, 0.3-3% of the board's 26.4-29.4 GB/s ceiling, so shrinking their weights cannot pay. The talker decode is the one component that streams its whole weight set per token: 196 Q8_0 matmuls plus codec_head = 471.3 MB per token, at 41.0 ms/token measured (36.5 ms/token documented) = 11.5-12.9 GB/s, i.e. 44-49% of the ceiling. Below it, and bytes convert to time sublinearly. Same conditions as above (307 Russian characters, greedy seed=42, --threads 4, conv_weight_type=f16, vocoder on the NPU), talker matmuls re-quantized with audiocpp_gguf --type orig --keep-type <name>=q6_k (q4_k_m is not a supported output type in this converter, so q4_k):

talker weights	MB/token	talker ms/token	RTF	peak RSS
Q8_0 (shipped)	471.3	41.0	2.820	3144
Q6_K	364.6	41.6	2.740	3044
Q4_K	251.0	31.4	2.519	2934

Q4_K is −47% bytes for −23% talker time and −10.7% RTF; Q6_K is −23% bytes for nothing — the dequant of a 6-bit K-quant eats the saving. No gate applies to these renders: a re-quantized talker changes the greedy argmax trajectory, so each variant is a separate valid synthesis (inter-run correlation 0.003/−0.012, per-sample SNR −2.5/−3.3 dB is trajectory divergence, not noise; the runs are 286/300/272 tokens, so RTF compares audio-seconds across syntheses). Listening verdict: Q4_K passes against the shipped render, Q6_K shows a slight split on the opening breath (not judged critical) — which is the weaker result of the two, and Q6_K is also the one that buys no time. So this is a measured lever, not a shipped default.

NPU core use

rknn.config in 2.3.2 has no core_num (it has single_core_mode), so an engine is built multi-core and the runtime core mask decides. RKNN_NPU_CORE_AUTO lets the driver pick, and it picks one core for every one of these engines: /sys/kernel/debug/rknpu/load stays at Core0: 55%, Core1: 0%, Core2: 0% for the whole qwen3_tts vocoder run and at Core0: 100% for the smaller ones. All three sidecars therefore ask for RKNN_NPU_CORE_0_1_2 explicitly and fall back to AUTO if the driver rejects it (single-core SoCs do). What the mask buys is per engine: piper_tts and kokoro_tts get real spread (kokoro Core0 95%, Core1 18%, Core2 19%), and the qwen3_tts vocoder gets spread too (41/15/16) for 1.8% on the stage (21766 → 21383 ms for 300 Russian characters, same binary with both masks, greedy). The reason the extra cores buy almost nothing is not memory bandwidth: measured on this board, the engines stream their weights at 0.08-0.10 GB/s (Kokoro 74.68 MB per window over 763 ms of window time, Qwen3 vocoder 259.69 MB per 64-token step over 3090 ms), against a DRAM read ceiling of 26.4-29.4 GB/s measured with a cache-bypassing STREAM triad on the A76 cluster (4 threads, 804 MB working set; sysbench with a 1 MB buffer reports a fake 61 GB/s because it never leaves L3). They sit at 0.3% of the ceiling, so they are compute- and scheduler-bound, and the headroom is in op scheduling, weight reuse across steps and tiling — not in shrinking the weights.

One context per sidecar, and one job per context: the mask spreads a single job over the cores (kokoro measures Core0 95%, Core1 18%, Core2 19% during a run). Running the windows concurrently on three rknn_dup_context contexts, which is what the reference Kokoro engine does, was built and measured here at 1.15x on the window loop (9650 → 8375 ms) with corrupted output (-7 dB SNR, rms 2x the reference): duplicated contexts share the runtime's tensor staging, and the gain is small because the windows are compute-bound (0.098 GB/s of weight traffic against a 26.4-29.4 GB/s ceiling, above), so three cores sharing one job beat three jobs on three cores.

The core mask is therefore also a numerics knob, and it is why every gate in this document is a tolerance gate. The reverse-engineered RK3588 notes record (erratum E7) that the DPU's fp16 results differ per core on lane 0 of each 16-channel group, so which cores an op is scheduled on can change the fp16 value it returns; the mask is not a pure scheduler decision. Consequence for this path: "bit-identical" is a claim that holds for the CPU greedy paths (the export gates, the gated-cast change on the f16 package), never for an NPU render. Every NPU number here is SNR and max|d| against the CPU render, and the observed NPU-to-NPU spread is of the same order as the NPU-to-CPU one (the two Kokoro sine engines differ by 28.2 dB from each other). Not characterized here: the per-core fp16 difference itself, which would need the same engine run with a single-core mask and the outputs compared.

The IOVA ceiling, i.e. how big an engine can be

An engine is mapped into the NPU's I/O address space, and on RK3588 that space is a 32-bit arena. A single mapping above ~4 GB is refused at rknn_init, before the first step, and the driver says so in dmesg:

rknn_init failed with code -4
RKNPU: failed to allocate IOVA: -12
rknpu_gem_get_pages: dma map 4710850560 fail
failed to malloc npu memory, size: 6189218496, flags: 0x2

Measured on this board (31 GiB RAM, 6.1.115-vendor-rk35xx) with a flow-matching DiT estimator compiled at fixed S=900, S=1100 and S=2000, fp16, optimization_level=3: the requests are 4.71 GB, 4.66 GB and 5.77 GB, and all three fail. It is not host memory (MemTotal 32 GiB; CmaTotal 256 MiB is not the pool the driver runs out of), and it is not the multicore workspace - AUDIOCPP_RKNN_CORE_MASK=1 and =7 fail identically, so the request is the engine's own mapping. The request grows with S, so the ceiling is a length ceiling for this shape: the same estimator loads and runs at S<=500, and above the ceiling the only path is the CPU graph, which costs the same S^2 it always did (1378 frames: 186.7 s of estimator for 26.9 s of audio).

Three consequences for anyone extending this path:

• Engine size is a design constraint, not a download-size constraint. The three shipped sidecars sit at 20-260 MB per engine and never approach the ceiling. A graph whose intermediates grow with the sequence length - attention at fixed S, so S^2 - reaches it in the low hundreds of frames, long before the weight set does.
• A ladder of engines has to be lazy. Four engines that each load on their own fail at rknn_init when all four are resident at once, so the ladder loads on demand and evicts. How many rungs share the aperture is a measurement, not a rule: the driver reports what a context will occupy (rknn:: engine_memory_mib: a RKNN_FLAG_COLLECT_MODEL_INFO_ONLY init plus RKNN_QUERY_MEM_SIZE, ~0.5 s per engine, no network built), and for this estimator that is 786 MiB at S=110, 1129 at 200, 1426 at 250, 2615 at 400, 3752 at 500 and 1039 at 900 -- all of them 661-714 MB images, so a context costs its graph and not its image, and the cost is not a function of the compiled length (S=900 costs less than S=500). Against a 3900 MiB per-process ceiling, measured by initializing the sets in one process, the sets that admit are 110+200+250 (3341), 110 x4 (3144), 200 x3 (3387), 250+250+110 (3638), 110+400 (3401) and 200+400 (3744), while 250+400 (4041), 110+500 (4538) and every set holding an S>=900 engine return -4. The accounting for that is shared (rknn::ApertureBudget, include/engine/framework/rknn/engine.h) and the numbers are not: the per-engine cost and the budget are measurements of one engine on one board, and they live with the sidecar that measured them.
• The failure is at init, so it is cheap to detect and cannot corrupt output: rknn_init returns -4 and dmesg names the byte count.

Who refused: the versions a failure names

Every RKNN_* failure a sidecar reports goes through one formatter, and once any context in the process has initialized, the message carries the versions the driver gave it. Measured on this board, with the aperture accounting switched off so the ladder pins every rung it declares:

CosyVoice3 flow estimator RKNN: rknn_init failed with code -6 (driver 0.9.8, api 2.3.2 (429f97ae6b@2025-04-09T09:09:27))

The code names what was refused and nobody: -13 is a target-platform mismatch, -4 and -6 are the driver refusing an init (the IOVA ceiling above, an aperture overrun here). api is the runtime library the sidecar links (librknnrt), driver is the kernel NPU driver, and Rockchip versions them apart -- the pair above is a working RK3588 on SDK 2.3.2, not a skew. The third version an operator has to line up, the rknn-toolkit2 that compiled the .rknn, is not queryable from the board, so this line is the half of the comparison the board can state; the other half is what the converter reported when it built the image.

The report is taken from the first context the driver builds (RKNN_QUERY_SDK_VERSION is per-context) and cached for the ones that fail to build, so the first engine of a process fails without it and every failure after it carries it.

What this path is not for

The talker and the code predictor stay on the CPU. Moving the code predictor to the same NPU was built and measured at 5.4x slower than the CPU path: the step is roughly 75 tiny S=1 GEMV operations, and the NPU matrix units only pay off on real GEMMs like the vocoder's 64-token chunk. The Mali GPU is 2.1x slower than the CPU for this workload.