voice_audio_multi/docs/maintainers/neon.md

NEON Library

The native library lives under framework/neon/. Its engine_neon target builds independently of model code and GGML. The runtime archive includes these objects, so consumers keep linking the same engine_runtime library.

Build with -DENGINE_ENABLE_NEON=ON to opt in. The option defaults to OFF. When disabled, the shared library, Moonshine native decoder and OpenWakeWord native embedding are excluded from compilation and linking. Their graph paths remain available; explicit native requests report the required build option. SanoTTS instead retains main's existing local NEON decoder and selection behavior. Enabling the option substitutes the shared-library SanoTTS implementation behind the same execution-plan contract. The two decoder implementations are mutually exclusive in each build.

Contracts

• tensor.h: F32 feature views, Int32 index views, and explicit channel-first or frame-major layout. Sequence dimensions are [channels, frames]; image dimensions are [channels, height, width]. Shapes use inline storage for up to four dimensions.
• operators.h: prepared convolution, linear/block-int8 linear, embedding, rotary embedding, attention, normalization, activation and tensor composition. Operators own immutable weights and can be shared by independent plans.
• execution.h: input/value ownership, shape validation, activation lifetimes, reusable packing/attention workspace and a persistent worker team.
• blocks.h: residual convolution and ConvNeXt compositions built from those ordinary operators.

Operator implementations contain the math. The executor runs a generic data-flow plan; it does not contain decoder/CNN stage enums or model-specific blocks. Intermediate storage is reused after its last consumer. Eligible elementwise operators run in place on owned buffers; borrowed inputs stay read-only. Convolution and attention can write frame-major output directly.

Models declare their architecture and retain weight-name mapping, reference/state management and model-specific transformations. For example, sanoTTS retains its acoustic-output/embedding folding; its convolution, normalization, activation and residual execution use library operators. OpenWakeWord retains its frontend transforms and streaming history.

Each selected model component has its own native architecture file: sanotts/neon_library_decoder.cpp, moonshine_asr/neon_decoder.cpp, and open_wake_word/neon_embedding.cpp. These compile with the repository's include/ path and ENGINE_HAVE_NEON=1; no GGML include path is needed. Weight names, loading, storage conversion and backend binding live outside those files.

#include "engine/framework/neon/execution.h"

namespace arm = engine::neon;

arm::ExecutionPlan plan;
const auto input = plan.input();
const auto output = plan.add(arm::Conv1d({2, 2, 1}, {1, 0, 0, 1}), {input});
plan.set_output(output, arm::Layout::FrameMajor);

std::vector<float> samples{1, 2, 3, 4};
const auto & result = plan.run({arm::TensorView(samples, {2, 2})}, 4);
// result.values == {1, 3, 2, 4}

Input storage must remain valid throughout run(). The returned tensor belongs to the plan and is valid until its next run. A plan is used serially; concurrent requests use independent plans and may share prepared operators. Changing shapes or thread counts replans storage or replaces the worker team before dispatch. Validation failures occur before workers execute, and the plan can be reused.

Use retain(value) for intermediate results that must survive the run; view(value) returns a read-only view after successful execution. Moonshine uses this for KV state, with cache ownership and advancement in the model. Linear and embedding constructors accept weights by value: lvalues are copied for intentional reuse, and rvalues transfer ownership into prepared operators.

Component Selection

Use the existing <family>.exec_plan session option and parse_exec_plan. Each model declares its component implementations; RuntimeSessionBase remains unchanged. The framework validates component names, implementation names, backends, artifacts and parameters. The model checks native availability and chooses between its existing graph and prepared native component.

• Omission, {}, and impl: "default" preserve the model's placement defaults.
• impl: "ggml" selects the existing graph.
• impl: "neon" requires an AArch64 NEON CPU session.
• Component backend overrides must match the resolved session backend.
• These implementations accept no artifacts or parameters.

For example: --session-option 'moonshine_asr.exec_plan={"decoder":{"impl":"neon"}}'. Existing embedded contracts can omit the option; model-local validation admits only this new key, and parse_exec_plan still validates its contents.

Current production selections are decoder for sanoTTS nano voices, decoder for Moonshine's complete autoregressive decoder, and embedding for OpenWakeWord's CNN. sanoTTS retains its existing ARM default; Moonshine and OpenWakeWord keep their graph defaults. There is no additional ARM-specific session selector.

Keep autoregressive caches, speaker state, weight folding tied to a model's architecture and application policy in the model. Add a framework block when its math, packing or workspace management can be reused, rather than moving a whole model implementation into the framework.

Validation

neon_library_test links only the native library. It checks scalar-reference math, branching, buffer reuse, layouts, shape/thread changes, invalid-input recovery and concurrent plans sharing weights. It includes ConvNeXt.

neon_blocks_test supplements this with kernel geometry, vector tails, layouts and component selection. The sanoTTS decoder test checks the full decoder against an independent double-precision implementation. The independently linked Moonshine decoder test checks its whole decoder against a double-precision reference, including cache reset and changing memory lengths.

The actual-model tests validate OpenWakeWord frontend/embedding/probabilities against official intermediates, including streaming reset and irregular input chunks.