Fabio
c36510c21e
Some checks failed
Linux Build / Linux cpu (push) Has been cancelled
Linux Build / Linux cpu-clang (push) Has been cancelled
Linux Build / Linux vulkan (push) Has been cancelled
mac-build / macOS CPU (push) Has been cancelled
Nix Build / Nix (cpu) (push) Has been cancelled
Nix Build / Nix (vulkan) (push) Has been cancelled
windows-build / Windows CPU (push) Has been cancelled
117 lines
6.1 KiB
Markdown
117 lines
6.1 KiB
Markdown
# NEON Library
|
|
|
|
The native library lives under `framework/neon/`. Its `engine_neon` target
|
|
builds independently of model code and GGML. The runtime archive includes these
|
|
objects, so consumers keep linking the same `engine_runtime` library.
|
|
|
|
Build with `-DENGINE_ENABLE_NEON=ON` to opt in. The option defaults to `OFF`.
|
|
When disabled, the shared library, Moonshine native decoder and OpenWakeWord
|
|
native embedding are excluded from compilation and linking. Their graph paths
|
|
remain available; explicit native requests report the required build option.
|
|
SanoTTS instead retains main's existing local NEON decoder and selection behavior.
|
|
Enabling the option substitutes the shared-library SanoTTS implementation behind
|
|
the same execution-plan contract. The two decoder implementations are mutually
|
|
exclusive in each build.
|
|
|
|
## Contracts
|
|
|
|
- `tensor.h`: F32 feature views, Int32 index views, and explicit channel-first or frame-major layout.
|
|
Sequence dimensions are [channels, frames]; image dimensions are
|
|
[channels, height, width].
|
|
Shapes use inline storage for up to four dimensions.
|
|
- `operators.h`: prepared convolution, linear/block-int8 linear, embedding,
|
|
rotary embedding, attention, normalization, activation and tensor composition.
|
|
Operators own immutable weights and can be shared by independent plans.
|
|
- `execution.h`: input/value ownership, shape validation, activation lifetimes,
|
|
reusable packing/attention workspace and a persistent worker team.
|
|
- `blocks.h`: residual convolution and ConvNeXt compositions built from those
|
|
ordinary operators.
|
|
|
|
Operator implementations contain the math. The executor runs a generic data-flow
|
|
plan; it does not contain decoder/CNN stage enums or model-specific blocks.
|
|
Intermediate storage is reused after its last consumer. Eligible elementwise
|
|
operators run in place on owned buffers; borrowed inputs stay read-only.
|
|
Convolution and attention can write frame-major output directly.
|
|
|
|
Models declare their architecture and retain weight-name mapping, reference/state
|
|
management and model-specific transformations. For example, sanoTTS retains its
|
|
acoustic-output/embedding folding; its convolution, normalization, activation and
|
|
residual execution use library operators. OpenWakeWord retains its frontend
|
|
transforms and streaming history.
|
|
|
|
Each selected model component has its own native architecture file:
|
|
`sanotts/neon_library_decoder.cpp`, `moonshine_asr/neon_decoder.cpp`, and
|
|
`open_wake_word/neon_embedding.cpp`. These compile with the repository's
|
|
`include/` path and `ENGINE_HAVE_NEON=1`; no GGML include path is needed. Weight names, loading, storage
|
|
conversion and backend binding live outside those files.
|
|
|
|
```cpp
|
|
#include "engine/framework/neon/execution.h"
|
|
|
|
namespace arm = engine::neon;
|
|
|
|
arm::ExecutionPlan plan;
|
|
const auto input = plan.input();
|
|
const auto output = plan.add(arm::Conv1d({2, 2, 1}, {1, 0, 0, 1}), {input});
|
|
plan.set_output(output, arm::Layout::FrameMajor);
|
|
|
|
std::vector<float> samples{1, 2, 3, 4};
|
|
const auto & result = plan.run({arm::TensorView(samples, {2, 2})}, 4);
|
|
// result.values == {1, 3, 2, 4}
|
|
```
|
|
|
|
Input storage must remain valid throughout `run()`. The returned tensor belongs
|
|
to the plan and is valid until its next run. A plan is used serially; concurrent
|
|
requests use independent plans and may share prepared operators. Changing shapes
|
|
or thread counts replans storage or replaces the worker team before dispatch.
|
|
Validation failures occur before workers execute, and the plan can be reused.
|
|
|
|
Use `retain(value)` for intermediate results that must survive the run;
|
|
`view(value)` returns a read-only view after successful execution. Moonshine
|
|
uses this for KV state, with cache ownership and advancement in the model.
|
|
Linear and embedding constructors accept weights by value: lvalues are copied
|
|
for intentional reuse, and rvalues transfer ownership into prepared operators.
|
|
|
|
## Component Selection
|
|
|
|
Use the existing `<family>.exec_plan` session option and `parse_exec_plan`.
|
|
Each model declares its component implementations; `RuntimeSessionBase` remains
|
|
unchanged. The framework validates component names, implementation names,
|
|
backends, artifacts and parameters. The model checks native availability and
|
|
chooses between its existing graph and prepared native component.
|
|
|
|
- Omission, `{}`, and `impl: "default"` preserve the model's placement defaults.
|
|
- `impl: "ggml"` selects the existing graph.
|
|
- `impl: "neon"` requires an AArch64 NEON CPU session.
|
|
- Component backend overrides must match the resolved session backend.
|
|
- These implementations accept no artifacts or parameters.
|
|
|
|
For example: `--session-option 'moonshine_asr.exec_plan={"decoder":{"impl":"neon"}}'`.
|
|
Existing embedded contracts can omit the option; model-local validation admits
|
|
only this new key, and `parse_exec_plan` still validates its contents.
|
|
|
|
Current production selections are `decoder` for sanoTTS nano voices,
|
|
`decoder` for Moonshine's complete autoregressive decoder, and `embedding` for OpenWakeWord's CNN.
|
|
sanoTTS retains its existing ARM default; Moonshine and OpenWakeWord keep their
|
|
graph defaults. There is no additional ARM-specific session selector.
|
|
|
|
Keep autoregressive caches, speaker state, weight folding tied to a model's
|
|
architecture and application policy in the model. Add a framework block when its
|
|
math, packing or workspace management can be reused, rather than moving a whole
|
|
model implementation into the framework.
|
|
|
|
## Validation
|
|
|
|
`neon_library_test` links only the native library. It checks scalar-reference
|
|
math, branching, buffer reuse, layouts, shape/thread changes, invalid-input
|
|
recovery and concurrent plans sharing weights. It includes ConvNeXt.
|
|
|
|
`neon_blocks_test` supplements this with kernel geometry, vector tails,
|
|
layouts and component selection. The sanoTTS decoder test
|
|
checks the full decoder against an independent double-precision implementation.
|
|
The independently linked Moonshine decoder test checks its whole decoder
|
|
against a double-precision reference, including cache reset and changing memory lengths.
|
|
|
|
The actual-model tests validate OpenWakeWord frontend/embedding/probabilities
|
|
against official intermediates, including
|
|
streaming reset and irregular input chunks.
|