Fabio
c36510c21e
Some checks failed
Linux Build / Linux cpu (push) Has been cancelled
Linux Build / Linux cpu-clang (push) Has been cancelled
Linux Build / Linux vulkan (push) Has been cancelled
mac-build / macOS CPU (push) Has been cancelled
Nix Build / Nix (cpu) (push) Has been cancelled
Nix Build / Nix (vulkan) (push) Has been cancelled
windows-build / Windows CPU (push) Has been cancelled
96 lines
5.3 KiB
Markdown
96 lines
5.3 KiB
Markdown
# Startup-selected runtime separation
|
|
|
|
Historical implementation/evidence report. The
|
|
[Parallel Server Behavior Contract](../maintainers/parallel_server_behavior_contract.md)
|
|
is authoritative for current expected behavior, tested scopes and known limits;
|
|
this report retains its recorded revision's results.
|
|
|
|
|
|
Implements the [maintainer's Stage 1 proposal](https://github.com/0xShug0/audio.cpp/pull/715#issuecomment-5963682706)
|
|
against upstream `2892ed3e94b6ccbdacc111fa2896aaae0a5dc8fc`.
|
|
This replaces the previous slot-count-selected guard inside a shared server runtime.
|
|
|
|
| Startup | Slots field | Result |
|
|
|---|---|---|
|
|
| Default | Absent | Original `ServerState` |
|
|
| Default | Any explicit value, including `1` or `null` | Startup error; dynamic registration returns HTTP 400 |
|
|
| `--parallel-jobs` | Absent | `ParallelServerState`, capacity one |
|
|
| `--parallel-jobs` | Integer 1-16 | Parallel runtime, subject to advertised model capacity |
|
|
|
|
The enabled process uses the new pool, leases and FIFO scheduler even with one
|
|
slot. Its model capacity can change through one without changing the runtime.
|
|
An unsupported count cannot publish a partial pool. Existing model adapters and
|
|
admission tables are unchanged: this foundation does not enable catalogue models
|
|
for multiple slots. Model support remains in the separate follow-up, PR #706.
|
|
|
|
## Source boundary
|
|
|
|
`runtime.h`, `busy_guard.h`, `http.h/.cpp`, `frontend.h/.cpp`, the existing session
|
|
interface, CLI files, model implementations and ggml match upstream. The sole
|
|
legacy `runtime.cpp` addition rejects explicit slots immediately after parsing
|
|
dynamic registration. Execution, ownership, loading, unloading, streaming,
|
|
batching, eviction and shutdown bodies retain upstream behavior.
|
|
|
|
`parallel_runtime.h/.cpp` owns the enabled process's registry, sessions, leases
|
|
and locks. Both classes implement the existing interfaces. Startup constructs
|
|
one stack-owned concrete class. The flagged default HTTP path now selects
|
|
`parallel_http.h/.cpp`, whose owned workers drain before runtime destruction;
|
|
default startup still selects the original listener. Custom frontend listeners
|
|
retain the existing interface and must drain their workers before returning.
|
|
The copied externally linked language-option helper has a distinct name.
|
|
|
|
The follow-up [self-audit](parallel_runtime_self_audit.md) records scoped fixes
|
|
for failure transitions, concurrent registration, transport-policy reads, bulk
|
|
barrier ordering and HTTP worker lifetime. The validation below describes the
|
|
initial separation revision; the audit has its own evidence/results.
|
|
|
|
JSON parsing synchronization is a startup-only process opt-in, enabled before
|
|
configuration parsing and concurrent users. Default startup and the CLI leave it
|
|
off. The only production direct cJSON parse caller found is the framework wrapper.
|
|
New direct callers must honor this policy.
|
|
|
|
Tensor storage and resource cache synchronization require separate explicit
|
|
construction options. Both resource-ID and canonical-path caches distinguish
|
|
source policies; indexed shards and prefix/folded/composite holders retain the
|
|
underlying source and its policy. Existing loaders keep their original calls.
|
|
|
|
Run `python tests/server/check_parallel_boundaries.py --base origin/main` after
|
|
carrying upstream updates into both runtime copies. Source boundary checks do
|
|
not certify model output or multi-slot admission.
|
|
|
|
## Local validation
|
|
|
|
Windows/MSVC, RTX 3090, CUDA 12.4 and Vulkan SDK 1.4.350.0. GPU builds select
|
|
Piper, Canary, NeuTTS and Nemotron diarization, with native model management.
|
|
|
|
| Check | Result |
|
|
|---|---|
|
|
| Protected-file and legacy-runtime boundary | Pass |
|
|
| CPU server build and focused CTests | 11/11 pass |
|
|
| Windows AddressSanitizer focused CTests | 8/8 pass |
|
|
| CUDA CLI and server build | Pass |
|
|
| Vulkan CLI and server build | Pass |
|
|
| Actual process startup and model response schema | 9/9 pass |
|
|
| Matched checkpoint integration, both backends | 96/96 requests pass |
|
|
|
|
Controlled lifecycle tests cover capacities 1, 2 and 4: cold load, queued
|
|
unload/reconfiguration, FIFO ordering, management barriers, idle-first bulk
|
|
unload, retirement under resident limits 1/2, metadata contention, deferred
|
|
stream/native-batch/generic-batch ownership, disconnect, and rejected preparation.
|
|
Additional coverage exercises count changes 1/2/1/4/1, invalid counts without
|
|
unloading existing state, concurrent valid/invalid JSON, synchronized storage
|
|
release/remapping and resource-cache policy separation.
|
|
|
|
Checkpoint integration compares current upstream, unflagged candidate, flagged
|
|
candidate with omitted slots, and flagged candidate with explicit slots=1.
|
|
Piper, Canary and Nemotron diarization have exact aligned output/metadata after
|
|
removing only top-level execution timing, across cold, two warm requests and
|
|
unload/reload on each backend. Model-list schema and drained slot counters are
|
|
checked too. This is representative capacity-one integration, not complete
|
|
catalogue admission or a throughput benchmark.
|
|
|
|
Local evidence is retained in `outputs/pr715-stage1-separation-20261003/`: proposal,
|
|
source/build/executable manifest, CMake caches and commands, CTest XML, build logs,
|
|
startup records, checkpoint harness/results/raw responses/server logs and the
|
|
boundary audit. Remote cross-platform and Linux TSan coverage is tracked by CI
|
|
on the pushed revision; local results do not substitute for those jobs.
|