Fabio
c36510c21e
Some checks failed
Linux Build / Linux cpu (push) Has been cancelled
Linux Build / Linux cpu-clang (push) Has been cancelled
Linux Build / Linux vulkan (push) Has been cancelled
mac-build / macOS CPU (push) Has been cancelled
Nix Build / Nix (cpu) (push) Has been cancelled
Nix Build / Nix (vulkan) (push) Has been cancelled
windows-build / Windows CPU (push) Has been cancelled
216 lines
12 KiB
Markdown
216 lines
12 KiB
Markdown
# Underdog Bark 3.0
|
|
|
|
Conway Research's **Standard, Plus and Pro are English ASR variants** of the
|
|
`underdog_bark3` community family, sharing the implementation under
|
|
`parakeet_tdt`. These are ASR models, not Bark speech-generation models.
|
|
All share the 0.6B FastConformer/TDT architecture, publisher's 250 ms prefix,
|
|
original-sample timestamps and native runtime. Stock Parakeet and Phonon keep
|
|
their behavior; no shared GGML kernels or parallel server runtime are needed.
|
|
|
|
| Variant | Pinned Hugging Face source / revision | Source weights | Converter | Recommended GGUF |
|
|
|---|---|---|---|---|
|
|
| Standard | [ConwayResearch/Underdog-Bark-3.0](https://huggingface.co/ConwayResearch/Underdog-Bark-3.0) / `85db2ad18ab41e534a222ee253c29191d77ed919` | 163 MB `model.bark` | `convert_underdog_bark3_standard.py` | `underdog-bark-3.0-mixed-f16.gguf` |
|
|
| Plus | [ConwayResearch/Underdog-Bark-3.0-Plus](https://huggingface.co/ConwayResearch/Underdog-Bark-3.0-Plus) / `d4fab2f8b95681e5ad3cfb4e8c5f4e0c009b3deb` | 180 MB `model.bark` | `convert_underdog_bark3_plus.py` | `underdog-bark-3.0-plus-mixed-f16.gguf` |
|
|
| Pro | [ConwayResearch/Underdog-Bark-3.0-Pro](https://huggingface.co/ConwayResearch/Underdog-Bark-3.0-Pro) / `4b5c51e04aa8d16dda9ecdff71bc29f7360149e4` | About 1.25 GB BF16 safetensors | `convert_underdog_bark3.py` | `underdog-bark-3.0-pro-q8_0.gguf` |
|
|
|
|
The compressed download sizes are not expanded GGUF/RAM/VRAM requirements.
|
|
All three packages select the `underdog_bark3` family. Checkpoint variant
|
|
labels describe the weights; they do not select runtime behavior. This guide covers GGUF; native compressed `.bark` loading
|
|
is proposed separately in PR #1133.
|
|
|
|
## Convert
|
|
|
|
Build shared binaries once. Disable both GPU flags for CPU-only builds; adjust
|
|
paths for your generator/platform (`.exe` on Windows).
|
|
|
|
```bash
|
|
cmake -S . -B build/bark3 -DCMAKE_BUILD_TYPE=Release \
|
|
-DAUDIOCPP_MODEL_SET=custom -DAUDIOCPP_MODELS=underdog_bark3 \
|
|
-DENGINE_ENABLE_CUDA=ON -DENGINE_ENABLE_VULKAN=ON \
|
|
-DENGINE_BUILD_TESTS=ON -DENGINE_BUILD_MODEL_TESTS=ON -DENGINE_BUILD_WARMBENCH=ON
|
|
cmake --build build/bark3 --config Release --target \
|
|
audiocpp_cli audiocpp_server audiocpp_gguf parakeet_warm_bench \
|
|
parakeet_bark3_session_test parakeet_variant_config_test
|
|
```
|
|
|
|
Conversion needs Python, `numpy`, `safetensors` and `huggingface_hub`; add
|
|
`constriction==0.5.0` for Standard/Plus or `torch` for Pro. Native inference
|
|
requires none of these Python dependencies.
|
|
|
|
### Standard or Plus
|
|
|
|
Use the Standard assignments below, or replace all five with the Plus values.
|
|
These assignments use Bash syntax; in PowerShell use `$NAME = '...'` syntax or
|
|
substitute the literal values.
|
|
|
|
```bash
|
|
NAME=Underdog-Bark-3.0
|
|
REVISION=85db2ad18ab41e534a222ee253c29191d77ed919
|
|
CONVERTER=convert_underdog_bark3_standard.py
|
|
OUTPUT=underdog-bark-3.0-mixed-f16.gguf
|
|
TYPE=mixed-f16
|
|
# Plus: NAME=Underdog-Bark-3.0-Plus
|
|
# REVISION=d4fab2f8b95681e5ad3cfb4e8c5f4e0c009b3deb
|
|
# CONVERTER=convert_underdog_bark3_plus.py
|
|
# OUTPUT=underdog-bark-3.0-plus-mixed-f16.gguf
|
|
# TYPE=mixed-f16
|
|
|
|
hf download "ConwayResearch/$NAME" --revision "$REVISION" \
|
|
--include model.bark bark_loader.py mlx/config.json NOTICE README.md coreml/LICENSE \
|
|
--local-dir "models/$NAME-source"
|
|
hf download nvidia/parakeet-tdt-0.6b-v3 \
|
|
--revision 541d1f99c6b0c3cd0b11a95167540bb8edefd82b \
|
|
--include config.json processor_config.json tokenizer.json tokenizer_config.json generation_config.json \
|
|
--local-dir "models/$NAME-source"
|
|
python "tools/community_models/$CONVERTER" \
|
|
--source "models/$NAME-source" --staging "models/$NAME-prepared" \
|
|
--gguf-tool build/bark3/bin/audiocpp_gguf --type "$TYPE" --output "models/$OUTPUT"
|
|
```
|
|
|
|
The converters verify checkpoint SHA256, pinned NVIDIA sidecars and vocabulary,
|
|
decode ANS/FP16 levels without executing downloaded Python, and prepare 699
|
|
inference tensors after dropping 24 training counters. Immutable profiles reject
|
|
staging from the other checkpoint. No NVIDIA model weights are downloaded.
|
|
|
|
### Pro
|
|
|
|
```bash
|
|
hf download ConwayResearch/Underdog-Bark-3.0-Pro \
|
|
--revision 4b5c51e04aa8d16dda9ecdff71bc29f7360149e4 \
|
|
--include model.safetensors config.json processor_config.json tokenizer.json \
|
|
tokenizer_config.json generation_config.json NOTICE README.md coreml/LICENSE \
|
|
runtime/transcribe_bark_pro.py runtime/requirements.txt \
|
|
--local-dir models/Underdog-Bark-3.0-Pro-source
|
|
python tools/community_models/convert_underdog_bark3.py \
|
|
--source models/Underdog-Bark-3.0-Pro-source --staging models/Underdog-Bark-3.0-Pro-prepared \
|
|
--gguf-tool build/bark3/bin/audiocpp_gguf --type q8_0 \
|
|
--output models/underdog-bark-3.0-pro-q8_0.gguf
|
|
```
|
|
|
|
All converters support `--prepare-only`, or reuse staging with `--pack-only`
|
|
and another `--type`/`--output`. `--overwrite` replaces generated GGUFs; sources
|
|
are never changed. Every GGUF embeds the English model spec v1, tokenizer/configs,
|
|
source/conversion provenance, NOTICE and full CC-BY-4.0 license. No unpublished
|
|
model-manager download is advertised. Projection scaling and BatchNorm variance
|
|
floor compensation match the existing loader; corrected gamma stays F32.
|
|
|
|
| Variant | Precision policy |
|
|
|---|---|
|
|
| Standard / Plus | `mixed-f16` preserves every staged FP32 value, keeping non-representable row-scale products, normalization and compensated projections in F32. Q8 keeps feed-forward/pointwise matrices in exact F16 and quantizes eligible attention matrices. Q8/BF16 are rounded derivatives. |
|
|
| Standard BF16 | Additionally retains predictor, encoder-to-joint projection and output head in F32 to protect case/duration decisions; Plus's BF16 default is unchanged. |
|
|
| Pro | BF16 is source precision; F32 is lossless expanded storage, not extra precision. Protected Q8 retains predictor/projector/output head in exact source BF16 and keeps convolutions floating point. |
|
|
|
|
Keep `matmul_weight_type`/`conv_weight_type` at `native`: explicit overrides can
|
|
round protected tensors again.
|
|
|
|
## CLI and server
|
|
|
|
Use any recommended GGUF with the same flags:
|
|
|
|
```bash
|
|
build/bark3/bin/audiocpp_cli --task asr --family underdog_bark3 \
|
|
--model-spec-override embedded --backend cuda --threads 8 \
|
|
--model models/underdog-bark-3.0-mixed-f16.gguf \
|
|
--audio input-16k-mono.wav --text-out transcript.txt --words-out words.json --metrics
|
|
|
|
ffmpeg -i input.wav -ar 16000 -ac 1 -f f32le - | \
|
|
build/bark3/bin/audiocpp_cli --task asr --family underdog_bark3 \
|
|
--model-spec-override embedded --backend cuda \
|
|
--model models/underdog-bark-3.0-mixed-f16.gguf \
|
|
--mode streaming --audio - --input-format f32le --input-rate 16000 --input-channels 1
|
|
```
|
|
|
|
Select the embedded contract explicitly; the family catalog otherwise describes
|
|
multilingual stock Parakeet. Use `--backend cpu` or `vulkan` for other devices and
|
|
`--device` to choose the intended GPU when needed.
|
|
|
|
The single [server example](../../examples/server/underdog-bark3-asr.json) has
|
|
offline and buffered-streaming entries for all three variants. Replace the paths,
|
|
or keep only the entries you use. Lazy loading and `max_loaded_models: 1` avoid
|
|
keeping all three resident together. Pro's existing IDs remain available.
|
|
|
|
| Variant | Offline model ID | Streaming model ID |
|
|
|---|---|---|
|
|
| Standard | `bark3standard` | `bark3standard-stream` |
|
|
| Plus | `bark3plus` | `bark3plus-stream` |
|
|
| Pro | `bark3` | `bark3-stream` |
|
|
|
|
```bash
|
|
build/bark3/bin/audiocpp_server --config examples/server/underdog-bark3-asr.json
|
|
curl http://127.0.0.1:8080/v1/audio/transcriptions/details \
|
|
-F model=bark3standard -F file=@input-16k-mono.wav
|
|
```
|
|
|
|
Streaming entries support the raw-PCM/SSE
|
|
[`/v1/audio/transcriptions/live`](../../app/server/api.md#post-v1audiotranscriptionslive)
|
|
route. These use the ordinary legacy server, without parallel-runtime flags.
|
|
|
|
## Input, context and timestamps
|
|
|
|
Supply original unpadded audio. The adapter adds **4000 zero samples at 16 kHz
|
|
(250 ms)** once per request/stream after resampling/downmixing. Word timestamps
|
|
subtract the prefix and stay within the recording; they follow native TDT emission
|
|
boundaries, not an independently validated forced aligner. For source parity use
|
|
identical mono 16 kHz inputs: the publisher uses a different resampler at other
|
|
rates. Streaming requires contiguous mono 16 kHz PCM chunks. Empty audio is
|
|
rejected; silence is valid.
|
|
|
|
Full context is the default. Bidirectional attention grows quadratically with
|
|
duration and retains the encoder frame limit. Explicit `offline_mode=auto`
|
|
switches to bounded windows after 30 seconds including the prefix; `long_form`
|
|
always uses windows. Windowing and offline VAD can change punctuation/case and
|
|
word spans. Enable offline-only VAD with:
|
|
|
|
```text
|
|
--request-option audio_chunk_mode=vad
|
|
--request-option audio_chunk_duration_sec=30
|
|
--session-option underdog_bark3.vad_model_path=assets/framework/models/silero_vad
|
|
```
|
|
|
|
The bundled default is Silero VAD. Each speech span receives its own prefix and
|
|
keeps the original sample clock. Optional `hotwords`/`hotwords_score` provide
|
|
native token-logit bias (also HTTP `prompt`), with comma-separated terms or JSON
|
|
word/spoken forms. Empty lists or zero score disable bias; this is an extension
|
|
to the original runner. Standard emits `Yeah.` on one-second silence; unfiltered
|
|
native inference preserves it, while VAD can skip non-speech.
|
|
|
|
Streaming is buffered, not native cache-aware: bidirectional attention and
|
|
whole-utterance normalization change earlier activations when future audio
|
|
arrives. The frontend only caches raw log-mel frames whose complete
|
|
FFT/preemphasis support fits both windows. Edges and F64 whole-window normalization
|
|
are recomputed and caches reset for every stream; encoder/context policy is unchanged.
|
|
|
|
## Validation
|
|
|
|
Evidence remains in the separate [Standard](../validation/underdog_bark3_standard.md),
|
|
[Plus](../validation/underdog_bark3_plus.md) and [Pro](../validation/underdog_bark3.md)
|
|
validation records, plus the [initial Pro comparison](../validation/underdog_bark3_original_comparison.md).
|
|
CPU/CUDA/Vulkan are tested; Metal/HIP and corpus WER are unvalidated. Fixture
|
|
matches do not imply universal rounded parity. Plus CPU Q8 omitted one comma on
|
|
the five-minute recording; mixed-F16 is recommended for Standard/Plus fidelity.
|
|
The CPU graph fix reduced the tested five-minute allocation from 35.2 GiB to
|
|
2.62 GiB without changing full-context recognition.
|
|
|
|
For references use a separate Python 3.11 environment with PyTorch 2.8.0,
|
|
Transformers 5.19.0, soundfile, soxr and librosa, plus conversion dependencies.
|
|
Original loaders may fetch processor/generation metadata, not NVIDIA weights.
|
|
The reference scripts are `bark3_standard_reference.py`, `bark3_plus_reference.py`
|
|
and `bark3_reference.py` for Standard, Plus and Pro respectively:
|
|
|
|
```bash
|
|
python tests/parakeet_tdt/bark3_standard_reference.py \
|
|
--source models/Underdog-Bark-3.0-source \
|
|
--audio tests/parakeet_tdt/assets/2086-149220-0033.wav \
|
|
--device cpu --check-official-transcribe --output build/bark3/reference.json
|
|
python tests/parakeet_tdt/validate_bark3.py --bin-dir build/bark3/bin \
|
|
--model models/underdog-bark-3.0-mixed-f16.gguf --backend cpu \
|
|
--reference build/bark3/reference.json --output-dir build/bark3/validation
|
|
python -m unittest discover -s tests/parakeet_tdt -p 'test_bark3*conversion.py'
|
|
ctest --test-dir build/bark3 -C Release \
|
|
-R 'parakeet_bark3_session_test|parakeet_variant_config_test' --output-on-failure
|
|
```
|
|
|
|
The harness reuses a session across input lengths and repeats. Unit/path tests
|
|
cover prefix clocks, streaming callbacks, request reuse and exception recovery.
|
|
Weights are CC-BY-4.0: credit Conway Research, NVIDIA Parakeet v3 and Moondream
|
|
Parakeet Ultra, retain NOTICE/license and disclose conversion/rounding changes.
|