Fabio
c36510c21e
Some checks failed
Linux Build / Linux cpu (push) Has been cancelled
Linux Build / Linux cpu-clang (push) Has been cancelled
Linux Build / Linux vulkan (push) Has been cancelled
mac-build / macOS CPU (push) Has been cancelled
Nix Build / Nix (cpu) (push) Has been cancelled
Nix Build / Nix (vulkan) (push) Has been cancelled
windows-build / Windows CPU (push) Has been cancelled
216 lines
11 KiB
Markdown
216 lines
11 KiB
Markdown
# Qwen3 ASR
|
|
|
|
For shared conversion guidance, see
|
|
[Qwen3 GGUF checkpoints](qwen3.md#gguf-checkpoints).
|
|
|
|
Qwen3 ASR transcribes speech audio. Word timestamps are produced by running the
|
|
recognized transcript through Qwen3 Forced Aligner, so `--words-out` requires
|
|
the forced aligner model path. Long audio is split inside the model session
|
|
before ASR inference; word timestamps are shifted back onto the original audio
|
|
timeline.
|
|
|
|
Streaming mode accepts live audio chunks and emits buffered transcript deltas.
|
|
Timestamp output remains offline-only because word alignment runs after the full
|
|
transcript is known.
|
|
|
|
Offline `audio_chunk_mode=auto` is the default. For transcript-only ASR, Qwen3 ASR uses
|
|
fixed chunks. When word timestamps are requested, it uses bundled Silero VAD
|
|
internally to choose speech-aware chunks before running ASR and alignment.
|
|
Streaming uses duration-based windows regardless of `audio_chunk_mode`.
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Family | `qwen3_asr` |
|
|
| Model directory | `models/Qwen3-ASR-0.6B` or `models/Qwen3-ASR-1.7B-hf` |
|
|
| Task | `asr` |
|
|
| Modes | `offline`, `streaming` |
|
|
| Input | Speech WAV through `--audio` |
|
|
| Output | Text to stdout or `--text-out`; optional word JSON through `--words-out` with a forced aligner |
|
|
|
|
```bash
|
|
audiocpp_cli --task asr --family qwen3_asr --model models/Qwen3-ASR-0.6B --backend cuda --audio speech_16k.wav --text "" --text-out transcript.txt
|
|
```
|
|
|
|
```bash
|
|
audiocpp_cli --task asr --mode streaming --family qwen3_asr --model models/Qwen3-ASR-0.6B --backend cuda --audio speech_16k.wav --text "" --request-option audio_chunk_seconds=5 --text-out transcript.txt
|
|
```
|
|
|
|
The native Hugging Face Transformers layout of `Qwen/Qwen3-ASR-1.7B-hf` is
|
|
also accepted directly. It uses `processor_config.json`, `tokenizer.json`, and
|
|
the `model.audio_tower` / `model.language_model` tensor namespaces; no model
|
|
conversion is required:
|
|
|
|
```bash
|
|
audiocpp_cli --task asr --family qwen3_asr --model models/Qwen3-ASR-1.7B-hf --backend cuda --audio speech_16k.wav --text "" --text-out transcript.txt
|
|
```
|
|
|
|
## Performance controls
|
|
|
|
Performance settings are session options. `qwen3_asr.perf_mode=off` preserves
|
|
existing decoder behavior. `cuda_perf` applies the preset below; explicit session
|
|
options always override its values, regardless of argument order.
|
|
|
|
| Session option (`qwen3_asr.` prefix) | Default / `off` | `cuda_perf` |
|
|
|---|---|---|
|
|
| `attention` | `mixed` | `flash` |
|
|
| `packed_projections` | `false` | `true` |
|
|
| `kv_cache_type` | `f32` | `f16` |
|
|
| `kv_cache_alignment` | `1` | `256` |
|
|
| `offline_batch_size` | `1` | `2` |
|
|
|
|
`mixed` retains manual prefill and flash cached decoding. `auto` uses the shared
|
|
backend attention policy; `eager` disables flash for prefill and decode; `flash`
|
|
selects flash for both. Attention selection does not affect the audio encoder or
|
|
a separately loaded forced aligner.
|
|
|
|
Packing combines QKV and gate/up projections without changing stored weight
|
|
precision. KV alignment above 1 uses cached attention graphs over an aligned
|
|
visible prefix; 1 attends over the full reserved cache. Offline batch size accepts
|
|
1 or 2 and pairs independent chunks without changing chunk boundaries, ordering,
|
|
or streaming scheduling. An unpaired tail is decoded individually. F16 KV,
|
|
attention lowering and packed/batched execution may change numerical results.
|
|
The preset can improve long-form throughput while slowing short requests; it is
|
|
opt-in.
|
|
|
|
Select the preset, including two-chunk batching:
|
|
|
|
```bash
|
|
audiocpp_cli --task asr --family qwen3_asr --model models/Qwen3-ASR-0.6B --backend cuda --audio speech.wav --session-option qwen3_asr.perf_mode=cuda_perf --log
|
|
```
|
|
|
|
Keep the preset but decode one chunk at a time:
|
|
|
|
```bash
|
|
--session-option qwen3_asr.perf_mode=cuda_perf \
|
|
--session-option qwen3_asr.offline_batch_size=1
|
|
```
|
|
|
|
Enable only packed projections, leaving the other controls at their defaults:
|
|
|
|
```bash
|
|
--session-option qwen3_asr.packed_projections=true
|
|
```
|
|
|
|
To recover default behavior while a preset is selected, explicitly set all five
|
|
controls to the Default / `off` column. Decoder scratch memory is reused whenever
|
|
the batched runtime is used; this allocation policy is independent of the preset
|
|
and does not change precision or model math.
|
|
|
|
## QwenCleo-ASR (Egyptian Arabic)
|
|
|
|
[QwenCleo-ASR](https://huggingface.co/mohammedaly22/QwenCleo-ASR) is a
|
|
Qwen3-ASR-1.7B fine-tune for Egyptian Arabic and Arabic-English code-switching.
|
|
It uses the same `qwen3_asr` family, loader, and options; only the weights
|
|
differ. The checkpoint is saved with tied word embeddings and therefore has no
|
|
`thinker.lm_head.weight`; the loader reuses `thinker.model.embed_tokens.weight`
|
|
for the output projection in that case. Model and integration by
|
|
[@Mohammedaly22](https://github.com/Mohammedaly22).
|
|
|
|
| Package | Format | Notes |
|
|
|---|---|---|
|
|
| `qwencleo_asr_q8_0` | GGUF Q8_0 | Standalone GGUF, recommended for local inference. |
|
|
| `qwencleo_asr_f16` | GGUF F16 | Closest to the original BF16 weights. |
|
|
| `qwencleo_asr_safetensors` | Safetensors | Original upstream checkpoint, loaded directly. |
|
|
|
|
Pass `--language Arabic`, as recommended by the model card; quantized packages
|
|
are less reliable at automatic language detection.
|
|
|
|
```bash
|
|
audiocpp_model_manager install qwencleo_asr_q8_0 --models-dir models
|
|
audiocpp_cli --task asr --family qwen3_asr --model models/QwenCleo-ASR-GGUF --backend cuda --audio egyptian_16k.wav --language Arabic --text "" --text-out transcript.txt
|
|
```
|
|
|
|
To build the GGUF packages from the upstream checkpoint:
|
|
|
|
```bash
|
|
audiocpp_gguf --input models/QwenCleo-ASR/model.safetensors --root models/QwenCleo-ASR --output models/QwenCleo-ASR-GGUF/qwencleo-asr-q8_0.gguf --family qwen3_asr --type q8_0 --overwrite
|
|
```
|
|
|
|
## Common Options (use directly)
|
|
|
|
| Option | Values | Default | Meaning |
|
|
|---|---|---:|---|
|
|
| `qwen3_asr.lora` | adapter directory or checkpoint file | not set | Load a PEFT LoRA at session creation. Encoder and decoder targets are supported; use an adapter trained for the same checkpoint size. Keep `adapter_config.json` beside the weights. |
|
|
| `qwen3_asr.lora_scale` | finite float | `1` | Strength multiplier applied after the adapter's alpha/r scaling. |
|
|
| `--audio` | WAV path | required | Speech audio; use 16 kHz WAV for the examples. |
|
|
| `--text` | text | empty string | Context prompt. |
|
|
| `--language` | language code | empty string | Recognition language hint. |
|
|
| `--max-tokens` | integer | checkpoint generation config; 512 fallback | Maximum decode tokens. |
|
|
| `--audio-chunk-seconds` | seconds | `30` for streaming/transcription; `15` for offline timestamps/VAD | Target/max chunk length; must be > 0 when chunking or streaming. Ignored with `audio_chunk_mode=none` offline. |
|
|
| `--audio-chunk-mode` | `auto`, `fixed`, `vad`, `none` | `auto` | Let the model choose, force fixed chunks, force internal VAD chunks, or disable model-side chunking. |
|
|
| `--text-out` | TXT path | not set | Transcript output. The transcript is also printed to stdout. |
|
|
| `--words-out` | JSON path | not set | Word timestamp output. Requires `qwen3_asr.forced_aligner_path`. |
|
|
|
|
The language, token limit, chunk mode, and chunk duration can also be supplied
|
|
through request options `language`, `max_tokens`, `audio_chunk_mode`, and
|
|
`audio_chunk_duration_sec`, respectively.
|
|
|
|
## Request Options (use with `--request-option`)
|
|
|
|
| Option | Values | Default | Meaning |
|
|
|---|---|---:|---|
|
|
| `return_timestamps` | bool | `false` | Offline only: run the configured forced aligner after ASR. Streaming rejects true, even with an aligner configured. CLI `--words-out` enables this automatically; server requests must enable it explicitly or through `default_request_options`. |
|
|
| `clamp_timestamps_to_audio` | bool | `false` | Opt-in guard for `--words-out`: keep repaired forced-aligner word spans inside each local audio chunk. Default preserves existing timestamp repair behavior. |
|
|
| `preserve_punctuation` | bool | `false` | Preserve ASR punctuation in timestamped, chunked transcript text instead of rebuilding text from aligned words. Legacy `qwen3_asr.preserve_punctuation` remains accepted. |
|
|
|
|
## Session Options (use with `--session-option`)
|
|
|
|
| Option | Values | Default | Meaning |
|
|
|---|---|---:|---|
|
|
| `qwen3_asr.forced_aligner_path` | model path | not set | Qwen3 Forced Aligner model used to generate word timestamps after ASR. |
|
|
| `qwen3_asr.vad_model_path` | model directory | `assets/framework/models/silero_vad` | Optional internal VAD model override for timestamp-safe chunking. Legacy `qwen3_asr.vad_path` remains accepted. |
|
|
| `qwen3_asr.weight_type` | `native`, `f32`, `f16`, `bf16`, `q8_0` | `native` | Text decoder weight storage fallback. |
|
|
| `qwen3_asr.audio_encoder_weight_type` | `native`, `f32`, `f16` | `native` | Audio encoder weight storage. |
|
|
| `qwen3_asr.text_decoder_weight_type` | `native`, `f32`, `f16`, `bf16`, `q8_0` | inherits `weight_type` | Explicit text decoder weight storage override. |
|
|
| `qwen3_asr.audio_encoder_graph_arena_mb` | positive integer | `128` | Audio encoder graph arena in MiB. |
|
|
| `qwen3_asr.text_decoder_prefill_graph_arena_mb` | positive integer | `256` | Text decoder prefill graph arena in MiB. |
|
|
| `qwen3_asr.text_decoder_decode_graph_arena_mb` | positive integer | `256` | Text decoder cached-step graph arena in MiB. |
|
|
| `qwen3_asr.text_decoder_weight_context_mb` | positive integer | `64` | Text decoder weight tensor context in MiB. |
|
|
|
|
Legacy session names `qwen3_asr.forced_aligner_model_path` (also
|
|
`qwen3_asr.aligner_model_path`) and `qwen3_asr.vad_model_path` remain accepted.
|
|
The normalized `qwen3_asr.text_decoder_weight_type`,
|
|
`qwen3_asr.text_decoder_prefill_graph_arena_mb`,
|
|
`qwen3_asr.text_decoder_decode_graph_arena_mb`, and
|
|
`qwen3_asr.text_decoder_weight_context_mb` also accept their existing `thinker_*`
|
|
spellings. Normalized names take precedence if both are supplied. These aliases
|
|
work with existing GGUFs without reconversion.
|
|
|
|
## Server: Transcript and Word Timestamps
|
|
|
|
Configure the aligner and VAD with paths visible to the server. Put request
|
|
defaults, including `language`, in `default_request_options`, not `session_options`.
|
|
|
|
```json
|
|
{
|
|
"id": "qwen3-asr",
|
|
"family": "qwen3_asr",
|
|
"path": "/path/to/qwen3-asr-0.6b-q8_0.gguf",
|
|
"task": "asr",
|
|
"mode": "offline",
|
|
"session_options": {
|
|
"qwen3_asr.forced_aligner_path": "/path/to/qwen3-forced-aligner-0.6b-q8_0.gguf",
|
|
"qwen3_asr.vad_model_path": "/path/to/assets/framework/models/silero_vad"
|
|
},
|
|
"default_request_options": {
|
|
"language": "English",
|
|
"return_timestamps": true
|
|
}
|
|
}
|
|
```
|
|
|
|
Add that entry to the server config's `models` array, then upload the recording:
|
|
|
|
```bash
|
|
curl http://127.0.0.1:8080/v1/audio/transcriptions/details \
|
|
-F model=qwen3-asr \
|
|
-F language=English \
|
|
-F file=@input.wav
|
|
```
|
|
|
|
The response includes `text`, `words` with `start_sample`/`end_sample`, and
|
|
`sample_rate`. Divide sample offsets by `sample_rate` for seconds. The ordinary
|
|
`/v1/audio/transcriptions` endpoint returns text and timing only, even when the
|
|
model generates word timestamps. Timestamp generation uses Silero VAD in the
|
|
default `audio_chunk_mode=auto` path.
|
|
|