voice_audio_multi/tests/whistle_asr/MULTILINGUAL_VALIDATION.md

# Whistle five-language synthetic spot checks

Measured on 2026-10-06 using the current locally converted FP32 GGUF.

These clips test French, Spanish, Italian, Dutch, and Polish through the actual
Whistle CLI. They are synthetic prompt-agreement checks, not a general accuracy
benchmark or independently human-annotated recordings. No runtime code changes
are required for these checks.

## Inputs and observed transcripts

ElevenLabs CLI 1.4.0 generated raw `pcm_16000`. Python standard-library `wave`
wrapped it as signed 16-bit mono WAV without changing samples or resampling.
Only one voice and one short prompt per language are covered.

All ten CLI invocations exit 0. Automatic and forced-language outputs agree.
Unicode NFKC, case folding, and punctuation removal produce zero word edits
against each synthesis prompt. Accents remain significant. The CLI does not
expose the resulting transcript language field, so automatic language-selection
execution is verified, not its internal language tag.

### fr.

Prompt: Bonjour. Aujourd'hui, le ciel est clair et la lumière est douce.

Observed automatic and forced transcript: Bonjour, aujourd'hui le ciel est clair et la lumière est douce.

Input duration is 4.040250 seconds. WAV SHA-256 is
`245e4386e74a2e137c7cea0635d0e41084033354c8972ea5dc53a826643f2edf`.

Automatic inference is 182.696 ms, RTF 0.045219,
peak process working set 434 MiB.
Forced inference is 184.586 ms, RTF 0.0456867,
peak process working set 433 MiB.

### es.

Prompt: Hola. Hoy el cielo está despejado y la luz es suave.

Observed automatic and forced transcript: Hola, hoy el cielo está despejado y la luz es suave.

Input duration is 3.436562 seconds. WAV SHA-256 is
`0f7b3b1ffe169852dd3443e92e9ef36cbac210af70bc34d95d7591104d4908a9`.

Automatic inference is 168.729 ms, RTF 0.0490982,
peak process working set 433 MiB.
Forced inference is 159.688 ms, RTF 0.0464672,
peak process working set 433 MiB.

### it.

Prompt: Ciao. Oggi il cielo è sereno e la luce è delicata.

Observed automatic and forced transcript: Ciao, oggi il cielo è sereno e la luce è delicata.

Input duration is 3.483000 seconds. WAV SHA-256 is
`0fa7b9f1b8a582abae94ca23b7914de4dc66e275b448e57def683a7ccc7becfd`.

Automatic inference is 163.765 ms, RTF 0.0470183,
peak process working set 433 MiB.
Forced inference is 161.245 ms, RTF 0.0462949,
peak process working set 433 MiB.

### nl.

Prompt: Hallo. Vandaag is de lucht helder en het licht is zacht.

Observed automatic and forced transcript: Hallo, vandaag is de lucht helder en het licht is zacht.

Input duration is 3.297250 seconds. WAV SHA-256 is
`d2c0089973b8d62ef093d81b93c60d0c5c044345e7da7acb1479143976771abe`.

Automatic inference is 131.029 ms, RTF 0.0397388,
peak process working set 432 MiB.
Forced inference is 129.424 ms, RTF 0.0392522,
peak process working set 433 MiB.

### pl.

Prompt: Dzień dobry. Dzisiaj niebo jest bezchmurne, a światło jest łagodne.

Observed automatic and forced transcript: Dzień dobry, dzisiaj niebo jest bezchmurne, a światło jest łagodne.

Input duration is 4.226063 seconds. WAV SHA-256 is
`8356c1fb3ff392a0f099a5f95e3ce1a6ee21cbb430f945a8be25d536395bc94b`.

Automatic inference is 241.06 ms, RTF 0.0570413,
peak process working set 434 MiB.
Forced inference is 225.295 ms, RTF 0.0533109,
peak process working set 434 MiB.

## Quantized reference comparison

The cached official `needle.exe` reference ran on all five WAVs with automatic
language selection and four threads. All five invocations exit 0. It reports
`fr`, `es`, `it`, `nl`, and `pl` respectively. French, Spanish, Italian, and Dutch
reference text exactly matches the FP32 CLI text recorded above.

The Polish reference output differs:

```text
Dzień dobry, dzisiaj nie bo jest bezchmurne, a światło jest łagodne.
```

The FP32 output uses `niebo`, matching the synthesis prompt. The quantized
reference splits this as `nie bo`, producing two normalized word edits.
This discrepancy is retained, not described as exact transcript parity.
Neither transcript is an independent human annotation of the synthesized audio.

The reference engine source revision remains unknown. The `.cact` bytes match
the pinned revision and hash recorded in the validation report. These are transcript
comparisons against quantized official weights, not FP32 numerical parity.
Exact argument lists, exit codes, raw JSON outputs, and timing remain in
`reference-runs/results.json`. Each reference call uses this shape:

```powershell
& C:/Users/adityasharma/Projects/ansible-windows/.runtime/whistle/reference/needle.exe --model C:/Users/adityasharma/Projects/ansible-windows/.runtime/whistle/reference/whistle.cact --audio C:/Users/adityasharma/Projects/audio.cpp/build/whistle/pr-prep/multilingual/fr.wav --threads 4
```

## Synthesis settings and commands

Measured generation uses voice `JBFqnCBsd6RMkjVDRZzb` and model
`eleven_multilingual_v2`, with seed 42, stability 0.5, similarity boost 0.75,
style 0, speed 1, and speaker boost enabled. No `language_code` is passed.
The service model revision is not pinned; seeded regeneration is not guaranteed
byte-identical. Recorded audio hashes identify the actual tested artifacts.

Exact generation invocations are preserved in `generation-manifest.json`.
The same commands with resolved output paths are:

```powershell
elevenlabs text-to-speech convert --voice-id JBFqnCBsd6RMkjVDRZzb --model-id eleven_multilingual_v2 --output-format pcm_16000 --seed 42 --voice-settings.stability 0.5 --voice-settings.similarity-boost 0.75 --voice-settings.style 0 --voice-settings.speed 1 --voice-settings.use-speaker-boost true --text "Bonjour. Aujourd'hui, le ciel est clair et la lumière est douce." --output "build/whistle/pr-prep/multilingual/fr.pcm"
elevenlabs text-to-speech convert --voice-id JBFqnCBsd6RMkjVDRZzb --model-id eleven_multilingual_v2 --output-format pcm_16000 --seed 42 --voice-settings.stability 0.5 --voice-settings.similarity-boost 0.75 --voice-settings.style 0 --voice-settings.speed 1 --voice-settings.use-speaker-boost true --text "Hola. Hoy el cielo está despejado y la luz es suave." --output "build/whistle/pr-prep/multilingual/es.pcm"
elevenlabs text-to-speech convert --voice-id JBFqnCBsd6RMkjVDRZzb --model-id eleven_multilingual_v2 --output-format pcm_16000 --seed 42 --voice-settings.stability 0.5 --voice-settings.similarity-boost 0.75 --voice-settings.style 0 --voice-settings.speed 1 --voice-settings.use-speaker-boost true --text "Ciao. Oggi il cielo è sereno e la luce è delicata." --output "build/whistle/pr-prep/multilingual/it.pcm"
elevenlabs text-to-speech convert --voice-id JBFqnCBsd6RMkjVDRZzb --model-id eleven_multilingual_v2 --output-format pcm_16000 --seed 42 --voice-settings.stability 0.5 --voice-settings.similarity-boost 0.75 --voice-settings.style 0 --voice-settings.speed 1 --voice-settings.use-speaker-boost true --text "Hallo. Vandaag is de lucht helder en het licht is zacht." --output "build/whistle/pr-prep/multilingual/nl.pcm"
elevenlabs text-to-speech convert --voice-id JBFqnCBsd6RMkjVDRZzb --model-id eleven_multilingual_v2 --output-format pcm_16000 --seed 42 --voice-settings.stability 0.5 --voice-settings.similarity-boost 0.75 --voice-settings.style 0 --voice-settings.speed 1 --voice-settings.use-speaker-boost true --text "Dzień dobry. Dzisiaj niebo jest bezchmurne, a światło jest łagodne." --output "build/whistle/pr-prep/multilingual/pl.pcm"
```

An additional `Hello.` synthesis probe tested CLI binary output. It is retained
as `probe.pcm` and excluded from the five-language corpus. No corpus clips were
regenerated to hide recognition errors.

## Validation commands and artifacts

Commands run from the repository root:

```powershell
python build/whistle/pr-prep/multilingual/wrap_pcm.py
python build/whistle/pr-prep/multilingual/run_cli_multilingual.py
python -X utf8 build/whistle/pr-prep/multilingual/compare_transcripts.py > build/whistle/pr-prep/multilingual/root-comparison.json
```

The wrapping script refuses to overwrite existing WAVs. The validation script
preserves exact CLI argument lists and separate stdout/stderr logs in `runs3/`.
Each automatic call uses this shape; each forced call adds `--language <code>`:

```powershell
build/whistle/bin/audiocpp_cli.exe --task asr --family whistle_asr --backend cpu --threads 4 --model build/whistle/pr-prep/model/whistle-f32.gguf --audio build/whistle/pr-prep/multilingual/fr.wav --metrics --log
```

The FP32 weights use upstream revision
`b358ddadd89b7a713b5aa131f23032d3cca1b251`. Hardware, OS, build configuration,
and model setup are recorded in [the validation report](VALIDATION.md).
`reference-runs/` retains the five quantized reference runs.
CLI timing excludes model/session creation and input loading, but includes
request preparation. Whole-process elapsed time is recorded separately in
`runs3/results.json`; it is not used as inference latency.

All audio, scripts, hashes, and logs stay in ignored
`build/whistle/pr-prep/multilingual/`. `audio-inputs.json` records each WAV format
and hash. `multilingual-validation.json` records outputs, metrics, and comparisons.
Initial PowerShell process-monitoring attempts remain in `runs/` and `runs2/`.
The successful ten-case Python run is `runs3/`. No recordings or weights are
added to source changes.

## Post-review regression check

After the byte-fallback decoder fix, the rebuilt CLI repeated all ten requests.
All exit 0 and preserve the exact earlier transcripts. The successful command and
artifact paths are recorded in [the validation report](VALIDATION.md).

## Longer seven-language clips (2026-10-07)

Measured on macOS 27.0.1, Apple M3 Ultra, with the GGML encoder revision and the
locally converted FP32 GGUF (SHA-256 `80c42e88...`). ElevenLabs CLI 1.4.0 generated
`wav_16000` directly (16 kHz, mono, signed 16-bit) with voice `JBFqnCBsd6RMkjVDRZzb`,
model `eleven_multilingual_v2`, seed 42, and stored voice settings:

```bash
elevenlabs text-to-speech convert --voice-id JBFqnCBsd6RMkjVDRZzb --model-id eleven_multilingual_v2 --output-format wav_16000 --seed 42 --text "<prompt>" -o build/whistle/pr-prep/longform/<lang>.wav -q
```

Each clip ran through `audiocpp_cli` with automatic and forced language on CPU with 1
thread, CPU with 4 threads, and Metal with 4 threads: 45 invocations including the
repository sample, all exit 0. For every clip the six transcripts are byte-identical.
Word error rate uses NFKC normalization, case folding, and punctuation removal, against
the synthesis prompt; accents remain significant. These are synthetic single-voice
prompt-agreement checks, not an accuracy benchmark. Wall times are `metrics.wall_ms`
(preparation plus inference) with RTF; "auto" and "forced" refer to language selection.

| Clip | Seconds | Prompt chars | WER | CPU 1 thread, auto (ms / RTF) | CPU 4 threads, auto | Metal, auto | Metal, forced |
|---|---:|---:|---:|---|---|---|---|
| `en` | 20.434 | 369 | 0.0% | 817 / 0.0400 | 413 / 0.0202 | 286 / 0.0140 | 285 / 0.0140 |
| `de` | 23.917 | 409 | 6.6% | 1274 / 0.0533 | 643 / 0.0269 | 512 / 0.0214 | 514 / 0.0215 |
| `fr` | 21.362 | 389 | 6.1% | 1111 / 0.0520 | 566 / 0.0265 | 474 / 0.0222 | 490 / 0.0229 |
| `es` | 26.239 | 417 | 0.0% | 1325 / 0.0505 | 694 / 0.0265 | 565 / 0.0215 | 569 / 0.0217 |
| `it` | 24.381 | 400 | 0.0% | 1303 / 0.0534 | 631 / 0.0259 | 525 / 0.0215 | 524 / 0.0215 |
| `nl` | 22.616 | 415 | 11.8% | 947 / 0.0419 | 472 / 0.0209 | 352 / 0.0156 | 378 / 0.0167 |
| `pl` | 22.941 | 369 | 5.8% | 1376 / 0.0600 | 742 / 0.0323 | 650 / 0.0283 | 689 / 0.0300 |

Peak RSS was 752 to 781 MiB across these runs. An earlier pass with a pre-review build
of the graph showed Metal automatic-language runs 70 to 110 ms slower than forced runs;
the final build shows no such gap. All 45 final transcripts equal the earlier pass.

Word-level differences (prompt versus transcript, normalized) are the same on every
backend, so they are model behavior, not conversion or backend drift:

- `de`: `kommende` -> `komende`; `bitte` -> `witte`; `fenster während` -> `fensterwähren`.
- `fr`: `aux projets` -> `au projet`; `doux` -> `d où`.
- `nl`: `goedemorgen` -> `goede morgen`; `bewaar jullie vragen alsjeblieft` -> `bevario lievragen als je blijft`; `ongewoon` -> `ongebone`.
- `pl`: `tę porę` -> `te pore`; `okna` -> `ochna`.

Prompts, with WAV SHA-256:

- `en` (`6bdd5211db28b7d4481773651019b850d34d4b856381bee153927c36195838ee`): Good morning everyone, and thank you for joining today's session. We will begin with a short summary of last week's results, then move on to the plans for the coming quarter. Please keep your questions until the end, because we have a lot of material to cover. The weather has been unusually warm for this time of year, so the windows will stay open during the meeting.
- `de` (`48c8a1c173cda31e50854037d73c303d8775d7d241116b05faf3ad83f75c101d`): Guten Morgen und herzlich willkommen zu unserer heutigen Besprechung. Wir beginnen mit einer kurzen Zusammenfassung der Ergebnisse der letzten Woche und sprechen danach über die Pläne für das kommende Quartal. Bitte stellen Sie Ihre Fragen erst am Ende, denn wir haben viel Material zu behandeln. Das Wetter ist für diese Jahreszeit ungewöhnlich warm, deshalb bleiben die Fenster während der Sitzung geöffnet.
- `fr` (`6ea0320089d52aa1d32ff86b59f344e9e2431e2fb1dc3a4a4f05df49bd4e3c77`): Bonjour à tous et merci de participer à la réunion de ce matin. Nous commencerons par un bref résumé des résultats de la semaine dernière, puis nous passerons aux projets pour le prochain trimestre. Merci de garder vos questions pour la fin, car nous avons beaucoup de sujets à traiter. Le temps est anormalement doux pour la saison, les fenêtres resteront donc ouvertes pendant la séance.
- `es` (`13a8fb01c15fc9b547aba68c67118f54afdeebcc26654cf19454ecc2a4663c21`): Buenos días a todos y gracias por acompañarnos en la reunión de hoy. Empezaremos con un breve resumen de los resultados de la semana pasada y después hablaremos de los planes para el próximo trimestre. Por favor, guarden sus preguntas para el final, porque tenemos mucho material que revisar. El tiempo ha sido inusualmente cálido para esta época del año, así que las ventanas permanecerán abiertas durante la sesión.
- `it` (`895dd794984b481806711fde2569081092a868dfcdd76ff7b1dff5c87807d15d`): Buongiorno a tutti e grazie per aver partecipato alla riunione di oggi. Inizieremo con un breve riepilogo dei risultati della settimana scorsa, poi passeremo ai piani per il prossimo trimestre. Vi chiedo di tenere le domande per la fine, perché abbiamo molto materiale da esaminare. Il tempo è insolitamente caldo per questo periodo dell'anno, quindi le finestre resteranno aperte durante l'incontro.
- `nl` (`2231149210543b1084a51722598898de8c5b31a1013938116ae0011c4379bd1a`): Goedemorgen allemaal en bedankt dat jullie bij de vergadering van vandaag zijn. We beginnen met een korte samenvatting van de resultaten van vorige week en gaan daarna verder met de plannen voor het komende kwartaal. Bewaar jullie vragen alsjeblieft tot het einde, want we hebben veel onderwerpen te bespreken. Het weer is ongewoon warm voor deze tijd van het jaar, dus de ramen blijven tijdens de bijeenkomst open.
- `pl` (`82087183a46396c1050ec2ca323ffb62c691dc403cc30c4a9371ce6818f96f65`): Dzień dobry wszystkim i dziękuję za udział w dzisiejszym spotkaniu. Zaczniemy od krótkiego podsumowania wyników z ubiegłego tygodnia, a następnie przejdziemy do planów na nadchodzący kwartał. Proszę zachować pytania na koniec, ponieważ mamy dużo materiału do omówienia. Pogoda jest wyjątkowo ciepła jak na tę porę roku, dlatego okna pozostaną otwarte podczas spotkania.

Audio, the generation manifest, and `results.json` stay in ignored
`build/whistle/pr-prep/longform/`. No recordings enter source changes.