voice_audio_multi/docs/models/nemotron_3_diar.md

Nemotron 3 Diarization

nemotron_3_diar provides native audio.cpp inference for NVIDIA Nemotron 3 Diarization. It identifies up to eight speakers by arrival order and supports offline, native-batch, and continuous streaming inference.

Model

Field	Value
Family	nemotron_3_diar
Task	diar
Modes	offline, streaming
Input	16 kHz WAV audio
Output	Speaker turns through --turns-out
Weights	BF16 GGUF (default), Q8_0 GGUF

The checkpoint is distributed under OpenMDW-1.1. GGUF weights are packaged in Nemotron-3-Diarization-GGUF.

Convert

Build audiocpp_gguf, then convert the original .nemo archive:

python tests/nemotron_3_diar/convert_gguf.py \
  --checkpoint /path/to/Nemotron-3-Diarization.nemo \
  --output-dir /path/to/staging \
  --converter /path/to/audiocpp_gguf \
  --gguf-output /path/to/Nemotron-3-Diarization-GGUF/nemotron-3-diarization-bf16.gguf \
  --type orig

The converter reads the original NeMo archive, writes canonical Safetensors staging files, and packages a self-contained GGUF with the v1 model spec.

Run

Offline:

audiocpp_cli --task diar \
  --family nemotron_3_diar \
  --model /path/to/nemotron-3-diarization-bf16.gguf \
  --backend cuda --audio meeting.wav --turns-out turns.json

The server exposes native batch inference through repeated multipart files:

curl -N http://127.0.0.1:8080/v1/batches/transcriptions \
  -F model=nemotron-3-diar \
  -F file=@/path/to/meeting-a.wav \
  -F file=@/path/to/meeting-b.wav

The SSE response emits each file's speaker_turns as soon as that result is ready. Use its index field to map results back to upload order.

Streaming with an official latency profile:

audiocpp_cli --task diar --mode streaming \
  --family nemotron_3_diar \
  --model /path/to/nemotron-3-diarization-bf16.gguf \
  --backend cuda --audio meeting.wav --turns-out turns.json \
  --session-option nemotron_3_diar.latency_profile=low

Latency Profiles

In server streaming mode, use /v1/audio/transcriptions/live for live PCM or /v1/audio/transcriptions with stream=true for a WAV upload. Configure the model with task: "diar", mode: "streaming", and the desired latency profile in session_options. See the server streaming API for request and response formats.

Streaming emits completed speaker turns. An ongoing turn appears when it ends or when the input stream closes, so time to the first turn also depends on the speech itself. The final response contains all turns, including the last open turn. The buffer latencies below describe model input buffering, not time to a completed speaker turn.

All geometry values use 80 ms encoder frames. Input-buffer latency is chunk_len + chunk_right_context.

Profile	Buffer latency	Speaker cache	FIFO	Chunk	Right context	Cache update
very_high	30.4 s	264	40	340	40	300
low	1.04 s	264	264	9	4	222
very_low	0.64 s	264	264	6	2	222
ultra_low	0.32 s	264	264	3	1	222
asr_la13	1.12 s	264	264	14	0	222
asr_la6	0.56 s	264	264	7	0	222
asr_la3	0.32 s	264	264	4	0	222
asr_la0	0.08 s	264	264	1	0	222

The asr_laN profiles match nemotron_asr speaker masking at ASR lookahead N: one ASR chunk of N + 1 frames per step and no right context, as in NeMo's multitalker pipeline. See speaker-tagged transcription. asr_la0 processes one 80 ms frame per step and is slow.

Use custom to set the five geometry controls directly.

Common Options

Option	Value	Default	Description
--audio	WAV path	required	Input audio for one request.
--turns-out	JSON path	not set	Save decoded speaker turns.
--mode	offline, streaming	offline	Select bounded offline or incremental streaming execution.
--batch-audio-dir	directory	not set	Submit all WAV files through native offline batching.

Request Options

Use these with --request-option.

Option	Value	Default	Description
speaker_threshold	0.0 to 1.0	0.5	Speaker activity threshold.
speaker_min_frames	integer >= 0	0	Minimum turn duration in 10 ms output frames.
speaker_pad_frames	integer >= 0	0	Padding around turns in 10 ms output frames.
return_frame_probabilities	true, false	false	Attach the raw 10 ms speaker-activity timeline as the speaker_probabilities artifact. With --out-dir, the CLI writes speaker_probabilities.safetensors. It holds one F32 tensor speaker_probabilities [frames, 8] and string metadata: format_version, source_family, frames, speakers, frame_hop_samples, sample_rate, latency_profile, and the five geometry values. nemotron_asr reads this file for speaker-tagged transcription.

Session Options

Use these with --session-option nemotron_3_diar.<name>=<value>.

Option	Value	Default	Description
latency_profile	very_high, low, very_low, ultra_low, asr_la0, asr_la3, asr_la6, asr_la13, custom	very_high	Streaming geometry preset.
spkcache_len	integer >= 16	checkpoint value	Speaker-cache length for custom.
fifo_len	integer >= 0	checkpoint value	FIFO length for custom.
chunk_len	integer >= 1	checkpoint value	Processing chunk length for custom.
chunk_right_context	integer >= 0	checkpoint value	Future context for custom.
spkcache_update_period	integer >= 1	checkpoint value	FIFO-to-cache update period for custom.
graph_arena_mb	integer >= 1	1024	GGML graph metadata arena size.
weight_context_mb	integer >= 1	1024	GGML weight metadata arena size.
weight_type	storage type	native	Runtime weight storage type.
attention	auto, flash, eager	auto	Encoder attention lowering. auto uses flash attention on GPUs that support it and eager attention on others, such as Volta and Turing.

Validation

The output was compared with NeMo Python (SortformerEncLabelModel.forward_streaming, FP32, CPU, NVIDIA-NeMo/Speech cf724ac) and with NVIDIA's C++ port (NeMo-Speech.cpp 8c15060), using the same .nemo checkpoint (SHA-256 867c53f5...). All numbers use the native 10 ms speaker probabilities.

Case	Build	Max |Δ| vs NeMo	Mean |Δ|	Frame flips at 0.5
AMI EN2002d 20 s (319,990 samples), very_high	CPU, F32 weights	8.7e-4	2.1e-5	0
AMI EN2002d 20 s (319,990 samples), low	CPU, F32 weights	1.6e-3	2.6e-5	1
Same clip, NVIDIA C++ port, very_high	CPU, F32	1.1e-1	9.4e-4	2

The model is sensitive to tiny input changes in long streams. On a 120 s meeting with the low profile, adding inaudible noise (σ = 1e-5) to the input changes the NeMo output by 1.1% DER-equivalent and this port's output by 3.2%. This port and NeMo differ by 3.6–4.2% on the same file, which is the same order. NVIDIA's C++ port shows the same difference. Against NeMo on that file, every build falls in one band: CPU F32 4.2%, CUDA F32 4.4%, CUDA BF16 5.1%, and CUDA Q8_0 4.3%. Q8_0 therefore shows no quality cliff.

On CUDA, flash attention converts K and V to F16. Compared with the CPU F32 output, eager attention reduces the difference from 74 to 2 flipped frames on the 120 s stream and costs about 35% more time (900 s of audio in 16.8 s instead of 12.5 s on an RTX 4070 Ti SUPER). The default stays auto.

Known deviation: NeMo pads mel features to a multiple of 16 frames, and its subpixel convolution reads those padded rows at the last real 80 ms frame. This port uses zero padding there, so the final 70 ms of a file can differ by up to 0.02.