"""Compare Kokoro GGUF output with matching extracted-F32 reference WAVs.""" import argparse import json import math import wave from pathlib import Path import numpy as np from pesq import pesq from pystoi import stoi from scipy.signal import resample, resample_poly, stft from scipy.spatial.distance import cdist def load(path: Path): with wave.open(str(path), "rb") as wav: if wav.getnchannels() != 1 or wav.getsampwidth() != 2: raise ValueError(f"expected mono PCM16: {path}") rate = wav.getframerate() audio = np.frombuffer(wav.readframes(wav.getnframes()), dtype=" left: filters[band, left:center] = np.arange(center - left) / (center - left) if right > center: filters[band, center:right] = np.arange(right - center, 0, -1) / (right - center) return np.log(np.maximum(filters @ power, 1e-10)).T def dtw_log_mel_cosine(reference, estimate, rate): ref = log_mel(reference, rate) est = log_mel(estimate, rate) # Per-frame cosine distance emphasizes spectral shape over loudness. ref -= ref.mean(axis=1, keepdims=True) est -= est.mean(axis=1, keepdims=True) costs = cdist(ref, est, metric="cosine").astype(np.float32) rows, cols = costs.shape accumulated = np.empty_like(costs) steps = np.empty((rows, cols), dtype=np.uint8) accumulated[0, 0] = costs[0, 0] accumulated[1:, 0] = np.cumsum(costs[1:, 0]) + accumulated[0, 0] accumulated[0, 1:] = np.cumsum(costs[0, 1:]) + accumulated[0, 0] steps[1:, 0] = 1 steps[0, 1:] = 2 for row in range(1, rows): previous = accumulated[row - 1] for col in range(1, cols): options = (previous[col - 1], previous[col], accumulated[row, col - 1]) step = int(np.argmin(options)) accumulated[row, col] = costs[row, col] + options[step] steps[row, col] = step row, col = rows - 1, cols - 1 path_cost = 0.0 length = 0 while row or col: path_cost += float(costs[row, col]) length += 1 step = steps[row, col] if step == 0: row -= 1; col -= 1 elif step == 1: row -= 1 else: col -= 1 path_cost += float(costs[0, 0]) return 1.0 - path_cost / (length + 1) def compare(reference_path, candidate_path): rate, reference = load(reference_path) candidate_rate, candidate = load(candidate_path) if candidate_rate != rate: raise ValueError("sample-rate mismatch") original_candidate_size = len(candidate) # Duration prediction changes by a few frames under quantization. Normalize the # global duration before signal metrics so they measure acoustic similarity. candidate = resample(candidate, len(reference)) reference_16k = resample_poly(reference, 2, 3) candidate_16k = resample_poly(candidate, 2, 3) eps = 1e-20 return { "reference_seconds": len(reference) / rate, "candidate_seconds": original_candidate_size / rate, "duration_delta_percent": 100 * (original_candidate_size - len(reference)) / len(reference), "rms_delta_db": 20 * math.log10(max(np.sqrt(np.mean(candidate * candidate)), eps) / max(np.sqrt(np.mean(reference * reference)), eps)), "correlation": float(np.corrcoef(reference, candidate)[0, 1]), "si_sdr_db": float(si_sdr(reference, candidate)), "log_spectral_distance_db": log_spectral_distance(reference, candidate, rate), "dtw_log_mel_cosine": dtw_log_mel_cosine(reference, candidate, rate), "stoi": float(stoi(reference_16k, candidate_16k, 16000, extended=False)), "pesq_wb": float(pesq(16000, reference_16k, candidate_16k, "wb")), } parser = argparse.ArgumentParser() parser.add_argument("--root", type=Path, required=True) args = parser.parse_args() records = [] for precision in ("bf16-gguf", "q8-gguf"): for index in range(7): record = {"precision": precision, "request": index} record.update(compare(args.root / "f32-directory" / f"request_{index}.wav", args.root / precision / f"request_{index}.wav")) records.append(record) (args.root / "quality.json").write_text(json.dumps(records, indent=2), encoding="utf-8") for precision in ("bf16-gguf", "q8-gguf"): subset = [r for r in records if r["precision"] == precision] print(precision) for key in ("duration_delta_percent", "rms_delta_db", "correlation", "si_sdr_db", "log_spectral_distance_db", "dtw_log_mel_cosine", "stoi", "pesq_wb"): values = np.array([r[key] for r in subset]) print(f" {key}: mean={values.mean():.6f}, min={values.min():.6f}, max={values.max():.6f}")