# LFM2.5-Audio LFM2.5-Audio is Liquid AI's end-to-end speech and text model. audio.cpp runs its speech recognition, text-to-speech and spoken chat (speech-to-speech) as the community family `lfm2_audio`, directly from the GGUFs Liquid AI publishes, for the English and the Japanese checkpoint. Upstream: [LiquidAI/LFM2.5-Audio-1.5B](https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B) · [LiquidAI/LFM2.5-Audio-1.5B-JP](https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B-JP) · reference implementation: [liquid-audio](https://github.com/Liquid4All/liquid-audio) | Field | Value | |---|---| | Family | `lfm2_audio` | | Tasks | `asr`, `tts`, `s2s` | | Modes | `offline`; `streaming` for TTS and S2S | | Languages | `en` (`LFM2.5-Audio-1.5B`), `ja` (`LFM2.5-Audio-1.5B-JP`); each checkpoint transcribes, speaks and chats in its own language | | Audio input (ASR, S2S) | WAV at any sample rate; channels are averaged and the audio is resampled to 16 kHz | | Speech output (TTS, S2S) | 24 kHz mono speech; S2S also returns the reply's text | | Voices | TTS: `us_male` (default), `us_female`, `uk_male`, `uk_female` for English; the Japanese checkpoint has one voice. S2S replies in the checkpoint's own voice | | Backends tested | CPU, CUDA, Metal | | License | LFM Open License v1.0 | ## Architecture - NeMo 128-bin log-mel frontend and a FastConformer encoder (17 layers, 8x depthwise-striding subsampling, relative-position attention) based on canary-180m-flash, followed by an MLP adapter. The frontend configuration is the same as `canary_asr`. - LFM2 hybrid backbone: 10 gated short-convolution blocks and 6 grouped-query attention blocks, with RMSNorm, QK-norm and SwiGLU. The text head is tied to the token embedding. - ASR: greedy decoding of the chat prompt liquid-audio builds: the system prompt `Perform ASR.` (`Perform ASR in japanese.` for the JP checkpoint), then the audio embeddings as the user turn, until `<|im_end|>`. - TTS: the system prompt `Perform TTS. Use the US male voice.` (one per voice; `Perform TTS in japanese.` for the JP checkpoint) and the text as the user turn. After `<|audio_start|>` each backbone step yields one 80 ms audio frame: a 6-layer depthformer predicts its 8 codes one codebook at a time, and the frame goes back into the backbone as the sum of its code embeddings, until a frame starts with the end-of-audio code. A frame that picks end-of-audio for another codebook goes back in too but has no sound, so, as in liquid-audio's demo, it is not decoded. - S2S: liquid-audio's interleaved generation. The system prompt `Respond with interleaved text and audio.` and the user's audio as the user turn; the reply alternates 6 text tokens and 12 audio frames (9 for the JP checkpoint, whose vocoder GGUF records its blocks) until `<|text_end|>`, continues with audio until end-of-audio, and ends at `<|im_end|>`. Text is greedy and audio is sampled at temperature 1.0 with top-k 4, as in liquid-audio's README and chat demo; `text_temperature` and `text_top_k` sample the text as `generate_interleaved`'s arguments of the same names do. A later turn of a conversation puts the earlier questions and replies before the new question, as liquid-audio's ChatState does ([Conversations](#conversations)). - An LFM2 detokenizer (8 layers, causal sliding-window attention over 30 steps) turns the mean of each frame's code embeddings, repeated 6 times, into log-magnitude and phase, and an ISTFT (n_fft 1280, hop 320) gives 24 kHz audio. ## Packages Liquid AI publishes each quantization as four llama.cpp-format GGUFs in [LiquidAI/LFM2.5-Audio-1.5B-GGUF](https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B-GGUF/tree/7d525f883a077e20afb782f2ff618edcae0e39e4) and [LiquidAI/LFM2.5-Audio-1.5B-JP-GGUF](https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B-JP-GGUF/tree/64b96718b341dbd5650f9e85627cecdcbd4ac61b). The packages point at those repositories, pinned to the linked revisions: | File | Contents | Used by | |---|---|---| | `-.gguf` | LFM2 backbone and text tokenizer | ASR, TTS, S2S | | `mmproj--.gguf` | Audio encoder and adapter; the embedding of generated audio codes | ASR, TTS, S2S | | `vocoder--.gguf` | Depthformer; the detokenizer's code embedding and ISTFT window | TTS, S2S | | `tokenizer--.gguf` | Audio detokenizer | TTS, S2S | | Package | Checkpoint | Weights | Download | |---|---|---|---| | `lfm2_audio_1_5b_q8_0` (default) | EN | Q8_0 | 1.8 GB | | `lfm2_audio_1_5b_f16` | EN | F16 | 3.3 GB | | `lfm2_audio_1_5b_q4_0` | EN | Q4_0 | 1.1 GB | | `lfm2_audio_1_5b_jp_q8_0` | JP | Q8_0 | 1.8 GB | | `lfm2_audio_1_5b_jp_f16` | JP | F16 | 3.3 GB | | `lfm2_audio_1_5b_jp_f32` | JP | F32 | 6.5 GB | | `lfm2_audio_1_5b_jp_q4_0` | JP | Q4_0 | 1.1 GB | ```bash python3 tools/model_manager_v2.py install lfm2_audio_1_5b_q8_0 --models-root models python3 tools/model_manager_v2.py install lfm2_audio_1_5b_jp_q8_0 --models-root models ``` All quantizations of a checkpoint install into one directory, `models/LFM2.5-Audio-1.5B-GGUF` or `models/LFM2.5-Audio-1.5B-JP-GGUF`, so components of different quantizations can be combined with the session options below. The WebUI lists both checkpoints under ASR, TTS and speech-to-speech with their Q8_0 and F16 packages; speech-to-speech has no text box, so replies use liquid-audio's system prompt. A package installed before audio.cpp had LFM2-Audio TTS holds only the backbone and mmproj GGUFs; TTS and S2S then name the missing files, and installing it again with `--overwrite` brings all four. The WebUI's speech-to-speech page holds a conversation: each Run sends the recorded or chosen question to `/v1/tasks/stream` with the earlier turns, as [Conversations](#conversations) describes, plays the reply as it streams in and adds the turn to a list, and New conversation starts over. With `--ui-management` the page loads the entry in streaming mode itself. With a config file, live playback needs the entry in `"mode": "streaming"`; an offline entry gives each reply when it is done. ## Run ```bash audiocpp_cli --task asr --family lfm2_audio \ --model models/LFM2.5-Audio-1.5B-GGUF --backend cuda --audio speech.wav audiocpp_cli --task tts --family lfm2_audio \ --model models/LFM2.5-Audio-1.5B-GGUF --backend cuda \ --text "The next train leaves in ten minutes." --voice-id uk_female --out speech.wav audiocpp_cli --task s2s --family lfm2_audio \ --model models/LFM2.5-Audio-1.5B-GGUF --backend cuda \ --audio question.wav --out reply.wav --text-out reply.txt ``` `--model` is the package directory: the published GGUFs do not embed an audio.cpp model spec, so a GGUF file path is rejected. With one quantization installed, its backbone is used with the `mmproj-`, `vocoder-` and `tokenizer-` files of the same name. With several, choose the backbone: ```bash audiocpp_cli --task asr --family lfm2_audio \ --model models/LFM2.5-Audio-1.5B-GGUF --backend cuda --audio speech.wav \ --session-option lfm2_audio.model_gguf=LFM2.5-Audio-1.5B-F16.gguf ``` For Japanese, pass `models/LFM2.5-Audio-1.5B-JP-GGUF` and no voice. `language` can be left out; a value other than the checkpoint's language is rejected. Speech is sampled like the README's example (temperature 0.8, top-k 64); pass `--seed` for repeatable audio, or `--temperature 0` for greedy decoding. S2S answers a spoken turn with a spoken reply and its text: the CLI prints the text (`text_output=`) and `--text-out` writes it. Alone, a request is a new conversation under liquid-audio's chat system prompt, `Respond with interleaved text and audio.`; `--text` replaces that prompt, which the checkpoints were trained with, so leave it out unless experimenting. The reply is sampled like liquid-audio's README and demo (temperature 1.0, top-k 4) and may run to 1024 steps, text tokens and audio frames together, about a minute of speech. Use the Q8_0 package, the default, or F16 for S2S: with Q4_0, 2 of 30 replies to one English question on CUDA and 2 of 30 on the M3 Ultra CPU (2 of 60 on the x86 CPU, none on Metal) repeated a sentence until `max_tokens`; with Q8_0 and F16 none did (see [Validation](#s2s)). A reply that reaches `max_tokens` is cut off there, its text and speech kept, as liquid-audio keeps what it generated, and a warning goes to stderr. Greedy audio (`--temperature 0`) can go on speaking after the text until `max_tokens`, as liquid-audio's greedy decoding does: to one Japanese question both gave the same text and 972 audio frames, all 1024 steps. Text tokens stand for bytes, not characters. Bytes that make no whole character, which replies in scripts such as Thai can have, come out as U+FFFD, as the Hugging Face tokenizer decodes them, in replies and transcripts alike; a character left open at the end of the text is dropped, as in a transcript cut at `max_tokens`. A request can also carry the turns before it; see [Conversations](#conversations). The question goes through the encoder in one pass, as in liquid-audio, so it can be at most `lfm2_audio.max_pass_duration_sec` long (120 s by default; see [Long audio](#long-audio)): a longer one is rejected before it is encoded. The server takes the same directory and session options: ```json {"models": [{"id": "lfm2-audio-asr", "family": "lfm2_audio", "task": "asr", "mode": "offline", "path": "models/LFM2.5-Audio-1.5B-GGUF", "session_options": {"lfm2_audio.model_gguf": "LFM2.5-Audio-1.5B-Q8_0.gguf"}}]} ``` ```bash audiocpp_server --backend cuda --config server.json curl http://127.0.0.1:8080/v1/audio/transcriptions -F model=lfm2-audio-asr -F file=@speech.wav ``` A TTS entry is the same with `"task": "tts"`, served at `/v1/audio/speech`: ```bash curl http://127.0.0.1:8080/v1/audio/speech -H 'Content-Type: application/json' \ -d '{"model": "lfm2-audio-tts", "input": "The next train leaves in ten minutes.", "voice": "us_female"}' -o speech.wav ``` An S2S entry (`"task": "s2s"`) answers at `/v1/tasks/run` with the reply's `text` and its audio as base64 WAV (`audio`): ```bash curl http://127.0.0.1:8080/v1/tasks/run -H 'Content-Type: application/json' \ -d '{"model": "lfm2-audio-s2s", "request": {"audio": "/path/to/question.wav"}}' ``` ### Streaming TTS With `--mode streaming` (CLI) or `"mode": "streaming"` (server entry), speech comes out as it is generated. Like liquid-audio's demo, each audio frame is decoded as soon as the depthformer picks it. The detokenizer is causal and carries each layer's state from frame to frame (the attention layers' last 29 keys and values and the short-conv layers' last inputs), so a frame costs only its own six steps, and the ISTFT releases a sample once no later window reaches it. Each event carries one frame (80 ms, `stream_frames_per_event` to change it), plus a last 20 ms event per text chunk; the events add up to the offline speech for the same seed, up to the detokenizer's arithmetic (below). A text chunk that reaches `max_tokens` ends its speech there, streamed as offline, with the same warning. ```bash curl -N http://127.0.0.1:8080/v1/audio/speech -H 'Content-Type: application/json' \ -d '{"model": "lfm2-audio-tts-stream", "input": "The next train leaves in ten minutes.", "voice": "us_female", "stream": true}' ``` The response is server-sent events with base64 16-bit PCM deltas at 24 kHz; `"stream_format": "audio"` returns the raw PCM instead. Time to the first audio and real-time factor through the server, for a two-sentence English text (6.5 s of speech) and a Japanese one (8.5 s); the M3 Ultra CPU's Q8_0 and Q4_0 figures are with ggml's repacked kernels, see [CPU](#cpu): | Backend | First audio, F16 / Q8_0 / Q4_0 / JP F32 | RTF, 1 frame per event | RTF, 4 frames per event | |---|---|---|---| | CUDA, NVIDIA A10 | 39 / 27 / 24 / 45 ms | 0.09-0.21 | 0.09-0.21 | | Metal, Apple M3 Ultra | 40 / 37 / 35 / 52 ms | 0.19-0.29 | 0.19-0.27 | | CPU, Apple M3 Ultra, 16 threads | 150 / 54 / 48 / 308 ms | 0.16-0.57 | 0.16-0.57 | A frame per event costs no more than four, and streaming about what offline speech does. The streamed audio differs from offline (relative RMS) by the detokenizer's arithmetic in graphs of other sizes. With F16 that is 5e-4 to 6e-3 on the test's short text across the backends; on CUDA, whose cuBLAS accumulates the offline products in half precision, it is 5e-3 to 3.2e-2 over English and Japanese texts of 1.5 to 18 s and 2e-2 over a minute of speech. Quantized packages round the activations to 8 bits on the CPU and CUDA, and graphs of other sizes round them differently: on CUDA, over texts of 2.5 to 19 s, the stream differs by 2.2e-2 to 4.6e-2 with Q8_0 and by 0.06 to 0.32 with Q4_0 (3.4e-2 and 0.22 over a minute). The stream decodes a frame at a time, and at that size ggml-cuda multiplies Q4_0 weights with its MMVQ kernel, whose dot product makes the rounding error two to three times larger than in the kernel offline decoding uses. By log-spectral distance, the offline Q4_0 audio on CUDA is the closer to exact arithmetic with the same weights (0.21 to 0.26 dB against 0.53 to 0.64 dB for the stream). ### Streaming S2S With `--mode streaming` (CLI) or `"mode": "streaming"` (server entry), the question comes in as audio chunks, live PCM included (`--audio -` on the CLI). The reply starts when the input ends and streams like TTS: each event carries the audio of one frame (`stream_frames_per_event`) and the text written since the last event, in whole characters. Together they are the offline reply for the same seed: the same text, and the same audio up to the detokenizer's arithmetic, as for TTS. On CUDA, streamed replies to `assets/resources/c.wav` differ from offline by 4.3e-2 with F16 (one reply), 2.2e-2 to 4.5e-2 with Q8_0 and 0.06 to 0.15 with Q4_0 (five replies each). A reply cut off at `max_tokens` streams as it does offline, with the same warning; a character left open at the cut is dropped in both. A stream finished before its reply is over returns the text its events carried and, with `return_codes`, the reply's steps so far. A question longer than `lfm2_audio.max_pass_duration_sec` fails as soon as that much of it has come, without waiting for the rest. The server's live route, `/v1/audio/speech/live`, takes the question as chunked raw PCM. Without `input` the reply uses liquid-audio's system prompt; an `input` replaces it, as `--text` does. ```bash ffmpeg -i question.wav -ar 16000 -ac 1 -f s16le - \ | curl -N -X POST -H 'Expect:' -T - \ 'http://127.0.0.1:8080/v1/audio/speech/live?model=lfm2-audio-s2s-stream&sample_rate=16000&channels=1&sample_format=s16le&return_text=true' ``` It returns the reply's audio as server-sent events and, with `return_text=true`, its text: `speech.text.delta` events carry what the reply wrote since the previous event, ahead of the audio that speaks it, and `speech.text.done` the whole text. The query also takes the request options `seed`, `temperature`, `top_k`, `text_temperature`, `text_top_k` and `max_tokens`. Time from the end of a 7.5 s English question, streamed in real time, to the first audio of the reply (the M3 Ultra CPU's Q8_0 and Q4_0 figures with ggml's repacked kernels, see [CPU](#cpu)): | Backend | F16 | Q8_0 | Q4_0 | |---|---|---|---| | CUDA, NVIDIA A10 | 107 ms | 106 ms | 73 ms | | Metal, Apple M3 Ultra | 183 ms | 168 ms | 153 ms | | CPU, Apple M3 Ultra, 16 threads | 478 ms | 228 ms | 205 ms | That covers encoding the whole question, the prompt, the first text block and the first frame. With the JP F32 package on CUDA, the first audio came 113 ms after a 2.6 s Japanese question. Offline, `/v1/tasks/run` returned a 13.1 s English reply in 2.1 s on CUDA (F16). ## Conversations liquid-audio's chat demo keeps every turn of a conversation and prefills all of it again for each reply. audio.cpp gives each reply the same prompt, but the client keeps the conversation: it sends the earlier turns with each new question, as the request's input artifacts, in conversation order. For each earlier turn, its question and then its reply: | Artifact id | Kind | Payload | |---|---|---| | `lfm2_audio.question` | `custom` | The question's audio as a WAV file, with the samples that turn's request had (a float WAV holds any exactly) | | `lfm2_audio.reply` | `acoustic_tokens` | The reply artifact that turn's result returned, payload and meta unchanged | The request's audio is the new question. With `return_codes=true` the result, offline or streamed, also returns the reply as an `lfm2_audio.reply` artifact for the next turn. The prompt is the one ChatState holds in liquid-audio's README multi-turn example: the system prompt once, each earlier question and its reply in the chat markup, then the new question. A reply goes back in step by step, its text as the token ids it generated and its frames as the sum of their code embeddings, as liquid-audio's `_prefill` takes them. This prompt differs from the chat demo's on purpose. `demo/chat.py`, as written, adds the system turn again after the open user turn on every turn, since its `len(chat.text) == 1` check always holds; audio.cpp keeps it once. In ten-turn English conversations run with liquid-audio, replies under the repeated system turn answered 1 of 21 questions about earlier turns, and 7 of 21 with the system prompt once. The reply artifact's payload is little-endian int32 values, one step after another: a text token is its id, and an audio frame is -1 followed by its 8 codes. It holds every step of the reply, `<|text_end|>`, the frame that ends the audio and frames with end-of-audio in another codebook included, as liquid-audio keeps them: about 470 bytes per second of reply audio. Its meta: | Key | Value | |---|---| | `format` | `lfm2_audio.reply/1` | | `language` | The checkpoint's language, `en` or `ja` | | `codebooks`, `audio_vocab_size`, `text_vocab_size` | The checkpoint's codebooks (8), codes per codebook with end-of-audio (2049) and text vocabulary (65536) | | `steps` | Text tokens and frames in the payload | | `ended` | `true` when the reply ended; `false` when `max_tokens`, or a stream finished early, cut it off | A reply that was cut off goes back in as it is, as liquid-audio keeps it. The artifact holds ids and codes only, so it replays on any quantization of the same checkpoint. A reply from the other checkpoint or in another format version is rejected, and so are artifacts out of turn, artifacts with other ids, meta the format does not have and a question that does not read as WAV. These are errors in the request, as are a `return_codes` that is neither true nor false and history or `return_codes` sent to ASR or TTS: the server answers each with 400 (`invalid_request_error`). The system prompt, `text_input` or the default, applies to the whole conversation, so send the same one every turn. A turn's reply depends only on the system prompt, the earlier turns, the new question and the request options, seed included: on a backend, the same request gives the same bytes whatever ran before it. A turn with earlier turns prefills in blocks of 256 steps, so its attention never holds more than 256 scores per prompt step and head. Each earlier question is held to `lfm2_audio.max_pass_duration_sec`, as the new one is. So that a turn does not run the whole conversation again, an S2S session keeps what the last turn of a conversation ran, for up to `lfm2_audio.conversation_cache_slots` conversations at a time (4 by default): - The questions of a conversation's last request with history or `return_codes`, as mono 16 kHz samples, and their encoder output: about 160 KiB per second of question. An earlier question with exactly the samples of a kept one takes the kept output instead of going through the encoder. The new question always goes through it. - The full 256-step blocks of the prefill of a conversation's last turn with history: the keys and values each block wrote, 24 KiB per step over the backbone's six attention layers, the conv state after it, 160 KiB per block, and its input, which holds 8 KiB per step of question audio. That is 6 to 8 MiB per block, the most for a block of question audio, and about 70 MiB at the tenth turn of a conversation of 2900 steps. A turn's prompt starts with the turn before's, so its prefill restores the blocks it starts with that have the same input, bit for bit, and runs the rest: the end of the turn before's last block, its reply and the new question. The last block always runs. At the 8192-step limit, a conversation that is nearly all question audio takes about 350 MiB of the session's host memory: about 190 MiB of keys, values and conv state, about 60 MiB of block inputs and about 100 MiB of questions. The session keeps up to that for each slot, the blocks that conversations share counted once, and a slot that no conversation has used takes nothing. A request whose questions are not the kept ones holds both until it has encoded them all, up to about 100 MiB more for that time. After each turn with history, the timing log (`--log` or `--log-file`) reports how many prompts' blocks are kept, one per conversation or two for one that changed or left out turns until the older is dropped, and the MiB they take (`lfm2_audio.prefill.kept_prompts` and `lfm2_audio.prefill.kept_mib`). A turn gives the same bytes with or without them; they only save time. They stay until a request replaces them or the session closes; resetting a stream does not free them. A request finds what was kept for its conversation by content: - A turn with history restores the blocks its prompt starts with from the kept conversation whose blocks start with the most of them. When it is that conversation's next turn, or the same turn again, its own blocks after them go on the end of that conversation's; otherwise they are kept as a new conversation's. So conversations that start the same, such as with one long system prompt or the same first turns, share those blocks, and each keeps its own from where they part. A turn that changes an earlier turn keeps its blocks as a new conversation's, and the one it came from is dropped when it is the least recently used. Until then it takes a slot, so another conversation's blocks can be dropped first; when clients leave out or change earlier turns, set one slot more than the conversations. A turn whose prompt is under 256 steps keeps no blocks. - A request with history or `return_codes` keeps its questions in place of those of its conversation: the kept questions whose last one it also asks, the ones that hold the most of its earlier questions if several do. Otherwise they are kept as a new conversation's. A new conversation drops the least recently used conversation's blocks or questions past the slots, each on its own: a first turn with `return_codes` keeps questions but no blocks. With 0 slots, nothing is kept. With more conversations taking turns than slots, they push each other out: with one slot, a turn of another conversation in between makes the next turn of this one encode its earlier questions again and run its prefill again from where the two differ. A request without history or `return_codes` keeps nothing and leaves what the session kept. A turn that leaves out the oldest turns moves every position after them, so its prefill runs in full and its blocks are kept as a new conversation's, as above, though its questions still take the kept encoder output; leave out several turns at once rather than one per turn, so that fewer turns run in full. A first turn runs as before and keeps no blocks, so the third turn of a conversation is the first to resume a prefill. Send each question as the samples its turn had, which a float WAV holds exactly: one that differs goes through the encoder again, and the prefill runs again from the block it starts in. A conversation's prompt and `max_tokens` may take 8192 steps together, where a step is 80 ms of a question's audio, a text token or an 80 ms reply frame. Past that the turn fails with a CapacityError as soon as the request shows it: when the request is read, from the replies and the chat markup, so that a stream fails at its start, before its audio, or else as the questions are encoded. The client then leaves out the oldest turns; audio.cpp never leaves out any itself. A first turn is held only to the context. Conversations were checked up to 10 turns, with prompts of up to about 5000 steps at the tenth: replies kept their speech and used what earlier turns said, but did not recall what users said about themselves, such as their name, and late Japanese replies sometimes repeated an earlier one. Past 10 turns, leave out the oldest turns. Through the C API (`include/audiocpp.h`), the reply artifact is the result's first artifact, from `audiocpp_session_run` or `audiocpp_stream_finish`, and goes into the next request with its meta: ```c /* Turn 1, with return_codes. */ audiocpp_request_set_audio(request_1, question_1, frames_1, sample_rate, channels); audiocpp_request_set_option(request_1, "return_codes", "true"); audiocpp_session_run(session, request_1, &result_1); /* Turn 2: the first question's WAV, then the reply artifact and its meta. */ audiocpp_artifact_kind kind; const char * id; const void * payload; size_t bytes, index; audiocpp_result_artifact(result_1, 0, &kind, &id, &payload, &bytes); audiocpp_request_add_artifact(request_2, AUDIOCPP_ARTIFACT_CUSTOM, "lfm2_audio.question", wav_1, wav_1_bytes, NULL); audiocpp_request_add_artifact(request_2, kind, id, payload, bytes, &index); for (size_t m = 0; m < audiocpp_result_artifact_meta_count(result_1, 0); ++m) { const char * key; const char * value; audiocpp_result_artifact_meta(result_1, 0, m, &key, &value); audiocpp_request_set_artifact_meta(request_2, index, key, value); } audiocpp_request_set_audio(request_2, question_2, frames_2, sample_rate, channels); audiocpp_request_set_option(request_2, "return_codes", "true"); audiocpp_session_run(session, request_2, &result_2); ``` A third turn sends both earlier questions and replies, in order, and so on. The server returns the reply artifact in the `artifacts` array of `/v1/tasks/run` and `/v1/tasks/stream` results (base64 `payload`, and `meta`) when the request's `options` have `"return_codes": true`. A later request sends the history back in its own `artifacts` array, in the same shape: per earlier turn, in order, an `lfm2_audio.question` of kind `custom` holding the question's WAV bytes, then the `lfm2_audio.reply` that turn returned, as it came back. Either can give its bytes as a base64 `payload` or a `path` to a file. The new question goes in `audio` as usual, and the reply's `meta` goes back exactly as the result returned it: ```json {"model": "lfm2-audio-s2s", "request": { "audio": "/path/to/turn2.wav", "options": {"return_codes": true}, "artifacts": [ {"id": "lfm2_audio.question", "kind": "custom", "path": "/path/to/turn1.wav"}, {"id": "lfm2_audio.reply", "kind": "acoustic_tokens", "payload": "", "meta": {"format": "lfm2_audio.reply/1", "language": "en", "codebooks": "8", "audio_vocab_size": "2049", "text_vocab_size": "65536", "steps": "", "ended": ""}} ]}} ``` The CLI's `--request-sequence` JSON takes the same `artifacts` array. The CLI writes result artifacts with `payload_hex`, which requests do not take, so write the bytes to a file and pass its `path`. The live route, `/v1/audio/speech/live`, takes no artifacts: each of its requests is a first turn. `/v1/tasks/stream` with `"stream_format": "sse"` sends each reply while it is generated: `task.stream.event` events carry its text and audio as in [Streaming S2S](#streaming-s2s), and the closing `task.stream.done` event carries the result with the `lfm2_audio.reply` artifact (see the [server README](../../app/server/README.md#post-v1tasksstream)). The entry needs `"mode": "streaming"`; an offline entry fails on this route. The done event repeats the whole reply's audio as one Base64 WAV on a single line. A client that closes the stream before `task.stream.done` gets no reply artifact, so it has to leave that turn out of its next request's history. The route does not read `audio_base64`, so the new question goes in `audio` as a path the server can read; the earlier questions can still be Base64 payloads. A turn the model turns away, such as one past the step limit, is an `error` event before any audio, with the message that the JSON response carries with HTTP 400. Every request to a server entry runs on that entry's one session, whatever its route, also with `--parallel-jobs`, which runs `lfm2_audio` requests one at a time. So clients that take turns on one S2S entry share its `lfm2_audio.conversation_cache_slots`: the default of 4 keeps four conversations. For more that go on at the same time, set it to at least their number: ```json {"id": "lfm2-audio-s2s", "family": "lfm2_audio", "path": "/path/to/LFM2.5-Audio-1.5B-GGUF", "task": "s2s", "mode": "streaming", "session_options": {"lfm2_audio.model_gguf": "LFM2.5-Audio-1.5B-Q4_0.gguf", "lfm2_audio.conversation_cache_slots": "8"}} ``` Each kept conversation adds the host memory above while the server runs: about 75 MiB at the tenth turn of a conversation of 2900 steps, and up to about 350 MiB at the step limit. The server's `min_free_memory_mb` guard estimates a model from its files when it loads, so it does not count that memory. ## Request Options (use with `--request-option`) | Option | Task | Default | Meaning | |---|---|---|---| | `language` | all | The checkpoint's | `en` or `ja`; must match the checkpoint. | | `max_tokens` | all | `512`; S2S `1024` | ASR: transcript tokens per audio chunk. TTS: 80 ms audio frames per text chunk. S2S: text tokens and audio frames of the reply together. A transcript, a text chunk's speech or a reply that reaches it is cut off there and kept, as liquid-audio keeps it, and a warning goes to stderr; the other chunks go on. Text tokens need not end on a character boundary, so a cut can fall inside a character: liquid-audio then shows U+FFFD for the partial character, while audio.cpp drops it and ends the transcript or reply at the last whole character. On the CPU, `max_tokens` can change an output at a near-tie even where it is not reached, so a cut transcript, speech or reply is not always the start of the uncut one: the decode cache's length follows `max_tokens`, and ggml's CPU flash attention splits a cache of 512 steps or more into one piece per thread. | | `audio_chunk_mode` | ASR | `auto` | `auto`, `vad`, `fixed` or `none`; see [Long audio](#long-audio). | | `audio_chunk_duration_sec` | ASR | `30`, or `lfm2_audio.max_pass_duration_sec` if lower | Longest chunk in seconds, at least 1. Legacy `audio_chunk_seconds` remains accepted. | | `temperature` | TTS, S2S | `0.8`; S2S `1.0` | Audio code sampling temperature; 0 is greedy. | | `top_k` | TTS, S2S | `64`; S2S `4` | Sample from the k most likely codes; 0 keeps all, 1 is greedy. | | `text_temperature` | S2S | `0` | Text token sampling temperature; 0 is greedy, as in liquid-audio's README and demo. | | `text_top_k` | S2S | `0` | With `text_temperature` above 0, sample from the k most likely text tokens; 0 keeps all, 1 is greedy. With 0, any token can be drawn, control tokens included, so keep a top-k with a high temperature. | | `seed` | TTS, S2S | Random | Sampling seed; TTS text chunk i uses seed + i. S2S text sampling draws from a stream of its own, so with the same seed a reply follows the greedy-text reply until a text token differs. | | `text_chunk_mode` | TTS | `japanese` for JP, else `default` | How long text is split; see [Long text](#long-text). | | `text_chunk_size` | TTS | `200` | Unicode codepoints per text chunk. | | `stream_frames_per_event` | TTS, S2S streaming | `1` | Audio frames (80 ms) per streaming event. | | `return_codes` | S2S | `false` | Also return the reply's text tokens and audio codes as an `lfm2_audio.reply` artifact, to send back as history; see [Conversations](#conversations). | Each task rejects the options it does not take. ## Session Options (use with `--session-option`) | Option | Default | Meaning | |---|---|---| | `lfm2_audio.model_gguf` | The directory's only backbone | Backbone GGUF, relative to the model directory. | | `lfm2_audio.mmproj_gguf` | `mmproj-`, else the only mmproj | Audio encoder GGUF, relative to the model directory. | | `lfm2_audio.vocoder_gguf` | `vocoder-`, else the only vocoder | TTS depthformer GGUF, relative to the model directory. | | `lfm2_audio.detokenizer_gguf` | `tokenizer-`, else the only tokenizer | TTS audio detokenizer GGUF, relative to the model directory. | | `lfm2_audio.vad_model_path` | `assets/framework/models/silero_vad` | Silero VAD model used to split long audio. | | `lfm2_audio.max_pass_duration_sec` | `120` | Most audio, in seconds, the encoder takes in one pass: an ASR chunk (the whole input with `audio_chunk_mode=none`) or the S2S question; at least 1. Longer audio is rejected before it is encoded; see [Long audio](#long-audio). Legacy `lfm2_audio.max_pass_seconds` remains accepted. | | `lfm2_audio.conversation_cache_slots` | `4` | S2S: how many conversations a session keeps the earlier questions' encoder output and prefill blocks of, for their next turns; the least recently used is dropped first, and `0` keeps none. Output never depends on it; see [Conversations](#conversations). | | `lfm2_audio.cpu_repack` | `true` | On the CPU, multiply the matrices with the kernels of ggml's CPU extra buffer types where those take the weight's type and shape: the repacked ones take Q4_0 on x86 with AVX2 and Q4_0, Q8_0 and Q6_K on Arm; where ggml's AMX buffer is built, it comes first and also takes F16, Q8_0 and K-quants. Where the token embedding's type is taken, the text head gets a copy of it. `false` keeps every weight in the plain CPU buffer. Other backends ignore it; see [CPU](#cpu). | ## Long Audio liquid-audio transcribes a whole file in one pass. The English model handles that up to about a minute; longer audio loses words and then repeats itself. The Japanese model holds to about 40 s; past that it starts to skip whole sentences, and at 60 s it leaves out 18 to 29% of the text. audio.cpp keeps short input whole and splits longer input: | Mode | Behavior | |---|---| | `auto` | Input up to `audio_chunk_duration_sec` is transcribed whole, as liquid-audio does. Longer input is split at pauses found by the bundled Silero VAD, and silence between speech is skipped. Without the VAD model, it falls back to `fixed`. | | `vad` | Always splits at pauses; fails without the VAD model. | | `fixed` | Cuts every `audio_chunk_duration_sec`; a last piece shorter than 1 s and the chunk before it split their audio evenly, so no chunk is longer than `audio_chunk_duration_sec`. | | `none` | One pass over the whole input, as liquid-audio does; input longer than `lfm2_audio.max_pass_duration_sec` is rejected. | Each chunk is transcribed on its own, and the transcripts are joined with a space between English words and without one in Japanese text. Audio the VAD does not mark as speech is not transcribed. That is usually silence, but it can be speech too quiet for the VAD: a sentence recorded over 30 dB below the others around it, or a whole 40 s file turned down by 60 dB (7 of 8 such files came back empty). A split can also land on a short pause inside a sentence, and the model may then leave out the words that open the next chunk (1 of 12 such sentences in the Japanese tests below). No pass takes more than `lfm2_audio.max_pass_duration_sec` of audio, 120 s by default. `none` rejects longer input. The other modes reject an `audio_chunk_duration_sec` above the limit on input longer than the limit, before any chunk is planned or encoded, so `vad` is refused on the setting even when its spans would be shorter; a request that leaves `audio_chunk_duration_sec` unset gets chunks of the limit when that is under 30 s. One pass is of little use past 120 s: `none` is at 29% WER on the 120 s English files below, and Japanese loses sentences from about 41 to 45 s. The encoder's buffer grows linearly, at about 6.46 MB per second of audio, up to about 240 s (775 MB at 120 s), and with the square of the length beyond that (8.8 GB measured at 600 s on an A10); without the limit, on the CPU or with unified memory, nothing stops a long pass before allocation. The server answers such a request with 400. The limit is a session option, so the operator can raise it (`--session-option` on the CLI, `session_options` in a server entry), and a task request cannot (with `--ui-management`, `/v1/models/load` sets session options like any server entry). Word error rate by input length, EN F16 on CUDA, 8 files per length built from consecutive LibriSpeech test-clean utterances joined with 0.3 s gaps (each file up to about 5 s longer than its length): | Length | liquid-audio (one pass) | `auto` | `fixed` | |---|---|---|---| | 30 s | 2.8% | 2.3% | 2.9% | | 40 s | 1.7% | 1.7% | 1.8% | | 50 s | 1.7% | 1.5% | 2.5% | | 60 s | 2.1% | 2.0% | 2.8% | | 90 s | 9.4% | 1.3% | 2.7% | | 120 s | 83% | 1.9% | 2.6% | | 180 s | 296% | 1.4% | 2.4% | `none` follows liquid-audio up to 90 s (2.6%, 1.7%, 1.7%, 2.0% and 9.4%). At 120 s it finishes all 8 files at 29%, 6 of them word for word as liquid-audio. At 180 s, 7 of the 8 reach `max_tokens` and come back cut off there, repetitions included, with a warning. Those runs predate the limit: with the default, `none` now rejects the 120 s files (up to 125 s long) and the 180 s ones. Character error rate by input length, JP F32 on CUDA, 8 files per length built from consecutive Common Voice ja test clips (one sentence each) joined with 0.3 s gaps. The clips keep their own silence at both ends, so sentences are about 1.5 s apart and `auto` cuts the files over 30 s about one sentence per chunk; each file is up to 3.7 s shorter than its length, so `auto` keeps the 20 and 30 s files whole. Punctuation and spaces are removed before scoring. The model's own errors on the single clips, often a different kanji or number form (1500円 for 千五百円), already come to 9 to 12%. In parentheses: sentences lost (less than half of their characters in the transcript) out of all sentences. | Length | liquid-audio (one pass) | `auto` | |---|---|---| | 20 s | 16.6% (1 of 32) | 16.6% (1 of 32) | | 30 s | 13.7% (1 of 46) | 13.7% (1 of 46) | | 40 s | 15.2% (1 of 61) | 12.7% (0 of 61) | | 45 s | 15.4% (4 of 65) | 10.8% (0 of 65) | | 50 s | 24.0% (10 of 73) | 11.3% (0 of 73) | | 60 s | 36.1% (26 of 89) | 10.3% (0 of 89) | | 90 s | 177% (81 of 131) | 10.0% (0 of 131) | The sentence lost at 20 to 40 s is the same recording in each file, over 30 dB quieter than the others; liquid-audio drops it from a 16 s file too. At 90 s one file repeats itself up to the test's 2,048-token cap. With `auto`, Q8_0 on CUDA and on Metal is within 0.3 points of F32. `none` gives liquid-audio's transcripts with F32 on CUDA (71 of 71 files at 20 to 90 s, compared without punctuation and spaces); on the repeating file it reaches `max_tokens` and returns the repetition cut off there, with a warning. The limit follows the seconds, not the amount of text. Files built other ways (FLEURS ja clips, or Common Voice clips in reverse order, with 0.8 s gaps, with their silence trimmed or played 1.3 times faster) lost no other sentence entirely (under 30% of it transcribed) in one pass up to 40 s, and those run further first lost one between 41 and 45 s; the fastest also kept under half of a few sentences from 34 s. For whole sentences, 30 s chunks leave about 10 s of margin. Where pauses are short, `auto` packs chunks of 26 to 30 s: with the clips trimmed to their speech and joined with pauses of about 0.4 s, it gives 12.1 to 12.9% at 31 to 35 s (one pass 11.6 to 12.6% on the same files) and 9.8 to 10.9% at 45 to 180 s, with no repeats, though it can still drop a sentence now and then. All of these files are recordings of read sentences; the Japanese TTS voice's speech lost sentences in one pass from 28 s. ## Long Text liquid-audio speaks a whole text in one turn. audio.cpp splits text longer than `text_chunk_size` codepoints at sentence boundaries with the framework's text chunker and speaks each chunk as its own turn in the same voice, then joins the audio. At the default 200 codepoints a chunk is about 13 s of English speech. The shared 6,000-character long-form test text (`tools/audiocpp_cli/audiocpp_cli_longform_tts_clone_cases.json`) comes out as about 345 s of speech that transcribes back through the ASR task at 1.5-1.6% WER with the EN F16, Q8_0 and Q4_0 packages on CUDA. Each chunk is its own take: the voice stays, but pitch, pace and loudness can shift from one chunk to the next (up to about 3 semitones, 20% and 4 dB on a 7-chunk English text), which can sound like a new recording. liquid-audio's chunks shift the same way, and the model cannot carry context across them. In one turn, liquid-audio speaks English reliably up to about 430 characters and starts to add or drop words from about 600; given the previous chunk as an earlier turn, it answered in text instead of speaking on up to 2 of 7 chunks. For English, a larger `text_chunk_size` means fewer joins: 300, 400 and 600 codepoints transcribed back as well as 200 (0.5-1.5% WER). Keep the default for Japanese: at 400 its chunks ran past `max_tokens`. ## Validation ### ASR Transcripts were compared with liquid-audio ([`19e65845`](https://github.com/Liquid4All/liquid-audio/tree/19e65845923a7f136442c95137884ec61eb386aa), fp32, greedy) on 200 English utterances from LibriSpeech test-clean and 200 Japanese utterances from the Common Voice 8.0 ja test set, for every package, on CPU and CUDA (NVIDIA A10) and on CPU and Metal (Apple M3 Ultra): | Weights | Transcripts identical to liquid-audio | |---|---| | F32 (JP), F16 | 198-200 of 200 | | Q8_0 | 193-197 of 200 | | Q4_0 | 160-170 of 200 | At F32 and F16, the few differences are near-ties in liquid-audio's own output, where its two best tokens are within 0.04 in log-probability. Below that, the quantized weights change close token choices. On the x86 CPU a Q8_0 transcript at such a tie can also depend on `--threads` and `max_tokens`: on one Japanese S2S reply whose two best tokens in liquid-audio are 0.04 apart, the Q8_0 package added a sentence at 4, 5, 7 and 8 threads but not at 1, 2, 3 and 6, nor at 4 threads with `max_tokens` 128 or 1024. F32 and F16 gave the same transcript at every thread count, and Q8_0 on the M3 Ultra CPU, Metal and CUDA added the sentence in every run. Stage by stage on CPU with the F32 weights, the adapter output matches liquid-audio within 1.4e-6 relative error and the first-step logits within 9e-7. `test_lfm2_audio_asr` (`ENGINE_BUILD_MODEL_TESTS`) checks the prompt, the tokenizer, the stage numbers (English F16) and the transcripts against liquid-audio: two LibriSpeech clips with the English packages and, with the Japanese ones, a Common Voice clip that every JP package transcribes exactly on the CPU, Metal and CUDA. It runs when `lfm2_audio_1_5b_f16` is installed in `models/` and skips otherwise; for a JP package, pass `--model models/LFM2.5-Audio-1.5B-JP-GGUF --model-gguf LFM2.5-Audio-1.5B-JP-Q8_0.gguf` (or the F16, F32 or Q4_0 file). The `lfm2_audio_*_test` unit tests run on small synthetic GGUFs. Real-time factor over each 200-utterance set (processing time divided by audio length, with the model loaded; the CPU runs used 16 threads; the M3 Ultra's Q4_0 figure is with ggml's repacked kernels and the x86 row without them, see [CPU](#cpu)): | Backend | EN F16 | JP F32 | EN Q4_0 | |---|---|---|---| | CUDA, NVIDIA A10 | 0.027 | 0.037 | 0.017 | | Metal, Apple M3 Ultra | 0.034 | 0.042 | 0.021 | | CPU, Linux x86-64 | 0.11 | 0.14 | 0.077 | | CPU, Apple M3 Ultra | 0.10 | 0.17 | 0.032 | ### TTS liquid-audio (fp32, CUDA with TF32 off) was dumped at every stage for three cases: "The quick brown fox jumps over the lazy dog." (US female), "What is this obsession people have with books?" (UK male) and a Japanese sentence, with greedy decoding (41, 32 and 45 frames): | Stage | JP F32, CPU | F16, Linux CPU and CUDA | |---|---|---| | Backbone output at `<|audio_start|>` | 1.4e-7 relative | 2e-4 to 9e-4 | | First frame's depthformer logits | 5e-7 | 4e-4 to 3e-3 | | Greedy frames identical to liquid-audio | all | all for UK male and Japanese; US female until a near-tie (top two logits 0.002 and 0.000 apart) | | Detokenizer head on the reference frames | 1.4e-6 | 3e-4 to 5e-3 | | Waveform | 6e-6 | 1.3e-3 to 1.9e-2 | The detokenizer's log-magnitudes reach 4.7, a magnitude of 114, which the framework's Vocos ISTFT would clamp at 100, so the ISTFT here is the reference's. Long audio is detokenized in chunks that overlap by the model's receptive field (97 steps); with F32 weights the chunked output matches one pass to 3e-5. A stream carries each layer's state instead, and with F32 weights its waveform matches one pass to 1e-6 (relative RMS). `test_lfm2_audio_tts` checks the prompt, the stage numbers, the greedy frames, the chunking, the stream, a round trip through ASR, greedy and sampled (two of three seeds have to come back word for word), and speech cut off at `max_tokens`, offline and streamed; it runs when `lfm2_audio_1_5b_f16` is installed in `models/`. Real-time factor through `audiocpp_server` (processing time divided by audio length, model loaded; short = three sentences of 3-7 s, long = the 345 s long-form text; the CPU runs used 16 threads; the M3 Ultra's short Q8_0 and Q4_0 figures are with ggml's repacked kernels, from [CPU](#cpu), on its three sentences of 2.5 to 5 s, and the x86 row is without them): | Backend | EN F16 short / long | EN Q8_0 short / long | EN Q4_0 short / long | JP F32 short | |---|---|---|---|---| | CUDA, NVIDIA A10 | 0.19 / 0.15 | 0.16 / 0.13 | 0.16 / 0.086 | 0.21 | | Metal, Apple M3 Ultra | 0.21 / 0.21 | 0.18 / 0.18 | 0.15 / 0.15 | 0.26 | | CPU, Linux x86-64 | 0.42 / 0.41 | 0.29 / 0.30 | 0.22 | 0.65 | | CPU, Apple M3 Ultra | 0.32 | 0.20 | 0.16 | 0.56 | Each frame takes a backbone step and eight small depthformer steps. On GPUs per-step overhead dominates, so the weights' size matters little; on the CPU each frame reads the backbone once and the depthformer eight times, which shows in F16. Transcribed back by the ASR task, the long-form speech scored 1.4-1.9% WER on every backend and package measured, mostly homophones and spellings such as "their" and "harbour"; the two Japanese test sentences came back exactly. With the Q4_0 package on Metal on an Apple M3 Max, the server's footprint was 1.17 GB and stayed within 4 MB of that over 12 requests alternating 2.5 s and 57 s of speech. ### S2S liquid-audio's `generate_interleaved` (fp32, CUDA with TF32 off) was dumped at every step for its README's two questions (English and Japanese) and for `assets/resources/c.wav`, greedy and sampled like the README (temperature 1.0, top-k 4, seed 0), and the same steps were replayed through audio.cpp: | Stage | JP F32, CPU | EN F16, CPU / CUDA | |---|---|---| | Prompt ids and audio positions | identical | identical | | Text logits at every text step | 4e-6 relative | 7e-4 / 3e-3 | | Backbone output at every audio step | 4e-7 | 5e-4 / 1.4e-3 | | Depthformer logits at every frame | 7e-6 | 1.5e-3 / 3e-3 | | Free-running steps identical to liquid-audio | all 277 greedy and 308 sampled | the whole 250-step greedy reply to `c.wav` on the CPU, 248 on CUDA; sampled replies until a near-tie | | Waveform of the reference frames | 5e-5 | 9e-3 / 0.12 | The sampled replies were compared through torch's own draws: fed liquid-audio's logits, the framework's torch-compatible sampler picks every code torch drew. audio.cpp itself samples with its own generator, so a seed does not reproduce a liquid-audio reply. The CUDA waveform differs more because CUDA accumulates F16 products in half precision. A reply's audio says what its text says, with rare exceptions. Over 30 seeds of replies to `c.wav` with the F16, Q8_0 and Q4_0 packages, on the CPU and Metal (Apple M3 Ultra) and on the CPU and CUDA (x86-64, NVIDIA A10), the ASR task heard the text at a median 4-7% WER, the names the model makes up aside. Whisper large-v3-turbo heard nearly every reply at 18% or less; one x86 CPU reply spoke a question its text did not have. The ASR task sometimes answers a reply, paraphrases it or runs on instead of transcribing it: on 0 to 6 of 30 replies per backend and package, and on 5 of 60 with liquid-audio fp32. It can also add a sentence that was never spoken to speech that sounds like an assistant. Over 450 Japanese replies to 13 questions, this happened only after the ending 「何かご質問があれば、いつでもお手伝いしますよ！」 (or 「ね！」), where the model is close to a tie between ending the transcript and going on. Of the 50 replies with that ending, liquid-audio fp32 added a sentence, mostly 「何かご質問はありますか？」, to 7, the F16 package to 6 or 7 of the same replies, and the Q8_0 package to 11 on every backend (the CPUs at 8 threads). liquid-audio with the Q8_0 weights added one to the same 11, so the four extra come from the quantized weights. With F16 and Q8_0, none of the other 400 replies gained a sentence. With Q4_0, 2 of 30 replies on the M3 Ultra CPU and 2 of 30 on CUDA repeated a sentence until `max_tokens` (2 of 60 on the x86 CPU, none on Metal). F16 and Q8_0 did not, though one Q8_0 reply (M3 Ultra CPU, 8 threads) finished its text but not its audio within `max_tokens`. `test_lfm2_audio_s2s` checks the prompt, the first text and audio blocks, the round trip of three replies through ASR (their median, a reply that fails or is cut off at `max_tokens` counting as a miss), well-formed text, streaming against offline, a stream finished early, a reply cut off at `max_tokens`, offline and streamed, that the first request, sent again after the others, gets the same reply (S2S sizes its decode cache from the request alone, as TTS does), that a reply with `text_temperature` is step for step the greedy-text reply of the same seed until a text token differs, and a conversation: the first reply's artifact holds its text and speech, a second turn (`assets/resources/a.wav`) sent with the first gets another reply than without it and the same bytes when sent again, and a third turn replays both; it runs when `lfm2_audio_1_5b_f16` is installed in `models/`. `lfm2_audio_chat_test` runs the S2S session on a synthetic package, offline and streamed, over three turns of a conversation, and up to the conversation limit. In one session, the four turns of a longer conversation, another conversation in between and a turn over the limit give the replies and errors of a fresh session, byte for byte, with the earlier questions and prefill blocks taken from the turn before. With two slots, two conversations taking turns, also two with one long system prompt and two that ask the same clips, give the bytes of sessions of their own and take at least what those take, and a third one drops what was used least recently; with one slot and with none, they give the same bytes. Without the option, four conversations taking turns each take what they take alone, and a fifth drops the questions of the least recently used. `lfm2_audio_backbone_test` checks that a chunked prefill restored from the ones before gives a fresh runtime's bits, at 1, 4 and 8 threads and with caches of other lengths, and with several kept prompts: prompts taking turns, the least recently used dropped, prompts that start the same, one that parts inside a kept one, prompts of whole blocks, a shorter one and one under a block. ### CPU On the CPU, the quantized packages run the matmul kernels of ggml's CPU extra buffer types where those take the weight's type and shape, as llama.cpp does (`lfm2_audio.cpu_repack`, on by default). ggml's repacked kernels take Q4_0 on x86 with AVX2, and Q4_0, Q8_0 and Q6_K on Arm. They cover the LFM2 layers of the backbone and the detokenizer, the depthformer and the encoder's linears. The depthformer's input projection, whose rows each codebook step reads apart, the embedding tables, which rows are gathered from, and the few matrices whose shape the kernels do not take keep the plain layout. The text head is tied to the token embedding, so on Arm it gets a repacked copy of it: 105 MiB of Q6_K with the Q4_0 packages, 136 MiB of Q8_0 with Q8_0. F16 and F32 weights, and Q8_0 on x86, run the plain, non-repacked kernels, unless ggml has its AMX buffer (below). ggml logs each weight it repacks on stderr as the model loads. ggml builds its AMX buffer for x86 CPUs with AMX (Xeon Sapphire Rapids and later) in a native build, and in the CPU variant for them that the release builds carry. That buffer comes first and also takes the F16, Q8_0 and K-quant matrices whose rows come in multiples of 32, so the F16 and Q8_0 packages run other kernels there too, and the text head gets its copy with them as well, 256 MiB with F16. No CPU with AMX was measured. On a Xeon 8358 server CPU (AVX-512, no AMX), a build like the portable release builds (`GGML_BACKEND_DL` with all CPU variants), which loaded the icelake variant there, repacked the same weights as a native build and gave its outputs byte for byte. Real-time factor with the repacked kernels / without them (`lfm2_audio.cpu_repack=false`) on an Apple M3 Ultra, the same binary at 16 threads, median of three runs that alternate the two: ASR over the 200-utterance sets as above, with the CLI, and TTS through `audiocpp_server` on three sentences of 2.5 to 5 s: | CPU | ASR EN Q4_0 | ASR EN Q8_0 | ASR JP Q4_0 | TTS EN Q4_0 | TTS EN Q8_0 | |---|---|---|---|---|---| | Apple M3 Ultra, 16 threads | 0.032 / 0.052 | 0.040 / 0.056 | 0.031 / 0.052 | 0.156 / 0.177 | 0.200 / 0.222 | On x86, an Intel Core i9-13900HK laptop CPU (AVX2, no AVX-512 or AMX; 6 performance cores with two threads each and 8 efficiency cores) ran with the repacked kernels / without them (main's code, whose outputs `cpu_repack=false` reproduces byte for byte), median of four runs that alternate the two: with the CLI, ASR of a 60 s clip and of ten short clips and TTS of a three-sentence text with three seeds, and through `audiocpp_server` the first audio of a reply on the S2S live route, measured as in its table above: | Intel Core i9-13900HK | 8 threads | 4 threads | |---|---|---| | ASR EN Q4_0, 60 s clip | 0.121 / 0.187 | 0.132 / 0.183 | | ASR EN Q4_0, short clips | 0.127 / 0.229 | 0.141 / 0.203 | | ASR JP Q4_0, 60 s clip | 0.106 / 0.192 | 0.115 / 0.161 | | ASR JP Q4_0, short clips | 0.120 / 0.234 | 0.136 / 0.198 | | TTS EN Q4_0 | 0.359 / 0.437 | 0.356 / 0.427 | | S2S EN Q4_0, first audio on the live route | 694 / 1494 ms | 775 / 1171 ms | Q8_0 moves no weight there, and its ASR ran within about 2% of main's. With 12 or 20 threads, which put a third to a half of the running threads on the efficiency cores (a fifth to a third at 8), each of these real-time factors was higher than at the better of 4 and 8 threads, with the repacked kernels and without. The encoder and the prefill multiply many steps at a time, where the kernels gain the most, so ASR gains more than speech, whose frames are a step each. On the M3 Ultra, measured as in the streaming tables above, the first audio of a TTS stream came after 48 ms with Q4_0 and 54 ms with Q8_0 (99 and 95 ms without), and that of a reply on the S2S live route after 205 and 228 ms (358 and 342 ms). There llama.cpp's `llama-server`, on the CPU only at 16 threads, transcribed the EN Q4_0 set at RTF 0.033 through its HTTP API, and at 0.043 with `--no-repack`, where audio.cpp's 0.032 is in process. Loading takes up to 0.7 s longer with the repacked kernels on the M3 Ultra, and at most 0.05 s longer on the i9-13900HK. The repacked kernels add up the same products in another order, so results move a little and can change where two choices are close. With them, 195 of the 200 EN Q4_0 transcripts were byte for byte the ones without them on the M3 Ultra and on the Xeon 8358 (both at 16 threads), and 193 on the i9-13900HK (at 8); with JP Q4_0, 196, 196 and 195; with EN Q8_0 on the M3 Ultra, 196. WER over the English set went from 1.48% to 1.51% (M3 Ultra), from 1.41% to 1.48% (Xeon) and from 1.43% to 1.53% (i9-13900HK) with Q4_0, and from 1.43% to 1.46% with Q8_0 (M3 Ultra); CER over the Japanese set went from 9.29% to 9.26%, from 9.21% to 9.36% and from 9.34% to 9.31%. Speech and replies part from those of the plain kernels within their first frames; a different thread count alone makes them part after 0.3 to 0.8 s (M3 Ultra). Transcribed back, 40 sampled takes of one sentence (seeds 1 to 40) came back exactly 40 times with them and 36 times without (Xeon, Q4_0), 37 and 39 times (M3 Ultra, Q4_0) and 39 and 40 times (M3 Ultra, Q8_0). The first text token of the S2S reply to `assets/resources/c.wav` is a near-tie, its two best logits 0.036 apart on the M3 Ultra CPU with Q8_0 and 0.048 on Metal, and there the repacked kernels swap them, so the greedy reply differs from its first word. The other CPU figures on this page that count transcripts or replies were measured without the repacked kernels. ### Memory Memory is dominated by the weights, about the package size. Between chunks and requests the encoder also keeps its graph and compute buffer, at most what a 30 s chunk needs: about 194 MiB, which took about 190 to 250 MiB of device memory on CUDA, depending on the requests before. The buffer of a longer chunk is freed after it. With the Q4_0 package on an Apple M3 Ultra, the server's memory footprint on Metal, kept buffer included, was 1168 MB after a first 3.5 s request and 1267 to 1310 MB over the next 59, alternating 70 s and 3.5 s of audio. The Q4_0 files store the token embedding as Q6_K, which CUDA cannot gather rows from, so on CUDA the backbone also keeps a 256 MiB F16 copy of it. On the CPU, where ggml's extra buffers take the token embedding's type (Q6_K and Q8_0 on Arm; with ggml's AMX buffer also F16), the backbone also keeps a copy of it in their layout as the text head (see [CPU](#cpu)). An S2S session also keeps the questions and prefill blocks of up to `lfm2_audio.conversation_cache_slots` conversations (four by default) in host memory, up to about 350 MiB each at the conversation limit; see [Conversations](#conversations). ## Limitations - The live route, `/v1/audio/speech/live`, answers each request as a first turn; conversations run through `/v1/tasks/run` and `/v1/tasks/stream` (which streams each reply with `"stream_format": "sse"`), the CLI's request JSON or the C API. - ASR is offline only; TTS and S2S also stream. - TTS speaks with the built-in voices only; there is no voice cloning. - On the CPU, F16 and F32 weights, and Q8_0 on x86, run ggml's generic matmul kernels, as it has no repacked ones for them. Only its AMX buffer, where built, takes F16 and Q8_0, and no CPU with AMX was measured (see [CPU](#cpu)).