HuggingFace Speech-to-Speech Pipeline: Go Local in 2026

By AIFoss · 10 min read
speech-to-speechollamaqwen3-ttsparakeetselfhostedvoice-ai

TL;DR: The HuggingFace speech-to-speech pipeline (announced July 1, 2026 with Cerebras) is an Apache 2.0 voice agent framework — VAD → Parakeet STT → LLM → Qwen3-TTS — that already runs on 9,000+ Reachy Mini robots. The LLM step defaults to Cerebras cloud, but it accepts any OpenAI-compatible endpoint, so you can point it at Ollama and keep every word on your own GPU. Budget roughly 6–9GB VRAM for a fully local real-time setup.

Fully local (Ollama)Default (Cerebras cloud LLM)Hybrid
Best forPrivacy, offline use, zero per-token costLowest latency, snappiest conversationLocal voice I/O, cloud brain
LLM speed~60–70 tok/s (12B on RTX 3090)1,851 tok/s (Gemma 4 31B, per HF)Depends on provider
PrivacyNothing leaves your machineTranscripts go to CerebrasTranscripts leave, audio stays
The catchNoticeable pause before replies on mid-range GPUsYour conversations are cloud-processedConfig juggling, two failure domains

Honest take: run the pipeline locally with a small, fast model — Gemma 4 12B QAT or a 7B-class Qwen — and it’s a genuinely usable private voice assistant on a 12GB card. Chase the Cerebras demo’s instant-response feel with a big local model and you’ll be disappointed; pick your model for tokens-per-second first, benchmark scores second.

HuggingFace and Cerebras shipped something rare on July 1, 2026: a real-time voice AI stack assembled entirely from open models, published as a normal repository (huggingface/speech-to-speech) instead of a waitlisted API. Speech goes in, speech comes out, and every stage in between is swappable. It’s not a lab demo either — the same pipeline powers the conversational mode on more than 9,000 Reachy Mini robots.

The launch coverage focused on the Cerebras side: Gemma 4 31B at 1,851 tokens per second, roughly 35× a typical GPU endpoint. That number is real, and it’s also the part self-hosters should ignore. The interesting part for anyone reading this site is that the LLM stage is just an OpenAI-compatible client. Swap the base URL for http://localhost:11434/v1 and the entire pipeline — ears, brain, and voice — runs on hardware you own.

What the pipeline actually is

The repo implements a four-stage loop, exposed through an OpenAI Realtime-compatible WebSocket API:

  1. VAD (voice activity detection) — decides when you’ve started and stopped talking.
  2. STT — NVIDIA Parakeet TDT by default; Whisper variants (faster-whisper, Lightning Whisper MLX) and Paraformer are drop-in alternatives.
  3. LLM — any OpenAI-compatible endpoint (chat-completions or responses API), local Transformers, or mlx-lm on Apple Silicon.
  4. TTS — Qwen3-TTS by default, with Kokoro-82M, Pocket TTS, ChatTTS, and MMS TTS as alternatives.

Each stage streams into the next, which is what makes it conversational rather than walkie-talkie-shaped: the TTS starts speaking the first sentence while the LLM is still generating the third.

If you’ve built voice assistants with Home Assistant’s Assist and the Wyoming protocol, the shape will feel familiar — the difference is that this pipeline puts a full LLM in the middle by default and speaks OpenAI Realtime on the outside, so anything that talks to OpenAI’s Realtime API can talk to your local stack instead.

The license check (better news than expected)

The trending discussion around this pipeline claimed a “mixed license” stack — Apache framework, but restricted models. Checking each component in August 2026, the reality is cleaner:

ComponentProjectLicenseCommercial self-hosting
Pipeline codehuggingface/speech-to-speechApache 2.0✅ Unrestricted
STT (default)NVIDIA Parakeet TDT 0.6B v3CC-BY-4.0✅ With attribution
LLM (default)Google Gemma 4Apache 2.0✅ Unrestricted
TTS (default)Qwen3-TTS (0.6B / 1.7B)Apache 2.0✅ Unrestricted

Two corrections to claims floating around Reddit threads on this. First, Parakeet is not Apache 2.0 — the model card says CC-BY-4.0. That’s still commercial-friendly, but it carries an attribution requirement, so a product built on this stack needs an “STT by NVIDIA Parakeet” line somewhere in its credits. Second, Gemma 4 shipped under plain Apache 2.0 in March 2026 — the first Gemma generation to drop the bespoke Gemma Terms of Use, something we verified in our open-source LLM license shootout. Qwen3-TTS is Apache 2.0 on both the code and the released weights.

So the stack is three parts Apache, one part CC-BY. For a home lab that distinction is academic. For a startup shipping a voice product, it means one attribution line — not a lawyer.

VRAM math: what “real-time” costs locally

The pipeline’s three model stages are small enough to share a single consumer GPU:

StageModelApprox. VRAM
STTParakeet TDT 0.6B~1GB
LLMQwen-class 7B–8B at Q4_K_M~4.5–5GB
LLM (better)Gemma 4 12B QAT Q4~7GB
TTSQwen3-TTS 0.6B~1GB

A 7B-class LLM keeps the whole pipeline around 6–7GB — comfortable on a RTX 3060 12GB. Stepping up to Gemma 4 12B QAT (the ~7GB quantization-aware build we covered in the Gemma 4 QAT self-hosting guide) pushes the total to roughly 9GB and noticeably improves answer quality; that’s 12GB-card territory with little headroom, or an easy fit on a RTX 3090 24GB. The Cerebras demo’s Gemma 4 31B (~18GB at QAT Q4) fits on a 24GB card too — but at local speeds it’s the wrong choice, as we’ll get to.

If you don’t have a GPU at all, a RunPod pod running Ollama gives you the same OpenAI-compatible endpoint per hour — you lose the privacy argument for the LLM stage but keep the swappable open stack.

Setup: pipeline plus Ollama on one machine

Get Ollama serving a tool-free chat model first:

ollama pull gemma4:12b-it-qat
ollama serve   # if not already running as a service

Then the pipeline:

git clone https://github.com/huggingface/speech-to-speech
cd speech-to-speech
pip install -r requirements.txt

The LLM stage takes an OpenAI-compatible base URL. The repo’s own docs show local serving with vLLM and llama.cpp (http://127.0.0.1:8080/v1); Ollama works the same way since it exposes http://localhost:11434/v1:

python s2s_pipeline.py \
  --llm chat-completions \
  --llm-base-url http://localhost:11434/v1 \
  --llm-model gemma4:12b-it-qat \
  --llm-api-key ollama

(Flag names evolve in a repo with 830+ commits — check python s2s_pipeline.py --help against your checkout rather than trusting a blog post, including this one.)

First run downloads Parakeet and Qwen3-TTS weights from the Hub. After that, the stack runs offline.

The problem you will actually hit: the silent context cutoff

Ollama’s default context window is 4,096 tokens unless the model or client raises it. A voice assistant burns context fast — system prompt, conversation history, and every transcribed turn accumulate — and when the limit is hit, Ollama silently truncates from the top. The symptom is distinctive: the assistant works beautifully for five minutes, then starts forgetting its instructions or losing the thread of the conversation mid-session, with no error anywhere.

Fix it server-side so every client benefits:

OLLAMA_CONTEXT_LENGTH=16384 ollama serve

or per-model with a Modelfile (PARAMETER num_ctx 16384). We’ve flagged this same trap in our Goose and OpenCode guides — it is the single most common “local AI feels broken” cause of 2026, and voice pipelines hit it faster than chat because nobody re-reads a spoken conversation to notice the truncation.

Latency: the honest local-vs-Cerebras story

Cerebras serves Gemma 4 31B at 1,851 tok/s. A RTX 3090 runs the same model class at maybe 25–35 tok/s, or a 12B at 60–70 tok/s. That sounds like a catastrophic gap, but conversational latency is dominated by two things: time-to-first-token and whether TTS streams.

Because the pipeline streams sentence-by-sentence into Qwen3-TTS, what you actually perceive is the pause before the first spoken word. With a 12B QAT model on a 24GB card, that pause is under a second and the assistant speaks continuously thereafter — the GPU generates text faster than the TTS speaks it. With a 31B model on the same card, the first-word pause stretches and long answers can outrun generation, producing mid-sentence gaps. That’s the practical rule: locally, pick the smallest model that answers well, because tokens-per-second is the user experience. Cerebras solves this with silicon; you solve it with model selection.

When NOT to use this pipeline

  • You only need voice commands, not conversation. Home Assistant Assist with Wyoming STT/TTS is lighter, and shoehorning an LLM into “turn off the lights” adds latency for nothing.
  • You need multilingual, low-resource STT. Parakeet TDT v3 covers 25 European languages; for broader coverage swap in a Whisper variant and accept the speed hit.
  • You want phone-call-grade instant responses on an 8GB GPU. Physics wins. Use the Cerebras/cloud LLM path, or temper expectations.
  • You need voice cloning. Qwen3-TTS in this pipeline does preset and designed voices; for cloning workflows see our audio.cpp setup guide, which covers PocketTTS and SeedVC.

For the GPU-buying side of this — which card sustains which model tier for real-time voice — runaihome.com’s local AI hardware guides cover the tradeoffs, and if your endgame is a voice interface for coding agents, aicoderscope.com has the tooling side covered.

Verdict

This is the most complete FOSS voice-agent starting point of 2026. The framework is Apache 2.0, the default models are Apache 2.0 plus one CC-BY, the whole stack fits on a mid-range card, and the OpenAI Realtime-compatible API means your local deployment speaks the same protocol the industry is standardizing on. The Cerebras partnership makes the headline demo fast, but the architecture is honest about being modular — and a Gemma 4 12B on your own GPU makes a private assistant that’s genuinely pleasant to talk to. Start with the 12B, raise the context window before your first long conversation, and only reach for bigger models if answer quality — not speed — is what’s bothering you.

FAQ

Can I run the whole pipeline offline? Yes, after the first run. Model weights (Parakeet, your LLM via Ollama, Qwen3-TTS) are cached locally; with the LLM base URL pointed at localhost, no stage needs internet access.

Does it work on Apple Silicon? Yes — the repo ships MLX paths: Lightning Whisper MLX / MLX Audio Whisper for STT and mlx-lm for the LLM stage. A 16GB M-series Mac handles a 7B-class model plus TTS; 32GB+ is more comfortable for 12B.

Why is my assistant slow to start answering even with a small model? Check that Ollama actually loaded the model onto the GPU (ollama ps should not say “cpu”), and that the model isn’t being evicted between turns by another workload. Cold-loading a 7GB model adds seconds to the first reply after idle periods; OLLAMA_KEEP_ALIVE=1h prevents mid-conversation unloads.

Sources

  • RTX 3060 12GB — the budget floor for a fully local real-time voice pipeline with a 7B-class LLM
  • RTX 3090 24GB — comfortable headroom for Gemma 4 12B QAT plus STT and TTS resident simultaneously

Was this article helpful?