Verified project record

huggingface/speech-to-speech

A modular voice-agent pipeline that connects VAD, STT, LLM, and TTS components via swappable backends selected through CLI flags. It exposes an OpenAI Realtime-compatible WebSocket API for building and deploying voice interactions.

Project overview

Project type: AI Agent · Model Runtime · Audio & Speech
Use cases: Automation
Deployment: Refer to project documentation
License: Apache-2.0

Best for

Developers and AI engineers building low-latency voice agents who need to swap between local or hosted STT, LLM, and TTS models.
Teams looking to connect existing OpenAI Realtime-compatible clients to a self-hosted, customizable voice pipeline.

Key capabilities

Runs VAD, STT, LLM, and TTS components in their own threads with swappable backends selected via CLI flags.
Exposes a WebSocket server at /v1/realtime implementing the OpenAI Realtime protocol for voice interactions.
Uses the machine's microphone and speakers to talk to the pipeline directly, without needing a separate client.
Streams raw audio bytes over a WebSocket for minimal custom clients without the Realtime protocol.
Streams raw PCM audio over a TCP socket for simple microphone/playback client setups.
Supports multi-language setups by pairing STT, LLM, and TTS backends covering the target languages.
Provides streaming TTS with voice cloning and custom voice files using Pocket TTS.
Supports LLM streaming text and tool calls during voice agent interactions.

Limitations and risks

TCP socket mode streams raw PCM audio but does not provide the full Realtime API feature set.

Getting started

Setup involves a simple pip install and running a single command to start the service, requiring no initial coding.

Evidence and sources

GitHub project description: Build local voice agents with open-source models
README: A low-latency, fully modular voice-agent pipeline: **VAD -> STT -> LLM -> TTS**, exposed through an **OpenAI Realtime-compatible WebSocket API**. Every component is swappable.
README: pip install speech-to-speech export OPENAI_API_KEY=... speech-to-speech
README: The pipeline is a cascade of four components, each running in its own thread and connected by queues: 1. **Voice Activity Detection (VAD)**: [Silero VAD v5](https://github.com/sna…
README: Requires Python 3.10+. ```bash pip install speech-to-speech ```

Reviewed at: 2026-07-16T20:21:18Z