Verified project record

huggingface/speech-to-speech

A modular voice-agent pipeline that combines VAD, STT, LLM, and TTS components in separate threads with swappable backends selected via CLI flags. It exposes an OpenAI Realtime-compatible WebSocket API and supports local microphone interaction, raw audio streaming, and multi-language configurations.

Project overview

The pipeline implements an OpenAI Realtime-compatible WebSocket server at /v1/realtime, allowing integration with voice clients while allowing developers to swap every component via CLI flags.

Project type: AI Agent · Model Runtime · Audio & Speech
Use cases: Meeting Notes · Translation & Subtitles · Audio & Speech · Automation
Deployment: Refer to project documentation
License: Apache-2.0

Best for

Developers and AI engineers building low-latency voice agents who need to swap VAD, STT, LLM, and TTS components individually.
Teams integrating voice interactions via the OpenAI Realtime protocol or custom raw PCM audio streaming.

Key capabilities

Runs VAD, STT, LLM, and TTS components in their own threads, with swappable backends selected via CLI flags.
Exposes a WebSocket server at /v1/realtime implementing the OpenAI Realtime protocol for voice interactions.
Uses the machine's microphone and speakers to talk to the pipeline directly, without needing a separate client.
Streams raw audio bytes over a WebSocket for minimal custom clients without the Realtime protocol.
Streams raw PCM audio over a TCP socket for simple microphone/playback client setups.
Supports automatic language detection and language switching by pairing STT, LLM, and TTS backends covering the target languages.
Provides streaming TTS with voice cloning and custom voice files using Pocket TTS.
Supports LLM streaming text and tool calls during voice agent interactions.

Limitations and risks

TCP socket mode streams raw PCM audio but does not provide the full Realtime API feature set.

Getting started

Setup is rated easy and involves a quickstart path of installing the package via pip and running the speech-to-speech command, requiring no coding.

Evidence and sources

GitHub project description: Build local voice agents with open-source models
README: A low-latency, fully modular voice-agent pipeline: **VAD -> STT -> LLM -> TTS**, exposed through an **OpenAI Realtime-compatible WebSocket API**. Every component is swappable.
README: pip install speech-to-speech export OPENAI_API_KEY=... speech-to-speech
README: The pipeline is a cascade of four components, each running in its own thread and connected by queues: 1. **Voice Activity Detection (VAD)**: [Silero VAD v5](https://github.com/sna…
README: Requires Python 3.10+. ```bash pip install speech-to-speech ```

Source updated at: 2026-07-15T18:18:53Z

Open original GitHub project