Verified project record

GetStream/Vision-Agents

A Python framework for building multi-modal real-time video and voice AI agents. It streams audio and video to model providers with low latency, orchestrates tool calls, and supports Kubernetes deployment.

Project overview

Project type: MCP · AI Agent · RAG
Use cases: Knowledge Q&A · Automation
Deployment: Refer to project documentation
License: Apache-2.0

Best for

Developers and AI engineers building low-latency, multi-modal real-time video and voice AI applications.

Key capabilities

Streams video directly to model providers for visual understanding and supports pluggable processor pipelines for YOLO, Roboflow, or custom PyTorch/ONNX models.
Enables natural conversation flow using Voice Activity Detection, diarization, and turn-taking.
Executes code and APIs mid-conversation, supporting Linear issues, weather, telephony, or any MCP server.
Provides retrieval-augmented generation with TurboPuffer/Qdrant vector search or Gemini FileSearch.
Supports inbound and outbound voice calls via Twilio or Telnyx with bidirectional audio streaming.
Agents recall context across turns and sessions via Stream Chat, and support silent text back-channel messaging during calls.
Includes a built-in HTTP server, Prometheus metrics, horizontal scaling, and Kubernetes deployment support.
Uses models like Decart Lucy to build virtual try-ons, stylized scenes, or give agents a visual identity.

Limitations and risks

Video AI struggles with small text, and models may hallucinate scores or signs.
Context degrades in longer sessions of approximately 30 seconds or more for continuous video understanding.
Real-time models require audio or text to trigger responses; video alone will not prompt output.
Most use cases require a mix of specialized models, such as YOLO or Roboflow, alongside larger LLMs.

Getting started

Install vision-agents via uv, obtain a Stream API key, and write a Python script using the Agent class to set up an edge, LLM, and processors.

Evidence and sources

GitHub project description: Open Vision Agents by Stream. Build voice and vision agents quickly with any model or video provider. Uses Stream's edge network for ultra-low latency.
README: [Vision Agents](https://visionagents.ai/) give you the building blocks to create intelligent, low-latency video experiences powered by your models, your infrastructure, and your u…
README: Multi-modal AI agents that watch, listen, and understand video.
README: - **Low Latency:** Join quickly (500ms) and maintain audio/video latency under 30ms using [Stream's edge network](https://getstream.io/video/?utm_source=github.com&utm_medium=refe…
README: **Step 1: Install via uv** `uv add vision-agents` **Step 2: (Optional) Install with extra integrations** `uv add "vision-agents[getstream, openai, elevenlabs, deepgram]"` **Step 3…

Reviewed at: 2026-07-16T12:28:53Z