An open-source framework providing drivers, APIs, and benchmarks for AI agents to autonomously operate computer interfaces in isolated sandboxes across macOS, Linux, and Windows. It enables background desktop automation and cross-OS virtualization management on Apple Silicon.
Project overview
It provides background drivers that allow agents to interact with native desktop apps without hijacking the user's cursor or focus, alongside a unified API for managing local or cloud virtual machines.
Project type
AI Agent · Evaluation & Observability · Infrastructure
Use cases
Automation
Deployment
Refer to project documentation
License
MIT
Best for
Developers and AI engineers building agents that need to autonomously operate computer interfaces and interact with native desktop applications.
Researchers requiring benchmarks and trajectory exports to evaluate and train computer-use agents.
Users on Apple Silicon needing to create and manage macOS/Linux virtual machines for isolated execution.
Key capabilities
Drive native desktop apps in the background. Agents click, type, and verify without stealing the cursor or focus.
One API to build agents that see screens, click buttons, and complete tasks autonomously across any VM or container image, cloud or local.
Evaluate computer-use agents on OSWorld, ScreenSpot, Windows Arena, and custom tasks. Export trajectories for training.
Create and manage macOS/Linux VMs with near-native performance on Apple Silicon using Apple's Virtualization.Framework.
Limitations and risks
The Sequoia macOS VM flow may still open the Accessibility step of Setup Assistant on its first display boot.
Linux supports X11 and compositor-specific Wayland routes with explicit limits for raw background input.
The optional cua-agent[omni] package includes ultralytics, which is licensed under AGPL-3.0.
Creating and managing macOS/Linux VMs with near-native performance using Lume requires Apple Silicon hardware.
Getting started
Install the package using pip (requiring Python 3.11+) or download Apple restore images for VMs. After installation, import and initialize the Sandbox or driver in code to execute commands or automate desktop apps.
Evidence and sources
GitHub project description: Scale computer-use 2.0 with open-source drivers, cross-OS fleets, and benchmarks for training, evaluation, and data generation.
README: Drive native desktop apps **in the background**. Agents click, type, and verify without stealing the cursor or focus.
README: Use the same CLI and MCP server on macOS, Windows, and Linux from Claude Code, Cursor, Codex, OpenClaw, and custom clients.
README: Linux supports X11 and compositor-specific Wayland routes with explicit limits for raw background input.
README: Build agents that see screens, click buttons, and complete tasks autonomously. One API for any VM or container image — cloud or local.