A hybrid edge-cloud AI engine providing OpenAI-compatible APIs for text, speech, and vision, with custom quantization for mobile devices. It enables low-latency inference on local mobile and wearable hardware using HuggingFace models.
Project overview
It applies a custom rotation-based quantization ranging from 4-bit to 1-bit across all weight tensors to compress models for mobile execution.
Project type
Model Runtime
Use cases
Chat Assistants · Automation
Deployment
Refer to project documentation
License
Apache-2.0
Best for
Developers and AI engineers seeking a framework to run AI inference on mobile and wearable devices using a hybrid edge-cloud engine.
Key capabilities
Provides C++ tensor operations for matrix multiplication, attention, normalization, and activations.
Applies custom quantization from 4-bit to 1-bit for all weight tensors.
Converts PyTorch models to a Cactus runtime graph for on-device inference.
Provides a terminal UI or print-based AI coding agent that can connect to a local server.
Performs live microphone transcription using a model.
Limitations and risks
Downloading and converting models requires access to the external HuggingFace service.
Setup difficulty is rated medium; while Mac installation is straightforward via Homebrew, other platforms require manual repository cloning and environment configuration.
The engine is documented as a hybrid edge-cloud system. The local runnable and self-hosted deployment options demonstrate that the component can run locally, but do not provide a direct decision that the system is free of cloud round-trips during all stages of operation.
The project documents a minimum hardware testing range from iPhone 15 Pro to Mac M5 Max. Performance on hardware outside this range is not documented.
Specific coding requirements and cost dependencies are not documented.
Getting started
On Mac, the framework can be installed via Homebrew (brew install cactus-compute/cactus/cactus) and started with the command 'cactus run'.
Installing on platforms other than Mac requires cloning the repository and configuring the environment manually.
Alternatives and comparisons
An SDK to run LLMs, vision, speech-to-text, and text-to-speech locally across iOS, Android, Flutter, React Native, and Web with optimized device routing.
A local or self-hosted LLM inference engine featuring zero-config Hugging Face model execution, multimodality, OpenAI and Anthropic API serving, and smart quantization.
A fast Apple-Silicon-native local AI engine providing drop-in OpenAI and Anthropic APIs, terminal chat, and agent integrations.