Needle is a 26m-parameter single-shot function calling model designed to run on small consumer devices such as mobiles, wearables, and smart home gadgets, achieving 1–6k tokens/sec. It can be finetuned locally for function calling on personal devices using synthetic data generated via Google Gemini.
Project overview
At 14mb in size with 26m parameters, the model reaches 6000 toks/sec prefill speed on Cactus, making single-shot function calling feasible on hardware as constrained as wearables and smart home gadgets.
Deployment
Refer to project documentation
License
MIT
Best for
Developers and AI engineers building single-shot function calling into mobiles, wearables, and smart home gadgets where a 14mb model with 26m parameters can fit on-device.
Key capabilities
A simple attention network designed for single-shot function calling to map text queries to tool calls.
Generates synthetic training data via Google Gemini for finetuning the model. Google Gemini API is a paid service used optionally for this step.
Provides a web interface to test and finetune the model on custom tools.
Limitations and risks
The model lacks capacity for conversational settings compared to larger models.
Small models can be finicky, so behavior may require careful tuning during finetuning.
Fewer than 120 examples per tool will overfit, resulting in perfect training metrics but poor generalization.
Getting started
Setup is rated easy because the quickstart involves a simple clone and source command. Run: git clone https://github.com/cactus-compute/needle.git, then cd needle && source ./setup, then needle playground.