Verified project record

FlashML-org/FreeToken

FreeToken is an inference runtime that runs 290B+ parameter frontier open-weight MoE models locally on consumer gaming PCs at interactive speeds. It treats GPUs, CPUs, host memory, and interconnects as a single elastic inference platform and exposes models through Anthropic/OpenAI-compatible APIs.

Project overview

It targets a concrete pain point—frontier-scale open-weight MoE models normally cannot run on personal consumer hardware at interactive speeds—and delivers local serving of 290B+ MoE models by unifying consumer hardware resources rather than requiring data-center GPUs.

Project type: Model Runtime
Use cases: Chat Assistants
Deployment: Refer to project documentation
License: Apache-2.0

Best for

Users with consumer gaming PCs who want to run 290B+ frontier open-weight MoE models locally at interactive speeds.
Developers building agent workflows that need a local MoE inference endpoint compatible with Anthropic/OpenAI APIs.

Key capabilities

Efficient MoE serving with bandwidth-adaptive CPU–GPU co-execution, full-layer double-buffered prefill streaming, global LRU expert caching, graph-compatible execution, and the FTW fast weight format.
Semantic anchor checkpoints for recurrent state and KV caches allow agentic context edits such as tool calls and thinking blocks to avoid redundant context recomputation.
Supports dynamic, runtime VRAM re-allocation between expert caches and KV memory without engine restarts or weight reloading.

Limitations and risks

Minimum hardware and GPU requirements are not documented; verify your consumer hardware can support the target model size before adopting.
Data boundary, telemetry, and cost dependency details are not documented; if these matter to your deployment, confirm them with the project before adoption.

Getting started

Install with a single documented command, uv pip install "freetoken[accel]", then run FreeToken with a supported open-weight MoE model per the quickstart docs. Setup difficulty is rated easy; coding is optional.

Evidence and sources

README: Run 290B+ frontier MoE models locally on your gaming PC at blistering interactive speeds.
README: FreeToken is an edge-native Mixture-of-Experts (MoE) serving engine designed for running frontier-scale open-weight models on personal and consumer hardware.
README: **Fast Edge-Native Runtime**: Provides efficient MoE serving with bandwidth-adaptive CPU–GPU co-execution ($q^\star$ policy), full-layer double-buffered prefill streaming, global…
README: **Semantic-Aware Caching**: Features semantic anchor checkpoints for recurrent state and KV caches, allowing agentic context edits (e.g., tool calls, thinking blocks) to avoid red…
README: **Elastic Memory Management**: Supports dynamic, runtime VRAM re-allocation between expert caches and KV memory without engine restarts or weight reloading.

Source updated at: 2026-11-09T18:36:38Z

Open original GitHub project