GPTCache reduces LLM API costs and improves response times by caching semantically similar query results. Developers can integrate it as a Python library or a Docker container to avoid redundant LLM API calls as application traffic grows.
Project overview
GPTCache reduces LLM API costs and improves response times by caching semantically similar query results. Developers can integrate it as a Python library or a Docker container to avoid redundant LLM API calls as application traffic grows.
Project type
RAG · Infrastructure
Deployment
Refer to project documentation
License
MIT
Best for
Developers and AI engineers looking to implement a self-hosted library for caching LLM API responses.
Key capabilities
Stores LLM responses and retrieves them based on semantic similarity to avoid redundant LLM API calls.
Caches and retrieves responses for exact identical queries.
Provides standardized interfaces to integrate and cache responses from various LLM APIs like OpenAI, LangChain, and Llamacpp.
Offers customizable components for embedding generation, cache storage, vector store, similarity evaluation, and eviction.
Integrates and caches responses from large multimodal models for image generation and audio transcription.
Supports horizontal scaling using distributed cache stores like Redis or memcached.
Provides a Docker image serving a server that allows any programming language to use GPTCache.
Limitations and risks
Eviction based solely on line count may cause out-of-memory errors.
The API may be subject to change at any time due to swift development.
Getting started
Installable via pip with automatic installation of optional libraries. Requires Python 3.8.1 or higher. First success path is running 'pip install gptcache'.
Evidence and sources
README: GPTCache : A Library for Creating Semantic Cache for LLM Queries
README: Slash Your LLM API Costs by 10x 💰, Boost Speed by 100x ⚡
README: we have created GPTCache, a project dedicated to building a semantic cache for storing LLM responses.
README: Before running the example, **make sure** the OPENAI_API_KEY environment variable is set by executing `echo $OPENAI_API_KEY`.