open-compass/opencompass

Project overview

OpenCompass is an open-source evaluation framework for large language models, offering a reproducible benchmarking platform across 70+ datasets. It supports both open-source and API models, distributed evaluation, and inference acceleration backends.

Project type

AI Agent · Model Development · Evaluation & Observability

Deployment

Refer to project documentation

License

Apache-2.0

Best for

• Researchers and AI engineers who need a reproducible, one-stop platform for benchmarking open-source and API models across a large number of datasets.

Key capabilities

• Assesses the capabilities of models in multiple dimensions using 70+ datasets and about 400,000 questions.
• Implements task division and distributed evaluation to complete the full evaluation of billion-scale models in just a few hours.
• Provides GenericLLMEvaluator for LLM-as-judge evaluations.
• Supports one-click switching between inference acceleration backends like LMDeploy and vLLM.
• Allows evaluating both open-source and API models in the same way without discrimination.
• Support various agent frameworks.

Limitations and risks

• The setup difficulty is rated medium because it requires configuring a conda environment and managing dataset downloads manually if not done automatically.
• GPU requirements, minimum hardware specifications, operating system constraints, and telemetry policies are not documented.

Getting started

• Setup involves creating a conda environment and installing the framework via pip. Users must manage dataset downloads manually if automatic fetching does not occur.
• The initial success path requires creating a conda environment, installing via pip, downloading datasets, and running an evaluation.

Evidence and sources

• GitHub project description: OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.
• README: OpenCompass will guide you through the complex landscape of evaluating large language models. With its powerful algorithms and intuitive interface, OpenCompass makes it easy to as…
• README: OpenCompass is a one-stop platform for large model evaluation, aiming to provide a fair, open, and reproducible benchmark for large model evaluation.
• README: We are delighted to announce that **the OpenCompass has been recommended by the Meta AI**, click [Get Started](https://ai.meta.com/llama/get-started/#validation) of Llama for more…
• README: CompassHub presents a pioneering benchmark browser interface, designed to simplify and expedite the exploration and utilization of an extensive array of benchmarks for researchers…

Reviewed at: 2026-07-16T23:27:24Z