Evaluation & Observability Open-Source Projects

Showing 32 indexable projects from the current snapshot.

• BerriAI/litellm

  LiteLLM provides a unified gateway and Python SDK for calling 100+ LLM providers using the OpenAI format, addressing the operational complexity of managing disparate SDKs and authentication patterns. It includes production-ready features s…

• langfuse/langfuse

  An open-source platform that helps teams collaboratively observe, evaluate, and debug LLM calls. It provides tracing, prompt management, and testing capabilities for AI engineering workflows.

• mlabonne/llm-course

  A free, structured educational course covering Large Language Model theory and practical application from fundamentals to deployment, organized into three progressive sections with interactive Colab notebooks. It is intended for researcher…

• netdata/netdata

  Netdata provides per-second, real-time infrastructure monitoring with auto-discovery and zero configuration. It is self-hosted, operates with low resource overhead, and includes edge-based ML anomaly detection.

• unslothai/unsloth

  Unsloth provides a local web UI and Python library for running and training large language models with reduced VRAM usage and faster speeds. It supports text, audio, embedding, and vision models, and operates via local or self-hosted execu…

• microsoft/promptflow

  PromptFlow is a development framework for building, testing, and deploying LLM-based applications by linking prompts, Python code, and other tools into executable flows. Teams adopt it to manage the cycle from prototyping to production dep…

• Arize-ai/phoenix

  Open-source AI observability and evaluation platform for LLM applications. Provides OpenTelemetry-based tracing, LLM evaluation, versioned datasets, and experiment tracking.

• SigNoz/signoz

  SigNoz provides a single platform for logs, metrics, traces, alerts, and dashboards. It is built on OpenTelemetry and ClickHouse, offering self-hosted deployment alongside a managed cloud service with usage-based pricing.

• aishwaryanr/awesome-generative-ai-guide

  A curated, hosted directory that organizes generative AI resources into goal-oriented learning paths, topic pages, and role-based interview preparation. It is consumed as a static web page on GitHub and requires no setup or coding.

• mlflow/mlflow

  MLflow is an open-source AI engineering platform that helps teams debug, evaluate, monitor, and optimize production AI applications including agents and LLMs. It provides tracing, evaluation, prompt management, an AI gateway, and tradition…

• promptfoo/promptfoo

  promptfoo is a CLI and library for declaratively testing prompts, models, and RAG systems, and for red-teaming LLM applications to identify vulnerabilities. Evals run locally so prompt data stays on your machine, while integration with ext…

• alirezarezvani/claude-skills

  This project provides 358 reusable skills, agents, and personas across 18 domains to add domain-specific expertise to 13 supported AI coding tools. It is intended for developers and teams using CLI-based coding agents who need modular, cro…

• recommenders-team/recommenders

  A repository providing Python utilities, examples, and best practices for prototyping, evaluating, and operationalizing recommendation systems. It covers classical and deep learning algorithms and supports a workflow from data preparation…

• comet-ml/opik

  Opik provides tracing, automated evaluation, and monitoring for generative AI applications, RAG systems, and agentic workflows. It supports local self-hosting via Docker Compose or a managed hosted service, and is built to handle up to 40…

• EthicalML/awesome-production-machine-learning

  A categorized directory of open source libraries for deploying, monitoring, versioning, scaling, and securing production machine learning systems. It includes a search toolkit to help users navigate the machine learning toolchain.

• trycua/cua

  An open-source framework enabling AI agents to autonomously operate computer interfaces and run in isolated sandboxes across operating systems. It provides background input drivers, a cross-OS sandbox API, Apple Silicon virtualization mana…

• steipete/CodexBar

  A macOS 14+ menu bar application that displays usage limits, countdowns to the next reset window, and spend dashboards for OpenAI Codex and Claude Code. It reuses existing provider sessions for authentication and includes a bundled CLI for…

• pydantic/pydantic-ai

  A Python agent framework that provides type safety, structured output validation via Pydantic, and model-agnostic provider support. Developers define agents programmatically in Python to build generative AI applications that validate LLM o…

• confident-ai/deepeval

  An open-source evaluation framework, modeled after Pytest, for unit testing and evaluating large language model applications. It provides ready-to-use metrics to help development teams assess LLM quality, compare model outputs, and detect…

• raga-ai-hub/RagaAI-Catalyst

  An SDK for evaluating, monitoring, and safeguarding LLM and agentic AI applications through project management, tracing, prompt management, and guardrails. It requires Python coding and external authentication to the RagaAI Catalyst Platfo…

• apache/doris

  apache/doris is a unified database built on MPP architecture for fast SQL analytics, lakehouse query acceleration, streaming ingestion, and hybrid search across structured, text, and vector data. It targets real-time analytics, lakehouse,…

• ConardLi/easy-dataset

  An application that converts domain-specific documents in formats such as PDF, Markdown, DOCX, TXT, and EPUB into structured datasets for LLM fine-tuning, RAG, and model evaluation. It provides multiple text splitting algorithms and a huma…

• millionco/react-doctor

  react-doctor deterministically scans codebases to find React issues across state, effects, performance, architecture, security, and accessibility. It works with React frameworks including Next.js, Vite, TanStack, React Native, and Expo, an…

• NVIDIA/SkillSpector

  skillspector is a security tool that inspects AI agent skills for vulnerabilities and malicious patterns before installation, using static analysis and optional LLM-based semantic evaluation. It accepts Git repositories, URLs, zip files, d…

• Data-Centric-AI-Community/fg-data-profiling

  fg-data-profiling is a Python library that delivers extended exploratory data analysis and data quality profiling for Pandas and Spark DataFrames through a single line of code. Generated profiling reports can be exported in multiple format…

• ShishirPatil/gorilla

  gorilla provides fine-tuned large language models for function calling, benchmarking, an execution engine, and a curated API dataset to build and evaluate LLM tool-calling while reducing hallucination.

• jacobgil/pytorch-grad-cam

  pytorch-grad-cam is a PyTorch library for producing pixel-attribution visualizations that help diagnose computer vision model predictions. It provides class activation map methods and related tools for evaluating and refining explanations.

• zalandoresearch/fashion-mnist

  Fashion-MNIST is a dataset that replaces the original MNIST benchmark with a more challenging image classification task while preserving the same image size and train/test structure. It loads directly in mainstream machine learning librari…

• langchain-ai/open_deep_research

  An open-source deep research agent that coordinates LLMs, search APIs, and MCP servers to produce automated research reports. It is configurable across many model providers and search tools and can be run locally or deployed to the Open Ag…

• Farama-Foundation/Gymnasium

  Gymnasium is a Python library that provides a standard API for developing and comparing reinforcement learning algorithms and environments, maintained as an official fork of OpenAI's Gym. It ships with families of reference environments an…

• kubeshark/kubeshark

  Kubeshark captures and indexes cluster-wide Kubernetes network traffic at the kernel level using eBPF, providing deep visibility without requiring code instrumentation. It includes built-in TLS decryption and an MCP server to enable AI age…

• KalyanKS-NLP/llm-engineer-toolkit

  llm-engineer-toolkit is a curated directory of over 120 LLM engineering libraries organized into 14 categories spanning the LLM lifecycle from training to monitoring. The repository serves as a read-only reference list to improve discovera…