Verified project record

unclecode/crawl4ai

Crawl4ai converts web pages into clean, LLM-ready Markdown and structured JSON for retrieval pipelines and AI agents. It provides controllable browser-based crawling and data extraction without mandatory API keys, operating as a self-hosted Python library or Docker service.

Project overview

Project type: AI Agent · RAG · Data Processing
Use cases: Knowledge Q&A · Search & Research
Deployment: Python environment
License: Apache-2.0

Best for

Developers and data teams needing self-hosted, controllable web crawling to generate Markdown for RAG pipelines without mandatory API keys.
AI engineers building systems that require structured JSON extraction or direct web context retrieval via MCP.

Key capabilities

Generates clean, structured Markdown with heuristic filtering such as BM25 and citation references for language models.
Extracts structured JSON from web pages using LLM-driven strategies or CSS/XPath schema-based selectors.
Provides full browser control with managed profiles, sessions, proxies, and stealth mode to avoid bot detection.
Supports deep crawling with BFS/DFS strategies, crash recovery, and full-page scanning for dynamic content.
Offers a Dockerized FastAPI server with a real-time monitoring dashboard, browser pooling, and an interactive playground.
Includes Model Context Protocol integration for direct connection to AI tools like Claude Code via Docker.

Limitations and risks

The synchronous version using Selenium is deprecated and will be removed in the future.
LLM-driven extraction requires an external or local LLM provider, such as OpenAI API or Ollama.

Getting started

Installation is rated easy. Users can install the package via pip or run a single Docker command. For pip installations, users should run the crawl4ai-setup post-installation tool.
After installation and setup, users can execute their first crawl via the command-line interface or the API.

Evidence and sources

GitHub project description: Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper.
README: pip install -U crawl4ai
README: Crawl4AI turns the web into clean, LLM ready Markdown for RAG, agents, and data pipelines.
README: Reliable, large-scale web extraction, now built to be _**drastically more cost-effective**_ than any of the existing solutions.
README: Full control, sessions, proxies, cookies, user scripts, hooks

Reviewed at: 2026-07-17T05:08:07Z