crawl4ai converts web pages into clean Markdown and structured JSON for retrieval-augmented generation and data pipelines. It supports self-hosted library and Docker deployments without mandatory API tokens, while LLM-driven extraction requires a separate local or remote model provider.
Project overview
The project combines controllable deep crawling, browser automation, and structured data extraction without mandatory API keys or paid subscriptions, addressing a common bottleneck in building data ingestion pipelines.
Project type
RAG · Data Processing
Use cases
Knowledge Q&A
Deployment
Python environment
License
Apache-2.0
Best for
Developers and data teams needing to convert web pages into clean Markdown or structured JSON without relying on mandatory API keys.
AI engineers building retrieval pipelines who require configurable browser sessions, proxies, and user hooks for data collection.
Key capabilities
Generates clean Markdown using heuristic filtering such as BM25 and includes citation references for LLM input.
Extracts structured JSON from pages using LLM-driven strategies or CSS/XPath schema-based selectors.
Provides browser control features including managed profiles, sessions, proxies, and stealth mode to navigate bot detection.
Supports deep crawling with BFS/DFS strategies, crash recovery, and full-page scanning for dynamic content.
Offers a Dockerized FastAPI server equipped with browser pooling, a real-time monitoring dashboard, and an interactive playground.
Includes Model Context Protocol integration to connect directly with AI tools such as Claude Code via Docker.
Limitations and risks
The synchronous version relying on Selenium is deprecated and scheduled for removal.
LLM-driven extraction requires configuring an external or local LLM provider; using proprietary models like OpenAI involves external API calls and does not guarantee a strictly offline data boundary.
A GPU is not required to run the base crawling and extraction operations.
Getting started
Install the package via pip or run the provided single Docker command.
After installing via pip, run the crawl4ai-setup tool to prepare the environment.
Execute crawl and extraction tasks using the command-line interface or the provided API.
Alternatives and comparisons
A data framework providing data connectors, data structuring, and retrieval interfaces for LLM applications.
An incremental data pipeline engine that turns codebases, notes, PDFs, and other sources into live context for AI agents.
Compiles raw documents into a persistent, interlinked wiki-style knowledge base using LLMs without a vector database.