Crawlee is a Node.js library for building reliable web scraping and browser automation crawlers that can avoid modern bot protections. It provides a unified interface for HTTP and headless browser crawling with built-in fingerprint generation and proxy rotation.
Project overview
The library addresses anti-bot protections by generating human-like browser fingerprints and replicating TLS fingerprints, consolidating HTTP and headless browser crawling behind a single interface.
Project type
Infrastructure
Use cases
Data Analysis · Automation
Deployment
Refer to project documentation
License
Apache-2.0
Best for
Developers building web scraping and browser automation crawlers in a JavaScript or TypeScript environment who need built-in routing, proxy rotation, and anti-bot fingerprint generation.
Key capabilities
Provides a single interface for HTTP and headless browser crawling, supporting Playwright, Puppeteer, Cheerio, and JSDOM.
Generates human-like browser fingerprints and replicates browser TLS fingerprints to fly under the radar.
Limitations and risks
To reduce install size, browser libraries like Playwright are not bundled with Crawlee and must be installed separately.
Getting started
Setup difficulty is rated medium because it requires writing JavaScript or TypeScript code. After installing the library in a Node.js project, developers can use the command line to navigate to the crawler directory (e.g., 'cd my-crawler') and run the start command (e.g., 'npm start').
Evidence and sources
GitHub project description: Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML,…
README: Your crawlers will appear human-like and fly under the radar of modern bot protections even with the default configuration.
README: The fastest way to try Crawlee out is to use the **Crawlee CLI** and choose the **Getting started example**. The CLI will install all the necessary dependencies and add boilerplat…
README: Because it uses `PlaywrightCrawler` we also need to install [Playwright](https://playwright.dev). It's not bundled with Crawlee to reduce install size.
README: By default, Crawlee stores data to `./storage` in the current working directory.