apify/crawlee

Project overview

Crawlee is an open-source Node.js library for building reliable web scraping and browser automation crawlers. It provides a single interface for HTTP and headless browser crawling, with built-in proxy rotation, routing, data extraction, and anti-bot fingerprint generation.

Project type

Infrastructure

Use cases

Data Analysis · Automation

Deployment

Refer to project documentation

License

Apache-2.0

Best for

• Developers working in a Node.js environment who need programmatic control over web scraping tasks using JavaScript or TypeScript.

Key capabilities

• Provides a single interface for HTTP and headless browser crawling, supporting Playwright, Puppeteer, Cheerio, and JSDOM.
• Generates human-like browser fingerprints and replicates browser TLS fingerprints to fly under the radar.

Limitations and risks

• Browser libraries like Playwright are not bundled with Crawlee to reduce install size and must be installed separately.

Getting started

• Requires a Node.js project environment where users write JavaScript or TypeScript code. The first success path involves navigating to the crawler directory and running npm start.

Evidence and sources

• GitHub project description: Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML,…
• README: Your crawlers will appear human-like and fly under the radar of modern bot protections even with the default configuration.
• README: The fastest way to try Crawlee out is to use the **Crawlee CLI** and choose the **Getting started example**. The CLI will install all the necessary dependencies and add boilerplat…
• README: Because it uses `PlaywrightCrawler` we also need to install [Playwright](https://playwright.dev). It's not bundled with Crawlee to reduce install size.
• README: By default, Crawlee stores data to `./storage` in the current working directory.

Reviewed at: 2026-07-17T10:47:24Z