apify/crawlee-python

Project overview

By offering a single unified interface that covers both lightweight HTTP scraping and full headless-browser crawling, crawlee-python provides integrated proxy rotation and session management for crawling workflows that target JavaScript-heavy sites and bot-protection mechanisms.

Project type

Data Processing

Use cases

Automation

Deployment

Refer to project documentation

License

Apache-2.0

Best for

• Developers who need a single library for both HTTP and headless-browser web crawling
• Data teams building scrapers that must handle modern bot protections

Key capabilities

• Downloads web pages using an HTTP library and parses HTML content for data extraction without launching a browser.
• Uses a headless browser to download web pages and provides an API for extracting data from JavaScript-heavy sites.
• Integrated proxy rotation and session management designed to bypass bot protections.
• Command line interface to quickly create crawlers from prepared project templates.
• Provides a CLI for creating crawler projects and selecting templates.

Limitations and risks

• The HTTP-based BeautifulSoupCrawler cannot execute client-side JavaScript; sites that require JavaScript rendering need the PlaywrightCrawler instead.
• Despite integrated proxy rotation and session management, bot protections may still block crawlers. Users should verify that their target sites remain accessible before committing to large-scale runs.

Getting started

• Setup is rated easy because crawlee-python installs as a standard Python package via pip.
• The CLI can be used to create crawler projects and select prepared templates, and the uv package manager is supported.