A Python library that uses LLMs and direct graph logic to build scraping pipelines for web pages and local documents from simple prompts. Developers configure an API key or local model runtime and define the target data to extract structured information.
Project overview
A Python library that uses LLMs and direct graph logic to build scraping pipelines for web pages and local documents from simple prompts. Developers configure an API key or local model runtime and define the target data to extract structured information.
Project type
RAG · AI Search · Data Processing
Use cases
Search & Research
Deployment
Refer to project documentation
License
MIT
Best for
Developers and data teams building scraping pipelines for websites or local documents using LLM-driven extraction.
Users who require local model execution via Ollama for on-premises data processing.
Key capabilities
Single-page scraper that extracts information given a user prompt and an input source.
Multi-page scraper that extracts information from the top n search results of a search engine.
Single-page scraper that extracts information from a website and generates an audio file.
Single-page scraper that extracts information from a website and generates a Python script.
Extracts information from multiple pages given a single prompt and a list of sources using parallel LLM calls.
Allows using different LLMs through APIs (OpenAI, Groq, Azure, Gemini, MiniMax) or local models via Ollama.
Creates scraping pipelines for local documents (XML, HTML, JSON, Markdown, etc.).
Collects anonymous usage metrics which can be disabled by setting an environment variable.
Limitations and risks
Requires managing browser rendering, proxies, and scaling when self-hosted.
Collects anonymous usage metrics by default; users must set an environment variable to opt out.
Getting started
Install via Python pip, run Playwright browser setup, configure LLM API keys or a local model runtime (Ollama), set up graph_config, and run SmartScraperGraph. The setup difficulty is rated medium.
Evidence and sources
README: [ScrapeGraphAI](https://scrapegraphai.com) is a *web scraping* python library that uses LLM and direct graph logic to create scraping pipelines for websites and local documents (X…
README: The reference page for Scrapegraph-ai is available on the official page of PyPI: [pypi](https://pypi.org/project/scrapegraphai/).
README: It is possible to use different LLM through APIs, such as **OpenAI**, **Groq**, **Azure**, **Gemini**, **MiniMax** and more, or local models using **Ollama**.
README: For OpenAI and other models you just need to change the llm config! > ```python >graph_config = { > "llm": { > "api_key": "YOUR_OPENAI_API_KEY", > "model": "openai/gpt-4o-mini", >…