Operate the just-scrape CLI to search the web, scrape URLs, extract structured JSON, crawl site sections, and monitor pages for changes using the ScrapeGraph AI API.
Tasks
Scrape a known URL and extract the page title and author into a JSON file using `just-scrape extract "<url>" -p "Extract title and author" --json > .just-scrape/extract-title-author.json`
Crawl a documentation site section restricted by depth and patterns using `just-scrape crawl "<url>" --max-pages 50 --max-depth 3 --include-patterns '["^https://example\\.com/docs/.*"]'`
Inputs
Specific URLs for scraping or extraction
Search queries and AI prompts for structured data extraction
Max pages, max depth, and URL include/exclude patterns for crawling
Outputs
Markdown, HTML, screenshots, links, images, summaries, or structured JSON
Past request IDs, statuses, and payloads via the history command
Remaining API credits and API health validation results
Limitations and checks
Operations consume API credits and high-cost features like stealth, crawling, and JS rendering increase usage
Not intended for integrating ScrapeGraph AI into application code or adding API keys to product `.env` files
Output files can be large; never read entire files at once, use incremental reads or tools like `rg` and `jq`
Confirm the CLI is installed and authenticated before working by running `just-scrape validate` and `just-scrape credits`
Quote all URLs in shell commands to prevent the shell from misinterpreting `?` and `&` characters
Verify setup with a small test request such as `just-scrape scrape "https://example.com" --json > .just-scrape/install-check.json` before executing bulk tasks
Check the `.just-scrape/` directory for existing data and review credit balance before parallelizing operations