Crawl from one or more starting web pages, fetch readable content, search within pages, follow relevant links, and stop when the requested information is found or a bounded limit is reached.
Tasks
Crawl from one or more start URLs, fetch readable content, search within pages, follow relevant links, and stop when the requested information is found or a bounded limit is reached
Fetch one page to extract compact readable text, links, and query snippets using scripts/fetch-page.mjs
Use scripts/crawl-web.mjs to fetch pages, search within content, and follow relevant links for broader crawling
Inputs
Start URLs
Search query or target phrase
Max pages
Max depth
Owner-provided URL as starting point
Outputs
textPreview: short readable preview
snippets: query matches with nearby context
links: normalized URLs with short labels and context
chunk.hasMore and chunk.nextOffset for fetching more text
Direct answer followed by concise source notes including decisive source URL and any remaining uncertainty
Limitations and checks
Default max pages: 8
Default max link depth from the starting page: 2
Default max pages from the same domain: 5
Do not build $LEON_HOME/bin/node/... or /bin/node/... paths manually
Avoid links that are likely unrelated, duplicated, navigational noise, ads, tracking links, login-only pages, or broad category pages unless they are the best available path
Confirm scripts are run with node and no manual $LEON_HOME/bin/node or /bin/node paths are constructed
Verify that visited URLs are tracked and the same page is not revisited
Check that crawling stops early when target information is found with enough context or when a bounded limit is reached