Crawls web pages from a URL and generates knowledge files formatted for uploading to OpenAI to create custom GPTs or assistants. Developers configure crawl parameters in a config.ts file or trigger crawling programmatically via an API.
Project overview
Automates the manual assembly and formatting of web content into knowledge files for custom OpenAI GPTs.
Project type
RAG
Use cases
Knowledge Q&A
Deployment
Refer to project documentation
License
ISC
Best for
Developers needing to convert web content into formatted knowledge files for custom OpenAI GPTs or assistants.
Key capabilities
Crawls web pages starting from a URL and follows matching links to collect content.
Generates an output file from crawled content formatted for uploading to OpenAI to create custom GPTs or assistants.
Can be run as an API to trigger crawling programmatically.
Limitations and risks
A paid ChatGPT plan may be required to access the custom GPT upload feature.
Getting started
Clone the repository, run npm install to install dependencies, edit the simple config.ts file to set parameters, and run the crawler.
Evidence and sources
README: Crawl a site to generate knowledge files to create your own custom GPT from one or multiple URLs
README: This project crawled the docs and generated the file that I uploaded as the basis for the custom GPT.
README: Be sure you have Node.js >= 16 installed.
README: npm i
README: Open [config.ts](config.ts) and edit the `url` and `selector` properties to match your needs.