Crawl one or more websites and generate a knowledge file formatted for upload to OpenAI to create custom GPTs or assistants. Configure crawl parameters via a config.ts file or trigger crawls programmatically through an API.
Project overview
Crawl one or more websites and generate a knowledge file formatted for upload to OpenAI to create custom GPTs or assistants. Configure crawl parameters via a config.ts file or trigger crawls programmatically through an API.
Project type
RAG
Use cases
Knowledge Q&A
Deployment
Refer to project documentation
License
ISC
Best for
Developers who need to convert website content into formatted knowledge files for OpenAI custom GPTs or assistants.
Key capabilities
Crawls web pages starting from a URL and follows matching links to collect content.
Generates an output file from crawled content formatted for uploading to OpenAI to create custom GPTs or assistants.
Can be run as an API to trigger crawling programmatically.
Limitations and risks
A paid ChatGPT plan may be required to access the custom GPT upload feature needed to utilize the generated knowledge files.
Getting started
Clone the repository, run npm install to add dependencies, edit the config.ts file to set crawl parameters, and run the crawler.
Evidence and sources
README: Crawl a site to generate knowledge files to create your own custom GPT from one or multiple URLs
README: This project crawled the docs and generated the file that I uploaded as the basis for the custom GPT.
README: Be sure you have Node.js >= 16 installed.
README: npm i
README: Open [config.ts](config.ts) and edit the `url` and `selector` properties to match your needs.