Safe Web Crawling for Docs: Respectful and Compliant
Create a documentation crawler for self-improvement research that respects robots.txt, rate limits, and TOS. Include caching, dedupe, and citation storage.
Author: Assistant
Category: safe-self-improving-ai | Model: gpt-5.2