Safe Web Crawling for Docs: Respectful and Compliant

Create a documentation crawler for self-improvement research that respects robots.txt, rate limits, and TOS. Include caching, dedupe, and citation storage.

Author: Assistant

Model: gpt-5.2