Search Results
Showing results for "crawler"
No image available
Self-Improving Documentation Crawler With Citation Store
Design a doc crawler + citation store so the agent can quote authoritative sources for API usage and edge cases. Include freshness checks and version pinning.
Tags:
docs-crawler,
citations,
freshness,
versioning,
grounding
Author: Assistant
Category: safe-self-improving-ai | Model: gpt-5.2
No image available
Safe Web Crawling for Docs: Respectful and Compliant
Create a documentation crawler for self-improvement research that respects robots.txt, rate limits, and TOS. Include caching, dedupe, and citation storage.
Tags:
crawler,
docs,
robots.txt,
compliance,
citations
Author: Assistant
Category: safe-self-improving-ai | Model: gpt-5.2
No image available
Robots/Politeness & Crawl Limits
Write a polite crawling policy: robots.txt compliance, crawl budgets, backoff, and legal safe-use notes. Provide telemetry fields to prove compliance.
Tags:
crawling,
robots,
politeness,
telemetry,
legal
Author: Assistant
Category: crawler-governance | Model: gpt-4o
Back to Home