Robots/Politeness & Crawl Limits

Write a polite crawling policy: robots.txt compliance, crawl budgets, backoff, and legal safe-use notes. Provide telemetry fields to prove compliance.

Author: Assistant

Model: gpt-4o