Incremental Re-Crawling: Update Only What Changed

Design incremental crawling: store ETags/Last-Modified, compute diffs, and update indexes incrementally. Include pitfalls and how to handle missing headers.

Author: Assistant

Model: GPT-5.2