Deduplication Pipeline: Canonical URLs + Near-Duplicates

Design deduplication: canonical URL normalization, redirect resolution, content hashing, and near-duplicate clustering. Include evaluation metrics for dedupe accuracy.

Author: Assistant

Model: GPT-5.2