Explore Prompts

Design snippet generation: pick supporting passages, highlight query terms/entities, and store offsets for UI rendering. Include how to avoid misleading snippets.

Design a ranking formula: lexical relevance, semantic similarity, trust score, freshness decay, and diversity penalty. Include how to tune weights and validate offline/online.

Design indexing: title/body, entities, dates, authors, domain, language, and reliability score as facets. Include query patterns and how to support analyst workflows.

Design storage: raw HTML, cleaned text, metadata, embeddings, and caches. Include schema, retention rules, and how to support reprocessing when extractors improve.

If browser automation is required, design a policy: strict allowlists, minimal interaction, rate limits, and recording of steps. Include risk assessment and a user-facing disclaimer template.

Create an ethical strategy for JS-heavy pages: prefer official APIs, use server-rendered alternatives, and gracefully degrade. Include guidelines for when to skip a site and explain why.

Design an extractor for structured data: schema.org/JSON-LD, OpenGraph, and meta tags. Include how to validate and store structured fields for downstream search.

Plan how to use sitemaps and RSS/Atom feeds to discover content efficiently. Include scheduling, change detection, and fallback to crawl when feeds are missing.

Design a URL frontier with ranking signals: relevance to query, freshness, link context, domain trust, and duplication risk. Include pseudocode-level description and metrics.

Design a politeness policy: per-domain rate limiting, crawl-delay handling, time windows, and exponential backoff. Include instrumentation to prove compliance.

Create a crawl plan: seed selection, depth limits, domain allowlists, URL patterns, and prioritization heuristics. Include a method to estimate crawl cost and coverage.

Plan an alert system: define watchlists, schedule searches, diff results, and send notifications. Include dedupe, false positive controls, and ‘significance’ scoring.