Showing results for "crawling"
Draft constraints so research/crawling never performs intrusive activity; only access allowed APIs and public docs. Include a compliance checklist and enforcement.
Author: Assistant
Category: safe-self-improving-ai | Model: gpt-5.2
Create a documentation crawler for self-improvement research that respects robots.txt, rate limits, and TOS. Include caching, dedupe, and citation storage.
Author: Assistant
Category: safe-self-improving-ai | Model: gpt-5.2
Create a crawl plan: seed selection, depth limits, domain allowlists, URL patterns, and prioritization heuristics. Include a method to estimate crawl cost and coverage.
Author: Assistant
Category: research-bot | Model: GPT-5.2
Create a selection matrix for available APIs: search/SERP providers, content extraction, crawling, and anti-bot friendly approaches. Compare cost, rate limits, coverage, and TOS constraints.
Author: Assistant
Category: research-bot | Model: GPT-5.2
Design incremental crawling: store ETags/Last-Modified, compute diffs, and update indexes incrementally. Include pitfalls and how to handle missing headers.
Author: Assistant
Category: research-bot | Model: GPT-5.2
Design how to expose search/fetch/extract as MCP tools: schemas, permissions, and audit logs. Include example tool definitions and a safety policy for tool use.
Author: Assistant
Category: research-bot | Model: GPT-5.2
Plan how to use sitemaps and RSS/Atom feeds to discover content efficiently. Include scheduling, change detection, and fallback to crawl when feeds are missing.
Author: Assistant
Category: research-bot | Model: GPT-5.2
Design a politeness policy: per-domain rate limiting, crawl-delay handling, time windows, and exponential backoff. Include instrumentation to prove compliance.
Author: Assistant
Category: research-bot | Model: GPT-5.2
Draft a compliance-first plan: robots.txt handling, site terms, user-agent identification, rate limiting, and respectful crawling. Include an ethics checklist and escalation rules.
Author: Assistant
Category: research-bot | Model: GPT-5.2
Build a cost model: per-query search API costs, per-page fetch cost, storage, embeddings, and LLM tokens. Provide formulas and a budgeting plan with guardrails.
Author: Assistant
Category: research-bot | Model: GPT-5.2
Design a URL frontier with ranking signals: relevance to query, freshness, link context, domain trust, and duplication risk. Include pseudocode-level description and metrics.
Author: Assistant
Category: research-bot | Model: GPT-5.2
Design PII handling: detection, redaction, retention limits, and access controls. Include how to avoid collecting unnecessary personal data and how to respond to removal requests.
Author: Assistant
Category: research-bot | Model: GPT-5.2
Design a time-decay model that boosts fresh, authoritative sources. Include publisher priors, sitemap/news bias, and re-crawl triggers. Output evaluation with time-split test sets.
Author: Assistant
Category: ranking-freshness | Model: gpt-4o
Design a DAG (Airflow/Prefect) for reproducible research: crawl→retrieve→synthesize→verify→export. Include artifact hashing and cache invalidation rules.
Author: Assistant
Category: ops-pipelines-research | Model: gpt-4o
Write a polite crawling policy: robots.txt compliance, crawl budgets, backoff, and legal safe-use notes. Provide telemetry fields to prove compliance.
Author: Assistant
Category: crawler-governance | Model: gpt-4o
You are a nightlife guide. Plan an evening: observatory view, Omoide Yokocho bites, Golden Gai bars, late ramen. Include etiquette (cover charges), last train reminders, and safe taxi fallback.
Author: Assistant
Category: nightlife | Model: gpt-4o
Act as a day-trip planner. Build a loop: Minato Mirai skyline walk, Red Brick Warehouse, Cup Noodles Museum, Yokohama Air Cabin ride, and evening in Chinatown with snack crawl.
Author: Assistant
Category: itinerary | Model: gpt-4o
Act as a food sherpa. Design a one-day food crawl across Tokyo: breakfast depachika, ramen for lunch, coffee kissaten, conveyor-belt sushi vs omakase, and izakaya night. Include reservation wording an...
Author: Assistant
Category: food | Model: gpt-4o
Plan a winter street‑food crawl in Japan around regional nabe (yose, chanko, motsu). Output a table with: rank, stall/name, exact spot (market/station), signature dish, spice/heat index (0–5), best ti...
Author: Inspire Search Corp.
Category: Culinary (Winter) | Model: gpt-5-thinking
Plan a winter street‑food crawl in China around northern dumplings & steamed buns. Output a table with: rank, stall/name, exact spot (market/station), signature dish, spice/heat index (0–5), best time...
Author: Inspire Search Corp.
Category: Culinary (Winter) | Model: gpt-5-thinking