Showing results for "evals"
Create a loop where production failures and near-misses become new eval tests. The agent should propose test additions with minimal reproductions and acceptance criteria.
Author: Assistant
Category: safe-self-improving-ai | Model: gpt-5.2
Design retrieval that avoids irrelevant context: recency weighting, scope filters, and contradiction detection. Include evals for context precision/recall.
Author: Assistant
Category: safe-self-improving-ai | Model: gpt-5.2
Build an eval harness for self-edits: deterministic tool mocks, seeded randomness, replayable runs, and stored artifacts for auditing decisions.
Author: Assistant
Category: safe-self-improving-ai | Model: gpt-5.2
Design a prompt library that the agent can improve safely: semantic versioning, eval gates, canary prompts, and rollback. Include prompt linting rules.
Author: Assistant
Category: safe-self-improving-ai | Model: gpt-5.2
Design an AgentOps platform: deployment, permissions, observability, evals, and compliance reporting. Include buyer personas and monetization (seat vs usage).
Author: Assistant
Category: future-monetization | Model: gpt-5.2
Design a plan to self-improve ranking/retrieval safely: offline eval sets, interleaving/AB tests, bias checks, and rollback on metric drops.
Author: Assistant
Category: safe-self-improving-ai | Model: gpt-5.2
Generate a model/system card template: intended use, limitations, safety mitigations, eval results, and known failure modes. Include a changelog section for each iteration.
Author: Assistant
Category: recursive-ai-safety | Model: GPT-5.2
Create a policy and technical controls for PII/secrets: detection, redaction, encryption, and safe storage. Include test cases and a plan to prevent secret leakage into training/evals.
Author: Assistant
Category: recursive-ai-safety | Model: GPT-5.2
Design an evaluation strategy that avoids overfitting: holdouts, rotating test sets, adversarial sets, and blind evaluation. Include rules for when to refresh benchmarks.
Author: Assistant
Category: recursive-ai-safety | Model: GPT-5.2
Create a safe synthetic task generation plan: avoid sensitive content, prevent leakage, and validate usefulness with human review. Include how to measure whether synthetic tasks improve real outcomes.
Author: Assistant
Category: agent-architecture | Model: GPT-5.2
Define specialist agent roles with distinct prompts, tools, and constraints. Provide handoff rules and an evaluation plan to ensure specialists outperform a monolith.
Author: Assistant
Category: agent-architecture | Model: GPT-5.2
Design a RAG pipeline for agents: retrieval, citation, cross-checking, and contradiction handling. Provide an evaluation plan to measure hallucination reduction and tool-call accuracy.
Author: Assistant
Category: agent-architecture | Model: GPT-5.2
Create a 90-day plan to build a high-capability agent system using MCP and A2A: milestones, architecture decisions, eval harness, safety gates, and a pilot deployment. Include staffing assumptions and...
Author: Assistant
Category: agent-architecture | Model: GPT-5.2
Design a safe self-improvement loop for prompts/tools: propose change, run evals, red-team, canary, deploy, monitor. Include stop rules and audit artifacts.
Author: Assistant
Category: agent-architecture | Model: GPT-5.2
Design an eval harness: deterministic replays, seeded randomness, fixed tool mocks, and artifact snapshots. Provide a folder structure and CI integration plan.
Author: Assistant
Category: agent-architecture | Model: GPT-5.2
Produce a 100-day plan: quick wins, foundational work (governance/evals), first agent rollout, security controls, and stakeholder cadence. Include a weekly operating rhythm and deliverables.
Author: Assistant
Category: ai-strategy-2026 | Model: gpt-4o
Create an eval-first LLMOps design: golden sets, adversarial tests, continuous regression, cost/latency tracking, and release gates. Include a ‘model change control’ policy.
Author: Assistant
Category: ai-strategy-2026 | Model: gpt-4o
Design an evaluation harness that measures relevance and faithfulness for IR+LLM answers. Include human labeling rubric and inter-rater checks.
Author: Assistant
Category: eval-framework-IR-LLM | Model: gpt-4o
Specify a training stack with SFT on curated data, preference optimization (DPO/ORPO), and optional RLHF. Include reward hacking tests, guardrails, and evals that predict production behavior.
Author: Assistant
Category: training-pipeline-LLM | Model: gpt-4o
Write a data governance spec: license screening, PII scrubbing, near-duplicate collapse, contamination checks vs eval sets, and audit trails. Provide rejection reasons and exception handling.
Author: Assistant
Category: data-operations-LLM | Model: gpt-4o
Outline a privacy strategy: DP-SGD variants for SFT, selective redaction layers, privacy evals (membership inference), and logging minimization.
Author: Assistant
Category: privacy-engineering-LLM | Model: gpt-4o
Distill a 70B teacher into a 7–13B student: loss mixing (logits+features+policies), curriculum, and temperature tuning. Provide downstream eval deltas.
Author: Assistant
Category: model-compression-training | Model: gpt-4o
Build an eval harness: recall@k, calibrated precision, answer faithfulness, and human-time-to-verify. Include topic-aware test buckets and data drift alarms.
Author: Assistant
Category: evaluation-frameworks-LLM | Model: gpt-4o
Create a senior-grade eval battery: reasoning (math/code), instruction-following, safety, multilingual QA, and tool-use. Include uncertainty intervals and power analysis for A/Bs.
Author: Assistant
Category: evaluation-design-LLM | Model: gpt-4o
Train an LTR model to order passages by usefulness. Define features (BM25, dense score, novelty, redundancy), labels, and offline/online eval plan.
Author: Assistant
Category: retrieval-ranking-ml | Model: gpt-4o
Create a pipeline that generates questions from documents, with required answer spans and citations. Provide filters for ambiguity and leakage checks.
Author: Assistant
Category: data-generation-eval | Model: gpt-4o
Build an eval that scores ground-truth attribution (exact passage match), answer faithfulness, and coverage. Provide dataset schema and a nightly regression plan.
Author: Assistant
Category: evaluation-frameworks | Model: gpt-4o
ChatGPT drafts prompts and adversarial tests; Cursor integrates an eval harness; Antigravity schedules nightly evals and posts regressions with diffs. Include versioning and approval flow.
Author: Assistant
Category: mlops-llm-quality | Model: gpt-4o
Create a lint checklist for prompts: brevity, unambiguous instructions, schema-first outputs, guardrails, and eval hooks. Return a checklist table with yes/no and examples. Add 5 before→after rewrites...
Author: Curioforce Corp. Corp.
Category: Prompt-Improvement | Model: gpt-5-thinking
List risks for {{AI_use_case}} (medical/fitness/environmental). Design mitigations: human-in-the-loop, disclaimers, confidence prompts, and fail-safes. Add an evaluation plan with red-team tests and m...
Author: Tsubasa Kato
Category: risk | Model: gpt-5-thinking
You are a robotics CI lead. Design a sim-in-the-loop pipeline for perception + planning stacks. Deliver: scenario library spec (lighting, occlusion, rare edge cases), metrics (success %, collision=0, ...
Author: Tsubasa Kato
Category: Engineering | Model: GPT-5 Thinking
Write an ITIL-aligned process for agent incidents and changes: severity matrix, rollback, shadow traffic, canary, approvals, comms templates, and post-mortems with eval deltas. Include regulator-ready...
Author: Tsubasa Kato
Category: Strategy | Model: GPT-5 Thinking
Architect secure RAG across lakehouse/DWH: metadata-driven retrieval, policy-aware chunks, per-record ACL, caching, eval sets by domain, hallucination controls, and can’t-answer routing. Deliver refer...
Author: Tsubasa Kato
Category: Strategy | Model: GPT-5 Thinking
Design an orchestration layer for 5–10 agents: routing, memory, tool auth, retries, tracing, and guardrails. Compare options (workflow engines vs. code), propose SLAs, queues, circuit breakers, and ob...
Author: Tsubasa Kato
Category: Strategy | Model: GPT-5 Thinking
Ship marketing and sales agents: content brief generator, SEO outline, ad variants, webinar follow-up, lead enrichment, meeting note sync. Define prompts, tool chain (CRM, MAP, docs), eval rubric (bra...
Author: Tsubasa Kato
Category: Strategy | Model: GPT-5 Thinking
Design a data-lite RAG plan without engineers. Sources: Google Drive/Notion/PDFs. Deliver: folder taxonomy, redaction rules, ingestion checklist, embedding strategy, update cadence, eval set of 25 Q&A...
Author: Tsubasa Kato
Category: Strategy | Model: GPT-5 Thinking