Explore Prompts

Create a balanced scorecard: utility metrics (task success), safety metrics (policy adherence), reliability (latency, uptime), and user trust (complaints). Include leading indicators and dashboards.

Design an evaluation strategy that avoids overfitting: holdouts, rotating test sets, adversarial sets, and blind evaluation. Include rules for when to refresh benchmarks.

Create decision criteria for retraining vs prompt tuning vs retrieval updates. Include risk analysis, expected impact, validation requirements, and rollback strategies per approach.

Create a safe-defaults checklist for the tool layer: deny-by-default, explicit allowlists, safe parameter validation, output filtering, and timeouts. Include common failure modes.

Create a least-privilege permission model for tools: scopes, rate limits, time bounds, and audit logs. Provide an authorization matrix and guidelines for granting elevated access.

Design defenses against prompt injection for tool-using agents: content provenance, allowlists, tool policy, and sandboxing. Include a suite of adversarial prompts for regression testing.

Create a policy and technical controls for PII/secrets: detection, redaction, encryption, and safe storage. Include test cases and a plan to prevent secret leakage into training/evals.

Design data governance for user feedback and logs: consent, retention, minimization, access controls, and audit trails. Provide a policy and an implementation checklist.

Create a canary strategy: cohort selection, metrics, guardrails, and automatic rollback conditions. Include steps to prevent canary contamination and to interpret results statistically.

Design a rollback system: model version pinning, feature flags, staged rollouts, and a kill switch. Include operational procedures and testing for rollback reliability.

Create an incident response plan specific to AI: detection, containment, user comms, rollback, forensic logging, and post-incident retraining rules. Include severity levels and example incidents.

Define monitoring signals: policy violations, anomaly detection, tool misuse attempts, unusual output distributions, and drift. Provide alert thresholds, runbooks, and an on-call playbook.