Prompt Cards

Create a post-mortem template tailored to AI regressions: data/prompt/model diffs, evaluation gaps, monitoring misses, and remediation tasks. Include a ‘lessons to tests’ section.

Run a pre-mortem for a recursive improvement project: list plausible failures, early warning signals, and prevention steps. Output prioritized mitigations and ‘watch items’.

Define a governance model: safety owner, release manager, red-team lead, privacy officer, and incident commander. Include RACI matrix and meeting cadence.

Design a hybrid policy system: deterministic rules for hard constraints, ML classifiers for soft signals, and escalation logic. Provide architecture, failure modes, and monitoring plan.

Create a prompting style guide for internal prompts: structure, tool usage rules, refusal patterns, and safety reminders. Include examples and a checklist reviewers can apply.

Design a hallucination reduction plan: retrieval augmentation, answer verification steps, consistency checks, and refusal behaviors. Include evaluation metrics and regression tests.

Create a truthfulness policy: source requirements, citation rules, and how to label speculation. Provide a checklist for editors/analysts and an automated linting concept.

Design a calibration approach: confidence estimation, abstention policies, escalation to human, and how to test calibration. Include UI patterns that communicate uncertainty responsibly.

Create a balanced scorecard: utility metrics (task success), safety metrics (policy adherence), reliability (latency, uptime), and user trust (complaints). Include leading indicators and dashboards.

Design an evaluation strategy that avoids overfitting: holdouts, rotating test sets, adversarial sets, and blind evaluation. Include rules for when to refresh benchmarks.

Create decision criteria for retraining vs prompt tuning vs retrieval updates. Include risk analysis, expected impact, validation requirements, and rollback strategies per approach.

Create a safe-defaults checklist for the tool layer: deny-by-default, explicit allowlists, safe parameter validation, output filtering, and timeouts. Include common failure modes.