Building AI-Powered Products

AI products vary in quality, incur cost with each task and require users to judge uncertain outputs. Specify those behaviors through evals, cost limits, action permissions and failure-handling UX.

Why this matters

AI changes three product assumptions:

• Quality varies. Specify acceptable performance across inputs and failure types, including confidently wrong outputs.
• Cost scales with work. Model calls and long agent loops can erode gross margin even when adoption grows.
• Trust depends on evidence. Users need to distinguish reliable answers from plausible mistakes and retain control over risky actions.

The PM defines quality, unit economics, failure UX and autonomy alongside engineering and domain experts. Regulatory requirements also shape early design. Chapter 32 covers using AI for PM work, and Chapter 33 covers responsible AI and regulation.

Does this need AI at all?

Google PAIR opens its People + AI Guidebook with a fit decision: establish that AI adds value before choosing how to build it.

AI is a good fit when the task involves judgment over messy inputs (language, images, logs, unstructured documents), when a probabilistic answer is useful even if it’s sometimes wrong, and when the cost of an error is low or can be caught. It’s a poor fit when the task has a correct, rule-based answer, when every error is expensive and hard to detect, or when users need to predict exactly what the system will do.

Compare false positives with false negatives. For Beacon, a wrong root cause can trigger a harmful rollback, while an uncertain diagnosis leaves the engineer with manual investigation. That asymmetry favors precision and an explicit “I don’t know” response. Use the error-cost table to set eval targets and controls.

Workflows, agents and autonomy

Workflows versus agents

Erik Schluntz and Barry Zhang’s Anthropic essay “Building effective agents” distinguishes workflows, with predefined orchestration paths, from agents, where models direct their own process and tool use. Both are agentic systems, with these patterns:

Start with the simplest effective pattern. Agents exchange additional latency and cost for open-ended capability. Fixed workflows are often easier to evaluate and audit. Record the choice and rationale in the PRD.

Design the agent-computer interface as carefully as a human interface. Tool descriptions and arguments should make intended use clear and errors difficult. For platform PMs, this extends API product work.

Autonomy is a dial, set per action

Kevin Feng, David McDonald and Amy Zhang’s 2025 paper “Levels of Autonomy for AI Agents” gives product teams a useful vocabulary. It defines five levels by the user’s role, not the agent’s capability:

The paper’s key point is that autonomy is a design choice, separate from capability. A model that can act autonomously shouldn’t necessarily be allowed to.

Set autonomy per action class and map reversibility. Reading logs may warrant L4 while production writes remain L2. Irreversible actions need safeguards such as approval, dry runs or two-step confirmation. Model capability alone cannot justify permission.

Connectors, MCP and agent protocols

MCP connects applications to tools and data. Anthropic introduced it in late 2024 and donated it to the Agentic AI Foundation in December 2025, reporting over 10,000 public servers. The foundation sits under the Linux Foundation with OpenAI and Block as co-founders. Google’s A2A, hosted by the Linux Foundation since mid-2025, supports communication between agents across organizations. The protocols address complementary needs.

Treat an MCP server as a product surface. Give it an owner, scope, authentication, rate limits, versioning, documentation and pricing position, just as you would a public API.

Evals: quality as a product decision

Why evals replace “it works”

An eval runs graded test cases to measure how often a feature works, for which inputs and with which failures. Track scores over time and use thresholds to gate releases.

The PM or designated domain expert owns the definition of “good.” Lenny Rachitsky’s 2025 conversation with Hamel Husain and Shreya Shankar highlighted evals as a product-builder skill because they make that definition executable.

Start with error analysis

Hamel Husain and Shreya Shankar, whose evals course and FAQ are a widely used practitioner reference, put it bluntly: error analysis is “the most important activity in evals.” Their method:

1. Read real traces. Pull about 30 real interactions (inputs, model outputs, tool calls). Read every one.
2. Write free-form notes on what went wrong in each. Don’t use a rubric yet; rubrics written in advance narrow what you notice.
3. Cluster the notes into a failure taxonomy: a named list of the ways the product fails.
4. Keep going until you stop finding new failure types. That’s your signal that the taxonomy is complete enough for now.

The failure taxonomy becomes the eval’s structure and, often, the team’s backlog. Each recurring failure mode gets a fix, and if it persists, a dedicated test.

Husain and Shankar add several rules that are worth adopting as team policy:

• Use binary pass/fail, not 1 to 5 scales. Rating scales produce inconsistent labels and drift toward the middle. If you need finer tracking, measure sub-components separately.
• One narrow judge per failure mode. When you use a model to grade outputs (an “LLM-as-judge”), give it one specific failure to check. Validate each judge against human labels, and report how often it catches real failures and how often it correctly passes good outputs, separately, rather than a single “agreement” number.
• Distrust generic metrics. “Helpfulness” scores and text-similarity metrics rarely track what users care about. Write evaluators for failures you’ve actually observed.
• Name one accountable domain expert. Someone has to make the final call on what “good” means. On most product teams, that’s the PM or a domain expert the PM designates.

Capability, regression and consistency

Anthropic’s January 2026 guide “Demystifying evals for AI agents” adds vocabulary that’s useful for planning.

Capability evals measure what the product can’t yet do well. They should start with a low pass rate: they’re the hill you’re climbing. Regression evals protect what already works, and should pass at nearly 100% on every release. When a capability eval saturates, it graduates into the regression suite.

The guide also distinguishes two ways of counting success over several attempts, and for customer-facing products the difference is large:

• pass@k is the probability that at least one of k attempts succeeds. It’s the right measure when a user can pick the best of several options.
• pass^k is the probability that all k attempts succeed. It’s the right measure when a customer expects the product to work every time.

At 75% per-trial success, three consecutive independent successes have a probability of about 42% (0.75 × 0.75 × 0.75). Repeated customer tasks expose consistency more clearly than a best-case demonstration.

Start with 20–50 tasks from real failures and user reports, balancing correct actions and refusals. Grade outcomes such as correct ticket updates or healthy services. Permit valid tool sequences the designers did not anticipate.

Are evals the new PRD?

The claim that “evals are the new PRD” points to executable acceptance criteria. Braintrust argues that evals specify target behavior and regression thresholds. Lisa Murkin’s Mind the Product counterpoint identifies overfit golden datasets, small samples, usage drift and product-only ownership as limits.

Use evals as executable PRD acceptance criteria. Keep the PRD’s customer, rationale, economics and risk decisions, and let the suite define measurable behavior.

The AI PRD

An AI PRD adds model, data, prompt, quality, guardrail and monitoring requirements. Published templates from Miqdad Jaffer and Paweł Huryn in The Product Compass and Product School cover many of these. This book also emphasizes inference economics, which Jaffer and Huryn’s template treats lightly.

The AI PRD addendum template at the end of this chapter covers every row.

Cost, latency and unit economics

The cost curve is falling, and your bill may still rise

Epoch AI’s 2025 analysis estimated annual price reductions of 9x–900x at fixed performance, depending on the benchmark and threshold. One GPT-4-level knowledge benchmark fell from about $37.50 per million tokens in March 2023 to $0.18 by early 2025. Andreessen Horowitz’s Guido Appenzeller estimated roughly 10x annually, calling it “LLMflation.”

Falling token prices can coexist with rising task cost. Reasoning consumes more tokens and agents make repeated calls. Budget for the completed task, including retries and tool loops.

Gross margin is the number to watch

Classic SaaS gross margins are roughly 75–80% or higher. ICONIQ’s 2026 report projected average AI-product margin near 53%. Kyle Poyar’s 2026 survey found a median target near 50%, with 12% targeting at least 80%. These are reported projections and targets, not a requirement for your product.

The levers a PM controls:

• Model routing: send easy requests to small, cheap models and hard ones to frontier models. Often the single biggest lever.
• Prompt caching: long, stable instructions and context can be cached so repeat calls cost a fraction of the original. Check your provider’s current pricing, because it changes often.
• Workflow caching for agents: once an agent solves a recurring task, convert the successful path into deterministic code and call the model only when something unexpected happens. Andreessen Horowitz’s 2026 analysis of computer-use agents describes this as the pattern that makes agents economical on stable, high-volume work.
• Budgets and stopping conditions: maximum steps, tokens, time and spend per task, with an escalation path when a limit is hit.
• Latency design: stream output, keep time to first token low on interactive paths, use small models where speed matters, and run long agent tasks asynchronously.

Pricing AI

Seat-based pricing breaks when AI does the work that seats represented. If one agent replaces part of five people’s work, charging per seat charges less as you deliver more. That’s why AI pricing is in flux.

Poyar’s 2026 survey of over 230 B2B software and AI companies found hybrid pricing most common at 37%, with 29% using AI credits. Intercom Fin has charged $0.99 per outcome. Salesforce moved from $2 per conversation to Flex Credits around $0.10 per action in May 2025, then added per-user options. Cursor’s June 2025 usage-pricing change led to surprise charges, July refunds and an apology. Ship metering visibility and spend controls with pricing (Chapter 15).

Trust and the UX of being wrong

Calibrated trust

The goal is calibrated trust: users trusting the system as much as its reliability warrants, and no more. PAIR’s guidebook builds its trust chapters around this idea, along with treating errors and graceful failure as part of the core flow rather than an edge case.

Microsoft’s Guidelines for Human-AI Interaction, published by Saleema Amershi and colleagues at CHI 2019 and validated with practitioners across twenty products, offer eighteen guidelines grouped into four moments. Paraphrased:

• Initially: make clear what the system can do, and how well it can do it.
• During interaction: time help to the context, show relevant information, and match social norms while mitigating bias.
• When it’s wrong: make it easy to invoke, dismiss and correct; scope down when uncertain; and explain why it did what it did.
• Over time: learn from behavior cautiously, invite granular feedback, provide global controls, and tell users when the system changes.

Use them as a launch-review checklist. Many AI trust failures map to one of them, often “make clear how well it can do it” or “scope services when in doubt.”

Patterns that work

• Show sources and confidence wherever the user must act on an output. A root cause with three linked log lines is checkable; a bare assertion isn’t.
• Make “I don’t know” a good answer. Design an explicit low-confidence state, and reward it in evals rather than penalizing it.
• Make correction cheap with inline edits, regenerate and undo, and capture every correction as a label for evals.
• Disclose AI. Users should know when they’re talking to an AI, especially in support. In the EU it’s increasingly a legal requirement, too.
• Degrade gracefully to search, templates or a human when confidence is low or the model provider is down.
• Design against rubber-stamping. “Human in the loop” fails silently when reviewers approve everything. Make wrong output easy to spot: highlight what changed, show the evidence, and sample approvals for audit.

What failure looks like in public

Four widely reported incidents illustrate the failure modes PMs must design against:

• Air Canada (2024). A support chatbot told a customer they could claim a bereavement fare retroactively. A Canadian tribunal held the airline responsible for what its chatbot said, rejecting the argument that the bot was a separate entity. Lesson: you own your AI’s statements. Ground policy answers in approved sources and show them.
• Cursor (2025). An AI support agent invented a login policy that didn’t exist, and users cancelled before a co-founder corrected it publicly. Lesson: disclose AI agents, and restrict policy statements to retrieved, approved content.
• Replit (2025). During a code freeze, an AI coding agent ran destructive commands against a user’s live database, then misreported whether recovery was possible. Replit separated development and production databases afterward. Lesson: least privilege, environment separation and approval for irreversible actions.
• Deloitte Australia (2025). A government report produced with AI assistance contained fabricated citations and a fabricated quote; the firm issued a corrected report and refunded part of the fee. Lesson: any AI-assisted deliverable needs citation verification and disclosure.

Data, flywheels and moats

A data flywheel needs three links: usage produces quality labels, labels feed evals or improvements, and users experience the gain. Define and verify each link before claiming that usage compounds.

Enterprise contracts may forbid training on customer data. Build permitted feedback loops from customer-specific context, your evals or consented aggregate signals. Specify the signal, consent, contract terms and storage in requirements.

Assess defensibility through workflow integration, switching costs, proprietary context, distribution and trust. Shared model access offers little exclusive advantage. Chapter 16 develops the tests.

Regulation: a product input now

Identify regulatory requirements early. Risk management, logs, oversight, accuracy testing and documentation need product capabilities and evidence built during development.

A snapshot as of late 2026, which Chapter 33 covers in depth. This is not legal advice, and dates should be confirmed with counsel before you rely on them.

• EU AI Act. The Act entered into force in August 2024. Prohibited practices and AI-literacy duties applied from February 2025, and obligations for general-purpose AI models from August 2025. Transparency obligations, such as telling people they’re interacting with AI, apply from August 2026. In July 2026 the EU adopted the “Digital Omnibus” amendment (Regulation (EU) 2026/1744), which postponed the high-risk obligations: stand-alone high-risk systems now from December 2027, and AI embedded in regulated products from August 2028.
• NIST AI Risk Management Framework. A voluntary US framework built on four functions (Govern, Map, Measure, Manage), with a 2024 Generative AI Profile listing risks specific to generative AI, such as confabulation. Widely used in enterprise risk programs.
• ISO/IEC 42001. The first certifiable AI management system standard, published in late 2023. It’s becoming the AI counterpart of ISO 27001 in enterprise security questionnaires.

Record provider/deployer roles, likely risk tier, transparency and documentation duties in the PRD. Customers may request this evidence during security reviews before the statutory deadline.

How to apply it

Here’s the lifecycle for building an AI-powered feature or product, from idea to scale.

1. Write the cost-of-error table. For each kind of mistake (false positive, false negative, harmful output, wrong action), write what it costs the user and the business.
2. Decide whether AI is the right tool. If a rule-based solution meets the need, use it. If not, state why AI adds value.
3. Choose the simplest pattern. Start at the top of the workflow-to-agent table and move down only when a simpler pattern fails. Record the decision.
4. Set autonomy per action class. Build the reversibility map. Default anything that changes a system of record to L2 (human approval) until evidence supports more.
5. Prototype and run error analysis. Put a prototype in front of real or realistic inputs, read at least 30 traces, and build the failure taxonomy.
6. Build the eval suite. Turn the taxonomy into 20 to 50 tasks with binary graders. Separate capability from regression evals. Validate any model judges against human labels. Set pass-rate and pass^k targets with your domain expert.
7. Build the cost model. Estimate tokens and model calls per task, cost per task at expected volume, and gross margin at the proposed price. Set a margin floor. Identify routing and caching levers.
8. Design the trust UX. Sources, confidence, “I don’t know,” correction, undo, escalation, disclosure. Review against the human-AI interaction guidelines.
9. Classify the regulatory exposure. Provider or deployer, risk tier, transparency duties, documentation. Bring counsel in now.
10. Roll out in stages. Shadow mode (the AI runs, users don’t see it, you compare to humans); assisted (users see suggestions); supervised (the AI acts with approval); autonomous for allowlisted actions. Move between stages on eval and production evidence, not on the calendar.
11. Monitor in production. Track quality on sampled production traffic, cost per task, latency, user corrections and escalations. Re-run regression evals on every model, prompt or provider change.
12. Close the loop. Every production failure becomes a new eval case. Review the failure taxonomy monthly.

The AI lens: using AI to build AI products

Use AI to draft the artifacts below while people retain domain judgment:

• Drafting eval cases. A model can generate dozens of candidate test inputs per failure mode, including adversarial and “should refuse” cases you didn’t think of. You and your domain expert curate them.
• Clustering error notes. After you’ve written free-form notes on traces, a model can propose a first failure taxonomy. Check it against your notes; the model will merge categories that matter.
• Writing judge prompts. A model can draft a narrow judge for one failure mode. You must still validate it against human labels. An unvalidated judge is a second model’s opinion, not a measurement.
• Building the cost model. Give a harness your expected volumes, token estimates and provider prices and it can produce a scenario model in minutes. Check every input yourself.
• Red-teaming. A model can play an adversarial user, a prompt injection in a retrieved document, or a confused customer. It finds different weaknesses than your team does.

The domain expert defines acceptable behavior. Validate model graders against independent human labels before using them as measurements.

The Beacon case: the wrong rollback

The immediate response. You pause the pilot’s ability to propose rollbacks, keeping diagnosis on. You write to the customer within 24 hours explaining what happened, what you’ve changed and what you’ll do next. The incident goes in the decision log and the risk register, and it becomes the first entry in a new AI incident review process (Chapter 33).

Error analysis. You and the applied AI engineers pull 120 pilot traces and read them. Free-form notes cluster into a failure taxonomy of seven modes. The top three:

1. Recency bias. The agent over-weights the most recent deploy, even when timing doesn’t fit.
2. Single-region blindness. The agent doesn’t check infrastructure events outside the affected service’s region.
3. Overconfident framing. Diagnoses are worded with the same certainty whether the evidence is strong or thin.

None of these was in the original PRD, which defined quality as “accurate root cause.”

The eval suite. You turn the taxonomy into 48 tasks built from real incidents, half “should diagnose” and half “should say I’m not sure.” Each has a binary grader: a code check for whether cited evidence exists and matches the timeline, and a narrow model judge for overconfident framing, validated against 150 labels from two SREs. Recency bias and region blindness become capability evals; the existing behaviors that worked become a regression suite.

The targets. With Priya’s team as domain experts, you set the bar: diagnosis accepted by responders in at least 60% of incidents; an “I’m not sure” rate between 15% and 30% (lower means overconfidence, higher means low value); zero rollback proposals where the cited evidence doesn’t match the timeline. Because responders see the agent in every incident, you track pass^3 on the regression suite, not just single-run accuracy.

Autonomy and UX. Rollback proposals move from L3 to L2: the agent can propose, but the UI shows the timeline evidence side by side with the change, and the approver must confirm the specific service and region. Every diagnosis now displays its evidence and an explicit confidence band.

Cost. The fix isn’t free. Checking cross-region infrastructure events adds three tool calls and about 40% more tokens per diagnosis. Your cost model shows Beacon Agents’ gross margin falling from 72% toward 66% at current pricing. You route the first-pass triage to a smaller model and cache the stable runbook context, which brings the projected margin back to about 71%. The margin floor goes in the AI PRD: 70%.

Pricing. The incident sharpens the per-resolved-incident pricing question. Marcus, the economic buyer, likes paying for outcomes, but his CFO asks who pays when the agent makes an incident worse. You add an outcome definition (incident resolved, with Beacon’s diagnosis accepted, and no reopening within 24 hours) and a credit policy for agent-caused incidents to the pricing proposal for Chapter 15.

The result. Six weeks later the pilot restarts with the new evals gating every release. Diagnosis acceptance reaches 64%, the “I’m not sure” rate settles at 22%, and there have been no mismatched rollback proposals. The retail customer stays in the pilot and, at Northlight Live, agrees to be a reference customer for diagnosis, though not yet for autonomous action.

Product 360 artifacts

# AI PRD addendum · [Feature] · v[n] · [Date]

Attach to the standard PRD. Every section is required for AI-powered features.

## 1. Why AI
- The user problem (link to PRD):
- Why a rule-based solution isn't enough:
- Cost-of-error table:

| Error type | Example | Cost to user | Cost to business | Detectable by user? |
|---|---|---|---|---|
| False positive | | | | |
| False negative | | | | |
| Harmful or inappropriate output | | | | |
| Wrong action (agents) | | | | |

## 2. Pattern and autonomy
- Pattern: augmented model / chain / routing / parallel / orchestrator / evaluator-optimizer / agent
- Why the simpler patterns weren't enough:
- Autonomy per action class:

| Action class | Reversibility (cheap / costly / irreversible) | Autonomy (L1 to L5) | Approval and confirmation design |
|---|---|---|---|
| | | | |

- Stopping conditions and budgets: max steps [n], max tokens [n], max time [n], max spend per task [$]
- Escalation path when a limit is hit:

## 3. Quality (link to eval plan)
- Failure taxonomy (top modes):
- Capability eval targets:
- Regression eval threshold (target near 100%):
- Consistency target (pass^k):
- Accountable domain expert:

## 4. Models and data
- Models and providers (link to model register):
- Fallback when the provider is unavailable or degraded:
- Grounding sources and how they're permissioned:
- Personal and customer data handled, retention, residency:
- Training and feedback use permitted by contracts: yes / no / with consent
- Feedback signals captured and where they're stored:

## 5. Cost and latency (link to AI cost model)
- Cost per task at expected volume:
- Gross-margin floor:
- Latency budget: time to first token [ms], end to end [s]
- Cost levers in use: routing / caching / workflow caching / budgets

## 6. Trust and failure UX
- How capability and limits are communicated:
- Sources and confidence display:
- "I don't know" state:
- Correction, undo and regenerate:
- AI disclosure:
- Graceful degradation:
- Anti-rubber-stamping design (for approvals):

## 7. Risk and regulation (link to AI risk assessment)
- Our role: provider / deployer
- Likely risk tier and jurisdictions:
- Transparency and documentation duties:
- Security threats considered: prompt injection, data exfiltration, credential misuse

## 8. Rollout
| Stage | Entry criteria (evals and production evidence) | Who sees it | Exit criteria |
|---|---|---|---|
| Shadow | | | |
| Assisted | | | |
| Supervised | | | |
| Autonomous (allowlisted actions) | | | |

- Kill switch: who can pull it, and how fast it takes effect:

# Eval plan · [Feature] · [Date]

**Owner:** [PM or domain expert]   **Engineering owner:** [Name]   **Last run:** [Date]   **Gates releases:** yes / no

## Error analysis
- Traces reviewed: [n] on [dates], from [source]
- Reviewers:
- Failure taxonomy:

| # | Failure mode | Description | Example trace | Frequency in sample | Severity (H/M/L) |
|---|---|---|---|---|---|
| 1 | | | | | |

## Eval suite
| Suite | Type | # tasks | Grader (code / model judge / human) | Target | Current |
|---|---|---|---|---|---|
| [Failure mode 1] | Capability | | | | |
| [Existing behaviors] | Regression | | | ≥ 98% | |
| Should refuse or say "I'm not sure" | Capability | | | | |
| Consistency | pass^[k] | | | | |

## Judge validation (for each model judge)
| Judge | Failure mode | Human labels | True positive rate | True negative rate | Last validated |
|---|---|---|---|---|---|
| | | | | | |

## Run policy
- Runs on: every prompt change / model change / provider change / release
- Regression failure policy: block release / require approval from [name]
- Production sampling: [n] traces per [week], reviewed by [name]
- New failures from production become eval cases within [n] days

## History
| Date | Change | Capability score | Regression score | Notes |
|---|---|---|---|---|
| | | | | |

# AI cost model · [Feature] · [Date]

**Owner:** [PM]   **Finance partner:** [Name]   **Prices as of:** [Date]

## Per-task cost
| Step | Model | Calls per task | Input tokens | Output tokens | Cached share | Cost per call | Cost per task |
|---|---|---|---|---|---|---|---|
| | | | | | | | |
| **Total** | | | | | | | **$** |

Include retrieval, tool and infrastructure costs per task.

## Volume and margin
| Scenario | Tasks per month | Inference cost | Other COGS | Revenue | Gross margin |
|---|---|---|---|---|---|
| Low | | | | | |
| Expected | | | | | |
| High | | | | | |

- Gross-margin floor: [%]
- Cost per task at which the floor is breached: [$]

## Levers
| Lever | Expected saving | Quality risk | Status |
|---|---|---|---|
| Route [step] to a smaller model | | | |
| Cache [stable context] | | | |
| Convert [recurring path] to deterministic code | | | |
| Lower step or token budget | | | |

## Sensitivities
- If tokens per task rise 50%:
- If the provider raises prices 20%:
- If usage doubles in a quarter:

# Model and vendor register · [Product] · [Date]

| Feature | Model and version | Provider | Hosting region | Data sent | Training on our data? | Contract and DPA | Fallback model | Deprecation date | Last eval run | Owner |
|---|---|---|---|---|---|---|---|---|---|---|
| | | | | | | | | | | |

## Change policy
- Any model, version or provider change requires a regression eval run and sign-off from [name].
- Deprecation notices are reviewed [monthly]; migrations are planned at least [n] months before the date.
- Customers are notified of material model changes per [contract terms].

# AI risk assessment · [Feature] · [Date]

**Owner:** [PM]   **Reviewers:** [security, legal, privacy, responsible AI]   **Next review:** [Date]

## System description
- What it does, for whom, and what decisions or actions it influences:
- Our role: provider / deployer
- Jurisdictions and likely risk classification (confirm with counsel):

## Risks
| # | Risk | Category (accuracy, bias, privacy, security, safety, IP, transparency, over-reliance) | Likelihood | Impact | Mitigations | Residual risk | Owner |
|---|---|---|---|---|---|---|---|
| 1 | | | | | | | |

## Controls
- Human oversight design:
- Logging and audit (what is retained, for how long, who can access it):
- Transparency to users:
- Security: prompt injection, data exfiltration, credential scope:
- Monitoring and incident response (link):

## Documentation customers may request
- [ ] System description and intended use
- [ ] Data handling and residency
- [ ] Evaluation results and known limitations
- [ ] Human oversight and controls
- [ ] Incident response process
- [ ] Relevant certifications or framework alignment (for example ISO/IEC 42001, NIST AI RMF)

## Sign-off
| Role | Name | Date | Decision |
|---|---|---|---|
| | | | |

Prompts

Goal: Get a fast first pass of error notes on real traces, to speed up (not replace) your own error analysis.
Paste in: 20 to 40 traces (input, output, tool calls, final outcome), with customer data removed or using an approved tool, plus a one-paragraph description of what the feature is supposed to do.

For each trace below, write one or two sentences of free-form notes on anything that went wrong or looks risky: wrong facts, unsupported claims, missed context, bad tool use, overconfident tone, unsafe action, poor format. If nothing went wrong, write "OK." Keep this first pass as ungrouped observations.

Then, separately, propose a draft failure taxonomy: 5 to 10 named failure modes, each with a one-line definition and the trace numbers that show it.

[feature description]
[traces]

Check the output: read at least 15 traces yourself before you look at the model's notes, then compare. Expect the model to miss domain-specific failures and to merge modes you'd keep separate. The final taxonomy is yours.

Goal: Build a balanced set of test cases quickly.
Paste in: the failure mode's name and definition, three real example traces that show it, and three that show correct behavior.

Generate 30 eval cases for this failure mode in [feature]: [name and definition].
- 15 cases where a correct system should succeed or act, and 15 where it should refuse, ask for more information, or say it isn't sure.
- Vary difficulty: include obvious cases and subtle ones.
- Include at least 5 adversarial cases (misleading context, conflicting evidence, an instruction hidden inside retrieved content).
- For each case give: the input, the context the system would have, the expected outcome (not the exact wording), and a one-line pass/fail criterion.

Base cases on capabilities shown in the real examples. Mark any needed capability information as a question.

[examples]

Check the output: have your domain expert review every case. Delete any whose expected outcome you'd argue about; ambiguous cases poison an eval.

Goal: Create a narrow LLM-as-judge for one failure mode, plus a plan to validate it.
Paste in: the failure mode's definition, 10 labeled examples (5 pass, 5 fail) with the reason for each label.

Write a judge prompt that checks one thing only: [failure mode]. The judge must return PASS or FAIL and a one-sentence reason quoting the part of the output that drove the decision. Include the definition, the labeled examples as few-shot guidance, and instructions to ignore every other quality dimension.

Then write a validation plan: how many human labels I need, how to split them into development and test sets, how to calculate the true positive rate and true negative rate, and what rates I should require before trusting the judge to gate releases.

[definition and examples]

Check the output: run the judge on a held-out set you labeled yourself. Don't use it to gate releases until you've measured both rates on data it hasn't seen.

Goal: See the gross-margin effect of an AI feature before it ships.
Paste in: the steps the feature performs, the model for each step, estimated input and output tokens per call, calls per task, current provider prices (with the date), expected monthly task volume and the planned price.

Build a cost model for [feature] with:
1. A per-task cost table by step, including caching assumptions.
2. Monthly cost, revenue and gross margin at low, expected and high volume.
3. The cost per task at which gross margin falls below [floor]%.
4. Three levers to reduce cost (routing, caching, converting recurring paths to deterministic code, budgets), each with an estimated saving and the quality risk it introduces.
5. Sensitivity to tokens per task rising 50% and prices rising 20%.

Show formulas and use the supplied prices. Label every assumed input and identify missing prices.

[inputs]

Check the output: recompute one scenario by hand. Confirm prices against the provider's current pricing page before sharing the model with finance.

Goal: Find autonomy and permission gaps before a customer does.
Paste in: the agent's list of actions and tools, the permissions each tool has, the proposed autonomy level per action, and the approval UX.

Review this agent design for [product] as a skeptical security reviewer and SRE.
1. For each action, classify reversibility (cheap, costly, irreversible) and say whether the proposed autonomy level (L1 operator to L5 observer) is appropriate. Recommend changes.
2. Identify any tool with broader permissions than its actions need.
3. List ways a prompt injection in retrieved content or tool output could cause a harmful action, and the control that would stop each.
4. Check stopping conditions and budgets. What happens when a limit is hit?
5. Assess whether the approval UX would let a tired reviewer rubber-stamp a wrong action, and suggest design changes.
Finish with the five changes you'd require before production.

[design]

Check the output: walk through the riskiest action end to end with your engineering lead and confirm the model's reading of the permissions is correct.

Goal: Find failure modes your team hasn't imagined.
Paste in: the feature description, the system instructions or a summary of them, the user types, and the tools and data it can access.

Act as a red team for [feature]. Generate 25 test scenarios across these categories: confused or mistaken users, adversarial users, prompt injection through documents or tool outputs, requests outside the intended scope, high-stakes situations where a wrong answer causes harm, attempts to extract other customers' data, and situations where the right answer is "I don't know." For each, give the scenario, what a safe response looks like and what failure would look like.

Rank the five scenarios you think are most likely to cause real harm in production, and explain why.

[feature details]

Check the output: run the top-ranked scenarios against the real system. Add every failure you find to the eval suite.

Goal: Make sure an AI PRD covers what classic PRDs miss.
Paste in: the PRD and its AI addendum.

Review this PRD for an AI-powered feature. For each of these areas, say whether it's covered well, covered weakly or missing, and quote the relevant text: cost of error; choice of pattern and autonomy; failure taxonomy and eval targets; consistency targets; model, fallback and data requirements; cost per task and margin floor; latency budget; trust and failure UX; regulatory classification; staged rollout and kill switch. Then list the three gaps most likely to cause a production incident or a margin problem, and the question I should ask the team about each.

[PRD]

Check the output: confirm each "covered well" against the PRD yourself. Bring the three questions to your next review.

Anti-patterns

Checklist

• A cost-of-error table exists, and it drives eval targets, autonomy and UX.
• The PRD explains why AI is the right tool, and why simpler patterns weren’t enough.
• Autonomy is set per action class, with a reversibility map and approval design for anything costly or irreversible.
• Stopping conditions and budgets (steps, tokens, time, spend) are defined, with an escalation path.
• Error analysis on at least 30 real traces produced a failure taxonomy.
• The eval suite has capability and regression sets, binary graders, validated judges and consistency (pass^k) targets, and it gates releases.
• A named domain expert owns the definition of “good.”
• The AI cost model shows cost per task, gross margin at three volumes and a margin floor.
• The model and vendor register lists every model, version, provider, data flow, fallback and deprecation date.
• The trust UX covers sources, confidence, “I don’t know,” correction, disclosure and graceful degradation, and it was reviewed against the human-AI interaction guidelines.
• Regulatory role, risk tier and documentation duties are recorded, and counsel has reviewed them.
• The rollout is staged (shadow, assisted, supervised, autonomous) with evidence-based entry and exit criteria and a kill switch.
• Production monitoring covers quality, cost, latency, corrections and escalations, and production failures become eval cases.
• Any MCP server or API that agents use to reach the product has an owner, scope, authentication, versioning and pricing position.
• The eval plan, cost model, model register and risk assessment are linked from the Product 360 index.