Navigating Technical Product Management

Translate architecture, APIs and operational constraints into customer and business choices. Match the development process to the risk, then frame and verify the evidence used to decide.

Why this matters

APIs, permissions, reliability, versioning and per-call costs shape the customer experience long before they appear in an interface. A technical PM must understand their consequences.

AI adds agent callers and faster implementation. Experiment analysis is also easier to generate than verify. Cagan’s productivity-paradox argument applies: faster output needs better choices and stronger evidence checks.

What makes a PM technical

A working definition

Companies define technical PMs differently. This book uses the product as the test:

A technical PM is a PM whose customers, product surface or dominant risks are technical, so they must reason about feasibility, architecture and operational tradeoffs directly instead of delegating them.

Cagan’s four risks remain value, usability, feasibility and viability. Technical products emphasize feasibility and operational viability. Guarantees, compatibility and per-call cost require product judgment alongside engineering judgment.

Four variants of the role

The craft is the same across technical products, but the customer and the dominant risk change. Name both before you choose a process or a metric.

Platform PMs. In Team Topologies, platform teams reduce complexity for stream-aligned teams through an internal product. Assess whether teams choose it for value or merely comply with a mandate. Chapter 14 covers external platforms.

API PMs. Treat observable behavior as a compatibility promise. Hyrum’s Law, Hyrum Wright’s observation popularized by Titus Winters, warns that someone will depend on any observable behavior at sufficient scale. Stripe’s 2017 account describes date-based versions, account pinning and reversed changes for older responses. Google’s API Design Guide covers resources, methods, errors and versioning. Yegge’s 2011 account recalls an Amazon mandate for service interfaces that could support external callers. Treat it as recollection rather than documented policy.

Infrastructure PMs. Here reliability is the product, and the Google SRE book’s chapter on service level objectives supplies the vocabulary. A service level indicator (SLI) is a precisely defined measure of one aspect of service, such as latency. A service level objective (SLO) is a target for an SLI. A service level agreement (SLA) is a contract with consequences, usually financial, for missing an SLO. The error budget is the failure the SLO allows. Setting an SLO is a product decision, because reliability and speed of change compete for the same engineering time.

AI/ML PMs. AI outputs vary from run to run and cost money on every call (Chapter 31). Quality has to be measured with evals before value is tested with experiments, and inference cost is a margin line you own. Beacon’s gross margin fell from 81% to 72% after agentic features launched, mostly because of inference cost.

Why products fail

Use three failure lenses to decide which risks the process must address.

What startup post-mortems say

CB Insights’ 2026 analysis of startup failures examined 431 venture-backed companies that shut down publicly since 2023 and identified reasons for 385. Companies could have several reasons, so the shares add up to more than 100%: 70% ran out of capital, 43% had poor product-market fit, 29% cited bad timing or macro conditions and 19% had unsustainable unit economics.

These are self-reported venture-backed-company failures after the zero-interest-rate era. Capital exhaustion often describes the final event. Use fit and economics as pre-mortem prompts, with limited causal inference.

Cagan’s root causes of failed product efforts

Cagan’s 2015 failure analysis in SVPG and Inspired identifies stakeholder-led ideas, speculative business cases, feature-list roadmaps, requirements-only PMs, late design and engineering involvement, delivery-only agile, project funding, late validation and opportunity cost. These practices concentrate risk after investment.

His alternative is continuous discovery alongside continuous delivery, with roadmaps of problems instead of features. Causes 6 and 9 are where a technical PM has the most leverage: bring engineers into discovery, and test feasibility early with spikes instead of finding a hard constraint in month five.

The feature factory

Cutler’s “12 Signs You’re Working in a Feature Factory” (2016) identifies unmeasured impact, unstable teams, launch theater, little scrapped work, weak metric links, missing decision retrospectives, unchecked prioritization, absent post-launch iteration, research handoffs, large batches, deal-driven complexity and hidden debt. His later Amplitude follow-up notes that good intentions, such as protecting overloaded designers, can create the pattern.

Cagan’s productivity paradox gives the 2026 twist: AI coding tools turn a feature factory into a faster one. For technical PMs the common symptom is the deal-driven connector, built to close one customer and then maintained, versioned and secured forever.

The pre-mortem

Klein’s “Performing a Project Premortem” (2007) asks participants to imagine failure, silently write plausible reasons, then share them in turn. The method builds on Mitchell, Russo and Pennington’s prospective-hindsight research. Include engineering, security and support to expose technical and commercial risks while changes remain cheap.

How products get built: processes compared

Every development process is a theory about where risk lives and when to retire it. Most enterprise organizations run several at once: funding on top, delivery underneath and, in good teams, discovery alongside.

Sequential (“waterfall”). Royce’s 1970 paper describes requirements through maintenance in sequence, then warns that it is risky because late testing forces redesign. He proposes preliminary design, documentation, a pilot, managed testing and customer involvement. Sequential practice often omits those safeguards.

Stage-Gate. Cooper separates evidence-gathering stages from management decisions to continue, stop or redirect investment. It supports funding governance. Agile Stage-Gate runs sprints within stages. Review whether gates genuinely stop weak projects.

Agile. The 2001 Manifesto for Agile Software Development values individuals and interactions over processes and tools, working software over comprehensive documentation, customer collaboration over contract negotiation, and responding to change over following a plan. Agile fixed the delivery problem Royce described by getting working software in front of people early and often. On its own it doesn’t decide what’s worth building, which is Cagan’s root cause 7.

Lean. Eric Ries’s The Lean Startup (2011), introduced in Chapter 01, makes validated learning the unit of progress through a build-measure-learn loop. In enterprise, a minimum viable product must clear a much higher bar for security and support, and “measure” often means a handful of design partners.

Dual-track. Sy’s 2007 work describes design at least one cycle ahead of development. Patton later emphasizes one team doing concurrent discovery and delivery: rapid learning alongside predictable, high-quality development. Discovery can change or stop ideas.

Cagan distinguishes discovery prototypes that test product risks from production delivery. As building costs fall, invest the saved capacity in testing what deserves delivery.

The technical fluency you need

Understand these five areas well enough to identify customer consequences and challenge tradeoffs.

Architecture. Draw components, external systems, data locations and synchronous versus asynchronous calls. A diagnosis waiting on five customer-system calls inherits their delays. Queued actions require the interface to distinguish approval from execution.

APIs. For any API you own or depend on, know its resources and methods, authentication, scopes, rate limits, versioning and what it has promised not to change. Know the difference between an additive change (a new field or endpoint, usually safe) and a breaking one (removing or renaming a field, changing a default, tightening validation). Because of Hyrum’s Law, ask “what observable behavior changes?” before “what does the contract say?”

Data models. Know the main entities, how they relate and what identifies each one. In B2B add tenancy (whose data is this, and how is it kept apart?) and residency (which region holds it?). “Show an MSP a combined view across its customers” is a tenancy question before it’s a screen.

System design tradeoffs. Four come up in almost every decision:

• Consistency, availability and latency. Eric Brewer’s 2012 retrospective on his CAP theorem argues that “pick two of three” misleads: the choice is made per operation and per piece of data, it’s a matter of degree, and it matters mainly during network partitions, which are rare but real. As he puts it, “CAP prohibits only a tiny part of the design space.” The product question is what each user should see, and what should be allowed to happen, when part of the system can’t reach another part.
• Reliability versus speed of change, set by the error budget.
• Cost per request, which for AI features scales with usage, context length and model choice.
• Reversibility. A public API shape, a data model or a pricing unit is expensive to undo. Spend review time in proportion.

Reading a design doc. Ubl’s 2020 Google account covers context, scope, goals, exclusions, design, alternatives and security, privacy and observability. Large documents may span 10–20 pages and mini designs 1–3. Review in five passes:

1. Goals and non-goals against the PRD (Chapter 04).
2. Alternatives considered. Were cheaper or more reversible options weighed?
3. Cross-cutting concerns. Which customer promises does it touch: security, residency, latency, cost?
4. Interfaces. Who calls this, and what breaks for existing callers?
5. What’s missing. Ask what would make the team regret this in a year.

Michael Nygard’s 2011 architecture decision record (ADR) uses one or two pages for title, context, decision, status and consequences. Supersede old decisions while preserving their history. Read it before requesting architectural changes.

Experiments and analysis

Chapter 05 covers instrumentation and readouts. These concepts help you frame a test and check its analysis.

Framing the question

Before launching a controlled experiment, write down:

• The decision it informs, who makes it, and the hypothesis.
• The primary metric. Ron Kohavi, Diane Tang and Ya Xu, in Trustworthy Online Controlled Experiments, call it the overall evaluation criterion (OEC): a quantitative measure of the experiment’s objective, ideally one that predicts long-term value.
• Guardrail metrics. They describe two kinds: organizational guardrails such as latency, errors and cost, which the change must not harm, and trust guardrails, which tell you whether the experiment itself is sound.
• The unit of randomization: user, account, team or tenant. In B2B, randomizing users inside one customer often contaminates the test, because colleagues share runbooks. Randomizing by account is cleaner but gives far fewer units.
• The minimum effect worth detecting: the smallest change that would alter the decision.
• The sample size and stopping rule, fixed in advance.

Getting the data into shape

Data preparation can invalidate an otherwise correct statistical analysis. Before viewing results, agree the eligibility, assignment, metric, outlier and missing-data rules:

• Which units count. Who was eligible, and which units are excluded: internal and test accounts, partner sandboxes, bots and synthetic monitors. In B2B a single load-testing tenant can outweigh dozens of real ones.
• How assignment joins to outcomes. Every metric row has to trace back to the group the unit was assigned to, not the group it happened to see. Ask what happens to units that switched groups or appear in both.
• Exact metric definitions. “Acceptance rate” needs a numerator, a denominator, a time window and a rule for repeats. Two reasonable definitions can produce opposite answers.
• How outliers are handled. One very large account can move an average on its own. Decide in advance whether to cap values or report a median as well.
• Gaps and duplicates. Ask for counts of missing and duplicated events by group. A logging gap that hits one group more than the other looks exactly like an effect.

Ask the harness to state each of these rules back to you, with the row counts before and after each exclusion, before it reports any effect.

Significance and power, as concepts

A p-value describes incompatibility between data and a specified model. Wasserstein and Lazar’s 2016 American Statistical Association statement warns that it measures neither the probability a hypothesis is true nor effect importance. Report fully, use judgment beyond thresholds, and read the effect size and confidence interval.

Power is the probability of detecting an effect of the specified size. Small B2B samples often miss real effects. An inconclusive test may require more time, a larger minimum effect, a different unit or another method.

Peeking inflates false positives when a fixed-sample test stops at first significance. Miller’s 2010 illustration reached about 26% false positives for a null change. Fix sample size or use a design with planned repeated checkpoints.

Interpreting results

Read a result in a fixed order, and stop if an early check fails.

1. Check the split. A sample ratio mismatch (SRM), as Aleksander Fabijan and colleagues define it in their 2019 KDD paper, is an observed ratio of units between groups that differs from the designed ratio, a sign that assignment, logging or eligibility is broken. Kohavi and colleagues treat the SRM check as a trust guardrail for every experiment. If it fails, the result can’t be trusted.
2. Read the primary metric: effect size, interval and significance, as planned.
3. Read the guardrails. A primary-metric win that breaks latency, cost or error rates isn’t a win.
4. Read only planned segments. Slice enough ways and something will look significant by chance. Treat unplanned segments as exploratory hypotheses.
5. Apply Twyman’s law. Kohavi and colleagues use it as a rule of thumb: a result that looks surprising or unusually good is more often a mistake than a discovery.

When you can’t A/B test

Many B2B decisions can’t be randomized cleanly, such as pricing under contract or admin features used by a few people per account. The alternatives are weaker but honest: staged rollouts with a comparison group, long-running holdouts, switching a change on and off over time, or before-and-after comparisons against a similar group. Name the method and its weaknesses in the readout.

For AI features, add a step first. Cagan’s October 2026 essay “Experimentation, AI and the Product Model” argues that because AI products are probabilistic, teams should use evals early to establish quality and reserve the costlier A/B tests for establishing value.

How to apply it

This method takes a significant technical bet (a platform capability, a public API, an AI feature) from idea to evidence. It assumes you have a product hypothesis (Chapter 01) and an outcome target (Chapter 02), and it runs over four to eight weeks alongside normal delivery.

1. Name the customer and the dominant risk. Write one sentence naming who calls or uses the capability: a person, a developer, an internal team or an AI agent. Rate Cagan’s four risks plus adoptability, the B2B lens from Chapter 01, then match the process: spikes and prototypes for feasibility risk, customer tests for value risk, a gate for a large funding ask.
2. Run a failure pre-mortem with the product trio plus security, support and a sales or partner lead. Announce that the bet failed a year from now, give everyone five silent minutes, then prompt for gaps with the three lenses. Rank the reasons and give each top risk an owner, an early-warning signal and a mitigation.
3. Get fluent on this bet. Rate yourself on the fluency self-assessment and close the two biggest gaps before the design review. Draw the system-context diagram yourself and compare it with engineering’s; the differences are your questions.
4. Review the design as a PM in five passes, using the technical design review checklist. Ask that every hard-to-reverse decision be recorded as an ADR.
5. Write the interface promise. If the bet exposes an API, platform service or MCP server, write the API and platform product brief, with an explicit section on what agents may call and what needs a human approval.
6. Plan the evidence. For AI capabilities, define the evals that establish quality (Chapter 31) and don’t expose customers until they pass. Write the experiment plan before launch: decision, hypothesis, primary metric, guardrails, unit, minimum effect, sample size and stopping rule. If you can’t randomize, name the alternative and its weaknesses.
7. Analyze with the harness, then check it. Send an experiment analysis request with the pre-registered plan and outputs in a fixed order, SRM check first. Reconcile group counts against the platform, ask the harness to critique its own analysis, and for important decisions have a fresh session or second model reproduce the headline numbers. Record the decision, evidence and limits in the decision log.

The AI lens

Building AI-powered and agent-facing products

Agents are a customer segment. Agents explore, retry, chain calls and act at machine speed for a human who may not be watching. Your API brief needs an agent section: permitted operations, scopes, rate limits, attribution to a human and revocation. The Model Context Protocol (MCP) has become the common way to expose a product to agents. Anthropic introduced it in November 2024 and in December 2025 donated it to the Agentic AI Foundation, a fund under the Linux Foundation co-founded by Anthropic, Block and OpenAI; by then there were more than 10,000 active public MCP servers. An MCP server is an API product with every obligation in this chapter, and agents will depend on its tool descriptions as surely as developers depend on field names.

Quality comes before value. AI behavior drifts when models or prompts change. Run evals before experiments, and treat a model or prompt change as a product change that needs its own evidence.

Cost belongs in the design review. For every AI capability, ask for cost per call and per outcome at expected and peak usage.

Experiments need AI-aware guardrails. Add guardrails for wrong or harmful outputs, cost per outcome and latency, and for agents, the rate of actions a human had to undo.

Doing the work with AI

Approved connectors let a harness reach experiment data, warehouses, tickets and repositories. Supply a clear brief, context and required checks.

Datadog announced its Eppo acquisition in May 2025. OpenAI announced a Statsig acquisition agreement in September 2025. Vendor agents also summarize results. Optimizely recommends specific variation and metric comparisons, significance and summary tables. Independent accuracy evidence is absent, so verify outputs.

A harness may choose a plausible metric or omit validity checks when the request is vague. Supply context, ask for critique and reproduce important conclusions. Cagan’s 2026 AI-coaching guidance emphasizes these checks. In practice:

• Supply the plan with the data. The pre-registered hypothesis, metric definitions, unit and stopping rule make the question checkable.
• Fix the order of outputs, SRM first, with an instruction to stop if it fails.
• Reconcile one number yourself. If the group sizes differ from the experiment platform’s, nothing else matters yet.
• Ask for assumptions and a critique. “What would a skeptical statistician say about this analysis?” tends to surface the assumptions a fluent summary hides.
• Reproduce important calls in a fresh session or with a second model.

The Beacon case: a technical review of Beacon Agents

The fluency check. On the self-assessment you rate yourself weak in two areas the launch depends on: data residency, because EU customers want AI Act documentation, and inference cost, because gross margin has already fallen from 81% to 72%. You spend an hour with the applied AI engineers on how cost per diagnosis scales with retrieved data, and ask the harness to summarize the ADRs behind Beacon’s regional data stores.

The pre-mortem. You run 60 minutes with the trio, a Northlight security architect, a support lead and the partner manager for Leo’s MSP. The premise: six months after Northlight Live, Beacon Agents has failed. The reasons cluster into four main stories:

Two Cutler signs also surface: a prospect’s demand for a bespoke connector, and no post-launch review of the last two correlation releases. Both go to the roadmap review.

The design review. Reviewing the Act team’s design doc for the approval gate with the checklist, you raise five questions:

• Scope: is the allowlist per tenant or global? Leo’s MSP runs operations for 60 customers with different risk tolerances, so it has to be per tenant.
• Reliability: what’s the SLO for approval latency? If Dana approves a rollback at 3 a.m. and execution waits in a queue, the UI must show “approved, executing” separately from “done.”
• Reversibility: what happens if the rollback itself fails? The doc covers the action, not undoing it.
• Data: where do EU tenants’ audit logs live, and for how long? The doc assumes one region.
• Alternatives: only a synchronous approval call was considered. You ask for the asynchronous option to be written up.

The regional audit-log decision becomes an ADR, with the cost of a second store under consequences.

The interface promise. Leo’s MSP wants to trigger approved runbook steps from its own tooling. Your brief for a Beacon Actions API and MCP server specifies tenant-scoped tokens, per-tenant rate limits, every action attributed to a named human approver, date-based versions in the style Stripe describes, and deprecation aligned with the Beacon Classic sunset (Chapter 30). Agents may read diagnoses and request actions; only a human approval or the customer’s allowlist can execute one.

The experiment. Design has a new diagnosis layout that puts the cited evidence above the proposed root cause. It first runs against the diagnosis evals to confirm it doesn’t change which root cause is shown. The plan names diagnosis acceptance rate as the primary metric, with MTTR, wrong-action rate and inference cost per incident as guardrails. Randomization is by tenant, because responders on one team would otherwise see both layouts. With about 420 customers, power is limited, so the plan fixes a longer run.

A quick look at the experiment dashboard shows a clear, significant lift. But your analysis request puts the SRM check first, and it fails: the planned 50/50 split of tenants came out noticeably uneven. The harness traces the cause to provisioning. One MSP onboarded its customers through a bulk path that bypassed the assignment service and defaulted every new tenant to the new layout. Those smaller tenants already had higher acceptance, so most of the “lift” came from who was in each group.

The team fixes assignment and reruns the test. The second run shows a smaller lift with healthy guardrails, and the layout ships. The decision log records both runs and the SRM failure’s cause.

Product 360 artifacts

These artifacts belong in the Build and Measure dimensions of your Product 360. The experiment analysis request works alongside the experiment design and readout templates in Chapter 05; the API and platform brief complements the developer platform brief in Chapter 14, which covers ecosystem strategy rather than the interface promise.

# Technical design review · [Design doc title] · [Date]

**Reviewer:** [PM]   **Doc owner:** [Tech lead]   **Linked PRD:** [link]   **Decision needed by:** [Date]

## 1. Problem and outcome
- [ ] The doc's context and goals match the PRD's problem and outcome metric.
- [ ] Notes:

## 2. Non-goals
- [ ] Every non-goal is acceptable to customers, sales, support and partners.
- [ ] Non-goals that need to be communicated externally: [list]

## 3. Who calls this
- [ ] Callers are named: human users / developers / internal teams / AI agents / partners.
- [ ] For each caller, what changes for them: [notes]

## 4. Alternatives considered
- [ ] At least one cheaper, simpler or more reversible option was weighed.
- [ ] The reason for rejecting each alternative is stated.

## 5. Data
- [ ] What is stored, where (region), for how long, under which tenant: [answer]
- [ ] Residency and retention match customer commitments.

## 6. Reliability
- [ ] Proposed SLIs and SLOs: [list]
- [ ] Error budget and what happens when it's spent.
- [ ] Failure behavior: what each user sees when a dependency is down.

## 7. Compatibility
- [ ] Observable behaviors that change for existing callers (ordering, timing, defaults, error text): [list]
- [ ] Versioning and migration plan for breaking changes.

## 8. Security and privacy
- [ ] Least-privilege access and scopes.
- [ ] Audit logging, including actions taken by agents and who approved them.
- [ ] Revocation path (kill switch, token revocation).

## 9. Cost
- [ ] Unit cost per request and per customer outcome, at expected and peak usage: [numbers]
- [ ] Margin impact: [estimate]

## 10. Observability and rollback
- [ ] How we'll know it works in production: [signals]
- [ ] How we undo it, including undoing a failed rollback.

## 11. Reversibility
- [ ] Decisions that are hard to reverse (public interface, data model, pricing unit) are flagged: [list]

## 12. Decisions to record
| Decision | ADR number | Status |
|---|---|---|
| | | |

## The regret question
What would make us regret this design in a year? [answer]

## Outcome
Approve / Approve with changes / Revise and re-review. Changes requested: [list]

# API and platform product brief · [Interface or service name] · v[n] · [Date]

**Owner:** [PM]   **Tech lead:** [name]   **Status:** Draft / In review / Published

## Customers
| Customer type | Who specifically | Their job to be done | Volume expected |
|---|---|---|---|
| External developers | | | |
| Partners | | | |
| Internal teams | | | |
| AI agents (via API or MCP server) | | | |

## The interface promise
- Resources and operations offered: [list]
- Events or webhooks emitted: [list]
- What we promise never to change without a new version: [list]
- Observable behaviors we know callers depend on (Hyrum's Law check): [list]

## Versioning and deprecation
- Versioning scheme: [date-based / numbered / other]
- What counts as a breaking change: [definition]
- Deprecation notice period and migration support: [policy]

## Reliability
- SLIs and SLO targets: [list]
- SLA, if any, and its consequences: [terms]
- Error budget policy: [what happens when it's spent]

## Limits and access
- Rate limits and quotas, by customer tier: [table]
- Authentication method and scopes: [list]
- Least-privilege defaults: [description]

## Agent readiness
- Operations agents may call: [list]
- Operations that require a human approval: [list]
- Agent rate limits and attribution (which human each agent acts for): [policy]
- MCP server: yes / no / planned. Tool descriptions owned by: [name]

## Commercials
- Pricing or internal chargeback unit: [unit]
- Cost to serve per unit: [estimate]

## Adoption and success
- Adoption metric: [active integrations, time to first successful call, share of target teams using it]
- Health metrics: [error rate, breakage reports, support tickets per integration]

## Documentation and onboarding
- Reference docs, samples and quickstart: [links]
- Time to first successful call target: [time]

## Open questions
1.
2.

# Failure pre-mortem · [Initiative] · [Date]

**Facilitator:** [name]   **Participants:** [product trio, security, support, sales or partner lead, others]

## The premise
It is [date, 6 to 12 months out]. [Initiative] has failed. Write down every plausible reason why, on your own, before anyone speaks.

## Prompts (use after silent writing, to find gaps)
- Market and product fit: did customers need it badly enough? Was the timing wrong?
- Unit economics: did cost to serve or pricing make it unsustainable?
- Process (Cagan's root causes): were ideas dictated by stakeholders? Did engineering or design arrive too late? Did all the risk sit at the end?
- Feature factory signs (Cutler): no measurement, deal-driven features, no tweaking after launch, hand-offs, invisible debt?
- Technical: compatibility breaks, reliability, security review, data residency, cost at scale?

## Raw reasons
| # | Reason | Raised by |
|---|---|---|
| 1 | | |

## Ranked risks
| Rank | Failure story | Likelihood (H/M/L) | Damage (H/M/L) | Early-warning signal | Mitigation | Owner | Review date |
|---|---|---|---|---|---|---|---|
| 1 | | | | | | | |
| 2 | | | | | | | |
| 3 | | | | | | | |

## Changes to the plan
1.
2.

## Revisit
At each milestone: which early-warning signals have moved?

# Experiment analysis request · [Experiment name] · [Date]

Use this brief as the prompt to an AI harness or the request to an analyst. If it includes customer data, use an approved tool.

## The decision
- Decision this informs: [ship / iterate / stop / roll back]
- Decision maker: [name]

## The plan, as written before launch
- Hypothesis: [text, copied from the plan]
- Unit of randomization: [user / account / team / tenant]
- Expected split: [for example, 50/50]
- Primary metric (OEC) and definition: [metric]
- Guardrail metrics and definitions: [list]
- Minimum effect worth detecting: [value]
- Stopping rule and planned sample size: [rule]
- Pre-registered segments: [list, or "none"]

## Data
- Sources (connector or system names): [list]
- Date range: [start to end]
- Known data issues: [list]
- Exclusions (internal, test and sandbox accounts, bots): [list]
- Outlier rule (cap, median or none): [rule]

## Required outputs, in this order
0. Data preparation: each exclusion applied, with unit counts by group before and after; counts of missing and duplicated events by group.
1. Sample ratio mismatch check: observed versus expected split, and whether it fails. If it fails, stop and report likely causes; do not report effects.
2. Primary metric: effect size, confidence interval and significance, as planned.
3. Guardrails: status of each, with intervals.
4. Pre-registered segments only.
5. Plain-language summary in under 150 words.
6. Assumptions made about data and method.
7. Data-quality caveats.
8. What result would change the conclusion.

## Not allowed
- Changing the primary metric or its definition.
- Reporting new segments unless labeled "exploratory."
- Declaring a winner if the SRM check fails or the stopping rule wasn't met.

## Verification (completed by the requester)
- [ ] Group counts match the experiment platform.
- [ ] Primary metric and segments are the pre-registered ones.
- [ ] Harness critiqued its own analysis; issues raised: [list]
- [ ] For major decisions, a second pass reproduced the headline numbers.

# Technical fluency self-assessment · [Name] · [Quarter]

Rate each area 1 to 4. 1 = I need someone to explain it to me. 2 = I can follow the discussion. 3 = I can ask the questions that expose tradeoffs, without help. 4 = I can teach it and spot a weak answer.

| Area | Behavioral anchor for a 3 | Rating (1 to 4) | Evidence |
|---|---|---|---|
| Architecture | I can draw our system-context diagram and explain which calls are synchronous | | |
| APIs | I can say which proposed changes are breaking and why | | |
| Data models | I can explain our main entities, tenancy and residency | | |
| Reliability | I can propose SLOs and explain the error budget tradeoff | | |
| Security and privacy | I can explain our access model, scopes and audit trail | | |
| Cost | I know our unit cost per request and per customer outcome | | |
| Experimentation | I can frame a test and spot SRM, peeking and segment fishing | | |
| AI systems | I can explain our evals, retrieval and agent permissions | | |
| Design docs | I can review a design doc and find the hard-to-reverse decisions | | |

## Two areas to grow this quarter
1. Area: [ ]   Plan: [who will teach me, what I'll read in our own docs, what I'll practice]   Evidence of progress: [ ]
2. Area: [ ]   Plan:   Evidence of progress:

## Review
Re-rate at the end of the quarter with your tech lead's input.

# Feature factory check · [Team] · [Quarter]

Score each sign 0 (not true), 1 (sometimes true) or 2 (true). Signs paraphrased from John Cutler.

| Sign | Score (0 to 2) | Evidence |
|---|---|---|
| We don't measure the impact of what we ship | | |
| People move between projects without a stable mission | | |
| Launches are celebrated; outcomes aren't checked | | |
| We rarely kill or scrap work | | |
| Work isn't connected to core metrics | | |
| We don't review past product decisions | | |
| We prioritize hard but never check whether priorities paid off | | |
| We don't iterate after launch | | |
| Hand-offs keep the team away from research and from customers after release | | |
| We ship in large batches | | |
| Deal-driven features ignore complexity cost | | |
| Technical debt and refactoring are invisible | | |

**Total:** [ ] of 24

## AI acceleration check
Has AI-assisted building increased our output without improving outcomes? [evidence]

## Two changes for next quarter
1.
2.

Prompts

Goal: Find the product decisions hidden in an engineering design doc before you approve it.
Paste in: the design doc, the PRD it implements and any customer commitments it touches (SLAs, residency, security terms). Use an approved tool if the doc is confidential.

Act as a senior product manager reviewing this design doc for [product]. Compare it with the PRD.

1. List any gap between the doc's goals and non-goals and the PRD's problem and outcome. Flag non-goals that customers, sales or partners would object to.
2. Summarize the alternatives considered and the stated reason each was rejected. If only one option is discussed, say so, and suggest one cheaper or more reversible alternative worth writing up.
3. List every customer promise the design touches: security, data residency and retention, latency, availability, cost. For each, say whether the doc addresses it.
4. List the callers of any interface (users, developers, internal teams, AI agents) and what changes for each. Name any observable behavior that changes, even if it isn't documented.
5. Identify the decisions that would be expensive to reverse and should be recorded as ADRs.
6. Write ten questions for the design review, most important first.

Quote the doc when you cite it. Don't assume facts that aren't in the material.

[design doc]
[PRD]

Check the output: read the alternatives-considered section yourself and compare it with the summary. Confirm every quoted line exists in the doc. Drop any question you can answer from the doc in under a minute.

Goal: Generate a broad, ranked list of failure stories for a major bet, as preparation for a team pre-mortem.
Paste in: the plan or PRD, the launch date, the target customers, the business model and known constraints.

It is [date, 6 to 12 months after launch] and [initiative] has failed. Write the most plausible failure stories, each in two or three sentences, in the past tense.

Cover all of these lenses:
- Market and fit: customers didn't need it badly enough, or timing was wrong.
- Unit economics: cost to serve, inference cost or pricing made it unsustainable.
- Process: ideas came from stakeholders, design or engineering joined too late, all the risk sat at the end, or the team worked as a feature factory.
- Technical: compatibility breaks, reliability, security review, data residency, cost at scale.
- Adoption: the buying group, security or procurement blocked it.

For each story, give a likelihood (H/M/L), damage (H/M/L), the earliest signal that would warn us, and one mitigation. Rank by likelihood times damage.

Write specific failure stories grounded in the plan. Label assumptions beyond the supplied evidence.

[plan]

Check the output: use it after the team's silent writing round, not before, so it widens the list rather than anchoring it. Delete generic stories that would apply to any product. The useful ones mention your customers, your architecture or your pricing.

Goal: Turn engineering's interface design into a product promise that customers, partners and agents can rely on.
Paste in: the API spec or design doc, the target customers, any SLOs and the existing versioning or deprecation policy.

Using the material provided, draft an API and platform product brief for [interface or service] with these sections: customers (developers, partners, internal teams, AI agents); the interface promise; observable behaviors callers are likely to depend on even though they aren't documented; versioning and deprecation; reliability (SLIs, SLOs, any SLA); limits and access (rate limits, scopes, least-privilege defaults); agent readiness (which operations agents may call, which need human approval, attribution, MCP server plan); commercials; adoption and success metrics; open questions.

Rules:
- Mark every field the material doesn't support as "Open question" rather than filling it in.
- List any change in the spec that would break existing callers.
- Flag any operation that lets an agent change customer systems without a human approval or allowlist.

[material]

Check the output: walk the "breaking changes" and "agent operations" lists with the tech lead. Confirm the rate limits and SLOs match what engineering can operate, not what sounds good.

Goal: Pre-register an experiment so the analysis can't drift after the data arrives.
Paste in: the hypothesis, the feature, the customer base (number of accounts and users, how they're grouped) and the metrics available.

Help me write an experiment plan for [feature]. Ask me questions until you can fill every field, then produce the plan:

- The decision this informs and who makes it.
- Hypothesis.
- Primary metric (one) with a precise definition.
- Guardrail metrics: business guardrails (latency, error rate, cost, and for AI features wrong-output and undo rates) and the sample ratio check.
- Unit of randomization, and why. Consider whether users within the same account would contaminate each other.
- Minimum effect worth detecting, in terms of the decision.
- Whether we have enough units to detect it in a reasonable time. Explain the tradeoff in plain language; don't show formulas.
- Stopping rule, fixed in advance.
- Pre-registered segments.
- If a randomized test isn't feasible, the best alternative method and its weaknesses.
- For AI features, the evals that must pass before exposure.

[hypothesis and context]

Check the output: make sure there's exactly one primary metric and the stopping rule is a rule, not "when it looks significant." If the plan says the test can't reach a useful answer, believe it and change the design.

Goal: Get a trustworthy analysis of a finished experiment, in an order that catches broken experiments first.
Paste in (or connect the harness to): the pre-registered experiment plan and the experiment platform or warehouse tables. Use only tools approved for customer data.

Analyze the experiment [name] using the data in [sources] for [date range]. The plan below was written before launch. Follow it exactly.

Produce, in this order:
1. Sample ratio mismatch check: expected versus observed split of [unit]. If the check fails, stop. Report the mismatch and the most likely causes, and don't report any effects.
2. Primary metric [name, definition]: effect size, confidence interval and significance.
3. Each guardrail [names]: effect and interval, and whether it breached its threshold.
4. Pre-registered segments only: [list]. Anything else must be labeled "exploratory."
5. A plain-language summary under 150 words for [decision maker].
6. Every assumption you made about the data and the method.
7. Data-quality caveats.
8. What result would change the conclusion.

Use the pre-registered primary metric. Label added segments as exploratory. Declare a winner only when the stopping and decision rules permit it.

[experiment plan]

Check the output: compare the group counts with the experiment platform yourself. Confirm the metric definitions match the plan. Then run the critique prompt in this chapter on the result.

Goal: Find the weaknesses in an experiment analysis, whether a harness, an analyst or a vendor tool produced it.
Paste in: the analysis, the pre-registered plan and the raw summary counts from the experiment platform.

Act as a skeptical statistician reviewing this analysis before a product decision. Identify weaknesses, deviations and unsupported conclusions.

1. Does the analysis follow the plan: same primary metric, unit, segments and stopping rule? List every deviation.
2. Was a sample ratio mismatch check done, and does the split in the analysis match the platform counts I've supplied?
3. Is there any sign of peeking (stopping early) or of segments chosen after seeing the data?
4. Is the effect size practically meaningful, and how wide is the interval? What decision would a result at each end of the interval support?
5. Does any result look too good or surprising? List checks to rule out a data or logging problem.
6. What else could explain the result besides the change being tested?

[analysis]
[plan]
[platform counts]

Check the output: for any deviation it finds, go back to the source data or the analyst before deciding. For a major decision, run the original analysis again in a fresh session or with a second model and compare the headline numbers.

Goal: Get up to speed on the architecture behind your product, using your own company's documents as the textbook.
Paste in (or connect the harness to): design docs, ADRs, architecture diagrams and runbooks for [system]. Use an approved tool for internal documents.

I'm the PM for [product] and I want to understand [system] well enough to review designs and spot product tradeoffs. Using only the documents provided:

1. Describe the system-context: main components, external systems, where data lives (including region and tenancy) and which calls are synchronous or asynchronous.
2. Summarize the ADR history: the five decisions that most shape the system today, what each gave up, and which are hard to reverse.
3. Explain the reliability targets and how close we run to them, if the documents say.
4. Explain what drives cost per request.
5. Quiz me with eight questions, one at a time, starting easy and getting harder. After each answer, tell me what I got right and what I missed, citing the source document.

Cite the document for every claim. If the documents don't cover something, say so rather than filling the gap from general knowledge.

Check the output: draw the system-context diagram yourself, then compare it with the tech lead's. Ask about every difference. Re-rate yourself on the fluency self-assessment afterward.

Anti-patterns

Checklist

• For each major capability, the customer (human, developer, internal team, agent) and the dominant risk are named, and the process matches the risk.
• A failure pre-mortem with engineering, security and support has produced ranked risks with owners, signals and mitigations.
• The team has scored itself on the feature factory check this quarter.
• You can draw the system-context diagram and explain the main data entities, tenancy and residency.
• You’ve completed the fluency self-assessment and are working on two named areas.
• Every significant design doc has a PM review covering goals, alternatives, data, reliability, compatibility, security, cost and reversibility, and hard-to-reverse decisions are recorded as ADRs.
• Every API, platform service or MCP server has a product brief with versioning, SLOs, limits, scopes and an agent section.
• AI capabilities pass evals before any customer experiment.
• Every experiment has a pre-registered plan with one primary metric, guardrails, a unit of randomization and a stopping rule.
• Every analysis starts with an SRM check, reports effect sizes with intervals and labels unplanned segments exploratory.
• AI-produced analyses are reconciled against platform counts and critiqued before anyone decides.