Questions That Drive Decisions

Start with the decision, then ask what evidence could change it. Define the question precisely, check the answer and review the decision separately from its outcome.

Why this matters

“How is adoption going?” leaves the population, period, metric and comparison undefined. A harness can answer fluently while choosing those details for you.

Connected analytics make access easier, but useful analysis still requires a named decision, an owner and an answer that could change the plan. Otherwise, data becomes decoration.

This chapter builds the question brief. Chapter 18 maps the evidence and reports. Chapter 19 covers connected analysis and verification.

Core concepts and frameworks

Start from the decision, not the data

Use this five-link chain:

1. Decision: what you’re choosing, between which options, by when and who decides.
2. Question: the one thing you need to know to choose well.
3. Evidence: the quantitative and qualitative sources that answer it.
4. Report: the view that puts the evidence in front of you (a funnel, a cohort table, a theme analysis).
5. Owner: the person who gets the answer, checks it and brings it to the decision.

Dashboards can reveal problems, but begin analysis with the decision the result will inform.

Douglas Hubbard’s How to Measure Anything treats measurement as reducing uncertainty enough to inform a choice. Ask: what answer would change what we do? Drop questions whose plausible answers all imply the same action. The decision threshold also tells you how precise the evidence must be.

Hubbard argues that intangibles become measurable when you define observable changes. Higher trust might mean fewer manual overrides or more production approvals. Specify those observations before measuring.

Decisions across the product lifecycle

Use seven decision types across the lifecycle:

A question bank for each decision

Adapt this question bank to your product. Each question identifies quantitative and qualitative evidence and the decision it informs.

Combine numbers with conversations. Eric Ries’s “metrics are people, too” principle asks you to trace results to real customers and speak with them. In B2B, analyze the account, then the role. Heavy use by three accounts can conceal low adoption across four hundred others.

Discover. Choose a problem and segment. Use Teresa Torres’s opportunity solution tree (Chapter 01) to rank customer needs under a desired outcome.

Invest. Choose funding, scale and sequence. Marc Andreessen’s product/market-fit framing asks whether a product can satisfy a worthwhile market. Look for evidence of customer pull.

Build. Decide whether to continue the solution. Test Marty Cagan’s four risks: value, usability, feasibility and business viability.

Launch. Choose ship, widen, hold or roll back. Kohavi, Tang and Xu recommend an overall evaluation criterion (OEC) tied to long-term value, plus guardrails. Set the questions and decision thresholds before launch (Chapter 05).

In enterprise products you often have too few accounts for a clean A/B test. The fallback is a staged rollout compared with similar accounts that haven’t received the change, plus a qualitative readout. It’s weaker evidence, so say so in the decision memo.

Grow. Choose retention and expansion investments. Rodden, Hutchinson and Fu’s HEART framework links goals to signals and metrics across happiness, engagement, adoption, retention and task success. In B2B, measure by account and role.

Price. Set or change pricing and packaging (Chapter 15). Van Westendorp’s four questions test prices that feel too expensive, suspiciously cheap, expensive but acceptable, or a bargain.

Retire. Choose sunset, migration or continuation (Chapter 30). Following Annie Duke’s Quit, set stopping and pause conditions before the decision becomes emotional.

Good questions and vanity questions

Eric Ries distinguishes vanity metrics, which flatter, from actionable metrics, which inform choices. His examples include cumulative totals that only rise. The Lean Startup offers three tests for reports:

• Actionable: the answer shows cause and effect clearly enough that you know what to do.
• Accessible: the people who need to act can understand it, and it’s expressed in terms of real customers and their behavior.
• Auditable: anyone can trace the number back to the underlying customers and records, and check it.

Vanity questions are the questions that produce vanity metrics. You can spot them with five tests:

1. Does the answer only go up? Totals (all users ever, all incidents touched, all agents deployed) rise with time and market size. Rewrite as a rate per account, per cohort or per period.
2. Could any plausible answer change what we do? If not, it’s a vanity question, however interesting.
3. Is there a comparison? A number without a baseline, a control, a segment or a target is trivia.
4. Can we name the customers behind it? If you can’t trace it to accounts you could call, you can’t audit it and you can’t learn why.
5. Can it be gamed once it’s a target? Charles Goodhart observed in 1975 that a statistical regularity tends to collapse once it’s used for control. Marilyn Strathern’s 1997 phrasing became the common version: when a measure becomes a target, it ceases to be a good measure. Pair every target with a guardrail that catches the gaming.

Here’s what rewriting looks like:

Jeff Bezos’s 2016 shareholder letter warns about proxies: teams begin managing a metric or process instead of the customer outcome it represents. Check whether the proxy still reflects customer reality.

Which report answers which question

Choose the report that answers the question. Chapter 18 explains each type:

Every one of these should state four things on its face: the population, the time window, the metric definition and the comparison. If a report you’re handed doesn’t, ask for them before you read the result.

Decision quality: judge the decision, not the dice

Annie Duke’s Thinking in Bets distinguishes decision quality from outcome quality. Luck can reward a weak decision or defeat a strong one. She calls outcome-based judgment resulting. It encourages risk avoidance and rewritten history.

The fix is to treat decisions as bets. State your confidence, list the possible outcomes, choose the option with the best odds and record what you knew at the time. Then, when you review the decision later, ask whether the process was sound given that information, separately from whether the result was good.

Ronald Howard and colleagues at Stanford and the Strategic Decisions Group developed a definition of decision quality with six elements, later set out by Carl Spetzler, Hannah Winter and Jennifer Meyer in Decision Quality:

1. An appropriate frame: the right problem, scope and perspective.
2. Creative, doable alternatives.
3. Relevant and reliable information.
4. Clear values and trade-offs.
5. Sound reasoning.
6. Commitment to action.

Decision quality follows a weakest-link principle. Better evidence improves information, but cannot repair a bad frame or an omitted alternative.

Kahneman, Lovallo and Sibony’s 2011 HBR checklist helps reviewers detect bias in proposals. Ask about attachment to the preferred option, dissent, credible alternatives, number sources and the downside. Apply it to your memo or ask the harness to challenge it.

One-way doors and two-way doors

Jeff Bezos’s 2015 shareholder letter distinguishes Type 1 decisions, consequential and hard to reverse, from Type 2 decisions, reversible at limited cost. The first need deliberation and consultation. Individuals or small groups can make the second quickly. Applying Type 1 process to everything slows experimentation.

His 2016 letter added a rule of thumb: most decisions should be made with about 70% of the information you wish you had. Waiting for 90% usually means you’re being slow, and being wrong is often cheaper than being slow if you’re good at noticing and correcting mistakes. The same letter describes disagree and commit: when there’s conviction without consensus, say you disagree, then commit fully to the decision.

Put the two ideas together and you get an evidence bar for each decision:

A skill worth practicing is converting Type 1 decisions into Type 2 ones. A staged rollout, a per-customer toggle, a grandfathering window or a pilot contract makes a one-way door partly reversible, and lowers the evidence bar to match.

Framing a question so a harness can answer it

A well-framed data question has four parts that a vague one leaves out. This book’s shorthand is PTMC:

• Population: which accounts, users or records count, and which are excluded. “Paying enterprise accounts live for 60 days or more, excluding internal and trial tenants.”
• Time window: the dates, the time zone and whether partial periods count. “Weeks starting 1 July to 30 September, UTC, complete weeks only.”
• Metric definition: the numerator, the denominator, the unit and the filters, or the name of a governed definition everyone already agrees on.
• Comparison: against what. A prior period, a control group, another segment or a target.

Add the decision and required output, including definitions and record counts. Anthropic’s prompting guidance recommends supplying the context a capable new colleague would need. Use that as the clarity test.

How to apply it

Use this method for any decision that matters enough to write down. For a Type 2 decision it takes an afternoon and most steps shrink to a sentence. For a Type 1 decision, expect one to three weeks, most of it waiting for evidence and conversations.

1. Name the decision. Write the options (at least three, including “do nothing” or “keep doing what we do”), the date it must be made, who decides and whether it’s a one-way or two-way door. If you can’t list the options, you aren’t ready to ask questions yet; go back to framing.
2. Draft the questions that could change it. Start from the question bank for the right lifecycle stage. Keep three to five. For each one, write the answer that would change your choice. Drop any question that fails the value test.
3. Run the vanity tests. Check each question for totals that only rise, missing comparisons, numbers you can’t trace to accounts and targets that could be gamed. Rewrite the failures as rates, per-account measures or cohort comparisons.
4. Define the metrics. For every metric a question uses, find the governed definition if your company has one. If it doesn’t, write a metric definition card: numerator, denominator, population, time grain, source, owner and the guardrail that catches gaming. Agree the definitions with whoever will challenge the result before you see the numbers.
5. Plan the evidence. For each question, list the quantitative sources, the qualitative sources and the report type that answers it. Decide the comparison now. If accounts are too few for an experiment, name the matched comparison and the qualitative follow-up you’ll use instead.
6. Ask the harness, then audit the answer. Frame each question with population, time window, metric definition and comparison, plus the decision it serves. Ask for the definitions, filters and record counts back. Reconcile the result with a number you already trust, or ask for it two different ways. Chapter 19 covers this step in detail.
7. Put people behind the numbers. Pull the list of accounts behind the key result and talk to a handful: some that moved, some that didn’t. This is where you find out whether the metric means what you think, and why it moved.
8. Write the decision memo. Lead with the recommendation. Cover the frame, the options, the evidence and your confidence in it, the trade-offs, the reasoning, the reversibility and the kill criteria. Run the bias checks against it before anyone else reads it.
9. Decide and commit. Bring the memo to the decision maker or forum. Record who disagreed and committed. Log the decision with its review date.
10. Review the decision, not the dice. On the review date, compare what happened with what you expected. Ask two separate questions: was the process sound given what we knew, and what did we learn about the world? Feed the answers back into the question bank.

Bezos’s 2017 letter describes narrative memos read silently in meetings and argues that good memos need substantial revision. Use the time AI saves on drafting to examine the reasoning.

The AI lens

Building AI-powered products

AI features add decisions the classic lifecycle doesn’t cover, and each brings its own questions.

• How much autonomy to grant. “Is the AI good?” is too broad to answer. Ask instead: for which kinds of task, at what accuracy and with what consequences when wrong, should it act without a person? That needs a measure of how often outputs are accepted, a measure of how often accepted outputs later turn out to be wrong, and a breakdown by severity. Autonomy decisions that touch production or customer data are usually one-way doors for the customer, even when they’re reversible for you.
• Whether quality is good enough to ship. For AI features, eval results are first-class evidence in the build and launch question banks, alongside usage and interviews. Chapter 31 covers evals. The question to ask is whether the eval set reflects the incidents, documents or requests your customers actually send.
• Whether the economics hold. Inference cost scales with usage. Every AI feature needs a cost-per-task question next to its value question, broken down by task type, because the average hides the expensive tail.
• Which metric to target. AI features are especially prone to Goodhart’s problem. Target an automation rate and the system (or the team tuning it) will find ways to automate more, including things it shouldn’t. Pair every AI target with a guardrail that measures the harm the target could cause.

AI features also create new evidence. Every accepted, edited or rejected suggestion is a labeled example of what users think good looks like. Questions such as “where do users most often edit the AI’s output, and what do they change?” are some of the most useful you can ask, and the answers are sitting in your logs.

Doing the work with AI

MCP connects assistants to analytics under existing permissions. Introduced in November 2024, it moved to the Agentic AI Foundation in December 2025, with more than 10,000 public servers reported then. Mixpanel’s server, for example, exposes funnels, retention and flow reports to natural-language requests.

Governed definitions remain essential. dbt’s semantic layer defines metrics once and exposes them through MCP. Snowflake recommends semantic models and verified-query evaluations for Cortex Analyst because database schemas lack business definitions.

Public benchmarks point the same way. On the BIRD benchmark, which tests natural-language questions against real-world databases, human data engineers and students reached about 93% execution accuracy. When this chapter was written in October 2026, the top entry on the leaderboard scored about 83%. So even the best systems get some well-posed questions wrong, before you add the ambiguity of a badly posed one.

Three rules follow:

• Name the definition or supply it. Ask whether a governed definition exists before trusting a number. If it doesn’t, put the definition in the question.
• Make the answer auditable. Ask the harness to show the population, filters, time window and record count it used, and to list its assumptions. Reconcile against a number you already trust.
• Keep the decision human. Analytics vendors now ship agents that watch metrics on a schedule and propose analyses (Amplitude’s AI agents are one example). Monitoring “what changed?” is being automated. Deciding “what should we do about it?” isn’t, and neither is judging whether the evidence is good enough.

When questions involve customer data, contracts or revenue, use only harnesses and connectors your company has approved for that data, and ask for aggregates rather than records about named individuals unless you need them.

The Beacon case: questions before the keynote

Rewriting the keynote question. “Incidents touched” fails three vanity tests: it only goes up, it rises with alert volume whether or not Beacon helps, and it has no comparison. You propose Beacon’s North Star instead, incidents resolved with Beacon’s help per week, reported per active account and shown with two guardrails: the wrong-action rate and the share of accepted diagnoses later contradicted by the post-incident review. It makes a smaller number and a better story, and it’s one you’d be comfortable having a customer’s SRE lead check.

Naming the decision. The rollback decision has three options: allow autonomous rollback for allowlisted services at launch, keep rollback supervised (human approval) and revisit later, or remove rollback from Agents until the diagnosis quality improves. For customers, letting an agent roll back production on its own is close to a one-way door. You classify it Type 1 and write a question brief for each critical unknown.

The question bank for the decision.

The metric definition card. The second question depends on diagnosis acceptance rate, and you find three teams computing it differently. You write the card: the numerator is incidents where a responder accepted Beacon’s proposed root cause; the denominator is incidents where Beacon proposed one; the population is production environments of paying customers, excluding internal and test tenants; the grain is weekly, in UTC. The guardrail is the share of accepted diagnoses later contradicted by the post-incident review. The gaming risk is an interface default that nudges responders toward “accept.”

Asking the harness. You send the first question to your harness, which is connected to product analytics, the incident store and the CRM, with the population, window, definition and comparison spelled out, and you ask it to return the account count and filters it used. The account count is higher than the number of design partners. Trial and internal tenants had slipped in because they carry the same feature flag. You re-ask with an explicit exclusion and reconcile the corrected count against the design-partner list from customer success. The corrected answer is less flattering, and it’s the one you can defend.

What the evidence says. In the design-partner data, supervised remediation shortened resolution for the allowlisted incident classes, and responders accepted most restart and scale-out suggestions. Rollback was different. Accepted rollback suggestions on services with shared dependencies were contradicted by post-incident reviews far more often than restart suggestions, the same pattern as the incident everyone remembers. Two security reviewers said they’d allowlist rollback only with per-action audit logs, scoped credentials and a kill switch, and one of those controls isn’t ready.

The decision memo. You recommend option two: launch with autonomous restarts and scaling for allowlisted services, keep rollback supervised, and build a per-customer rollback toggle so the later change is a two-way door for Northlight even though it isn’t one for customers. The kill criterion is written in advance: any autonomous action with customer impact pauses autonomy expansion until a review is complete. The review date is set for after Northlight Live. The product marketing manager disagrees, because autonomous rollback would make a stronger keynote demo, and commits; the memo records that. The keynote shows supervised rollback with the approval step on screen.

The review. When the review date comes, you’ll judge two things separately: whether the process was sound given what you knew in October, and what the post-launch data says about rollback reliability. If autonomous rollback would have been fine in hindsight, that’s information about the world, not evidence that the decision was wrong.

Product 360 artifacts

# Decision question bank · [Product] · v[n] · [Date]

**Owner:** [PM]   **Reviewed with:** [analytics lead, product ops]   **Next review:** [Date]

One row per standing question. Keep the questions you ask every quarter here, with the evidence that answers them, so nobody reinvents them.

| # | Stage (Discover / Invest / Build / Launch / Grow / Price / Retire) | Decision | Question (with population, window, metric, comparison) | Quantitative evidence | Qualitative evidence | Report type | Answer that would change the decision | Owner | Verified question? (Y/N) |
|---|---|---|---|---|---|---|---|---|---|
| 1 | | | | | | | | | |
| 2 | | | | | | | | | |
| 3 | | | | | | | | | |

## Retired questions
| Question | Why retired (failed value test, vanity, decision no longer live) | Date |
|---|---|---|
| | | |

Rules: every question names a decision. Totals that only rise are rewritten as rates, per-account measures or cohort comparisons. A question marked "verified" has a reviewed definition and a known-good answer the harness can be checked against.

# Question brief · [Short name] · [Date]

**Owner:** [Name]   **Reviewer:** [Name]   **Answer needed by:** [Date]

## The decision
- Decision: [what we are choosing]
- Options: 1. [ ] 2. [ ] 3. [do nothing / keep current approach]
- Decision maker or forum: [ ]
- Decision date: [ ]
- Type: one-way door (Type 1) / two-way door (Type 2). Why: [ ]

## The question
[One sentence.]

## Why it matters
- What answer would change our decision: [ ]
- What we'll do if the answer is [A]: [ ]  If [B]: [ ]
- Confidence needed to act: [for example, "a clear direction across most comparison accounts"; roughly 70% of the information we'd like is enough for Type 2]

## Framing (PTMC)
- **Population:** [who or what counts; exclusions such as internal, trial or test tenants]
- **Time window:** [dates, time zone, partial periods in or out]
- **Metric definition:** [governed definition name, or numerator / denominator / unit / filters; link the metric definition card]
- **Comparison:** [prior period / control / matched accounts / segment / target]

## Evidence plan
| Evidence | Type (quant / qual) | Source or connector | Report type | Status |
|---|---|---|---|---|
| | | | | |

## Output requested
Format: [table, chart, one paragraph]. Show the definitions, filters, time window and record counts used. List assumptions and anything that would change the answer.

## Answer
- Result: [ ]
- Reconciled against: [known number or second method]
- Accounts behind the result we spoke to: [ ]
- Confidence: low / medium / high, because [ ]
- Effect on the decision: [ ]

# Metric definition card · [Metric name]

**Owner:** [Name]   **Last reviewed:** [Date]   **Status:** Draft / Agreed / Governed (in semantic layer)

## Meaning
- In plain language: [one sentence a new hire would understand]
- Decision it serves: [ ]
- Good direction: up / down   **Target or threshold:** [ ]

## Formula
- Numerator: [ ]
- Denominator: [ ]
- Unit: [percent, count per account per week, currency]

## Scope
- Population: [ ]
- Exclusions: [internal, test, trial tenants; other]
- Time grain and time zone: [ ]
- Unit of analysis: [user / account / environment / incident]

## Source of truth
- Governed definition: [name in the semantic layer or metrics catalog, or "none yet"]
- Underlying sources: [systems, not tables]
- Verified question and known-good answer: [a question with an answer the team has checked, for testing harness answers]

## Integrity
- Paired guardrail metric: [the harm this target could cause, measured]
- Gaming risks: [how people or systems could move the number without moving the outcome]
- Known caveats: [seasonality, data gaps, definition changes and their dates]

## Change log
| Date | Change | Approved by |
|---|---|---|
| | | |

# Decision memo · [Decision] · [Date]

**Author:** [PM]   **Decision maker:** [Name or forum]   **Type:** one-way door / two-way door   **Review date:** [ ]

## Recommendation
[Two or three sentences: what we should do, and the single strongest reason.]

## Frame
- The problem we are solving: [ ]
- In scope: [ ]   Out of scope: [ ]
- What happens if we do nothing: [ ]

## Options considered
| Option | What it means | Main upside | Main risk |
|---|---|---|---|
| 1. | | | |
| 2. | | | |
| 3. Do nothing / keep current approach | | | |

## Evidence
- What we know (with question briefs linked): [ ]
- How confident we are and why: [ ]
- What we don't know, and why we're deciding anyway: [ ]

## Values and trade-offs
[The criteria we weighed and which mattered most.]

## Reasoning
[Why this option beats the others, in prose.]

## Pre-mortem and bias check
- It's a year from now and this failed. The most likely reasons: [ ]
- Have we fallen in love with this option? Were dissenting views and credible alternatives explored? Where did the key numbers come from? Is the worst case bad enough? [answers]

## Reversibility and kill criteria
- How we could reverse or limit it: [toggle, staged rollout, contract terms]
- Kill or pause criteria, set now: [ ]

## Commitment
- Owner and resources: [ ]
- Who disagreed and committed, and on what: [ ]

## Post-decision review (complete on the review date)
- What happened: [ ]
- Was the process sound given what we knew then? [ ]
- What did we learn about the world? [ ]
- Changes to our question bank or metric cards: [ ]

# Vanity question rewrite · [Team or review] · [Date]

Run every question in a review, dashboard or deck through these five tests. Rewrite any that fail.

| # | Original question | Only goes up? | Could change a decision? | Has a comparison? | Traceable to accounts? | Gameable without a guardrail? | Rewritten question | Decision it informs |
|---|---|---|---|---|---|---|---|---|
| 1 | | Y/N | Y/N | Y/N | Y/N | Y/N | | |
| 2 | | | | | | | | |
| 3 | | | | | | | | |

A question passes when it can change a decision, includes a comparison, traces to named accounts and has a guardrail if it will become a target. Totals that only rise become rates, per-account measures or cohort comparisons.

# Decision review · [Period] · [Date]

**Facilitator:** [Name]   **Decisions reviewed:** [number]

For each decision whose review date has passed, judge the process and the outcome separately.

| Decision | Date made | Type (1 / 2) | Expected outcome and confidence | Actual outcome | Process quality (frame, alternatives, information, trade-offs, reasoning, commitment: weakest link) | Decision or dice? | Lesson | Change to question bank |
|---|---|---|---|---|---|---|---|---|
| | | | | | | | | |

## Patterns
- Two-way doors that waited too long: [ ]
- One-way doors decided too lightly: [ ]
- Questions that kept failing the value test: [ ]

Prompts

Goal: Turn a pending decision into three to five questions that could change it, each with the evidence that answers it.
Paste in: the decision, its options, the decision date, who decides, and a short description of the product and its customers.

I'm a product manager deciding [decision] for [product]. The options are [options, including do nothing]. The decision is due [date] and [person or forum] decides.

1. Classify the lifecycle stage (discover, invest, build, launch, grow, price, retire) and say whether this is a one-way door or a two-way door, with your reasoning.
2. Propose up to eight candidate questions. For each, state the answer that would change the decision. Drop any question where every plausible answer leads to the same choice, and tell me which you dropped and why.
3. For the three to five questions that remain, give: the quantitative evidence, the qualitative evidence, the report type that would show it (trend, funnel, cohort, segment breakdown, distribution, experiment readout, NRR bridge or theme analysis) and the population, time window, metric definition and comparison it needs.
4. Use accounts and roles as the unit of analysis where it matters, not only users.

Use supplied data and results. Where availability is uncertain, name the source I should check.

Check the output: for each question, read the "answer that would change the decision" aloud. If it doesn't actually flip the choice, cut the question. Make sure at least one question has qualitative evidence you'll gather by talking to customers.

Goal: Replace questions that produce feel-good numbers with questions that inform decisions.
Paste in: the questions or metrics from a dashboard, review deck or status update.

Review these questions and metrics from [team or review]. For each one, test:
- Does the answer only go up over time (a cumulative total)?
- Could any plausible answer change a decision? Which decision?
- Is there a comparison (prior period, control, segment, target)?
- Can the number be traced to named accounts we could call?
- If it became a target, how could it be gamed, and what guardrail would catch that?

Then rewrite each failing question as a rate, a per-account measure or a cohort comparison, and name the decision it informs. Keep any question that passes and say why.

[questions or metrics]

Check the output: confirm each rewrite names a real decision your team makes. If a rewrite needs data you don't collect, decide whether the question is worth instrumenting or should be answered qualitatively.

Goal: Get a result with an auditable method and sources.
Paste in: the question brief (decision, question, population, time window, metric definition, comparison). Run it in a harness connected only to approved data sources.

I'm deciding [decision] by [date]. Answer this question: [question].

- Population: [who counts; exclude internal, trial and test tenants and anything else listed]
- Time window: [dates, time zone; complete periods only unless stated]
- Metric: use the governed definition [name] if it exists. If not, use: numerator [ ], denominator [ ], unit [ ], filters [ ]. Tell me which you used.
- Comparison: [prior period / matched accounts / control / segment / target]

Return:
1. The answer in a short table, with the comparison.
2. The exact population, filters, time window and record counts you used.
3. Any assumptions you made where my instructions were ambiguous.
4. The list of accounts behind the result, so I can follow up with some of them.
5. What would most likely change this answer.

If the data can't answer the question as framed, say so rather than approximating silently.

Check the output: compare the record count with a number you already trust (for example, the count of design partners or paying accounts). Ask for one key figure a second way. If the population is wrong, fix the framing and re-ask before reading the result.

Goal: Find out how a metric is really being calculated across the company, and agree one definition.
Paste in (or connect the harness to): dashboards, reports, decks and documents that mention the metric, plus any semantic-layer or metrics-catalog entry for it. Use a harness your company has approved for internal business data.

Find every place [metric name] is defined or reported in the material provided. For each, extract the numerator, denominator, population, exclusions, time grain and source, and quote the definition with its location.

Then:
1. Show where the definitions differ, and how much each difference could change the number.
2. Propose one definition as a metric definition card: plain-language meaning, decision it serves, formula, population and exclusions, time grain and time zone, source of truth, owner, paired guardrail metric, gaming risks and known caveats.
3. List the people or teams who'll need to agree to the change.

Choose the definition by business meaning and documented ownership.

Check the output: open two of the cited sources and confirm the extracted definitions. Circulate the card to the teams that use the metric before you use it in a decision.

Goal: Find the weakest link in a decision before the decision forum does.
Paste in: the decision memo and the question briefs behind it. Remove confidential details your tool isn't approved for.

Act as a skeptical reviewer of this decision memo. Assess it in two passes.

Pass 1, decision quality. Rate each element as strong, adequate or weak, with a reason: the frame, the alternatives, the information, the values and trade-offs, the reasoning and the commitment. Name the weakest link, since a decision is only as good as its weakest element.

Pass 2, bias check. For each, say whether the memo shows signs of it and quote the passage: self-interest, falling in love with the option, groupthink or missing dissent, an overly convenient analogy, missing credible alternatives, anchoring on a number of unclear origin, an overly optimistic base case, a worst case that isn't bad enough, and excessive caution.

Then list the three questions the decision maker is most likely to ask, and whether the memo answers them.

Return a candid review of the supplied memo, with specific evidence for each concern.

[memo and question briefs]

Check the output: fix the weakest link first, even if it isn't the easiest. If the review says an alternative is missing, add it to the options table rather than arguing it away.

Goal: Stop treating every decision as a one-way door, or every one-way door as routine.
Paste in: a list of pending decisions with a line of context each (from a roadmap review, planning doc or decision log).

For each decision below, classify it as a one-way door (consequential and hard to reverse) or a two-way door (changeable and reversible), from the perspective of both our company and our customers. For each:
1. Explain the classification in one sentence.
2. Recommend the evidence bar: number of questions, quantitative and qualitative evidence, who decides and which artifact (decision log line or full memo).
3. For each one-way door, suggest a way to make it more reversible (staged rollout, per-customer toggle, pilot terms, grandfathering) and how that would change the evidence bar.
4. Flag any two-way door that appears to be waiting on a heavyweight process.

[decisions]

Check the output: challenge any one-way door the model classified as reversible only for us. If customers can't easily undo it, treat it as Type 1.

Goal: Learn from past decisions without judging them by their outcomes alone.
Paste in: decision memos or decision-log entries whose review dates have passed, with what actually happened. Use a harness approved for confidential business data.

For each decision:
1. Summarize what was expected at the time, with the stated confidence and the evidence available then.
2. Summarize what happened.
3. Assess the process separately from the outcome: given only what was known then, was the frame right, were the alternatives adequate, was the information relevant and reliable, were trade-offs clear, was the reasoning sound and was there commitment? Name the weakest element.
4. Classify the result: good decision and good outcome, good decision and bad outcome, bad decision and good outcome, or bad decision and bad outcome. Explain.
5. Suggest changes to our question bank or metric definitions based on what we learned.

Use only information that existed at the decision date when judging the process. Mark anything you can't determine.

[memos and outcomes]

Check the output: look hardest at "bad decision, good outcome," because a lucky result hides the flaw and the team is likely to repeat the process. Confirm the model judged the process only on information dated before the decision.

Anti-patterns

Checklist

• Every analysis request names a decision, its options, its date and its decision maker.
• Each decision is classified as a one-way or two-way door, and the evidence bar matches.
• Each question states the answer that would change the decision.
• Questions are checked against the five vanity tests, and totals are rewritten as rates, per-account measures or cohort comparisons.
• Every data question specifies population, time window, metric definition and comparison.
• Every metric used in a decision has a definition card or a governed definition, with a paired guardrail.
• Each question has both quantitative and qualitative evidence, or a stated reason why one is enough.
• Harness answers show their population, filters and record counts, and at least one number is reconciled against a trusted source.
• Someone has spoken to customers behind the key results.
• One-way doors have a decision memo with options, evidence, trade-offs, a pre-mortem, kill criteria and a review date.
• Dissent is recorded, and disagree-and-commit is explicit.
• Decisions are reviewed on their review dates, judging process separately from outcome.
• The decision question bank is updated after each review.