Product Analytics and Iteration

Define success before launch. Instrument the questions that test it, read account-level behavior, check experiment validity and turn the evidence into the next iteration.

Why this matters

Launch makes a hypothesis testable when measurement is ready. Write success criteria, implement the events and set the review date before release.

Enterprise analytics has three constraints: limited account volume, workflows spanning multiple people, and productivity goals where less time in the tool can be better. AI adds variable quality and drift. Pair usage with outcomes and correctness checks.

Instrumentation and event design

Start from questions

Tracking every possible event often leaves decision questions unanswered. Design measurement backward from the questions.

Amplitude recommends objectives, critical paths and actions, then events and properties. HEART’s Goals, Signals, Metrics process follows the same logic.

Before you name a single event, write the five to ten questions the launch must answer. “Are new accounts reaching first value within two weeks?” “Do responders accept the agent’s diagnosis?” “Where do admins abandon connector setup?” Each question points to a metric, and each metric to a small set of events. Anything left over is a candidate for deletion.

Naming and properties

Twilio Segment recommends Object Action names such as “Diagnosis Accepted,” with Proper Case events and snake_case properties. Consistency matters most. Amplitude notes that capitalization differences create separate events.

Three rules do most of the work:

• Put variable detail in properties, never in names. One “Order Completed” event with a payment-method property beats a family of near-duplicates. A name that embeds an ID or a customer creates an unbounded list of events nobody can analyze.
• Avoid events so generic they answer nothing. “Page Clicked” tells you activity happened, not what the person was trying to do.
• Separate event properties from user and account properties. Event properties describe one occurrence (which diagnosis, what confidence). User properties describe who did it and persist (role, plan). In B2B, add account and, where relevant, workspace or partner properties.

B2B and platform additions

Enterprise identity spans users, accounts, workspaces and sometimes partners. Include an account identifier on every event so workflows can be analyzed across people. Check coverage before launch.

Two more additions matter for platform and AI products. Log actor type (human, agent or automation) whenever a machine can trigger the same action as a person; otherwise an agent that starts acting looks like a surge in user engagement. And fire events that feed revenue or trust metrics from the server, where a browser can’t block them; client events are fine for interface behavior.

The tracking plan as a contract

The tracking plan defines events, firing conditions, properties, types, questions, metrics and owners. Segment treats it as a living contract for data validation. Engineering implements it, data validates it, and product owns its meaning.

Add a privacy column. The security reviewer will ask what leaves the customer’s environment. Properties should never carry customer content (log lines, ticket text, prompts) or personal data without a documented reason and an approval.

Core analytics: activation, engagement, retention, funnels

Activation

Activation is first value for a user or account. In enterprise products, define a completed action that delivers the purchased outcome. A login rarely qualifies.

In B2B, activation usually belongs to the account and takes several people: the admin connects the data sources, the lead configures the workflow, a user completes the core task once. Track time to activation as well as the rate, because a slow path to first value is where enterprise deals turn into shelfware.

Engagement

Engagement measures frequency, intensity or depth of use over a period. Kerry Rodden, Hilary Hutchinson and Xin Fu recommend reporting it as an average per user, never as a total, because totals rise with headcount. Their Gmail example is the model: the share of active users who visited on five or more days in the past week, which they found strongly predictive of longer-term retention.

The same authors warn that engagement may not be meaningful in an enterprise context where people use the product as part of their job, and suggest focusing on happiness or task success there. For an incident tool, an approval workflow or a compliance product, less time spent can be the goal.

Retention cohorts and curves

Retention measures the share of users or accounts active in a later period. Group customers by starting period into cohorts and compare their retention curves. Ries emphasizes this because acquisition can hide deteriorating retention in aggregate totals.

Sequoia’s data science team describes three shapes worth recognizing on sight:

• Declining to zero. Every cohort eventually leaves. The product hasn’t found a group for whom it’s essential.
• Flattening. The curve drops early, then levels off. A core keeps coming back, and a higher plateau is healthier.
• Smiling. The curve dips and then rises as churned users return.

Match the retention window to natural usage and define active by a meaningful event. For incident-driven work, normalize by opportunity: share of incidents handled can reveal value when weekly usage reflects whether incidents occurred.

In B2B, read retention at two layers. Logo retention and net revenue retention arrive at renewal, so they lag. Product retention, the share of an account’s eligible work that runs through your product cohort by cohort, leads; it can warn you about a renewal two quarters early.

Funnels

A funnel is an ordered set of steps with the conversion rate between each pair. It shows where people drop out; recordings, tickets or a conversation with the people who dropped tell you why.

Enterprise funnels cross people and time. The admin installs a connector, the lead configures rules a week later, an on-call engineer uses the result a week after that. Build these funnels at account level, measure the time between steps as well as the conversion, and attribute each step to the role that owns it.

Product-market fit signals

Fit shows up in what people say about losing a product and in what they do over time. Use both.

The Sean Ellis test

Sean Ellis’s survey asks users how they would feel if they could no longer use the product: very disappointed, somewhat disappointed or not disappointed. As Rahul Vohra reported in First Round Review in 2018, Ellis benchmarked nearly a hundred startups and found that those struggling for traction almost always had fewer than 40% answering “very disappointed.”

Superhuman began at 22% “very disappointed.” Segmenting around those users raised the result to 33% in that segment. After roughly three quarters it reached 58%. Vohra added questions on who benefits, the main benefit and improvements. He focused on enthusiasts and somewhat-disappointed users seeking the same benefit.

In B2B, adapt it three ways. Survey active users inside active accounts and report the respondent count, because a few dozen answers are directional. Segment by persona, because the daily user and the champion experience the product differently. And ask the economic buyer about renewal intent and the business case instead, since the buyer doesn’t use the product.

Retention flattening

A flattening cohort curve shows sustained use. Compare it with the survey. High survey scores with declining retention may reflect enthusiast response bias. Flat retention with low enthusiasm may reflect mandatory use and renewal risk.

Measurement frameworks: AARRR and HEART

AARRR pirate metrics

Dave McClure’s AARRR framework, named for the sound a pirate makes, describes five stages of a customer’s life with a product:

Use the lifecycle order to diagnose growth constraints. Increasing acquisition while activation and retention fail wastes investment.

HEART and Goals, Signals, Metrics

Rodden, Hutchinson and Fu developed HEART at Google and published it at the CHI conference in 2010:

• Happiness: attitudes such as satisfaction, perceived ease of use and likelihood to recommend, tracked with a well-designed recurring survey.
• Engagement: frequency, intensity or depth of use per user over a period.
• Adoption: new users of a product or feature in a period.
• Retention: the share of one period’s users still present in a later one.
• Task success: efficiency (time to complete), effectiveness (share completed) and error rate.

They built HEART to complement what they called PULSE metrics (page views, uptime, latency, seven-day active users and earnings), which run the business but are indirect and easy to misread as measures of experience: more page views can mean people are lost.

For each relevant HEART category, state the Goal, observable Signals, and normalized Metrics. Filter automated traffic. Abandonment and undo may expose failure sooner than success measures. Enterprise workflows often emphasize task success and happiness. Select categories deliberately.

One more lesson from the paper applies to every launch. After a redesign of iGoogle, satisfaction dropped and then recovered as users adjusted. The authors call this change aversion. Read the trend over enough weeks to see past an early dip.

Experimentation

What a controlled experiment is

A controlled experiment randomly assigns units to a control and one or more treatments. With valid assignment and measurement, outcome differences beyond chance support a causal conclusion.

Ron Kohavi, Randal Henne and Dan Sommerfield’s 2007 KDD paper, subtitled “Listen to Your Customers not to the HiPPO” (the highest paid person’s opinion), set out the practice that Kohavi, Diane Tang and Ya Xu expanded in their 2020 book Trustworthy Online Controlled Experiments.

The OEC and guardrails

The Overall Evaluation Criterion (OEC) is a quantitative measure of what the experiment is trying to achieve. Kohavi and his co-authors recommend agreeing on a single metric, possibly a weighted combination of objectives, because it forces the tradeoffs to be made once, before the test. They also warn against short-term OECs: long-term goals such as repeat use belong inside the definition.

Guardrail metrics must not get worse even if the OEC improves: latency, error rates, cost, trust signals, revenue. Kohavi, Tang and Xu treat them as a distinct class of metric. Declare them before the test starts; a treatment that lifts the OEC and breaks a guardrail doesn’t ship.

How experiments go wrong

Sample ratio mismatch (SRM). An unexpected assignment ratio signals a potential validity problem. Microsoft uses a chi-square threshold below p = 0.0005 and reported SRM in about 6% of tests in 2020. Investigate assignment, execution, logging, processing and rollout before trusting results.

Peeking. Stopping a fixed-sample test at its first significant result inflates false positives. Evan Miller’s 2010 example reached 26.1% for a null change. Fix sample size and duration, or use a sequential method designed for repeated checks. Include complete weeks.

What experiments can’t tell you

Experiments estimate which variant performs better and by how much. Qualitative research explains why. Short tests can miss long-term effects. Newness and primacy may require longer runs or new-user analysis. In B2B, shared work can cause interference between variants.

Expect most ideas to fail. Kohavi and Thomke reported that only about 10% to 20% of experiments at Google and Bing produce positive results, and that across Microsoft roughly a third are positive, a third neutral and a third negative. If every readout your team writes is positive, look at how the tests are chosen, run and reported.

When not to A/B test

Skip the controlled experiment when:

• Volume is too low. Work out the minimum detectable effect first. With hundreds of accounts it’s often larger than any plausible result; if so, don’t start.
• Units interfere. Teammates in one account share work, so if one sees the treatment and another the control, behavior leaks across. Randomize by account or not at all.
• The change is contractual. Pricing, terms and entitlements differ by customer only through negotiation.
• Safety is at stake. Don’t randomize whether a risky automated action runs.
• The decision is cheap and reversible. Ship it, watch the guardrails, roll back if needed.

Alternatives, from stronger to weaker evidence

When a clean test isn’t possible, you still want causal evidence. For each option below, ask the same question: what else could explain this change?

CUPED, published by Deng, Xu, Kohavi and Walker in 2013, adjusts for pre-experiment behavior to reduce noise. Bing reported roughly halved variance. Ask whether pre-period adjustment is available and how it changes the minimum detectable effect.

The iteration loop

Build, measure, learn

Plan Ries’s Build-Measure-Learn loop backward: identify the learning, choose its measurement, then build the minimum needed. Reduce time across the whole loop.

Ries’s innovation accounting establishes a baseline, tunes performance through experiments, and tests whether to pivot or persevere. Metrics should be actionable, accessible and auditable. Small interface changes often suit split tests. Larger bets need cohorts and milestones.

Validating the hypothesis after launch

Hold a post-launch review at the pre-agreed date, often 30, 60 or 90 days. Judge each assumption as validated, invalidated or unclear against the original criteria.

Planning the next iteration

The iteration plan turns the review’s open assumptions into the next tests. For each, write the next experiment or measurement, the evidence that would change your mind, an owner, a date and an explicit stop condition, and tie it to a dashboard metric. Without stop conditions, the plan is just a backlog.

How to apply it

This method runs from about four weeks before a launch to the first iteration decision after it. For a feature, the same steps fit into a sprint or two.

Before launch: decide what you’ll believe.

1. Restate the hypothesis and success criteria. Take the solution hypothesis from Chapter 01 and the North Star and input metrics from Chapter 02. Write down, with numbers and a date, what result would mean validated, invalidated and unclear, and get your manager and engineering lead to agree in writing.
2. Write the questions. List the five to ten the launch must answer, including at least one each on activation, task success or quality, retention and a guardrail.
3. Draft the tracking plan. Map each question to a metric, events and properties. Use Object Action names, put the account identifier on every event, log actor type wherever an agent can act, and add a privacy note to every property.
4. Fill in a HEART grid with goals, signals and metrics. Leave rows blank on purpose and say why.
5. Spec the dashboard. North Star at the top, three to five inputs, guardrails, definitions (numerator, denominator, window, unit) and the action each chart should trigger. Cut any chart that doesn’t connect to a decision.
6. Choose how you’ll get causal evidence. If volume supports an A/B test, write the experiment design doc now. If it doesn’t, pick the strongest alternative (usually a staged rollout with a holdout) and write down its assumption.
7. Validate instrumentation before launch. Run the critical paths in a test environment and check every event against the plan. Data lost in the launch window doesn’t come back.

First weeks after launch: watch, don’t conclude.

8. Check data quality daily for the first week: event volumes by account, missing properties, duplicates, and the sample ratio if an experiment is running.
9. Watch guardrails, not the OEC. A guardrail breach justifies a rollback at any time. The OEC waits for the planned duration.
10. Collect the why in parallel. Talk to five to ten users in the rollout, read support tickets and, for AI features, read real traces.

At the review point: decide.

11. Write the readout or post-launch review: validity first, then the primary result with its uncertainty, guardrails, segments (labeled pre-registered or exploratory) and qualitative evidence.
12. Call each assumption validated, invalidated or unclear against the criteria from step 1. “Unclear” is an honest verdict; it means the next iteration must produce better evidence.
13. Run a fit check if the product is new: the Ellis survey by persona, next to the retention cohorts.
14. Write the iteration plan, with a stop condition for every item and a dated pivot-or-persevere checkpoint.
15. Record the decision (ship, iterate or kill) in the decision log, and update the Measure dimension of your Product 360 with the current tracking plan, dashboard spec and readout.

The AI lens

Measuring AI-powered products in production

Offline evaluations need production counterparts. Inputs, models, prompts and retrieval change. Anthropic’s January 2026 agent-evaluation guide combines automated evaluations, monitoring, experiments, feedback, transcript review and human studies.

Behavioral signals of quality. The user’s response to an AI output is your richest quality signal. Instrument each as an event: accepted, edited or rejected (with a reason if you can ask cheaply); the size and type of corrections; reversions of an accepted output, which for agents that act is the most important failure signal you have; regenerations and retries; escalation back to the manual path; and time to complete the task with and without AI on similar cases.

Acceptance measures perceived value. Ziegler and colleagues’ 2022 Copilot study found acceptance tracked perceived productivity better than code persistence. Acceptance alone cannot establish correctness. Pair it with reversions, downstream outcomes or graded samples.

Consistency for agents. Anthropic’s guide distinguishes pass@k (at least one success in k tries) from pass^k (success on all k). An agent that succeeds 75% of the time on one try succeeds three times in a row only about 42% of the time. Users of an agent acting on production systems experience every run, so pass^k is the honest view.

Review production errors. Husain and Shankar recommend free-text expert notes on representative traces, grouped into failure modes and counted. Review at least 100 traces and annotate about 30 yourself before AI proposes categories. A domain expert owns the final quality bar. Derive binary checks from observed failures and reserve weekly review time after launch.

Cost, latency and context. Inference cost per completed task belongs on the dashboard next to quality, because a feature can improve one and destroy margin at the same time; latency belongs among the guardrails. Every AI event should carry the model version, prompt or agent version, number of retrieved sources, tool calls, latency, cost and the human decision that followed. OpenTelemetry maintains an evolving set of semantic conventions for generative AI (spans, metrics, events and MCP calls) that lets engineering traces and product events share identifiers, so you can go from a product metric to the exact traces behind it.

Model changes are product changes. Where volume allows, run a model or prompt swap as an experiment with quality, cost and latency as guardrails. Where it doesn’t, replay past cases offline first, then stage the rollout with a holdout.

Doing the work with AI

Use approved analytics, warehouse, ticketing and CRM connectors to draft tracking plans, detect event drift, examine funnels, check sample ratios and draft readouts. The harness can also cluster rejection reasons.

Own questions, definitions, validity and decisions. A fluent response can hide a wrong question or broken data. Use four checks:

• Ask for definitions with every number. Numerator, denominator, window and unit, checked against the dashboard spec.
• Ask for validity before results. Sample ratio, duration served and data quality come first.
• Ask what else could explain it. Seasonality, a concurrent launch, one large account’s outage, a shift in customer mix.
• Label exploratory findings. A harness can slice data a hundred ways in a minute, which makes chance wins easy to find. Pre-register the segments you care about.

Analytics prompts often carry customer names and usage details. Use only tools approved for that data, and keep traces with customer content inside approved systems.

The Beacon case: did the diagnosis agent work?

The tracking plan. You start with seven questions, among them: do responders accept the diagnosis; how long from page to accepted diagnosis (Dana needs under five minutes); do accepted diagnoses hold up; and what does each resolved incident cost in inference. Ten core events follow, all Object Action: Incident Correlated, Diagnosis Proposed, Diagnosis Accepted, Diagnosis Rejected (with a reason), Diagnosis Edited, Action Proposed, Action Approved, Action Executed, Action Reverted and Incident Resolved. Each carries account_id, partner_id (for Leo’s MSP tenants), actor_type and model_version; actions also carry risk_tier and whether they were allowlisted. Sam adds one rule: no log content or personal data in any property, and the customer-facing audit trail stays separate from product analytics.

HEART and the dashboard. The trio leaves Engagement blank, citing the HEART authors’ own caveat: for Dana, less time in Beacon at 3 a.m. is success. Task success carries the weight (acceptance rate, page to accepted diagnosis, action revert rate). Adoption is accounts with a first accepted diagnosis within 14 days. Retention is the share of each account’s incidents touched by Beacon, by cohort month. Happiness is a quarterly Ellis-style survey. The dashboard puts the North Star on top; inputs are compression ratio, acceptance rate, autonomous resolution rate and weekly active responders; guardrails are revert rate, wrong-root-cause reports, inference cost per resolved incident and 95th percentile diagnosis latency; MTTR and NRR (118%) sit at the bottom as lagging indicators.

The experiment that wasn’t. Design wants to test inline versus collapsed evidence citations. Responders in an account share incidents, so randomization has to be by account, and the harness’s estimate shows that with about 420 accounts the minimum detectable effect is far larger than anyone expects. You choose a staged rollout to half the accounts with a matched holdout, a pre-period adjustment on each account’s acceptance rate, and an offline replay on 300 past incidents first. The move to per-resolved-incident pricing isn’t tested at all, because prices differ by contract and testing them would damage trust; that evidence comes from design partners and a cohort comparison.

An experiment that was. The alert grouping screen in self-serve mid-market tenants does have volume, so the team runs a responder-level 50/50 test. The first readout shows a strong win, and the harness’s draft calls it. The sample ratio check fails. MSP tenants had been bucketed by partner_id, so whole partners landed in one arm. You hold the readout, fix assignment and rerun. The rerun shows a smaller real improvement with every guardrail passing.

The 30-day review. Acceptance is above the 60% target, and on the dashboard the hypothesis looks validated. Then the revert rate at one large customer spikes. It traces to the failure Chapter 31 warns about: the agent proposed a confident, wrong root cause, the responder accepted it and rolled back the wrong service.

The review separates trust from correctness. Add an accepted-diagnosis/action-reverted-within-an-hour signal. An SRE reviews 100 rejected or reverted traces. The largest cluster combines recency bias with stale deploy data from one connector. Define success as acceptance without reversion. Record the verdict as “validated for acceptance, unclear for correctness.”

The fit check. In Beacon’s survey data, 46% of SRE leads like Priya would be very disappointed to lose Beacon, against 31% of on-call engineers like Dana. Dana’s group asks for one thing: faster time to a trustworthy diagnosis. The cohorts agree: accounts where on-call engineers accept diagnoses in their first two weeks flatten, and the rest decline.

The iteration plan. Three items, each with a stop condition. Fix the stale connector and add a freshness check, measured by the reverted-after-accept rate. Speed up diagnosis for Dana, measured against the five-minute goal. For fully autonomous low-risk actions, use Ries’s milestones: the baseline is the autonomous resolution rate at Northlight Live, tuning covers allowlist templates and the approval experience, and the pivot-or-persevere decision is dated for the end of next quarter with a written rule. Marcus gets one page with the North Star, cost per resolved incident and the decision, and the readout goes into the decision log.

Product 360 artifacts

These are Product 360 formats, synthesized for this book from the practices described above. Keep the tracking plan, HEART grid and dashboard spec as living artifacts in the Measure dimension; experiment designs, readouts and post-launch reviews are event artifacts that feed the decision log.

# Tracking plan · [Product or feature] · v[n] · [Date]

**Owner:** [PM]   **Engineering owner:** [Name]   **Data reviewer:** [Name]   **Status:** Draft / Implemented / Validated

## Questions this plan must answer
1. [Question] → Metric: [metric name]
2.
3.

## Conventions
- Event names: Object + past-tense Action, Proper Case (for example, "Report Exported").
- Property names: snake_case.
- Variable detail goes in properties, never in event names.
- Every event carries: account_id, user_id, actor_type (human / agent / automation), [workspace_id or partner_id if relevant].
- Trust- or revenue-critical events fire server-side.

## Events
| Event name | Fires when (exact moment) | Question it answers | Metric(s) it feeds | Event properties (name: type) | Source (server/client) | Owner | Status | Privacy note |
|---|---|---|---|---|---|---|---|---|
| | | | | | | | Proposed / Implemented / Validated | |

## User and account properties
| Property | Level (user/account/workspace/partner) | Type | Allowed values | Source |
|---|---|---|---|---|
| | | | | |

## AI events (AI features only)
Each AI output event also carries: model_version, prompt_or_agent_version, sources_retrieved, tool_calls, latency_ms, cost, and the human decision (accepted / edited / rejected / reverted).

## Privacy and security
- Customer content (logs, tickets, prompts, documents) is never sent as a property unless approved by [reviewer] on [date].
- Personal data in properties: [none / list with approval].

## Validation
- [ ] Every critical path run in test and checked against this plan
- [ ] Every event has account_id
- [ ] No duplicate or near-duplicate events against the existing taxonomy
- [ ] Changes logged below

## Change log
| Date | Change | Reason | By |
|---|---|---|---|

# HEART grid · [Product or feature] · [Date]

**Owner:** [PM]   **Unit of analysis:** user / account / unit of work ([which])

| Category | Goal (what we want for the user) | Signal (behavior or attitude that shows success or failure) | Metric (normalized, with window) | Source (logs / survey) | Logged today? | Target | Owner |
|---|---|---|---|---|---|---|---|
| Happiness | | | | | | | |
| Engagement | | | | | | | |
| Adoption | | | | | | | |
| Retention | | | | | | | |
| Task success | | | | | | | |

## Rows left blank on purpose
| Category | Why we are not measuring it |
|---|---|
| | |

## Failure signals
List the signals that would show the feature failing (abandonment, undo, rejection, reversion, escalation):
-
-

## Notes
- Engagement is reported per user or per account, never as a total.
- Plan to read Happiness over several weeks after a change, to see past change aversion.

# Metrics dashboard spec · [Product] · v[n] · [Date]

**Purpose:** [the decision or review this dashboard serves]   **Audience:** [who looks at it, how often]   **Owner:** [PM]   **Data freshness:** [daily / hourly]

## Layout
1. North Star: [metric] (Chapter 02)
2. Input metrics (3 to 5)
3. Guardrails
4. Lagging indicators

## Metric definitions
| Metric | Tier (North Star / input / guardrail / lagging) | Numerator | Denominator | Window | Unit (user / account / work item) | Segments available | Target or threshold | Action this metric triggers |
|---|---|---|---|---|---|---|---|---|
| | | | | | | | | |

## Guardrails
| Guardrail | Threshold | Who is alerted | What happens on breach |
|---|---|---|---|
| | | | |

## Standard segments
[Persona or role, plan tier, region, account size, partner, cohort month, model_version]

## Charts we decided not to include
| Chart | Why it was cut |
|---|---|
| | |

## Review rhythm
Reviewed in [weekly team review / monthly metric review]. Definitions changed only through [owner], with a dated note.

# Experiment design · [Name] · [Date]

**Owner:** [PM]   **Analyst:** [Name]   **Decision maker:** [Name]   **Status:** Draft / Approved / Running / Read out

## 1. Decision this informs
[What we will do differently depending on the result.]

## 2. Hypothesis
We believe [change] for [segment] will [move OEC by at least X] because [reason]. We will know when [metric and threshold].

## 3. Unit of randomization
User / account / tenant / other: [choice]. Why: [interference risk, how the experience is shared].

## 4. Metrics
- OEC: [one metric or weighted combination, with definition]
- Secondary metrics: [list]
- Guardrails: [metric: must not degrade by more than X]

## 5. Power and duration
- Baseline and variance of the OEC: [values and source]
- Minimum detectable effect: [value]
- Sample size per arm: [n]
- Duration: [whole weeks]
- Variance reduction: [pre-period adjustment used? yes / no]
- Stopping rule: fixed duration / sequential method [name]. No early stopping on the OEC.

## 6. Population and ramp
Included: [who]. Excluded: [who and why]. Ramp: [percent by date].

## 7. Validity checks
- [ ] Sample ratio check before readout
- [ ] Pre-period or A/A comparison of arms
- [ ] Novelty or primacy watch: read results by week
- [ ] Data quality: event volumes by arm

## 8. Decision rule (written before launch)
| Result | Decision |
|---|---|
| OEC up, guardrails pass | |
| OEC up, a guardrail fails | |
| No detectable change | |
| OEC down | |

## 9. Risks and communication
Privacy review: [status]. Customer notice required by contract? [yes / no]. Safety: [any action that must not be randomized].

## If we cannot run a controlled experiment
Method chosen: [staged rollout with holdout / matched comparison / time series / offline replay / design partners]. Assumption we are making: [state it]. What else could explain a change: [list].

# Experiment readout · [Name] · [Date]

**Decision:** [Ship / iterate / kill], recommended by [PM], decided by [Name] on [date].

## Validity
- Sample ratio: configured [x/y], observed [x/y], check [pass / fail]
- Duration served: [dates, whole weeks]
- Data quality issues: [none / list]
- Deviations from the design doc: [none / list]

If any validity check failed, stop here: the result is not interpretable. State the cause and the rerun plan.

## Primary result
OEC: control [value], treatment [value], difference [value] with [95%] interval [low to high].

## Guardrails
| Guardrail | Threshold | Result | Pass / fail |
|---|---|---|---|
| | | | |

## Secondary metrics
| Metric | Difference | Interval | Notes |
|---|---|---|---|

## Segments
| Segment | Pre-registered or exploratory | Result |
|---|---|---|

## What we learned
[What moved, and the qualitative evidence for why. Include what surprised us.]

## Next step
[Action, owner, date]

# Post-launch review · [Feature or product] · [Review point: 30 / 60 / 90 days] · [Date]

**Owner:** [PM]   **Attendees:** [trio, data, stakeholders]

## The hypothesis as written before launch
[Paste it verbatim, with its date.]

## Success criteria set before launch
| Criterion | Target | Date set |
|---|---|---|

## What happened
| Area | Metric | Result | Against target |
|---|---|---|---|
| Activation | | | |
| Task success or quality | | | |
| Retention | | | |
| Guardrails | | | |
| Revenue or business signal | | | |

## What customers said
[Themes from conversations, tickets and, for AI features, trace review. Link the evidence.]

## Verdict by assumption
| Assumption | Validated / invalidated / unclear | Evidence |
|---|---|---|

## Decisions
1.
2.

## What we would instrument differently
[Events or properties we missed, definitions that caused confusion.]

# Iteration plan · [Product or feature] · [Period] · [Date]

**Owner:** [PM]   **Linked review:** [post-launch review or readout]

## Open assumptions and next tests
| Assumption | Current status | Next test or measurement | Evidence that would change the plan | Dashboard metric | Owner | Due | Stop condition |
|---|---|---|---|---|---|---|---|
| | | | | | | | |

## Pivot-or-persevere checkpoint
- Baseline: [metric and value, date]
- Tuning work: [what we will change]
- Decision date: [date]
- Persevere if: [result]
- Pivot or stop if: [result]

## Not doing this iteration
[Items considered and deliberately deferred, with the reason.]

Prompts

Goal: Get a first tracking plan that starts from questions, not events.
Paste in: the PRD or feature spec, the main user flows, the North Star and input metrics, and the current event taxonomy (or a list of existing event names). Unreleased plans are confidential; use an approved tool.

You are helping a PM instrument [feature] in a B2B product before launch.

1. List the 5 to 10 questions this launch must answer. Include activation, task success or quality, retention and at least one guardrail.
2. For each question, name the metric (numerator, denominator, window, unit: user, account or work item).
3. Propose the events and properties each metric needs. Name events as Object plus past-tense Action in Proper Case; name properties in snake_case. Put variable detail in properties, never in names.
4. Every event must carry account_id and actor_type (human, agent, automation). Mark events that should fire server-side.
5. Compare your events with the existing taxonomy. Flag duplicates, near-duplicates and casing differences.
6. Add a privacy note to every property. Flag any property that could carry customer content or personal data.

Output the plan as a table with: event, fires when, question, metric, properties, source, privacy note.

Check the output: every event should trace to a question. Delete events that don't. Walk one critical path with an engineer and confirm the "fires when" moments are precise enough to implement.

Goal: Choose user-centered metrics for one feature and decide which categories to leave out.
Paste in: the feature description, the personas who use it, the product hypothesis and any existing metrics.

Using Rodden, Hutchinson and Fu's HEART framework and their Goals, Signals, Metrics process, build a HEART grid for [feature] in a B2B product.

For each category (Happiness, Engagement, Adoption, Retention, Task success):
- State the goal for the user.
- List candidate signals, including failure signals such as abandonment, undo, rejection or reversion. Say whether each is likely logged today.
- Turn the best signal into a normalized metric (rate, share or per-user or per-account average) with a window.

Then recommend which categories to leave blank and why. Consider whether engagement is meaningful when use is part of the person's job, and whether more time spent would be good or bad here.

Check the output: no metric should be a raw total. If the grid uses Engagement for a workflow where less time is the goal, push back.

Goal: Read cohort retention honestly and find what separates customers who stay.
Paste in (or connect the harness to): cohort retention data by start month, the definition of "active" and the window, and account attributes (segment, size, plan, persona mix, activation date). Use only tools approved for customer data.

Analyze retention for [product] by cohort.

1. State the definition of active, the unit and the window you are using. Tell me if the window seems mismatched to how often customers naturally need the product.
2. Describe the curve shape for each cohort: declining toward zero, flattening (and at what level) or smiling.
3. Compare recent cohorts with older ones. Are newer cohorts better or worse?
4. Find which early behaviors (in the first 14 to 30 days) most separate accounts that stay from accounts that decline. Show the size of each group.
5. List alternative explanations for any pattern (customer mix, seasonality, a large account, a pricing change).

Label these comparisons as associations to test. State the main alternative explanations.

Check the output: confirm the definition of active matches the dashboard spec. Look at the group sizes; a striking difference between two groups of eight accounts is a lead, not a finding.

Goal: Find out whether a controlled experiment is feasible and, if it is, write the design doc.
Paste in: the change, the decision it informs, the candidate OEC with its baseline and variance, traffic or account counts, how users share work within an account, and any contractual or safety constraints.

Act as an experimentation reviewer for [product], a B2B product.

1. Recommend the unit of randomization (user, account, tenant) and explain the interference risk.
2. Estimate the minimum detectable effect for a test of [duration] with this volume, with and without a pre-period adjustment (CUPED-style). Show your assumptions.
3. Say plainly whether the test is worth running. If the detectable effect is larger than any plausible result, or the change is contractual or safety-critical, say so.
4. If it is worth running, draft an experiment design doc: decision, hypothesis, unit, OEC, guardrails with thresholds, sample size, duration in whole weeks, stopping rule (no peeking), validity checks including a sample ratio check, and a decision rule for each outcome including a null.
5. If it isn't, recommend the strongest alternative (account-paired pilot, staged rollout with holdout, comparison group, time series, offline replay, design partners) and state the assumption it relies on.

Check the output: have an analyst confirm the power estimate before you commit. Make sure the decision rule was written before the test, and that a null result has a named action.

Goal: Produce a readout that can't announce an invalid result.
Paste in (or connect the harness to): the experiment design doc and the results: counts per arm, OEC, guardrails, secondary metrics and segments.

Draft an experiment readout for [experiment] using this order:
1. Validity: compute the sample ratio against the configured split and say whether it passes a strict test. Report duration served and any data quality issues. If validity fails, stop and write only the cause and rerun plan.
2. Primary result: the OEC difference with a confidence interval, not only a p-value.
3. Guardrails: each one, pass or fail against its threshold.
4. Segments: label each as pre-registered (from the design doc) or exploratory.
5. Week-by-week view of the OEC, to check for novelty or primacy effects.
6. A recommendation (ship, iterate or kill) that follows the decision rule in the design doc. If the result conflicts with the rule, say so.

Label exploratory segments as hypotheses. Report null or inconclusive results against the pre-agreed decision rule.

Check the output: recompute the sample ratio yourself or with an analyst. Read the recommendation against the pre-written decision rule; if they differ, the readout needs an explicit reason.

Goal: Find the real failure modes of an AI feature in production and turn them into checks.
Paste in: a sample of production traces (inputs, retrieved context, outputs, tool calls and the user's response), plus your own free-text notes on the first 30 or so. Traces often contain customer data; use only an approved tool and remove identifiers where you can.

You are assisting a domain expert with error analysis on [AI feature].

1. Using my notes as the starting vocabulary, propose a taxonomy of failure modes. Keep categories specific (for example, "used stale deploy data") rather than generic ("low quality").
2. Assign each remaining trace to zero or more categories, with a one-line justification quoting the trace.
3. Count traces per failure mode and show which modes most often lead to the user rejecting or reverting the output.
4. Tell me when new traces stop producing new categories.
5. For the top three failure modes, propose a binary pass or fail check that an automated grader or a reviewer could apply.

Use specific failure categories and binary checks grounded in the traces.

Check the output: read 10 traces the model categorized and confirm you agree. The domain expert has the final say on every category and every check. Compare the proposed checks with human judgment on a fresh sample before trusting them.

Goal: Call each assumption honestly and turn the open ones into dated tests.
Paste in: the product hypothesis and success criteria as written before launch, the dashboard results, any experiment readouts, Ellis survey results by persona, retention cohorts and a summary of customer conversations and trace review. Customer names and conversation notes are confidential; use an approved tool.

Draft a post-launch review for [feature] at [30/60/90] days.

1. Restate the pre-launch hypothesis and success criteria verbatim. Don't adjust them.
2. For each assumption, give a verdict (validated, invalidated or unclear) with the evidence. Where a metric met its target but another signal contradicts it, say so and mark it unclear.
3. List what we would instrument differently.
4. Draft an iteration plan: for each open or invalidated assumption, the next test, the evidence that would change our plan, the dashboard metric, an owner placeholder, a date and a stop condition.
5. Propose a pivot-or-persevere checkpoint with a baseline, the tuning work and a written rule for each outcome.

Check the output: compare the success criteria with the originals word for word. If any verdict reads more positive than the evidence, rewrite it yourself. Every iteration item needs a stop condition.

Anti-patterns

Checklist

• The product hypothesis and success criteria, with numbers and a date, were written and agreed before launch.
• A tracking plan starts from questions, uses Object Action names, puts account_id and actor_type on every event, and has a privacy note on every property.
• Instrumentation was validated in a test environment before launch.
• A HEART grid exists for the feature, with blank rows explained.
• The dashboard spec defines every metric’s numerator, denominator, window and unit, and names the action each chart triggers.
• Activation is defined as a completed action that delivers value, measured per account with time to activation.
• Retention is read by cohort, with a window matched to natural usage and a meaningful definition of active.
• Every experiment had its OEC, guardrails, sample size, duration, stopping rule and decision rule written before it started, and every readout reports validity before the result.
• Where volume is too low for an A/B test, the alternative method and its assumption are written down.
• AI features log model version, cost, latency and the human decision, and pair acceptance with a correctness signal.
• Error analysis on at least 100 production traces has been done, with a named domain expert owning the quality bar.
• The post-launch review calls each assumption validated, invalidated or unclear against the original criteria.
• The iteration plan has an owner, a date and a stop condition for every item, plus a dated pivot-or-persevere checkpoint.