The experiment dashboard has fourteen metrics. Checkout conversion is up 3%. Time-on-page is down 12%. Support tickets are flat. Revenue per user is " trending positive" but not significant. Which numbers decide the launch?
Without a clear hierarchy — one primary metric, a small set of guardrails, everything else diagnostic — teams ship on whichever line turns green first. That is how you optimize clicks while eroding margin, or boost sign-ups while churn quietly rises six weeks later.
This guide defines each metric role, how many guardrails to carry, how to handle tradeoffs, and why blurring categories reintroduces the multiple comparisons problems you thought you had under control.
Three roles: primary, guardrail, diagnostic
Primary metric — the one that decides
The primary metric is the single outcome your hypothesis predicts will move. It is pre-registered, powers your sample size calculation, and defines success before the test runs. There should be exactly one primary per experiment (per decision, if you run a family of related tests).
Good primary metrics are sensitive to the change you made, measurable in the test window, and aligned with the business goal of the experiment — not merely easy to track.
Guardrail metrics — protect the business
Guardrails are metrics you are not trying to improve in this test but cannot afford to harm. They set boundaries: "We will not ship if latency p95 worsens by more than 10%" or "Refund rate must not increase significantly."
Guardrails are veto powers, not optimization targets. A winning primary with a breached guardrail means do not ship — or redesign.
Diagnostic metrics — explain the mechanism
Diagnostics help you understand why the primary moved: funnel step completion, scroll depth, feature adoption. They inform the next iteration but do not gate the launch unless promoted to guardrail status upfront (which you should avoid mid-test).
Simple rule: one metric to decide, a few to protect, many to learn — but only the first two categories drive the ship/kill call.
How many guardrails is enough?
More guardrails feel safer but inflate false alarm rates. Each additional hypothesis test at \( \alpha = 0.05 \) adds another chance to block a good launch by noise alone. As a practical default:
- 2–4 guardrails for most product experiments — covering revenue/margin, retention or quality, and one operational constraint (latency, errors, support load).
- Stricter thresholds on high-stakes guardrails (e.g., \( \alpha = 0.01 \) or equivalence bounds) instead of adding many loose ones.
- Pre-register which guardrails can veto a ship — no adding "actually, NPS matters too" after seeing results.
If you monitor ten guardrails uncorrected, expect roughly one spurious "breach" by chance. That is the same family of problems as testing many variants or segments — see our multiple comparisons guide.
Handling tradeoffs honestly
Real experiments often improve the primary while slightly hurting a secondary metric that matters — but was not elevated to guardrail. Example: simplified checkout raises conversion but reduces average order value by 1%, not enough to trip a pre-set guardrail.
Three ways teams handle this well:
- Composite or north-star primaries when tradeoffs are inherent (e.g., revenue per session instead of conversion alone).
- Explicit guardrails on downstream metrics you refuse to sacrifice — set before the test, not after a scary dashboard read.
- Follow-up experiments when diagnostics suggest a Pareto improvement is possible — ship is not always binary.
Anti-pattern: declaring whichever metric improved as "the real primary" post hoc. That is metric shopping — the softer cousin of p-hacking.
A pre-flight checklist
Before code ships, document:
- Primary: one metric, direction of expected change, MDE used for sample size.
- Guardrails: list, direction of harm, threshold or significance rule for veto.
- Diagnostics: exploratory metrics named as non-gating.
- Decision rule: "Ship if primary significant and positive and no guardrail breached" — written in the experiment brief.
At readout, walk the list in that order. Primary first. Guardrails second. Diagnostics last, for storytelling and backlog ideas.
Common mistakes
- Multiple primaries — "conversion and revenue are co-primary" without multiplicity correction doubles your effective alpha.
- Guardrails that are really diagnostics — tracking twelve "guardrails" guarantees noisy vetoes.
- No guardrails on revenue tests — conversion wins that lose money are a classic failure mode.
- Changing roles mid-flight — promoting a diagnostic to guardrail when control looks better on that line.
A shared metrics catalog — with definitions, owners, and default roles — keeps teams aligned. AB-Labz Workbench lets you attach metrics from a catalog to each experiment and mark primary vs guardrail before analysis, so readouts follow the plan instead of the loudest chart.
Summary
Pick one primary metric to optimize and decide the experiment. Add a small, pre-registered set of guardrails to block launches that harm the business. Treat everything else as diagnostic. Respect tradeoffs upfront, apply multiplicity thinking to guardrails, and never retroactively swap which metric "really mattered." Clear roles turn a wall of charts into a decision.
Define metrics before you run
AB-Labz Workbench links experiments to a metrics catalog — primary, guardrail, and diagnostic — so analysis matches the decision plan.