Quick definition: A binary metric records whether a defined event happened for each analysis unit: it has two possible values, normally 1 for the event and 0 for no event. Its average is an event rate.
What is a binary metric?
A binary metric translates an outcome into a yes-or-no measurement. Examples include purchase within seven days, account activation within 24 hours, subscription cancellation during a billing cycle, delivery delay, and successful completion of a task. One unit receives one value. A user who purchases is coded 1; an eligible user who does not is coded 0.
Binary does not mean simplistic. The event definition encodes product judgment: which action counts, whose action counts, how long the team waits, and whether a reversal such as refund changes the result. “Converted” is not yet a metric specification. “Completed a paid order within seven days of eligibility, net of canceled orders, per assigned user” is much closer to one.
Binary metrics are widely used because their rates are intuitive and often align with decisions. However, they discard information beyond the threshold. “At least one order” does not distinguish a $5 order from a $500 order or one order from five. Pair a binary metric with a continuous or count metric when those distinctions affect value.
Formula and interpretation
Let Yi equal 1 if unit i meets the event rule and 0 otherwise. Across n eligible units:
binary metric rate = ΣYi / n
Because zeros remain in the denominator, the rate estimates the probability that an eligible unit experiences the event in the stated window. For a control and treatment comparison, use:
absolute difference = mean(Y treatment) − mean(Y control)
For an undesirable event such as a payment failure, a negative difference is an improvement. State directionality in the analysis plan so a dashboard does not encourage accidental sign reversal. Report both the observed counts and denominators; “conversion rose to 4.5%” is incomplete without knowing from what population and baseline.
Designing a reliable binary metric
| Metric component | Example decision | Why it matters |
|---|---|---|
| Unit | One account, not one page view | Must align with randomization and decision. |
| Eligibility | New trial accounts created on supported plans | Sets the denominator before treatment. |
| Event | Completed first project with a teammate | Avoids optimizing a shallow click. |
| Window | Within 14 days of assignment | Controls maturation and comparability. |
| Data source | Server-confirmed event | Reduces missing or duplicate client events. |
Define eligibility before the variant changes behavior. If a new onboarding path causes fewer users to reach screen three, measuring “completion among screen-three viewers” selects a treatment-affected subgroup. For the total onboarding effect, include all eligible assigned accounts. This is an intent-to-treat approach and avoids a common post-treatment selection error.
Use stable identity rules. If a person can act on several devices, user- or account-level consolidation may be necessary. If the treatment is displayed per session, session-level measurement may be suitable, but repeated sessions from one person can create dependency. The unit of analysis is a design choice, not an implementation detail.
Binary metrics in A/B testing
In an A/B test, a binary metric commonly follows a Bernoulli outcome model. Treatment and control rates are compared with a confidence interval and a hypothesis test or regression suited to proportions. Randomization, not the binary coding itself, supports the causal interpretation. Reliable assignment, exposure, eligibility, and event logging are all required.
Choose one primary metric before launch. Supporting binary metrics can explain mechanism: a variant may increase activated accounts because it reduces setup errors, for example. Guardrails prevent a local gain from masking harm: a change that raises completed orders could also raise refund, fraud, or support-contact rates. See how primary and guardrail metrics work.
Do not stop a fixed-horizon experiment as soon as a binary rate looks significant. Repeated unplanned looks inflate false-positive risk. Follow a planned sample or a valid sequential approach; test duration must include both the required information and the metric’s attribution window.
Worked example: activation metric
A collaboration product tests a guided invitation prompt. The unit is a new workspace, assigned at creation. The event is “at least two members complete their first shared project within 14 days.” Every eligible workspace receives a 1 or 0 after its full window ends.
Control has 1,440 activated workspaces among 24,000, a 6.0% rate. Treatment has 1,632 among 24,000, or 6.8%. The treatment-control difference is +0.8 percentage points; the relative lift is 13.3%. The team reports an interval around that difference, not just the uplift.
It also monitors invitation delivery failures, product latency, and 30-day retention. The prompt could create fast two-member projects that do not persist. It validates the server event and confirms that every assigned workspace, including those never opening the invitation screen, remains in the denominator. That makes the result about offering the prompt, which is the rollout decision.
Assumptions and edge cases
A binary cutoff can be arbitrary. “Activated within seven days” may classify a user who activates on day eight as a failure even though the product effect is positive. Choose the window based on product logic and examine time-to-event analysis when timing itself matters. Do not change the cutoff after observing which one favors the variant.
Rare safety events may have too few observations for a short experiment. Zero observed incidents does not prove zero risk. Use appropriate intervals, longer monitoring, historical evidence, or a staged rollout. For frequent events near 100%, focus on absolute headroom; a large relative claim can correspond to negligible practical change.
Do not redefine an event merely to improve the rate. Adding automatic events, weakening completion rules, or counting duplicates can create an apparent win without increasing customer value.
Common mistakes
- Using a post-treatment denominator: it can bias the treatment comparison.
- Counting one unit multiple times: define one outcome per analysis unit.
- Ignoring delayed outcomes: read results only after the declared window matures.
- Reporting percentages without counts: small samples can produce unstable rates.
- Optimizing one binary proxy: include value and harm metrics where relevant.
Frequently asked questions
Is a binary metric the same as a conversion rate?
A conversion rate is a common binary metric: the average of a conversion indicator. Binary metrics can also measure failures, retention, opt-out, or any two-outcome event.
Should I use users or sessions as the unit?
Use the unit matching assignment and the product question. If users are randomized, user-level analysis is often clearest; repeated-session data may require adjusted inference.
Can a binary metric have more than one event?
It can define “at least one” qualifying event. If the number of events matters, use a count metric as a companion or primary outcome.
How should I report lift?
Report event counts, denominators, rates, absolute percentage-point difference, uncertainty interval, and relative lift when it adds clarity.
Why include non-converters?
They are part of the eligible population exposed to the decision. Removing them changes the question and can introduce bias.
Summary
A binary metric assigns one of two values to each clearly defined unit, and its mean is an event rate. It is central to conversion and reliability experiments when eligibility, event rule, window, and identity are explicit. In A/B testing, retain the full assigned population, quantify uncertainty, and complement the rate with metrics that capture value and harm.
Sources
- OpenIntro Statistics
- NIST/SEMATECH e-Handbook: Type I and Type II errors
- Kohavi, Tang, and Xu: Trustworthy Online Controlled Experiments