Quick definition: Aggregation is the process of combining detailed records—events, sessions, orders, or users—into a defined analysis unit and summary metric such as a count, mean, total, or rate.
What is aggregation?
Aggregation turns raw observations into a number that can support a decision. A product event stream may contain page views, clicks, purchases, refunds, and timestamps. An analyst can aggregate those records to sessions, users, accounts, stores, days, or experimental clusters; then calculate a conversion rate, average revenue, retention rate, or total. The chosen level is not a technical afterthought. It determines whose behavior is weighted, which population is represented, and the question the metric actually answers.
For example, “checkout conversion” could mean orders divided by sessions, users with an order divided by eligible users, or orders divided by checkout exposures. Each is a defensible ratio in the right context, but they are different estimands. A change that encourages repeat sessions may raise a session-weighted rate while leaving the proportion of users who purchase unchanged. Clear metric definitions make these choices visible before results arrive.
Choose the analysis unit first
The analysis unit is the entity that contributes one observation to an analysis: often a user, account, device, store, household, or geographic area. It should be chosen from the decision question and aligned with the unit of analysis and randomization design. If a feature is assigned by user and the business question is “does this improve a customer’s likelihood of purchase?”, a user-level outcome usually gives each assigned person equal weight.
Events are rarely independent when they belong to the same person. A user may open ten sessions, and those ten rows share preferences, context, and treatment exposure. Treating every session as an independent replicate can make an effect look more precise than it is and can overweight highly active users. If assignment is by store or region, analyzing individual transactions without accounting for clustering can create the same false certainty on a larger scale.
| Aggregation level | Typical question | Weighting consequence |
|---|---|---|
| Event | Did the event itself succeed? | People with more events contribute more rows. |
| Session | Did a visit lead to a desired action? | Frequent visitors receive more weight. |
| User | Did an eligible person convert or generate value? | Each eligible user can contribute equally. |
| Account or household | Did the customer organization change behavior? | Multiple users are combined into one customer outcome. |
| Day, store, or region | What happened in a clustered or geo experiment? | Variation between clusters, not rows, drives uncertainty. |
There is no universally correct level. Session-level conversion is appropriate when the product decision concerns visits, such as search-result quality. User-level conversion is often better for a user-randomized onboarding test. The requirement is to say which population and weight are intended, then use a method whose uncertainty calculation respects that choice.
Numerators, denominators, and rate construction
A ratio metric has a numerator, a denominator, and an eligibility rule. Write all three. For a user conversion rate:
\(\text{Conversion rate} = \frac{\#\{\text{eligible users with at least one qualifying conversion in window}\}}{\#\{\text{eligible users}\}}\)
The numerator is not simply “all purchase events.” It may require a successful payment, deduplicated order ID, first exposure before conversion, a 14-day observation window, and a rule for cancellations. The denominator may be assigned users, exposed users, qualified visitors, or users who completed a prerequisite. Changing it changes the meaning of the rate.
Aggregate counts before calculating an overall rate. Given groups \(i\), compute \( \sum_i n_i / \sum_i d_i \), not the unweighted average of group rates \( \frac{1}{k}\sum_i (n_i/d_i) \). The latter gives a tiny group the same influence as a large group. The same warning applies to averaging daily conversion rates when daily traffic differs.
Ratios also demand compatible levels. A user-level numerator divided by sessions is neither a user conversion rate nor a session conversion rate. A common symptom is a percentage that looks plausible but cannot be reconciled to raw records. Maintain a lineage table that specifies source event, entity key, deduplication rule, filters, time zone, window, numerator, and denominator.
Why ratios are easy to misinterpret
Rates can change because their numerator changes, their denominator changes, or both. A treatment that makes checkout slower could reduce the number of users who reach checkout; conversion among those who do reach checkout might rise, even while purchases per assigned user fall. This is a post-treatment selection problem: conditioning on an action affected by treatment can distort the comparison. Prefer a denominator defined before treatment for a primary causal metric, such as all eligible assigned users, and use downstream rates as diagnostic metrics.
Also check whether a group average represents the intended weight. Average revenue per user is total revenue divided by users, while average order value is total revenue divided by orders. An unweighted average of daily or segment rates can give small groups too much influence.
Aggregation in A/B tests
In an A/B test, randomization creates comparable groups at the assignment unit. Preserve that connection. Join outcomes to assignment using a stable identity, restrict analysis to the pre-specified population, aggregate each unit’s outcomes over the same observation window, and then compare arm-level summaries. This is particularly important for exposure logging: analyze the assigned population for an intention-to-treat-style question, or define an exposure-based population consistently and explain what effect it estimates.
Suppose users are randomized but revenue arrives as order rows. A robust user-level dataset contains one row per assigned user: assignment arm, eligibility status, exposure time, number of qualifying orders, total net revenue, and a zero for users with no order. That zero is information. Dropping non-buyers changes average revenue per assigned user into average revenue among buyers, a selection that can make a treatment look profitable simply by reducing the buyer population.
For cluster-randomized designs, aggregate or model at the cluster level and account for within-cluster correlation. For switchback tests, the relevant unit may be a location-time block rather than an individual event. For repeated measures, decide whether the outcome is cumulative per user, an average per active day, or a time series, and use the corresponding estimator. The test’s hypothesis should name the outcome and unit, while sample-size planning must match the variance at that level.
Practical example: a search-results experiment
A marketplace tests a new ranking algorithm. Users are randomized at the user level. The product team initially reports “purchase conversion” as orders divided by search sessions. Variant B is 3% higher. That metric answers whether a search session produces an order, but the new ranking also causes users to perform fewer searches before deciding. The ratio rises partly because its denominator falls.
The analyst rebuilds the dataset at user level. For every eligible assigned user, they sum qualifying orders and net revenue for seven days, retaining zeros. Purchasers per assigned user increase by 0.2 percentage points, orders per assigned user are unchanged, and revenue per assigned user is slightly lower because the variant shifts orders toward lower-priced items. The original session rate was not false; it was incomplete for the decision. The revised aggregation shows that a rollout decision should consider revenue and user outcomes, not only efficiency per session.
A decision workflow for aggregation
- Name the decision and estimand. State whether success means value per assigned user, purchase per visit, account retention, or another outcome.
- Set the unit and key. Identify the entity that should receive one contribution and how events map to it. Document identity-resolution limits.
- Specify the formula. Define numerator, denominator, eligibility, attribution, time window, deduplication, and zero-value handling before analysis.
- Build a unit-level audit table. Retain enough detail to trace a summary back to source records and verify row counts, joins, and missingness.
- Reconcile alternatives. Compare totals, unique entities, and a small sample of manual traces. Explain deliberate differences between user-, session-, and event-level metrics.
- Interpret with uncertainty and guardrails. Evaluate effect size and confidence intervals, plus denominator movement and complementary metrics. See primary and guardrail metrics for a decision framework.
Data quality and interpretation issues
Aggregation can conceal data defects. Duplicate events inflate totals, late events create apparent treatment differences if one arm is ingested sooner, and missing IDs prevent outcomes from joining to assignments. Time-zone boundaries can place a conversion in a different day than exposure. Refunds and cancellations need a consistent net-revenue rule. Bot traffic and internal users can produce abnormal event frequency that dominates event-level summaries. Monitor these rates by arm as well as overall.
Inspect arm counts, eligibility, exposure, missing outcomes, and numerator and denominator separately. A confidence interval quantifies random error, but cannot repair biased selection, broken tracking, or a mismatched unit.
Common aggregation mistakes
- Mixing units. Combining user numerators with session denominators creates a metric with no clear interpretation.
- Averaging percentages blindly. Recompute rates from summed compatible counts unless equal group weighting is explicitly intended.
- Excluding zeros. Removing users without activity changes the population and can introduce treatment-driven selection.
- Ignoring clustering. Many rows from the same user, store, or region do not provide the same information as independent rows.
- Hiding denominator changes. Always report numerator and denominator movement alongside a rate lift.
Frequently asked questions
Should I aggregate before or after joining experiment assignment?
Join to assignment early enough to preserve the randomized population and audit missing matches. Aggregate outcomes to the planned analysis unit after applying the documented attribution rules.
Is a user-level metric always best for an A/B test?
No. It is often appropriate for user randomization and user decisions, but a session, account, store, or time-block metric may better match the intervention and decision. The design and analysis must align.
Why not average daily conversion rates?
Days usually have different traffic volumes. An unweighted average gives a low-traffic day the same impact as a high-traffic day. Sum eligible conversions and eligible units first unless each day intentionally has equal weight.
What is the difference between conversion rate and conversions per user?
Conversion rate usually counts whether each user converted at least once. Conversions per user counts all qualifying conversions divided by users. Repeat behavior makes them different metrics.
How do I handle users with no events?
If they are in the pre-defined eligible population, retain them with zero outcomes. Exclude them only when the metric definition explicitly targets an upstream population and that rule applies identically to both arms.
Summary
Aggregation defines what an experiment measures. Choose an analysis unit that matches randomization and the decision, write compatible numerator and denominator rules, retain zero outcomes when they belong in the population, and inspect composition and data quality before interpreting a lift. A well-labeled rate can be useful; an ambiguous aggregate can lead to a confident but wrong rollout.
Sources
- Microsoft Research: Trustworthy Online Controlled Experiments
- NIST/SEMATECH e-Handbook: Measurement Process Characterization
- Encyclopaedia Britannica: Simpson’s Paradox