Quick definition: A cluster-randomized experiment assigns groups—such as stores, classrooms, accounts, or geographic areas—to variants rather than assigning individual users independently.
What is a cluster-randomized experiment?
In a cluster-randomized experiment, all eligible units inside a cluster receive the cluster’s assigned condition. A retailer might assign stores to a new staffing policy, a B2B platform might assign customer organizations to a workflow, and a social product might assign communities to a feature. The causal comparison is still randomized, but the number of independently assigned units is the number of clusters, not the number of users recorded in the dataset.
This design is appropriate when individual assignment is impossible, would cause contamination, or does not match the intervention. It is not a way to avoid careful design. Clusters often differ substantially in baseline volume, mix, and outcomes, and users within a cluster tend to resemble one another. Both facts make ordinary user-level A/B calculations misleading.
Design mechanics and assumptions
Choose the cluster boundary, eligibility rule, allocation ratio, and outcome before assignment. Randomize enough clusters to balance important baseline factors, often using blocking or matched allocation. Keep treatment and control clusters concurrent; assigning all treatment regions later confounds treatment with seasonality, campaigns, and operations.
Within-cluster correlation is summarized by the intracluster correlation coefficient, ICC. With average cluster size m, a rough design effect is DE = 1 + (m − 1)ICC. Divide the nominal individual-level information by this design effect to understand the loss of precision. Unequal cluster sizes can worsen it, so plan with cluster-count and size distributions rather than a single optimistic average.
Valid inference must reflect assignment. Analyze cluster-level outcomes, use mixed models, generalized estimating equations, randomization inference, or cluster-robust standard errors appropriate to the number of clusters and outcome. With few clusters, standard robust errors can be unreliable; use small-sample methods or randomization-based analysis. State whether the estimand weights clusters equally or weights people equally.
Application to A/B testing
Cluster randomization is common in marketplace, logistics, education, sales, and networked-product experiments. If a new dispatch rule affects every driver in a depot, individual driver assignment can cause spillovers. If a customer success manager changes behavior for an entire account, assigning users inside the account to different scripts is not credible. Cluster assignment aligns delivery with the intervention.
Log assignment at the cluster and individual level, cluster membership at exposure, eligibility, and outcome maturity. Check allocation and exposure by cluster, not only overall users. A few large clusters can make user totals look balanced while treatment has very few independent assignments. The data-quality checks behind sample ratio mismatch remain useful but must be interpreted at the randomization level.
Worked scenario: store-level checkout training
A chain randomizes 40 stores, 20 per arm, to current or revised checkout training. The outcome is weekly transactions per eligible store over six weeks. It aggregates each store’s outcome using a predeclared window and compares treatment and control using a cluster-aware regression that adjusts for pre-period store volume and blocked region. It does not treat 200,000 transactions as 200,000 independent assignments.
If treatment stores average 3% higher transactions with a wide interval, the decision should reflect 40 randomized stores, baseline imbalance, costs, customer complaints, and whether the interval excludes an important loss. More transaction rows can improve measurement, but they do not create more randomized clusters.
Interpretation
Report cluster count, allocation, cluster-size distribution, outcome aggregation, ICC or design effect where relevant, estimator, weights, effect, and interval. A significant cluster-aware result supports a causal claim for the defined cluster population if randomization and measurement were valid. It does not establish that every store or customer benefits.
Limitations and common mistakes
- Planning from user traffic instead of the number of clusters.
- Ignoring unequal cluster sizes and high ICC.
- Assigning treatment and control in different time periods.
- Using user-level standard errors for cluster assignment.
- Changing cluster membership after assignment without a rule.
- Over-interpreting a few clusters or exploratory subgroups.
Frequently asked questions about cluster-randomized experiments
Why not randomize each user?
Individual randomization is preferable when feasible, but it can cause contamination or fail to match a group-delivered intervention.
How many clusters are enough?
It depends on ICC, cluster sizes, effect size, and analysis. A power calculation must use clusters, not only people.
Can I use a t-test?
A cluster-level comparison can be appropriate in simple designs; choose the method before launch and account for blocks and unequal sizes.
Does more data within each cluster solve low power?
Only partly. Correlated observations have diminishing independent information; additional clusters are usually more valuable.
Summary
Cluster-randomized experiments assign groups to conditions when delivery or interference makes user-level assignment unsuitable. Their credibility depends on concurrent randomization, enough clusters, cluster-aware analysis, and reporting that recognizes clusters—not raw event rows—as the independent experimental units.
Sources
- Campbell and Walters, How to Design, Analyse and Report Cluster Randomised Trials
- Eldridge et al., CONSORT extension for cluster trials
- Cochrane Handbook: Cluster-randomized trials