Quick definition: Cluster-level randomization assigns treatment at a group level, so every eligible member of a store, account, region, household, or other cluster receives the group’s assigned experience.
What is cluster-level randomization?
Cluster-level randomization is the assignment procedure used in a cluster-randomized experiment. It differs from user-level randomization because the independent draw occurs once for each cluster. A product team may assign an entire organization to a new admin interface, a delivery depot to routing software, or a city to a marketing policy. Individual users are outcomes within that randomized group, not separately randomized treatments.
Use it when the intervention is delivered to a group, when individual variants would interfere with each other, or when operational systems cannot reliably serve mixed conditions. It should be chosen because it protects a valid comparison, not because it makes implementation easier. The loss of independent sample size requires more planning and a cluster-aware analysis.
How cluster assignment works
Define a stable cluster identifier, determine eligibility before assignment, and randomize clusters with a recorded seed or auditable allocation mechanism. Specify whether allocation is simple, blocked by geography or baseline size, or constrained to improve balance. Every active cluster should have a known chance of each variant unless the protocol explicitly excludes it.
Members of one cluster often share exposure, context, and outcomes. The intracluster correlation coefficient means that 1,000 customers in one store provide less information than 1,000 independently assigned customers. A planning approximation is 1 + (m − 1)ICC for the design effect, where m is cluster size. Unequal cluster sizes require more conservative calculations.
Keep assignment concurrent. A region treated in December cannot be cleanly compared with a control region observed in January when demand and staffing changed. Log cluster assignment, cluster membership, assignment timestamp, allocation probability, exposure, and outcome availability. These are essential diagnostics, not administrative detail.
Choose a cluster boundary that follows the way treatment can spread. A household may be the appropriate cluster for a family subscription offer, while a store is appropriate for a staff-training intervention. An overly narrow boundary leaves spillovers across assigned groups; an overly broad boundary reduces the number of independent units unnecessarily. Document how a user with multiple accounts, a customer served by two stores, or a depot that changes territory will be handled before enrollment begins.
Allocation can be simple randomization when many similar clusters are available, but constrained or blocked randomization is often safer with a small number. A team can form pairs by baseline volume and region, then randomly assign one cluster in each pair to treatment. This prevents an avoidable allocation in which most high-volume clusters land in one arm. The algorithm, baseline data cutoff, fallback for late clusters, and random seed should be reproducible. Do not rerun the assignment until the balance table looks favorable.
Assumptions and analysis requirements
Randomization must occur at the cluster level, outcomes must be measured comparably, and clusters must remain identifiable. Analyze using a cluster-level estimator, mixed model, generalized estimating equation, randomization inference, or appropriately corrected cluster-robust method. User-level standard errors are usually too optimistic because they ignore correlation.
Define the target estimand. An equal-weighted cluster effect answers a different question from a person-weighted effect when large and small clusters differ. Prespecify handling of cluster entry, exit, mergers, and missing outcomes. Blocking on baseline cluster characteristics can improve balance; see blocking.
For a simple continuous outcome, calculate one prespecified outcome per cluster and compare those outcomes according to the randomization scheme. For example, average weekly on-time-delivery rate within each depot over a fixed post-assignment period, then compare depot averages. A regression can adjust for pre-treatment cluster outcomes and blocking variables, but should not condition on variables changed by the intervention. For binary individual outcomes, mixed-effects logistic models or population-average models can be useful, though a team should still present an effect measure that product stakeholders can interpret.
Few clusters are a special risk. Standard large-sample cluster-robust errors may produce overly confident p-values with, for example, ten regions per arm. Randomization inference, permutation tests consistent with the allocation, small-sample corrections, or a cluster-level analysis are often more defensible. Select the approach before results and report the number of clusters that actually contributed a mature outcome. There is no universal numerical cutoff: the appropriate method depends on allocation, cluster-size imbalance, and the estimand.
Cluster randomization also requires positivity. Each cluster type represented in the target population must have a meaningful opportunity to receive each arm under the design. If every rural region is assigned control and every urban region treatment, geography and treatment are inseparable even if the rule was operationally convenient. Similarly, a cluster-level holdout cannot support conclusions about clusters excluded from eligibility.
A/B testing application
Suppose a B2B product tests a new permissions model. Assigning people inside the same company to different permissions can cause support burden and cross-user contamination, so the company is randomized. The primary outcome might be organization-level activation, with user events aggregated under a consistent window. The dashboard should show randomized company counts and sizes by arm, not merely total users.
A sample-ratio check must match assignment. A 50/50 user total can hide 70/30 allocation of organizations if treatment happened to receive more large accounts. Review cluster counts, baseline outcomes, exposure rates, and missingness before interpreting lift; sample ratio mismatch is a relevant diagnostic framework.
Operational dashboards should show both a cluster view and a member view. The cluster view contains assigned counts, allocation by block, exposure status, baseline volume, and missing outcomes. The member view shows whether the intervention was actually delivered to intended people and whether any eligibility path is broken. A user-level funnel alone can conceal a cluster that never activated treatment, while an organization-only dashboard can conceal a treatment-specific event-loss bug.
Plan the decision rule around the unit that can be rolled out. A positive person-weighted average may be driven by a handful of large accounts, whereas a policy that must work reliably for every account type may need cluster-level distributional evidence and guardrails. Keep a primary outcome, define acceptable loss thresholds, and specify whether a limited expansion, a follow-up experiment, or full deployment follows each plausible result. Guidance on primary and guardrail metrics applies equally to cluster tests.
Worked scenario: depot routing software
Twenty-four depots are blocked by region and baseline deliveries, then randomized 1:1 to old or new routing software. The team measures weekly on-time delivery per depot for eight concurrent weeks and adjusts for pre-period delivery rate. It reports the mean depot-level difference and a cluster-aware interval. Thousands of deliveries support precise depot measurement, but the randomized evidence still comes from 24 depots.
Before launch, the team defines the eligible depot population, excludes two depots already using incompatible hardware, and records the baseline eight-week on-time rate and average daily parcel volume. It creates region-by-volume blocks, randomizes within them, and starts both conditions on the same Monday. The primary outcome is each depot’s proportion of eligible parcels delivered on time during weeks two through nine; week one is reserved for implementation stabilization and is retained only as a protocol diagnostic. Secondary outcomes are delivery cost, driver overtime, customer complaints, and scan-event completeness.
At analysis, treatment depots average 92.4% on-time delivery and control depots average 91.1%, a 1.3-point unadjusted difference. A prespecified model including baseline rate and block indicators produces a similar adjusted estimate with an interval of −0.4 to 3.0 points. The team cannot claim a reliable improvement at its decision threshold, even though millions of parcels were observed. It checks that all 24 depots supplied outcome data, that allocation was balanced inside blocks, and that the scan-event definition did not change. A further test with more depots or a staged deployment may be appropriate if 1.3 points would be valuable.
Interpretation
Report the cluster definition, number assigned, allocation method, size distribution, outcome aggregation, estimator, effect, uncertainty, and protocol deviations. An observed average does not prove every member benefits, and a statistically significant result can still be operationally too small to justify rollout.
Limitations and common mistakes
- Counting users rather than clusters in power calculations.
- Using individual-level standard errors.
- Assigning arms in different calendar periods.
- Ignoring changed cluster membership.
- Allowing cross-cluster spillovers without measuring them.
- Claiming user-level personalization evidence from a cluster-level intervention.
- Weighting clusters differently without saying which population the estimate represents.
- Adjusting for post-treatment cluster behavior, such as staff adoption, and thereby changing the causal comparison.
- Using an attractive baseline balance result as permission to discard the randomized allocation or add unplanned exclusions.
Frequently asked questions about cluster-level randomization
Is it the same as a cluster-randomized experiment?
It is the assignment method; the experiment is the full design and analysis built around it.
Can clusters have unequal sizes?
Yes, but plan and analyze for that inequality because it affects precision and weighting.
Does blocking replace cluster-aware analysis?
No. Blocking improves allocation; correlation still requires appropriate inference.
What is the analysis unit?
It depends on the estimand, but inference must recognize the cluster as the randomized unit.
How should I estimate sample size for cluster randomization?
Start with the number of clusters needed for the minimum useful effect, then account for ICC, expected cluster sizes, unequal-size inflation, allocation, baseline adjustment, and outcome availability. An individual-level A/B calculator is not sufficient.
Can a cluster switch variants during a test?
A mid-test switch generally complicates the estimand and may create carryover. Avoid it unless the protocol explicitly uses a crossover or switchback design with suitable washout and time-aware analysis.
What if one cluster has missing outcome data?
Follow the prespecified missing-data rule and investigate why it is missing. Dropping it only because its result is inconvenient can bias the comparison, particularly when missingness differs by arm.
Summary
Cluster-level randomization assigns groups, not individual users, to treatment. It enables valid tests of group-delivered or interference-prone changes when clusters are randomized concurrently, logged carefully, sufficiently numerous, and analyzed as correlated units.