Blocking

What is blocking?

In an experiment, a block is a set of units expected to resemble one another on an outcome-relevant characteristic measured before treatment. Instead of assigning every eligible unit from one pooled population, the experiment assigns units separately inside each block. A subscription product might block by country and existing plan; a marketplace test might block by region and merchant size; a business-to-business trial might block accounts by industry and prior usage. Within each block, treatment is still determined at random.

Blocking is sometimes called stratified randomization. Its purpose is not to manufacture a positive result or to make groups identical on every feature. It ensures that the arms are compared among similar units on a small, important set of known baseline variables. Because those variables often explain part of the natural variation in the outcome, the treatment-effect estimate can be more precise. It also prevents a chance imbalance in a small but consequential subgroup from dominating the overall result.

Blocking differs from post-hoc segmentation. The blocks, allocation rule, and analysis plan are chosen before assignment. Looking at results and then creating favorable segments is exploratory analysis, not blocking. It also differs from matching observational data: random assignment remains the source of causal identification. Blocking organizes that assignment; it does not replace it.

When blocking is useful in experimentation

Blocking is most useful when four conditions hold: a baseline attribute is available before randomization, it is measured reliably, it predicts the outcome or materially affects the treatment effect, and each block will contain enough units for the planned allocation. Common candidates include geography, platform, acquisition channel, tenure, baseline conversion behavior, account tier, and calendar period. The value is especially high when the experiment has a limited sample or clusters rather than individuals are randomized.

For example, a pricing-message experiment for enterprise accounts may include a few large customers whose revenue is much larger than the rest. Blocking accounts into prespecified size bands makes it less likely that one arm accidentally receives most of those accounts. In a consumer A/B test with millions of independently randomized users and stable 50/50 assignment, blocking may add little beyond simple randomization. The decision should be based on expected precision and operational risk, not on a rule that every experiment needs strata.

A block must use information fixed before exposure. A user’s number of pages viewed after seeing a new navigation, a merchant’s sales after a fee change, or an account’s post-launch eligibility is not a baseline attribute. Conditioning assignment or analysis on such variables can introduce post-treatment bias. When in doubt, record the timestamp and data source used to create the block.

How blocked randomization works

Suppose there are H blocks. In block h, let n_Th and n_Ch be the treated and control counts, and let ŷ_Th and ŷ_Ch be their mean outcomes. The within-block estimate is:

τ̂_h = ŷ_Th − ŷ_Ch.

For an average effect in the eligible population, combine the block estimates using a population weight w_h, usually the block’s share of the target population:

τ̂ = Σ_h=1^H w_hτ̂_h, where Σ_h=1^H w_h = 1.

The analysis must respect the design. A regression that includes block indicators and treatment-by-block interactions where warranted is often convenient, but the reported estimand and weights should be explicit. Do not simply pool ratios from blocks with different denominators unless that pooling represents the stated estimand. For a binary outcome, a stratified risk difference, a regression with appropriate standard errors, or a prespecified generalized model may be suitable depending on the decision question.

Assumptions and validity conditions

Blocking retains the core assumptions of a randomized experiment. Assignment must be random within every block, each eligible unit must have a known positive chance of receiving each relevant arm, and treatment delivery and outcome measurement must be comparable across arms. If assignment is deterministic within a block—for example, every account in one region receives treatment—then region and treatment are confounded; this is not blocked randomization.

The block definition must be stable enough to implement consistently. If country is derived from a changing IP lookup, plan status comes from a delayed warehouse table, or different services apply different tier boundaries, units may be assigned under the wrong rule. Log the raw baseline value, block label, assignment timestamp, allocation probability, and experiment version. These logs allow an analyst to distinguish a true treatment effect from a randomization or pipeline error.

Blocking does not assume treatment effects are identical across blocks. An overall estimate is interpretable if its weights match the target population, even when effects differ. But a strongly heterogeneous effect may make an overall average a poor rollout guide. Prespecify a small number of decision-relevant interaction analyses and treat broad subgroup mining cautiously. A blocked design improves balance; it does not provide unlimited power for every subgroup.

Practical scenario: testing an onboarding checklist

A software company tests a new onboarding checklist. Its primary metric is activation within 21 days. New users arrive from paid campaigns, organic search, and partner referrals; those sources have very different activation rates. The team blocks on acquisition channel and platform (web or mobile), producing six planned blocks. Within each block, eligible users are assigned 1:1 to the current or new checklist.

Before launch, the team discovers that partner-mobile traffic is too sparse for stable 1:1 assignment every day. It writes a fallback rule: accumulate that block until an assignment batch has an even number of users, then randomize in balanced pairs; if the experiment ends first, include the available users in a design-aware overall analysis without claiming a precise partner-mobile effect. The launch dashboard shows counts, exposure rate, and 21-day outcome maturity by arm and block. It does not show a single blended conversion rate as the only quality check.

At the end, the new checklist lifts activation in most blocks, but paid-mobile users have a near-zero effect. The overall effect is weighted to the eligible population, not equally across the six labels. The product team can roll out broadly while investigating whether the mobile paid funnel needs a different intervention. That is a valid use of heterogeneity discovered through a planned structure, provided the individual block estimates are not overstated as definitive findings.

Analysis and decision workflow

1. State the target population, unit of randomization, primary outcome, and rollout decision.
2. List candidate baseline variables and assess their availability, timing, predictive value, and likely block sizes.
3. Select the smallest set of variables that offers meaningful protection or precision; merge rare categories before launch.
4. Specify allocation ratios, batching behavior, treatment availability, and a rule for undersized or newly appearing blocks.
5. Run an A/A test for QA or assignment dry run, examining arm counts and exposure rates within each block.
6. Lock the analysis: estimand, weights, model or stratified estimator, missing-data handling, and a limited set of interaction checks.
7. Report overall and block-level counts, effects, uncertainty, protocol deviations, and the rule used to make the rollout decision.

Use the result alongside a practical threshold, not only a significance label. A precise average lift may be too small to justify implementation cost. Conversely, a useful but uncertain effect may justify a staged rollout or follow-up test. For sample planning, see how to calculate sample size; the calculation should reflect the planned allocation and the outcome variance left after blocking.

Limitations

• Blocking only controls known, measured baseline variation. It cannot repair unmeasured confounding created by broken randomization.
• Many small blocks increase code complexity, create empty cells, and can reduce rather than improve precision.
• Changing blocks over time can correlate treatment with calendar effects; preserve concurrent assignment in all active blocks.
• For interference, network effects, or cluster-level assignment, blocking alone does not solve spillovers or the need for cluster-aware inference.
• An overall weighted effect can hide material differences that matter for product policy or fairness.

Common mistakes

• Calling any covariate adjustment “blocking” when assignment was actually pooled and unstratified.
• Creating blocks from post-treatment behavior, such as purchasers or exposed sessions.
• Crossing so many attributes that nearly every block contains one user.
• Ignoring blocks in the final analysis and reporting an unweighted average that targets a different population.
• Using global 50/50 assignment as evidence that every block received balanced traffic.
• Interpreting noisy block estimates as proof that every subgroup needs a separate experience.

Frequently asked questions

Is blocking the same as stratification?

In randomized experiments, the terms are often used interchangeably: both mean creating baseline groups and randomizing within them. “Stratification” can have other meanings in sampling or reporting, so a protocol should state exactly how assignment occurs.

Should we block on every variable we know?

No. Use a limited set of variables that are reliably available before assignment and likely to matter. More variables create more cells, smaller samples, and more ways for implementation to fail.

Can we analyze a blocked experiment with a simple overall conversion rate?

Sometimes the pooled comparison is close to a valid estimate, but it may not use the design efficiently and can target the wrong population when allocation or block sizes differ. Use the prespecified, design-aware analysis for the decision.

Does blocking eliminate the need for randomization checks?

No. Check assignment, exposure, and outcome availability within blocks. An overall balance check can miss a defect localized to one platform, region, or eligibility path.

What if a new category appears during the experiment?

Follow a prewritten fallback, such as assigning it under a designated catch-all block or pausing its enrollment. Do not invent a favorable treatment rule after observing its outcomes.

Summary

Blocking is pre-treatment grouping followed by random assignment within each group. Used sparingly and analyzed as designed, it improves balance, can reduce variance, and makes important operational checks visible. It is not a substitute for randomization, clean measurement, or a clearly defined decision; its value comes from protecting a comparison that already has those foundations.

Sources

• Schulz and Grimes, “Generation of Allocation Sequences in Randomised Trials”
• Bugni, Canay, and Shaikh, “Inference Under Covariate-Adaptive Randomization”
• Stark, Randomized Experiments lecture notes

Related terms

• Multivariate Testing
• Multivariate A/B Test
• Factorial Design
• Full Factorial Design
• Fractional Factorial Design