Quick definition: Bucketing is the deterministic process of assigning each eligible experimental unit to a mutually exclusive group that governs which experience, configuration, or analysis condition it receives.
What is bucketing?
Bucketing is the allocation layer of an experimentation system. When a product test says “50% of eligible users see A and 50% see B,” bucketing is the procedure that turns that rule into a stable decision for each user. It generally takes an experiment identifier and an assignment key such as a user or account ID, produces a pseudo-random value, and maps that value into an allocation range. The output is a bucket label that feature-delivery systems can use consistently.
The word is sometimes used casually to mean grouping users for any reason. In A/B testing, it has a stricter role: buckets should be mutually exclusive within an experiment, assignments should be independent of potential outcomes, and each unit should retain its assignment for the period required by the treatment. Those properties help create comparable groups. They do not make a test valid alone; eligibility, exposure, logging, metric definitions, and analysis all remain necessary.
Bucketing commonly powers a two-arm A/B test, but the same mechanism can allocate several variants, holdout groups, phased rollouts, or an A/B/n test. It can operate in a client SDK, an edge service, a feature-flag platform, or a server-side decision engine. The location matters because it affects latency, privacy, what identifiers are available, and whether an assignment can be verified from raw logs.
Bucketing versus randomization, targeting, and segmentation
Randomization is the design principle that assignment should not systematically favor units with different likely outcomes. Bucketing is the implementation mechanism that often realizes randomization. A deterministic hash is usually “random enough” for this purpose when its input and ranges are correctly chosen; the important test is whether assignments are balanced and unrelated to pre-treatment characteristics in the eligible population. A flawed hash input can make deterministic bucketing predictably biased.
Targeting determines who may enter an experiment. For example, an experiment may target active iOS subscribers in Canada. Bucketing happens after that eligibility decision and decides which eligible units receive control or treatment. If targeting conditions differ by bucket, the groups may no longer represent the same population. This can happen accidentally when one variant requires a capability that the other does not, or when an exposure event is used as a late eligibility filter.
Segmentation divides data for analysis, such as comparing effects for new and returning customers. Segments are not necessarily assignment buckets and should not be invented after a result merely to search for a winner. A bucket is assigned before an outcome; a segment can be a pre-treatment attribute, a planned audience, or an exploratory analytic view. Post-treatment segments—for example, “people who clicked the new banner”—are especially risky because treatment can affect membership.
| Concept | Timing | Purpose |
|---|---|---|
| Targeting / eligibility | Before assignment | Defines which units can enter the experiment. |
| Bucketing | At assignment | Maps each eligible unit to an experience group. |
| Randomization check | During or after run | Tests whether observed assignment behaves as intended. |
| Segmentation | Analysis | Describes or estimates effects for defined subpopulations. |
| Feature delivery | After assignment | Serves the configuration associated with the bucket. |
Precise bucketing mechanics
A typical implementation constructs a stable string such as experiment-key:account-id, hashes it, converts part of the result to a uniform integer, and assigns ranges. For an 80/20 control/treatment allocation over 10,000 slots, slots 0–7,999 might map to A and 8,000–9,999 to B. Given the same account ID and experiment key, the account receives the same result. Changing the experiment key intentionally creates a new randomization; changing it accidentally can reassign everyone mid-test.
Choose the assignment key to match the treatment’s scope. If a pricing experience is shared by members of an organization, account-level bucketing keeps them aligned. If a personalized feed is individual, user-level bucketing may be appropriate. Session bucketing may suit an isolated page experiment but permits returning people to see multiple versions. Device-level IDs can reduce cross-device consistency. No key is universally correct: decide whose potential outcome the experiment is meant to measure, then choose a key that avoids unwanted mixing and captures expected exposure.
Allocation is an explicit product and statistical choice. A 50/50 split maximizes information for a simple equal-cost comparison. A 95/5 ramp limits the number of people receiving a riskier new implementation, though it gives the treatment less precision. The desired allocation must be reflected in power calculations and monitoring expectations. For multiple buckets, set each range before launch and decide whether the aim is treatment-versus-control estimation, selection among variants, or operational rollout. More buckets do not make a fixed volume of traffic more informative.
Assignment should be logged with the experiment key, bucket label, allocation version, time, and enough privacy-compliant identifiers to reconcile counts. Separately log exposure when the intended experience is actually available. A decision system may assign an account to B, yet a stale client cache, entitlement failure, or server fallback can deliver A. Without both records, an apparent treatment effect may be a delivery artifact. A periodic A/A test is useful for exercising assignment, delivery, and measurement without a real product difference.
Never derive buckets from a business attribute with outcome meaning. Assigning even account IDs to treatment and odd IDs to control is only safe if ID generation is demonstrably unrelated to acquisition time, region, plan, or behavior. Use a tested hash and verify balance instead.
Worked scenario: an account-level billing flow test
A B2B product wants to test a redesigned billing portal. The experience can be used by several finance contacts in one company, so it selects account ID—not user ID—as the bucketing key. Eligibility is accounts with an upcoming invoice and a supported contract type. The experiment key is billing-portal-2026q3. A stable hash maps 50% of eligible accounts to A, the current portal, and 50% to B, the redesigned portal.
The first request from an eligible account resolves the bucket on the server. The response includes a configuration version, and both the server and client emit assignment and rendered-exposure records. The primary metric is invoices paid by the due date per eligible account. Guardrails include payment failures, time to payment, billing-support tickets, error rate, and page latency. Because billing outcomes can take weeks, the test’s decision window includes the invoice due date plus a documented grace period rather than only the first portal visit.
After launch, a monitoring query shows an apparent 52/48 split in exposures. Assignment logs are near 50/50, but B has fewer client exposures on older browsers. Investigation reveals that an unsupported browser triggers a fallback to the old portal after B assignment. The team has three possible questions: the effect of being assigned to the new delivery path, the effect of actually seeing the redesign among compatible browsers, or the effect of a revised implementation that includes reliable support. It does not choose among them after looking at payment rates. It pauses interpretation, repairs compatibility or adjusts eligibility prospectively, then runs a documented clean test. Bucketing made the defect observable because the assignment and exposure populations were distinguishable.
Bucketing decision workflow
- Name the estimand and decision. Clarify whether the goal is an assignment effect, an exposure effect, a variant selection, or a safe rollout decision.
- Set eligibility before assignment. Ensure all buckets use the same population unless a deliberate design states otherwise.
- Choose the assignment unit and key. Account for repeat visits, households, teams, cross-device use, and interference.
- Define stable ranges. Version the experiment key and allocation rule; make ranges mutually exclusive and collectively complete for eligible traffic.
- Instrument the funnel. Record eligibility, assignment, delivery or exposure, outcomes, and key failure paths.
- QA and monitor. Verify persistence, inspect sample allocation, compare pre-treatment attributes, and watch platform-specific delivery.
- Analyze only after quality checks. Apply the prespecified denominator, metric window, stopping rule, and treatment comparison.
Layered bucketing requires special care. A user can belong to multiple independent experiments, but the platform must decide whether their combined variants can coexist. Experiments that change the same page or metric can interact. A global namespace or mutual-exclusion layer can prevent incompatible tests from overlapping, while deliberate factorial designs can estimate interactions when enough traffic exists. Simply hashing each experiment independently does not prove the experiments are independent in customer experience.
Limitations and common failures
Bucketing provides an assignment rule, not immunity from causal problems. Shared inventory, network effects, team communication, and capacity constraints can make one group’s treatment alter another group’s outcomes. Missing identifiers can exclude a nonrandom subset. A user who clears cookies may be assigned again. In a client-side implementation, flicker or delayed configuration can change the experience before the recorded exposure. Each issue requires design or engineering work beyond choosing a hash function.
- Assignment instability: changing a salt, key, ID source, or SDK behavior moves repeat units between groups.
- Sample-ratio mismatch: allocation ranges, eligibility order, bot filtering, logging, or joins yield an unexplained observed split.
- Cross-experiment collisions: two experiments alter the same surface and make the delivered experience ambiguous.
- Inappropriate allocation: a tiny treatment range cannot answer a decision-relevant question within the planned runtime.
- Identity fragmentation: anonymous and logged-in states assign the same person inconsistently.
- Exposure mismeasurement: the log says B was assigned even though a fallback served A, or the event fires only after treatment-affected behavior.
Operational controls reduce these risks: immutable configuration history, code review for assignment changes, runtime dashboards that reconcile the funnel, raw-event audits, and clear ownership for anomalies. Do not “repair” a mismatch by reweighting or excluding records until its cause is understood. Those techniques may be appropriate in a documented analysis, but they cannot turn an unknown delivery failure into a trustworthy randomized comparison.
Frequently asked questions
Is deterministic bucketing truly random?
It is deterministic for a given key but can produce randomized-looking assignment across a population when the hash is well designed and the key is not correlated with outcomes. Validate balance empirically.
Should I bucket by user or account?
Use the unit that receives and can influence the treatment. Shared experiences usually need account or organization assignment; individual experiences may use user assignment.
Can I change traffic allocation mid-experiment?
Only with a preplanned procedure or a documented redesign. Changing ranges affects precision and may complicate interpretation, especially if it responds to early outcomes.
What is a sample-ratio mismatch?
It is an observed bucket split that materially differs from the intended allocation beyond expected chance. It is a diagnostic signal, not a result to ignore.
Can two experiments bucket the same user at once?
Yes, if their experiences can safely coexist and their interaction is understood. Use mutual exclusion for conflicting tests or design a factorial experiment when interactions are the question.
Summary
Bucketing maps eligible units to persistent experiment groups and is the practical foundation of controlled delivery. Reliable bucketing uses an appropriate stable key, explicit allocation ranges, consistent eligibility, verified exposure, and monitoring for assignment and identity failures. It supports randomization but does not replace thoughtful design, measurement quality, or safeguards against interference and overlapping experiments.
Sources
- Kohavi, Tang, and Xu, Trustworthy Online Controlled Experiments.
- Fabijan et al., “Experimentation Growth: Evolving Trustworthy A/B Testing Capabilities”.
- AB Labz: Sample-ratio mismatch in A/B testing.