What is bucket testing?
In a bucket test, an experimentation system maps a unit to a bucket and uses that bucket to decide which experience the unit receives. A bucket can represent a control, treatment, holdout, rollout percentage, or one arm of an A/B/n experiment.
Eppo, VWO, Optimizely, and Crazy Egg describe bucket testing as a synonym or close implementation of A/B/split testing: users are divided into groups, shown different versions, and compared on a defined outcome [1] [2] [3].
The technical emphasis is important. A reliable bucket assignment should be random-like, stable, reproducible, independent across experiments where required, and consistent with the analysis unit.
Bucket testing vs. A/B testing
| Term | Emphasis | Typical audience |
|---|---|---|
| Bucket testing | How units are assigned to experiment groups | Engineers, data scientists, platform teams |
| A/B testing | Comparing two experiences, A and B | Product, CRO, marketing teams |
| Split testing | Splitting traffic or audience between experiences | Marketing and CRO practitioners |
| A/B/n testing | One control plus multiple treatment variants | Experimentation teams with more traffic |
| Multivariate testing | Testing combinations and interactions of factors | Advanced experimentation programs |
These labels are often used interchangeably in content. In an implementation specification, state the allocation, randomization unit, assignment method, persistence rule, and variant mapping explicitly.
How bucket testing works
- Define the experiment objective, hypothesis, population, and primary metric.
- Choose the randomization unit and a stable identifier.
- Combine the identifier with an experiment-specific salt or seed.
- Hash the input or use a tested random assignment service.
- Map the resulting range to buckets and variant weights.
- Persist or deterministically recompute the assignment.
- Log assignment and exposure separately.
- Check allocation, SRM, identity stability, and event completeness.
- Run the test to the planned sample and analyze the result.
Deterministic bucketing with hashing
A common implementation hashes a stable identifier plus an experiment-specific salt, then maps the hash to a numeric range. For example, a hash can produce a number from 0 to 99:
if bucket_number < 50 → Control
otherwise → Treatment
With a 50/50 allocation, values 0–49 map to control and 50–99 to treatment. With a 70/30 allocation, values 0–69 map to control and 70–99 to treatment. For A/B/C, define cumulative bands such as 0–32, 33–65, and 66–99.
Optimizely documents a similar approach using a deterministic hash of user ID and experiment ID mapped into 10,000 buckets [4]. Hashing is not used here for cryptographic security; its useful properties are deterministic output and an approximately uniform distribution.
Persistence, identity, and unit choice
The same unit should normally remain in the same bucket for the duration of the experiment. A user who sees control on Monday and treatment on Tuesday has a contaminated experience, and the analysis may attribute behavior to an experience the user did not consistently receive.
| Randomization unit | Use when | Risk if chosen incorrectly |
|---|---|---|
| User ID | Experience spans sessions and devices | Anonymous-to-authenticated identity changes can reassign |
| Account or organization | Members share workflows or resources | Colleagues see conflicting variants if randomized per user |
| Device ID | Experience is device-specific or user is anonymous | One user sees different variants across devices |
| Session | Only the current session is the treatment unit | Returning behavior becomes cross-exposed |
| Request or page view | Each request is independent and the metric is request-level | Repeated users violate independence for user-level outcomes |
| Geo or cluster | Spillover or shared resources exist | Fewer independent units and lower power |
Optimizely notes that a bucketing ID can decouple assignment from user identification—for example, a shared device can receive one experience while events remain attributed to individual users [5]. Choose deliberately and document the reason.
Traffic allocation and ramps
Bucket ranges define both exposure and variant allocation:
| Design | Control | Treatment / variants | Purpose |
|---|---|---|---|
| Standard A/B | 50% | 50% | Efficient two-arm comparison |
| Risk-first ramp | 90% | 10% | Limit exposure while validating a risky change |
| A/B/C | 33% | 33% / 34% | Compare two alternatives |
| Holdout | 95% | 5% holdout | Long-term incremental effect |
| Multi-arm exploration | 25% | 25% / 25% / 25% | Explore three treatments alongside control |
Increasing the total exposure of an already-running experiment can preserve existing bucket assignments when implemented correctly. Decreasing traffic to zero and later increasing it may reset bucket ranges or reassign users. Optimizely warns that rebucketing can cause sample-ratio mismatch and invalidate results [4]. Treat allocation changes as design changes, not harmless configuration edits.
Bucket quality and SRM
A sample-ratio check compares observed assignments with the intended allocation. If a 50/50 test produces a small 50.2/49.8 difference, that may be ordinary variation. At scale, a statistically significant 55/45 split is a warning.
SRM = test observed counts against expected counts
Typical check = chi-square goodness-of-fit test
Harness uses a chi-square goodness-of-fit check and documents a p-value threshold of 0.001 for flagging meaningful mismatch [6]. The threshold should be standardized by the organization; a p-value alone does not identify the root cause.
| Observed symptom | Possible bucketing issue | Investigation |
|---|---|---|
| One arm is too small | Allocation or targeting rule excludes it | Compare assignment log to exposure log |
| Users switch variants | ID instability, cookie reset, or rebucketing | Trace assignments by user over time |
| Mobile-only imbalance | App routing, startup, or telemetry failure | Run segment-level SRM by platform |
| SRM after flag dependency | Upstream flag changes evaluation frequency | Review flag dependencies and rule order |
| Balanced assignment, unbalanced exposure | One variant fails to render or log | Compare assignment, exposure, and render success |
When SRM is confirmed and unexplained, do not trust the impact result. Fix the assignment or data path and restart with fresh assignments. Harness recommends an A/A test to distinguish a systemic SDK problem from a treatment-specific issue [6].
Worked example: bucket testing a checkout CTA
An ecommerce team wants to test a new checkout CTA label. Its hypothesis is: “If the CTA describes the next step clearly, checkout initiation will increase because users have less uncertainty.”
| Item | Specification |
|---|---|
| Unit | Persistent user ID |
| Salt | checkout-cta-2026-08 |
| Bucket range | 0–99 |
| Control | 0–49: “Continue” |
| Treatment | 50–99: “Review order” |
| Primary metric | Checkout initiation per eligible user |
| Guardrails | Payment errors, completion rate, refund rate, latency |
After the test reaches its planned sample:
Treatment: 10,000 users → 980 checkout starts = 9.8%
Absolute difference = 0.8 percentage points
Relative uplift = (9.8% − 9.0%) / 9.0% = 8.89%
The treatment looks positive on the primary metric. Before rollout, the team checks whether the 80 additional starts complete payment, whether errors rose, whether assignment was balanced, and whether the result is statistically and practically meaningful.
Bucket testing vs. multivariate and bandit testing
| Method | Allocation behavior | What it estimates | Best fit |
|---|---|---|---|
| Bucket / A/B test | Usually fixed allocation | Effect of one or more predefined variants | Ship/iterate decision with interpretable effect |
| Multivariate test | Fixed allocation across combinations | Main and interaction effects | High-traffic factor exploration |
| Multi-armed bandit | Shifts traffic toward apparently strong arms | Reward during learning and exploitation | Time-sensitive optimization |
| Feature rollout | Progressive exposure, not necessarily random | Operational risk and release health | Safe delivery and rollback |
A bucket test is designed for a clean comparison. A bandit optimizes allocation while learning, which can reduce exposure to weak variants but changes the statistical question. A feature flag controls delivery; it becomes an experiment only when randomization, metrics, and analysis are added.
Concurrent experiments and bucket independence
Hash-based assignment makes bucketing scalable, but a platform must also manage overlap. If every experiment uses the same salt or bucket map, assignments can become correlated. If two treatments interact, a user’s experience may depend on both flags even when the team analyzes only one.
Use experiment layers, mutual exclusion groups, independent salts, or explicit factorial designs. Log all active assignments so analysts can identify overlap. Depop Engineering describes how simple modulo-based bucket allocation becomes difficult to manage as concurrent experiments grow, while salted hashing scales more cleanly [7].
Common bucket-testing mistakes
- Using an unstable identifier: users switch buckets across sessions or devices.
- Reusing a salt: different experiments inherit correlated assignments.
- Changing bucket ranges: existing users are rebucketed mid-test.
- Logging only exposure: assignment and render failure remain invisible.
- Assuming 50/50 is perfect: normal random variation is expected; test for meaningful mismatch.
- Ignoring multi-flag interactions: concurrent features contaminate the comparison.
- Using page views for user outcomes: repeated units violate the analysis assumption.
- Calling the largest observed rate a winner: significance, uncertainty, and practical impact still matter.
- Changing allocation when results look good: introduces a new decision rule and potential bias.
- Ignoring privacy: use a stable, appropriate identifier without exposing unnecessary personal data.
Bucket testing checklist
- Objective, hypothesis, population, and primary metric are documented.
- Randomization and analysis units are compatible.
- Identifier is stable and privacy-appropriate.
- Experiment salt is unique and versioned.
- Hash function and bucket mapping are shared across platforms.
- Variant weights sum correctly and are tested before launch.
- Assignment is persistent and rebucketing behavior is documented.
- Assignment, exposure, render, outcome, and error events are logged.
- SRM and segment-level allocation checks are automated.
- Concurrent experiment layers and flag dependencies are reviewed.
- Sample size, MDE, duration, stopping, and multiplicity rules are planned.
- Rollout, rollback, and cleanup plans exist.
FAQ
Is bucket testing the same as A/B testing?
Usually. Bucket testing emphasizes assignment into groups; A/B testing emphasizes comparing variants. In common product and CRO usage, they describe the same controlled experiment.
Why are users called buckets?
Users are placed into logical groups or “buckets,” and each bucket maps to a treatment, control, holdout, or rollout condition.
What is deterministic bucketing?
It is assignment generated as a reproducible function of a stable identifier and experiment salt. The same inputs produce the same bucket without requiring a mutable assignment record.
Can bucket testing use more than two variants?
Yes. A/B/C or A/B/n experiments map separate bucket ranges to each variant. More variants require more traffic and multiple-comparison planning.
What happens if a user changes buckets?
The experience and analysis may be contaminated. Investigate identity, cookies, devices, allocation changes, and flag dependencies. For an important experiment, restart with clean assignment.
How do I know whether buckets are balanced?
Compare observed assignment counts with the planned weights using an automated sample-ratio check, commonly a chi-square goodness-of-fit test. Also check important segments and exposure—not only raw assignment.
Summary
Bucket testing is the infrastructure layer behind many A/B tests. A good bucket is not merely a random label: it is stable, reproducible, appropriately allocated, independent where necessary, and connected to trustworthy exposure and outcome data. When bucketing is designed well, product teams can test changes safely across websites, apps, APIs, and feature flags without losing the causal comparison.
Sources
- Eppo: Bucket Testing
- VWO: Bucket Testing
- Crazy Egg: Bucket Testing
- Optimizely: How bucketing works
- Optimizely: Bucketing IDs
- Harness: Sample ratio check
- Depop Engineering: Hash bucketing
- Eppo: A/B testing vs split testing