Quick definition: Bot traffic is activity generated by automated software rather than the people or organizations an experiment intends to measure. It can distort eligibility, assignment, exposure, metrics, and sample-size calculations unless it is identified and handled by a prespecified data-quality policy.
What is bot traffic?
Bot traffic includes requests, sessions, events, and transactions generated automatically by crawlers, monitoring agents, scripts, headless browsers, scrapers, fraudulent click systems, QA tools, and application integrations. Some bots are useful and expected: search engines index pages, uptime monitors verify availability, and internal synthetic checks detect regressions. Others are malicious or commercially harmful, such as credential-stuffing programs, ad-fraud networks, and scrapers. For an experiment, the important distinction is not whether an agent is “good” or “bad”; it is whether its activity belongs in the population represented by the metric.
A bot may never render a page yet inflate request counts. It may execute JavaScript and trigger exposure events, appearing as a real visitor. It may create thousands of accounts, repeatedly submit forms, or call an API at a rate no intended customer would. Some automation deliberately imitates ordinary browsing, so no single user-agent string, IP address, or behavioral rule perfectly separates humans from bots.
Bot traffic becomes an experimentation problem when its rate differs by arm, changes during the test, or interacts with the treatment. A new page implementation may block a crawler in one variant but not another. A feature flag endpoint may assign synthetic monitors to treatment. A fraud campaign may arrive disproportionately through a newly targeted channel. In each case, a comparison that looks like product impact may instead be a difference in automation.
Why bots can invalidate an experiment
Experiments estimate effects for a defined analysis population. If the intended population is eligible human customers, automated activity should usually be excluded from the primary outcome and denominator. Including it can dilute a real effect, increase noise, or create a false effect. The direction is not predictable: a crawler may generate many exposures with no conversions, an ad-fraud bot may generate clicks in only one campaign, and a load test may produce errors that make a performance guardrail look worse.
Bot contamination can affect each stage of the experiment lifecycle:
| Stage | Potential distortion | Diagnostic question |
|---|---|---|
| Eligibility | Automated accounts or requests enter the randomization population. | Which rules identify the intended human or customer population? |
| Assignment | Repeated or unstable identifiers create many apparent units. | Can the same agent receive several assignments? |
| Exposure | Scripts fire an exposure event without a meaningful view. | Does logged exposure represent actual delivery to an eligible unit? |
| Outcome | Clicks, forms, purchases, or API calls are generated automatically. | Is the outcome verified as a human-relevant action? |
| Analysis | Filters remove different proportions by arm or after outcome observation. | Was the exclusion rule fixed and audited before reading results? |
These issues are closely related to A/A testing for quality assurance. An A/A test cannot prove that traffic is human, but it can show whether bot filters, identifiers, exposure logs, and metrics produce unexpected arm differences when experiences should be identical. It is also a useful way to test a new classification rule before it affects a business-critical experiment.
Define bot treatment before the test
The protocol should define the target population first, then the bot policy. “Remove bots” is too vague to audit. State which signals are available at assignment time, which are available only later, which source is authoritative, the date a classification becomes final, and whether a category is excluded, separately reported, or retained. A policy might exclude known internal monitors and verified crawlers from eligibility; retain unclassified traffic but flag it; and run primary analysis only on users meeting a pre-treatment human-verification criterion.
Be careful with post-treatment classification. If a variant makes automation more likely to fail, then filtering everyone who fails later could erase a real performance or abuse-resistance effect. Conversely, if the product decision concerns people, automated failures should not dominate a human-conversion metric. The solution is not a universal filter; it is defining the estimand. Report a main human-eligible estimate and, when material, an all-traffic operational metric or a bot-sensitive guardrail separately.
Classification uncertainty should be acknowledged. Preserve raw signals and the classification version. Do not silently rerun a test with today’s bot model if the model did not exist at launch; that changes the analysis population and is a clearly labeled sensitivity analysis.
Signals used to detect automation
Each signal has false positives and false negatives. Corporate users may share an IP; privacy tools may resemble proxies; accessibility tools can automate interactions; power users may work rapidly; sophisticated bots can execute full browser flows. Treat a threshold as a measurement model, not a fact. Estimate how much traffic is classified, how classifications differ by source and arm, and how sensitive the effect estimate is to plausible alternative rules.
Important: Never choose a bot filter because it makes the treatment look better. The filter must be specified from pre-treatment signals or justified as a blinded, reproducible data-quality correction that is applied equally to all arms.
Practical scenario: testing a signup flow
A software company tests a shorter signup form. Its primary metric is completed verified signup per eligible visitor, and its guardrails include abuse reports and API latency. During the test, the treatment appears to increase form submissions by 18%, but verified signups rise by only 2%. The team finds that one acquisition source has a surge in scripted form submissions. The scripts load the treatment page more frequently because a cached redirect sends them there after an implementation change.
The team does not delete low-converting treatment sessions after seeing the discrepancy. It freezes the raw data, verifies the redirect behavior, and checks exposure and assignment by source. The protocol’s existing rule excludes verified internal synthetic tests but not the newly observed scripts. For the primary decision, the team reports the all-eligible result as collected and a prespecified-like sensitivity analysis that applies a blinded behavioral classification built from pre-exposure request patterns. It labels the latter exploratory because the classifier was developed after the test began.
The immediate conclusion is operational: fix the redirect and prevent the automation from entering future experiment eligibility. The product conclusion is more limited: there is insufficient clean evidence that the shorter form increases human verified signup. A follow-up test launches with corrected routing, known-bot exclusion before assignment, and a separate metric for suspicious submission attempts.
Analysis and decision workflow
- Define the intended population and whether the decision concerns humans, authenticated customers, API clients, or all requests.
- Inventory known automated sources: internal QA, load tests, partner integrations, crawlers, security tools, and fraud controls.
- Specify pre-treatment exclusion and labeling rules, their data sources, ownership, versioning, and how late-arriving labels are handled.
- Instrument assignment, exposure, and outcomes with stable identifiers and timestamps; record the filter decision without discarding raw events.
- Monitor counts, exposure rates, conversion, errors, and suspicious-traffic share by arm, channel, platform, and time period.
- Investigate unexpected differences while blinded where possible. Confirm whether they reflect randomization, delivery, classification, or a real treatment interaction.
- Run the primary analysis under the locked population definition, then report clearly separated sensitivity analyses for reasonable alternative classifications.
- Use the result to decide on rollout, remediation, or a clean retest; do not turn a data-quality incident into a claim of product lift.
Bot contamination can also affect planning. If 30% of nominal visitors are automated and generate no meaningful outcome, planning an experiment from raw traffic overstates effective sample size. Sample-size calculations should use the expected count and variability of eligible analysis units, as explained in the sample-size calculation guide, rather than headline request volume.
Assumptions and failure modes
The primary assumption is that the filtering and labeling process does not create differential selection that is mistaken for treatment effect. A rule based wholly on data available before assignment is usually safer. A rule based on later activity needs special scrutiny because treatment may affect the activity used to classify the unit. Keep a no-filter operational view so an implementation change that affects automation is not hidden from engineers.
Another failure mode is identifier fragmentation. Bots often clear cookies, rotate IPs, or create many accounts, but legitimate users can do the same. If the randomization unit is a browser cookie while the outcome is an account event, the same automation can populate multiple apparent units or cross arms. Define identity resolution, deduplication, and the unit of analysis before launch. A broad filter that removes legitimate users can introduce representativeness and fairness problems, especially when proxy use varies by region or community.
Finally, security and privacy constraints may limit which signals may be collected or retained. Do not use sensitive device fingerprints, identity data, or third-party reputation services without a documented legal and privacy basis. The need for clean experiments does not override data-minimization requirements.
Limitations
- No detector perfectly identifies every bot or every human; estimates may remain sensitive to classification error.
- Vendor labels and reputation feeds change over time and may not be reproducible retrospectively.
- Stricter filters can reduce sample size, alter population coverage, and remove legitimate high-value users.
- Automation may be part of the intended product population for an API or partner workflow, so exclusion can answer the wrong question.
- Bot handling cannot compensate for broken exposure logging, assignment, or an ambiguous metric definition.
Common mistakes
- Equating a user-agent string or a data-center IP with definitive bot status.
- Applying an outcome-informed filter only to the arm with an unfavorable result.
- Deleting raw traffic rather than preserving labels and an audit trail.
- Counting automated requests as independent users in a conversion denominator.
- Ignoring internal monitoring and load tests because they are “friendly” bots.
- Using a human-only metric while failing to monitor security, latency, or abuse effects on all traffic.
Frequently asked questions
Should bots always be excluded from an A/B test?
No. Exclude them when the estimand concerns people or customers and the bot is outside that population. Retain or separately analyze automation when it is an intended API client, a security concern, or an operational outcome relevant to the decision.
Can a bot filter be added after a test starts?
It can be added for future data-quality protection, but using it for the current primary result needs care. If it was not prespecified, disclose it and treat the analysis as sensitivity or exploratory unless it is a demonstrably blinded correction to a known logging defect.
What if bot traffic is balanced between variants?
Balanced contamination may mainly reduce precision, but it can still distort denominator meaning and hide an interaction with the treatment. Check balance by time, source, and outcome—not just one aggregate percentage.
Is CAPTCHA completion proof that traffic is human?
No. It is one signal with usability and accessibility costs, and some automated systems can complete or outsource challenges. Do not use it as the sole classifier without evaluating error rates and population impact.
How should we report bot exclusions?
Report counts and proportions excluded by arm, the rule and version, data availability timing, and the effect estimate with and without material exclusions. This lets readers assess whether the conclusion depends on the policy.
Summary
Bot traffic is automated activity that may not belong in an experiment’s target population. It can contaminate every stage from assignment through analysis, and its impact is often arm- or time-specific. A credible response starts with a precise population definition, pre-treatment and auditable classification rules, preserved raw data, arm-level diagnostics, and transparent sensitivity analyses—not with deleting inconvenient sessions after results appear.