Quick definition: Audience eligibility is the explicit, evaluable set of conditions that determines whether a unit could enter an experiment or receive a feature decision. It establishes the denominator before assignment, delivery, and outcome measurement.
What is audience eligibility?
Eligibility answers a narrow but essential question: “Which users, accounts, sessions, or requests were able to be randomized?” Conditions may include a supported app version, account state, country, consent status, feature entitlement, technical capability, or a prespecified data-quality requirement. A unit can be eligible without being assigned, assigned without receiving a rendered treatment, or exposed without producing an outcome.
That separation distinguishes eligibility from audience targeting, assignment, and exposure. Targeting describes the operational selection or routing of an audience. Assignment selects an arm among eligible units. Exposure indicates that a unit received a defined treatment. Keeping these events separate prevents the denominator from changing invisibly as delivery conditions change.
Eligibility is neither a convenience filter nor a result-driven exclusion list. It is part of the experiment specification and defines the population to which an estimate applies. A test of a feature available only in app version 8.4 and above can make a claim about eligible users of those versions; it cannot by itself estimate what would happen for unsupported versions.
Designing eligibility rules
Begin with necessity. Include a condition only when it protects users, makes a feature functional, satisfies a policy requirement, or defines the product question. Use data known before the intervention when possible. Good rules are precise: “accounts with an active paid subscription at evaluation time” is clearer than “valuable users.” Define the data source, evaluation moment, missing-data behavior, and owner for each condition.
Choose the correct unit. An account-wide billing feature should generally determine eligibility and assignment at account level; a user-level notification can use user-level rules. Identity resolution matters: a visitor may later log in, a user may belong to several accounts, and a device identifier may be reset. State how those transitions affect eligibility and prevent a unit from re-entering unexpectedly.
Implement rules as versioned, testable configurations or code. Evaluate inclusion and exclusion in a documented order, produce an eligibility result and reason code, and retain the rule version with experiment events. Boundary tests should cover missing consent, stale app versions, internal accounts, malformed attributes, time-zone changes, and a unit that becomes eligible during a session.
Why eligibility matters in experimentation
Eligibility defines an intent-to-treat denominator: the set of units that could have received an assigned experience. That denominator often answers the practical product question, because it includes delivery failures and noncompliance as part of operating the feature. A per-exposure analysis can answer a different question, but must not be presented as though it used the same population.
Do not condition primary analysis on an event influenced by treatment. For example, excluding users who did not open a treatment-specific modal selects on post-assignment behavior. Similarly, changing a rule after an early outcome looks favorable can make the reported sample incomparable with the planned one. If circumstances require a rule change, timestamp it, preserve the old version, and assess separate phases or a revised protocol.
Check eligibility counts before evaluating a metric. Compare assignment proportions within the eligible population and investigate unexpected differences by app version, geography, or account type. A sudden change can reveal a broken attribute feed or caching issue. Use a sample ratio mismatch check where its assumptions fit, but also inspect the upstream eligibility funnel rather than treating a statistical alert as a complete diagnosis.
Concrete engineering and product scenario
A mobile team tests a shorter onboarding flow. The experiment is relevant only to first-time users running app version 8.4 or newer who have provided the required analytics consent. It excludes employees and users already marked as having completed onboarding. Eligibility is evaluated when the app opens the onboarding entry point; user assignment is persisted so an eligible user does not alternate flows after a relaunch.
The client logs an eligibility decision with a reason code, assignment, flow render, first screen visible, completion, and fallback if the remote configuration cannot load. The dashboard reports all stages by arm and app version. When consent status changes, the app stops measurement according to policy rather than silently treating the user as a normal missing event. The analysis describes its results for consented, first-time users on supported versions, with a separate operational review of those unable to receive the configuration.
Monitoring and diagnostics
Monitor eligible volume, reason-code distribution, missing or stale attribute rates, assignment split after eligibility, persistence failures, and transitions into and out of eligibility. Compare expected and actual counts after releases, app-version changes, or new markets. Retain compact diagnostic metadata such as rule version and evaluation timestamp; do not retain extra personal attributes solely because they might someday be useful.
Investigations should trace a small sample from source attributes through rule evaluation, assignment, and delivery. Look for time-zone errors, inconsistent backend and client definitions, delayed profile updates, and cached entitlements. Treat consent and privacy rules as product constraints, not as optional analytics filters. When a unit cannot legally be measured, document that limitation rather than attempting to reconstruct it from unrelated identifiers.
Trade-offs and limitations
Tight eligibility can protect users and create a more coherent hypothesis, but it reduces sample size, statistical power, and external validity. Broad eligibility increases reach but may include devices or states where delivery is unreliable. Real-time evaluation improves freshness but may add latency; cached evaluation improves speed but can use stale facts. The appropriate balance follows the product risk and the decision the experiment is meant to inform.
Eligibility does not guarantee exposure, data quality, or comparable treatment delivery. It is the first gate in a measurement chain. A clear rule cannot compensate for missing exposure logging, poor randomization, or an outcome metric that does not represent user value.
Common failures
- Exposure used as eligibility: delivery failures disappear from an intent-to-treat denominator.
- Post-treatment filtering: a variant can alter who remains in the analyzed sample.
- Undocumented defaults: missing attributes silently widen or narrow the audience.
- Unversioned rules: analysts cannot reconstruct which population existed at launch.
- Inconsistent units: account features assigned by user can produce conflicting experiences.
FAQ
When should eligibility be evaluated?
At a prespecified decision point using the information the product can legitimately use then. Record the time because attributes can change.
Can an eligible user be excluded from analysis?
Possibly for a clearly defined analysis, but explain the criterion and its consequences. Do not remove users based on treatment-influenced behavior from the primary comparison.
Do consent rules belong in eligibility?
They may determine whether an experience or measurement is permitted. Model the product and measurement consequences explicitly rather than assuming consent is only an analytics detail.
What if eligibility changes during a test?
Version and date the change, quantify affected traffic, and decide whether the original and revised populations should be analyzed separately.
Summary
Audience eligibility defines who could enter an experiment before assignment and exposure. Make the rule necessary, pre-treatment, versioned, observable, and evaluated at the correct unit. Report results for the population it actually defines, while monitoring every subsequent delivery stage.
Sources
- CONSORT: Randomised trial reporting guidance
- NIST Privacy Framework
- FDA: Adjusting for Covariates in Randomized Clinical Trials