Global holdout groups

A global holdout group is where a subset of your traffic receives the control variation (the baseline). Meanwhile, the remaining traffic can participate in A/B tests and multi-armed bandit optimizations (collectively called experiments). Holdout groups help you measure the cumulative impact of your experimentation program by comparing those bucketed into test variants to those excluded.

Global holdout groups are

• utilized only during a specified timeframe.
• comprised of a set percentage or a fixed portion of traffic.
• implemented to assess the cumulative impact of experiments by comparing users in the control group to those in test variants.

Feature Experimentation is stateless and uses deterministic bucketing to make decisions. Because of this, global holdout groups are only directly supported out of the box using the stateless global holdout strategy. You can also define a global holdout with a persistent data store using the stateful global holdout strategy. However, you need a third-party or self-hosted infrastructure to implement this strategy.

Each strategy has its advantages and drawbacks, so evaluating which approach best aligns with your specific needs and architecture capabilities is important. No matter what solution you choose, you must include a default variation on each flag for users in the global holdout group to receive.

Stateless global holdout (recommended)

The easiest way to create a global holdout group is to create an user attribute and audience and add that audience to all experiments.

Pros

• Does not require third-party or separate hosted infrastructure.
• The global holdout group naturally scales as the user base grows (which can account for seasonality effects.
• The global holdout group is randomly sampled.

Cons

• The global holdout group is not static, requiring data analysts to know how the group is defined to determine whether a user is a member.

Example implementation steps

Example 1 – Create a 5% global holdout.

This example uses the last two digits of the user id to determine if the user is in the global holdout.

1. Create a user attribute named lastTwoOfId that you can use to determine if a user is in a global holdout group. See Define attributes to create attributes in the UI or the Create an Attribute endpoint in the API reference documentation.

2. Create an audience named Not in Global Holdout. See Target audiences for the UI or Create an Audience endpoint.

3. Drag and drop the lastTwoOfId attribute.

4. Configure the Audience Conditions drop-down lists to check if the Visitor matches the lastTwoOfId attribute where Number is greater than 4.

   

📘

Note

Users with lastTwoOfId = 00, 01, 02, 03, or 04 do not qualify for the Not in Global Holdout audience, as they are in the global holdout group.

5. Add the Not in Global Holdout audience to all experiments using an and condition with any other audiences you want to target to ensure the experiment considers only users who are not in the global holdout group.
6. After you pass the data into the user attribute, the previous steps create a global holdout group. Users in that group receive the default variation for each flag because they do not qualify for the Not in Global Holdbout audience.

Example 2 – Use a boolean to determine if the user is in the global holdout group.

1. Create an attribute named inGlobalHoldout that you can use to determine if a user is in a global holdout group.
2. At the SDK level, use your own logic to determine if the user is part of the global holdout group and pass true as the value for the boolean user attribute inGlobalHoldout.
3. Add the Not in Global Holddout audience to all experiments using an and condition with any other audiences you want to target to ensure the experiment considers only users who are not in the global holdout group.
4. After you pass data into the new user attribute, the previous steps create a global holdout group. Users in that group receive the default variation for each flag because they do not qualify for the Not in Global Holdbout audience.

Stateful global holdout

Instead of relying solely on user attributes, the stateful approach uses a dedicated experiment to determine holdout membership and dynamically applies that decision across all other experiments.

You can create a static list of users to comprise a global holdout group. You must persist that data using a third-party or self-hosted data store and pass it into a user attribute.

Pros

• A data analytics team, for example, can statically define the global holdout in advance and does not grow unless edited.
• You can define users in the global holdout group in a way that avoids bias (or introduces it, so be careful).
• It may be easier to query for global holdout membership with a persisted datastore, for example, in a data lake.

Cons

• Third-party or customer-hosted infrastructure is required for persisting global holdout membership, as it is not defined deterministically (on read).
• Developer effort is required to safely expose the static list of global holdout member IDs to all Feature Experimentation SDK implementations, like through an API, and implement the new user attribute.
• Assuming the user base grows while the global holdout group remains static, the global holdout group shrinks proportionally.

Implementation steps

Expose a static list of global holdouts in a way accessible to all Feature Experimentation SDK implementations. First, create and run an experiment to determine if the user is in the global holdout group. Assign users to an attribute and use this attribute in future experiments to filter out users from the holdout group.

1. Create a global holdout a/b test flag. This experiment determines whether a user is in the holdout. Evaluate this flag as early as possible in the user journey (for example, at login or first visit).

   • 5% of users are assigned to holdout (do not see any experiments).

   • 95% of users proceed as normal (eligible for experiments).

     

2. For any future experiments, you run Global holdout flag before the new experiment. Then create an attribute named eligibleForTests based on the global holdout experiment result. Which variation the user receives determines if they are eligible for the experiment.

   • On variation – User is eligible for experiments.

   • Off variation – User is in holdout and should not see any experiments.

     

3. Create an audience named Eligible for tests using the eligibleForTests attribute as an audience filter.

   • Configure the Audience Conditions drop-down lists to check if the Visitor matches the eligibleForTests equals on.

   • If the user is in the On variation, they are eligible for future experiments.

     

4. Any future flags use the the Eligible for tests audience. This audience filters out users that are in the global holdout.

   For example, create an A/B test flag rule named experiment_flag that uses the Eligible for tests audience like the following:

   

The following JavaScript sample code demonstrates how to call the decide() method on the global_holdout_experiment flag and set the eligibleForTests attribute based on the decision. Then, it retrieves a decision for experiment_flag, which uses eligibleForTests as an audience condition.

const user = optimizely.createUserContext("user123");

/* -- Uncomment below to force "off" decision for global holdout flag -- */
result = user.setForcedDecision(
  { flagKey: "global_holdout_experiment" },
  { variationKey: "off" }
);

/* -- Make decision for global holdout first -- */
const holdoutDecision = user.decide("global_holdout_experiment");
console.log(
  "\nGlobal holdout decision:",
  holdoutDecision.variationKey,
  "\n"
);

user.setAttribute("eligibleForTests", holdboutDecision.variationKey);
console.log("user attributes: ", user.getAttributes(), "\n");

/* -- If eligibleForTests=off for the user, they do not qualify for any experiment audiences -- */

experimentDecision = user.decide("experiment_flag");
console.log("Audience evaluation: ", experimentDecision.reasons[1]);
console.log(
  "\nexperimentDecision:",
  experimentDecision.variationKey,
  "\n"
);