Global holdback groups

A global holdback group is where a subset of your traffic receives the control variation (also called the baseline). Meanwhile, the remaining traffic can participate in A/B tests and multi-armed bandit optimizations. Holdback groups help you measure the cumulative impact of your experimentation program by comparing those bucketed into test variants to those excluded.

Global holdback groups are:

• utilized only during a specified timeframe.
• comprised of a set percentage or a fixed portion of traffic.
• implemented to assess the cumulative impact of experiments by comparing users in the control group to those in test variants.

Feature Experimentation is stateless and uses deterministic bucketing to make decisions. Because of this, global holdback groups are only directly supported out of the box using the stateless global holdback strategy. You can also define a global holdback with a persistent data store using the stateful global holdback strategy. However, you need a third-party or self-hosted infrastructure to implement this strategy.

Each strategy has its advantages and drawbacks, so it is important to evaluate which approach best aligns with your specific needs and architecture capabilities. No matter what solution you choose, you must include a sensible default variation on each flag for users in the global holdback group to receive.

Stateless global holdback (recommended)

The easiest way to create a global holdback group is to use a user attribute and audience and add that audience to all experiments.

Pros

• The only developer effort required is to implement the user attribute, but you must do this in all implementations of the Feature Experimentation SDKs in your apps.
• Does not require third-party or customer-hosted infrastructure.
• The global holdback group naturally scales as the user base grows (which can account for seasonality effects).
• The global holdback group is randomly sampled such that the make-up of this sample resembles the make-up of visitors not in the holdback.

Cons

• The global holdback group is not static, requiring data analysts to know how the group is defined to determine if a user is a part of it.
• If there is a structural reason that a sample of IDs following a given pattern is not random, this could introduce a bias.

Example implementation steps

📘

Note

See Define attributes and Build audiences from attributes for instructions to create them.

Example 1 – Create a 5% global holdback.

This example uses the last two digits of the user id to determine if the user is in the global holdback.

1. Create a user attribute named lastTwoOfId that you can use to determine if a user is in a global holdback group.

2. Create an audience named Not in Global Holdback.

3. Drag and drop the lastTwoOfId attribute.

4. To configure your audience, check if the visitor matches the lastTwoOfId where Number is greater than 4.

   

Users with lastTwoOfId = 00, 01, 02, 03, or 04 do not qualify for the audience, as they are in the global holdback group.

5. Add the Not in Global Holdback audience to all experiments using an and condition with any other audiences you want to target to ensure the experiment considers only users who are not in the global holdback group.
6. After a developer passes the correct data into the user attribute, the previous steps create a global holdback group. Users in that group receive the default variation for each flag with the Not in Global Holdback audience.

Example 2 – Use a boolean to determine if the user is in the global holdback group.

1. Create an attribute named inGlobalHoldback that you can use to determine if a user is in a global holdback group.
2. At the SDK level, use your own logic to determine if the user is part of the global holdback group and pass true as the value for the boolean user attribute inGlobalHoldback.
3. Add the Not in Global Holdback audience to all experiments using an and condition with any other audiences you want to target to ensure the experiment considers only users who are not in the global holdback group.
4. After a developer passes the correct data into the new user attribute, the previous steps create a global holdback group. Users in that group receive the default variation for each flag with the Not in Global Holdback audience.

Stateful global holdback

You can create a static list of users to comprise a global holdback group. You must persist that data using a third-party or self-hosted data store and pass it into a user attribute.

Pros

• A data analytics team, for example, can statically define the global holdback in advance, and it will not grow unless edited.
• You can define users in the global holdback group in a way that avoids bias (or introduces it, so be careful).
• It may be easier to query for global holdback membership with a persisted datastore, which could be in a data lake, for example.

Cons

• Third-party or customer-hosted infrastructure is required for persisting global holdback membership, as it is not defined deterministically (on read).
• Developer effort is required to safely expose the static list of global holdback member IDs to all Feature Experimentation SDK implementations, like through an API, and implement the new user attribute.
• Assuming the user base grows while the global holdback group remains static, the global holdback group shrinks proportionally.

Implementation steps

Expose your static list of global holdbacks in a way that is accessible to all Feature Experimentation SDK implementations, such as through an API.

Example 1 – Check if the user is in a global list.

1. Create an attribute named inGlobalHoldback that you can use to determine if a user is in a global holdback group.

2. Create an audience named Not in Global Holdback.

3. Drag and drop the inGlobalHoldback attribute.

4. Configure the audience targeting condition to check if the visitor matches where the inGlobalHoldback boolean is false.

   

5. Add the Not in Global Holdback audience to all experiments using an and condition with any other audiences you want to target to ensure the experiment considers only users who are not in the global holdback group.

6. After a developer passes the correct data into the new user attribute, the previous steps create a global holdback group. Users in that group will receive the default variation for each flag with the Not in Global Holdback audience.

Example 2 – Use an array in your code.

1. Create an array in your code named 3rdPartyAudiences with a list of audiences, such as inGlobalHoldback.

2. Create an audience named Not in Global Holdback.

3. Drag and drop the inGlobalHoldback attribute.

4. Configure the audience targeting conditions to check if the visitor does not match where inGlobalHoldback boolean is true.

   

5. Add the Not in Global Holdback audience to all experiments using an and condition with any other audiences you want to target to ensure the experiment considers only users who are not in the global holdback group.

6. After a developer passes the correct data into the new user attribute, the previous steps create a global holdback group. Users in that group will receive the default variation for each flag with the Not in Global Holdback audience.