Interactions between rollouts and feature tests
Feature tests take precedence over feature rollouts. This means that if your feature has both a feature test and rollout, the feature test is evaluated first. If a user does not qualify for the feature test, the rollout is evaluated.
The table lists the differences between feature tests and rollouts.
|Feature test||Feature rollout|
|Purpose||Lets you gather metrics data so you can compare multiple variations.||Lets you deploy a feature that you've already tested or measured some other way, such as qualitative feedback.|
|User impressions||When a user is assigned to a feature test, Optimizely sends an impression so that information is recorded in your test results.||When a user is assigned to a rollout, Optimizely does not send an impression. This means that extra network traffic and test results aren't generated.|
Note: Rollouts don't count against your impressions limit.
|Variations||Test multiple variations—on versus off, or a multivariate test of different configurations||Only toggles the feature on or off.|
To understand how this works in practice, imagine the following scenario. You've deployed a feature to your application and you've created both a feature test and rollout for that feature:
- Feature test running with an audience and a traffic allocation
- Rollout running with an audience and a traffic allocation
A series of users are handled by your application and are evaluated according to their audience attributes and your feature test and feature rollout rules. The table depicts all possible outcomes.
|User||Feature Test Audience||Feature Test Traffic Allocation||Feature Rollout Audience||Feature Rollout Traffic Allocation||Result|
Is Feature Enabled method
Feature tests and rollouts are both triggered using the Is Feature Enabled method.
Interactions in multiple feature tests
In each environment, you can only run one feature test on each feature at a time (meaning, you can't run two tests on the same feature simultaneously) unless your feature is assigned to a mutually exclusive group. If you try to launch a second feature test while another feature test is running, you’ll see a warning that only one feature test is allowed at a time—unless the tests are mutually exclusive.
Use mutually exclusive groups to run concurrent production feature tests. If feature tests are running in a mutually exclusive group, Optimizely evaluates a user’s mutually exclusive group first in the order of operations after calling the Is Feature Enabled method. After the mutually exclusive group is assigned, Optimizely evaluates the feature test first. If a user does not quality for the feature test, the feature rollout is evaluated.
Note that you can have as many draft, paused, and archived tests on a feature as you like.
Updated almost 4 years ago