Interactions between feature tests and rollouts
This topic describes the execution order of feature tests and rollouts and what will happen with your audiences.
Feature test configuration evaluates before feature rollouts configuration. This means that if your feature runs both a feature test and rollout simultaneously, Optimizely first evaluates if a given user qualifies for the experiment. If they do not, then Optimizely evaluates them for the rollout. Likewise, total traffic first evaluates for experiments, then for rollouts.
For example, if you configure 80% of total traffic to a test, and set 50% traffic for a simultaneously running rollout with the same audience, then 80% of traffic goes to the test, and half of the remaining 20%, or 10%, ends up in the rollout.
The table lists the differences between feature tests and rollouts.
| Feature test | Feature rollout | |
|---|---|---|
| Purpose | Lets you gather metrics data so you can compare multiple variations. | Lets you deploy a feature that you have already tested or measured some other way, such as qualitative feedback. |
| User impressions | When a user is assigned to a feature test, Optimizely sends an impression so that information is recorded in your test results. | When a user is assigned to a rollout, Optimizely does not send an impression. This means that extra network traffic and test results are not generated. |
| Variations | Test multiple variations—on versus off, or a multivariate test of different configurations | Only toggles the feature on or off. |
| Triggered by | Is Feature Enabled method | Is Feature Enabled method |
Example scenario
To understand how this works in practice, imagine the following scenario. You have deployed a feature to your application and you have created both a feature test and rollout for that feature:
- Feature test running with an audience and a traffic allocation
- Rollout running with an audience and a traffic allocation
A series of users are handled by your application and are evaluated according to their audience attributes and your feature test and feature rollout rules. The table depicts all possible outcomes.
| User | Feature Test Audience | Feature Test Traffic Allocation | Feature Rollout Audience | Feature Rollout Traffic Allocation | Result |
|---|---|---|---|---|---|
| user1 | pass | pass | N/A | N/A | Feature test |
| user2 | pass | fail | pass | pass | Feature rollout |
| user3 | fail | N/A | pass | pass | Feature rollout |
| user4 | fail | N/A | pass | fail | No action |
| user5 | fail | N/A | fail | N/A | No action |
Updated about 2 years ago