Run A/B tests

How to set up a simple A/B or ON/OFF test in Optimizely Feature Experimentation.

If you are new to experimentation, you can get a lot done with a simple ON/OFF A/B test. This configuration has one flag with two variations:

One "flag_on" variation
One "flag_off" variation

Restrictions

In your rulesets for a flag, your experiment must always be the first rule and must be the only experiment in the ruleset. In other words, you can only run one experiment at a time for a flag.

📘
Note
Multiple experiments per flag are only available for select plans.

Setup overview

To configure a basic A/B test:

(Prerequisite) Create a flag.
(Prerequisite) Handle user IDs.
Create and configure an A/B Test rule in the Optimizely app.
Integrate the example decide code that the Optimizely Feature Experimentation app generates with your application.
Test your experiment in a non-production environment. See QA and troubleshoot.
Discard any QA user events and enable your experiment in a production environment.

Create an experiment

Create A/B test rule

Select a flag from the Flags list.
Select the environment you want to target.
Click Add Rule.
Select A/B Test.

Configure your A/B test rule

Enter a Name
The Key will be automatically created based on the Name. You can optionally update it.
(Optional) Click Add description to add a description. It is a good practice to add your hypothesis for your A/B test rule as the description.
(Optional) Search for and add audiences. To create an audience, see Target audiences. Audiences evaluate in the order in which you drag and drop them. You can choose whether to match each user on any or all of the audience conditions.
Set the Ramp percentage to allocate the percentage of your audience to bucket into the experiment.

📘
Note
If you plan to change the traffic's Ramp percentage after running the experiment or select Stats Accelerator for the Distribution Mode, you will need to implement a user profile service before starting the experiment.
For more information, see Ensure consistent user bucketing.

Add Metrics based on tracked user events. See Create events to create and track events. For more information about selecting metrics, see Choose metrics.
Choose how your audience will be distributed using Distribution Mode. Use the drop-down list to select either:
1. Manual – By default, variations are given equal traffic distribution. Customize this value for your experiment's requirements.
2. Stats Accelerator – Stats Accelerator automatically manipulates traffic distribution to minimize time to statistical significance using . For more information, see Stats accelerator.
Choose the flag variations to compare in the experiment. For a basic experiment, you can include one variation in which your flag is on and one in which your flag is off. For a more advanced A/B/n experiment, create variations with multiple flag variables. No matter how many variations you make, leave one variation with the feature flag off as a control. For more information about creating variations, see Create flag variations.
(Optional) Click Allowlist: Force up to 10 users into any variation(s) and enter the User ID. See Allowlisting.
(Optional) Add the experiment to an Exclusion Group.
Click Save.

Implement the experiment using the decide method

Flag is implemented in your code

If you have already implemented the flag using a Decide method, you do not need to take further action (Optimizely Feature Experimentation SDKs are designed so you can reuse the exact flag implementation for different flag rules).

Flag is not implemented in your code

If the flag is not implemented yet, copy the sample flag integration code into your application code and edit it so that your feature code runs or does not run based on the output of the decision received from Optimizely.

Use the Decide method to enable or disable the flag for a user:

// Decide if user sees a feature flag variation
user := optimizely.CreateUserContext("user123", map[string]interface{}{"logged_in": true})
decision := user.Decide("flag_1", nil)
enabled := decision.Enabled

// Decide if user sees a feature flag variation
var user = optimizely.CreateUserContext("user123", new UserAttributes { { "logged_in", true } });
var decision = user.Decide("flag_1");
var enabled = decision.Enabled;

// Decide if user sees a feature flag variation
var user = await flutterSDK.createUserContext(userId: "user123");
var decideResponse = await user.decide("product_sort");
var enabled = decision.enabled;

// Decide if user sees a feature flag variation
OptimizelyUserContext user = optimizely.createUserContext("user123", new HashMap<String, Object>() { { put("logged_in", true); } });
OptimizelyDecision decision = user.decide("flag_1");
Boolean enabled = decision.getEnabled();

// Decide if user sees a feature flag variation
const user = optimizely.createUserContext('user123', { logged_in: true });
const decision = user.decide('flag_1');
const enabled = decision.enabled;

// Decide if user sees a feature flag variation
$user = $optimizely->createUserContext('user123', ['logged_in' => true]);
$decision = $user->decide('flag_1');
$enabled = $decision->getEnabled();

# Decide if user sees a feature flag variation
user = optimizely.create_user_context("user123", {"logged_in": True})
decision = user.decide("flag_1")
enabled = decision.enabled

// Decide if user sees a feature flag variation
var decision = useDecision('flag_1', null, { overrideUserAttributes: { logged_in: true }});
var enabled = decision.enabled;

# Decide if user sees a feature flag variation
user = optimizely_client.create_user_context('user123', {'logged_in' => true})
decision = user.decide('flag_1')
decision.enabled

// Decide if user sees a feature flag variation
let user = optimizely.createUserContext(userId: "user123", attributes: ["logged_in":true])
let decision = user.decide(key: "flag_1")
let enabled = decision.enabled

For more detailed examples of each SDK, see:

Adapt the integration code in your application. Show or hide the flag's functionality for a given user ID based on the boolean value your application receives.

The goal of the Decide method is to separate the process of developing and releasing code from the decision to turn a flag on. The value this method returns is determined by your flag rules. For example, if the current user is assigned to a control or "off" variation in an experiment, the method returns false.

Remember, a user evaluates each flag rule in an ordered ruleset before being bucketed into a given rule variation or not. See Interactions between flag rules for more information.

Test with flag variables

Once you have run a basic "on/off" A/B test, you can increase the power of your experiments by adding remote feature configurations or flag variables.

Flag variations enable you to avoid hard-coding variables in your application. Instead of updating the variables by deploying, you can edit them remotely in the Optimizely Feature Experimentation app. For more information about flag variations, see flag variations.

To set up an A/B test with multiple variations:

Create and configure a basic A/B test. See previous steps.
Create flag variations containing multiple variables.
Integrate the example code with your application.

Updated 5 days ago

Run flag deliveries