Decisions and impressions

Optimizely Experimentation records a decision event when a user is exposed to an experiment variation provided by Optimizely. Specifically, Optimizely logs a decision through the following sequence:

1. An experiment is activated with the Activate, Is Feature Enabled or Decide methods.
2. As a result, an SDK decision event is sent asynchronously to the Optimizely Event API. If the decision event indicates that the user is bucketed into an experiment, Optimizely logs the impression.

Decision events are used to generate the visitor counts shown on the Results page. They also serve as the denominator for measuring conversion rates and are used to determine your impression total.

Generally speaking, decision events are triggered automatically when you call Activate, Is Feature Enabled, Get Enabled Features for a user that Optimizely determines qualifies for a running experiment. No decision events are triggered for flag rollouts.

How the SDK triggers impressions

Decision events are sent

• When the user is bucketed into a feature test with the Is Feature Enabled or Get Enabled Features methods, regardless of whether the feature is enabled or disabled in the feature test variation.
• When a user is bucketed into an A/B test with the Activate method.

Examples:

• A new visitor lands on a page and is exposed to an experiment.
• A visitor refreshes a page and is exposed to the experiment again.
• A return visitor is exposed to the experiment again.

Decision events are not sent

• Any time the Activate method returns null because the user did not qualify for any experiment.
• If the user is only bucketed into a feature rollout (with the Is Feature Enabled or Get Enabled Features methods) and there is no running feature test associated with the feature.
• If neither a test nor rollout is running.

📘

Note

You must make sure that your local datafile has been updated after pausing or archiving a test to ensure you no longer send impressions to Optimizely.

Impressions deduplication

Depending on your application logic, your users may trigger rapid bursts of decision events. To ensure that your billing impression consumption is aligned to your end users' experience, Optimizely uses fixed-interval deduplication of decision events to calculate impression consumption. All decision events received within fixed 5-second intervals count as one impression for each user in an experiment. Optimizely buckets events into the fixed time windows based on when Optimizely received the event.

📘

Note

For more information on how impressions are triggered specifically, see Impression in Optimizely Experimentation.

Impressions and billing

Starting September 2020, Optimizely has introduced a simplified usage billing component: Monthly Active Users (MAUs), which replaces impressions. Monthly Active Users (MAUs) is the number of unique users used in a decision or tracking event. It is a measure of the overall traffic where you are using the snippet, APIs, or SDKs to do something, specifically:

• Experiment Evaluation
• Personalization Campaign Evaluation
• Feature Flag/Rollout Evaluation
• Event Tracking

Unlike impressions, it is not a measure of what percent of that traffic you are experimenting on—every user that gets evaluated is counted. This allows you to run large-scale experiments at 100% traffic in order to reach statistical significance more quickly.

See Monthly active users (MAUs) in Optimizely.