HomeDev GuideAPI Reference
Dev GuideAPI ReferenceUser GuideGitHubNuGetDev CommunitySubmit a ticketLog In
GitHubNuGetDev CommunitySubmit a ticket

Optimizely A/B testing (legacy)

Describes how to create variations for a number of page elements (blocks, images, content, buttons, and form fields), then compare which variation performs best.

A/B testing (legacy) lets editors create variations for a number of page elements (blocks, images, content, buttons, and form fields), then compare which variation performs best. It measures the number of conversions from the original (control) versus the variation (challenger). The one generating the most conversions during the test period is promoted to the design for that page. A/B testing has several predefined conversion goals, and you can also create custom conversion goals.

See also Delivery & Experimentation for information on the Optimizely delivery and experimentation platform for websites, mobile apps, smart devices, and back-end code. With this, you can A/B test everything from search results and promotions, to recommendations and payment options. You can also A/B test user interface variations, to optimize and personalize website messaging.

Requirements

  • No additional license fee.
  • An Optimizely Content Management System (CMS) installation.
  • See App platform compatibility for package and version information. 

Install

Documentation

Technical limitation

The underlying tool used by A/B testing to render preview images of the changes under test has the following technical limitations:

  • Media queries defined on link elements are ignored
  • Media queries defined on import rules are ignored

This limitation results in the styles, which would ordinarily be restricted to particular types of media, being applied to screen media and therefore appearing in the preview images that are generated.

However, the functionality respects media queries that are defined directly within the CSS. So, a workaround for these situations is to wrap the existing contents of the linked CSS source file in a print media query.

For example, let's imagine that a file named print.css is linked to a page with a media attribute for print.

<link src="print.css" rel="stylesheet" media="print" />

And imagine that print.css currently contains the following:

html, body { margin: 0; }
    img { display: none; }

Applying such a workaround would require you to simply wrap the existing statements in a media query:

@media print {
        html, body { margin: 0; }
        img { display: none; }
    }

The attribute can remain on the element as well, if that is preferable for other technical concerns.

Key performance indicator

A key performance indicator (KPI) in Optimizely records when a visitor on a website performs a desired action, such as clicking on an ad or a button, or completing a sale. The functionality is used for example in A/B testing of content.

You can create custom KPIs, using goal objects in the KPI framework, to use in Optimizely A/B testing or in any other package that relies on creating instance-based goal objects.

To create a custom KPI, implement the IKpi interface located in the EPiServer.Marketing.KPI.Manager.DataClass namespace.

Cookie usage in A/B testing

The AB testing package has one cookie per test the user visits in the format of:

EPI-MAR-

"Content GUID" is the Optimizely GUID for the content under test. Inside the cookie, the state of the visitor against the running test is stored. That is, the version the visitor should see if returning to a test item while the test is still running, if they have yet to view the content in question, and the various goals the visitor has converted on, or has yet to convert on.

Cookie NamePurpose
EPI-MAR-{xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx:xx}Analytical/performance functionality. Records a visitors interaction with a running website optimization test, to ensure that a visitor has a consistent experience. Persistent (variable according the test). Typically optimization tests are short-lived (one week).The cookie is removed after the test has completed.

See Cookies for information about other cookies in the Optimizely platform.

IKpi methods

 IKpi methods Description
Guid IdThe Id of the Kpi instance. Used to identify between multiple instances of the same type of Kpi.
string FriendlyNameDisplay name of the Kpi. Used for any calling package to identify the KPI type in a user-friendly manner.
string DescriptionDescribes what the KPI does so that it can be displayed to the user.
string UiMarkupThe HTML form inputs required to create an instance of the KPI. The Validate method uses this to store relevant information the KPI needs to properly evaluate. Inputs require a name attribute to retrieve the value and pass it to the Validate method.
string UiReadOnlyMarkupThe HTML fragment used for displaying the user-generated input that created the instance of the KPI.
void Validate(Dictionary\<string, string> kpiData)Reads a dictionary of input generated from the UiMarkup to create a KPI instance. Keys in the kpiData Dictionary match the name property of the inputs in the UiMarkup. Store values that need to be retrieved later for use in the Evaluate method as class properties. Generate a KpiValidationException when the data is not in a valid format so that the calling method can handle validation errors.
event EventHandler EvaluateProxyEventCalled by packages using this Kpi, the EvaluateProxyEvent attaches the passed-in event proxy to the .Net event the Kpi cares about. When the proxy is attached, it calls Evaluate with the proper sender and event arguments so that the KPI can determine if a conversion occurred.
IKpiResult Evaluate(object sender, EventArgs e)Determines if a conversion occurred when the proper .Net event is executed. The passed-in EventArgs should be cast to the expected EventArgs type to check the Kpi’s instance data against the event specific data.
DateTime CreatedDateDetermines when the Kpi instance was created.
DateTime ModifiedDateDetermines when the Kpi instance is modified.
ResultComparisonIndicates how the Kpi Evaluation result object should be interpreted when trying to decide if a result is better than another instance of the result. Used specifically for indicating whether a greater or lesser result is desired.
void InitializeCalled by external packages when the KPI needs to set up dependencies that occur outside of construction. Designed to do any extra setup for the KPI such as listening for setup events outside of the normal Evaluate event handler.
void UninitializeCalled by external packages when the KPI needs to clean up dependencies that are set up during the Initialize method.
string KpiResultTypeInforms external packages what type of KPI result evaluate will return.

IClientKpi

IClientKpi is an interface for marking a custom KPI that should be run on the client browser to convert an A/B test. It consists of only one method for retrieving the client JavaScript that needs to be presented in the browser to indicate when a conversion takes place.

IClientKpi methodDescription
string ClientEvaluationScriptReturns a JavaScript function to be executed in a visitor’s browser to indicate when a conversion has occurred. This function’s first parameter should be a success callback function to execute when a conversion has occurred.

📘

Note

The function returned by this method should be immediately invoked when loaded in the target page.

Example script to return:

function(success) {if( mySuccessConditionWasMet() ) {	success(); // Invoke the passed in success callback}
    }

Serialization

Calling packages such as A/B testing package rely on Microsoft’s System.Runtime.Serialization’s attributes to inform the user interface of a Kpi type’s information. The [DataContract] attribute should be on the class implementing the IKpi interface and the [DataMember] attribute on fields that should be sent to the user interface.

Example: Custom KPI multiple target page conversion

using System;
    using System.Collections.Generic;
    using System.Runtime.Serialization;
    using EPiServer.Marketing.KPI.Manager.DataClass;
    using EPiServer.Marketing.KPI.Results;
    using EPiServer.ServiceLocation;
    using EPiServer;
    using EPiServer.Core;
    using EPiServer.Marketing.KPI.Exceptions;
    using EPiServer.Marketing.KPI.Manager.DataClass.Enums;
    
    namespace Devsite.KPI
    {
        [DataContract]
        public class DemoKPI : IKpi
        {
            private string _description = "Conversion goal is activated when a user lands on one of the pages specified for the test.";
            private string _friendlyName = "Demo Landing Section Goal";
            private string _uiMarkup = "<div><input name=\"PageIds\" type = \"text\" placeholder = \"Pages\" /><span> Comma separated list of page ID's</span></ div>";
            private string _readonlyMarkup = "<div>You chose some pages that I am not going to display, but could</div>";
    
            private IServiceLocator _serviceLocator;
    
            public DemoKPI()
            {
                _serviceLocator = ServiceLocator.Current;
            }
    
            [DataMember]
            public Guid Id { get; set; }
    
            [DataMember]
            public List<Guid> PageGuids;
    
            [DataMember]
            public DateTime CreatedDate { get; set; }
    
            [DataMember]
            public DateTime ModifiedDate { get; set; }
    
            [DataMember]
            public string Description { get { return _description; } }
    
            [DataMember]
            public string FriendlyName { get { return _friendlyName; } }
    
            [DataMember]
            public string UiMarkup { get { return _uiMarkup; } }
    
            [DataMember]
            public string UiReadOnlyMarkup { get { return _readonlyMarkup; } }
    
            [DataMember]
            public ResultComparison ResultComparison
            {
                get
                {
                    return ResultComparison.Greater;
                }
            }
    
            [DataMember]
            public string KpiResultType
            {
                get
                {
                    return typeof(KpiConversionResult).Name.ToString();
                }
            }
    
            private EventHandler<ContentEventArgs> _eventHander;
    
            /// <summary>
            /// EventHandler that tells the code using this KPI what CMS event this KPI wants to attach to in order to properly evaluate
            /// </summary>
            public event EventHandler EvaluateProxyEvent
            {
                add
                {
                    _eventHander = new EventHandler<ContentEventArgs>(value);
                    var service = _serviceLocator.GetInstance<IContentEvents>();
                    service.LoadedContent += _eventHander;
                }
                remove
                {
                    var service = _serviceLocator.GetInstance<IContentEvents>();
                    service.LoadedContent -= _eventHander;
                }
            }
    
            /// <summary>
            /// Method that should get called when the CMS event we attach to in EvaluateProxyEvent that checks to see if a particular condition is met
            /// </summary>
            /// <param name="sender">The caller of the event</param>
            /// <param name="e">Generic event arguments that should be cast to the specific event arguments defined by the event attached to in EvaluateProxyEvent</param>
            /// <returns></returns>
            public IKpiResult Evaluate(object sender, EventArgs e)
            {
                var retval = false;
                var ea = e as ContentEventArgs;
                if (ea != null && PageGuids != null)
                {
                    retval = PageGuids.Contains(ea.Content.ContentGuid);
                }
    
                return new KpiConversionResult() { KpiId = Id, HasConverted = retval };
            }
    
            /// <summary>
            /// Validates that the passed in data is able to be used to create an instance of the KPI
            /// </summary>
            /// <param name="kpiData">dictionionary of data used to validate and create instances of the KPI for use in other classes (like AB Tests)</param>
            public void Validate(Dictionary<string, string> kpiData)
            {
                var ids = kpiData["PageIds"];
                if (!string.IsNullOrEmpty(ids))
                {
                    PageGuids = GetPageGuidsFromContentIds(ids);
                }
                else
                {
                    throw new KpiValidationException("Empty Page Id's");
                }
            }
    
            private List<Guid> GetPageGuidsFromContentIds(string ids)
            {
                var retList = new List<Guid>();
                var aContentLoader = _serviceLocator.GetInstance<IContentLoader>();
                foreach (var id in ids.Split(','))
                {
                    int aId;
                    if (int.TryParse(id, out aId))
                    {
                        var aContentReference = new ContentReference(aId);
                        var aPage = aContentLoader.Get<IContent>(aContentReference);
                        retList.Add(aPage.ContentGuid);
                    }
                    else
                    {
                        throw new KpiValidationException("unable to parse the Ids - they should be a comma separated list of integers");
                    }
                }
    
                return retList;
            }
    
            public void Initialize()
            {
                // not needed in this example
                // a hook to attach to external events other than the evaluate event to set up data in the KPI instance here
                // particularly useful for KPI's that rely on multiple user actions to happen before evaluate should return a converted result
            }
    
            public void Uninitialize()
            {
                // not needed in this example
                // a hook to clean up any objects set up during the intialize event
            }
        }
    }