Implementation checklist: Prepare Full Stack for production

When preparing to implement Full Stack in a production environment, it is a good idea to thoroughly familiarize yourself with the configuration details and best practices that will streamline the entire process.

Datafile management

A datafile is a JSON representation of your Full Stack project. It contains all the instructions needed to run your experiments. Learn more about the datafile.

Optimizely’s Full Stack SDKs automatically manage the datafile, handling the fetching of the latest version at whatever frequency you set when you instantiate the SDK. If you need to customize the way you access the datafile for your implementation, you can. It’s recommended to set up a separate datafile synchronization service that will be responsible for datafile storage, refresh frequency, and fetch method.

If you customize the way you access the datafile, be sure to handle datafile versioning and management. When that’s done, make sure the datafile itself is up to date.

You can fetch the datafile using the CDN or REST API. Requests to the REST API must be authenticated with a token. After you can access the datafile, choose a strategy for synchronizing the datafile with your servers. In production, we recommend polling at 5-minute intervals, although you can also use a “push” model based on webhooks and configured to point to a synchronization service.

The synchronization service should contain an endpoint that is responsible for pulling down the datafile and re-instantiating the Optimizely object. This should be set up as a one-off service (meaning, a microservice), and the object should be stored in this service. This service should also be responsible for propagating the update to other servers by pushing or notifying subscribes to re-instantiate the object. Otherwise, a synchronizer behind a load balancer could cause servers in the fleet to become out of sync.

📘

Note

To ensure webhook requests originate from Optimizely, secure your webhook using a token in the request header.

SDK configuration

The Full Stack SDKs are highly configurable and can meet the needs of any production environment, but adequate scaling may require overriding some default behavior to best meet the needs of your application.

Event dispatcher

Each SDK provides a reference event dispatcher implementation out-of-the-box. For blocking languages like PHP, Ruby, and Python, the reference dispatcher is synchronous. For languages that support asynchronous operations, the reference dispatcher is asynchronous.

Regardless of the SDK you are using, we recommend customizing the event dispatcher you use in production to ensure that you queue and send events in a manner that scales to the volumes handled by your application. Customizing the event dispatcher allows you to take advantage of features like batching, which makes it easier to handle large event volumes efficiently or to implement retry logic when a request fails. You can build your dispatcher from scratch or start with the provided dispatcher.

📘

Note

It is important to customize your event dispatcher when using an SDK for a blocking language (PHP, Ruby, and Python) to ensure that you can retrieve variations without waiting for the corresponding network request to return.

Consider creating a separate service that works within your networking requirements and would be responsible for queuing and flushing events. In this scenario, the SDK acts as the producer and writes all events to a datastore (for example, a queue). The microservice—now acting as the consumer—then builds a single-event object containing all items in the datastore and dispatches it with a single request to Optimizely. The dispatch frequency can be based on the number of events in the queue or time-base, whichever comes first. After the request is successfully received by the logging servers, it’s safe to flush the events.

For more information, see our documentation on the bulk dispatcher reference implementation.

If you are using a machine which includes a firewall, you may notice events being blocked. Our logs most likely won't show events being blocked by a firewall. If this is the case, you will need to configure Optimizely to use a proxy through a custom event dispatcher. Your firewall might be blocking requests to logx.optimizely.com. If this is the case, the events will not come through to Optimizely.

One other way around this would be to whitelist logx.optimizely.com within your firewall. This will allow our events through.

Logger

Verbose logs are critical. The default no-operation SDK logger gives you the scaffolding to create a customer logger. It’s fully customizable and can support use cases like writing logs to an internal logging service or vendor. However, it is intentionally non-functional out-of-the-box. Create a logger that suits your needs and pass it to the Optimizely client.

For more information, see the documentation for the logger and the SimpleLogger reference implementation.

Error handler

In a production environment, errors must be handled consistently across the application. The Full Stack SDKs allow you to provide a custom error handler to catch configuration issues like an unknown experiment key or unknown event key. This handler should cause the application to fail gracefully to deliver a normal user experience. It should also ping an external service, like Sentry, to alert the team of an issue.

🚧

Important

If you do not provide a handler, errors will not surface in your application.

User profile service

Building a User Profile Service (UPS) helps maintain consistent variation assignments between users when test configuration settings change.

The Full Stack SDKs bucket users via a deterministic hashing function, so as long as the datafile and user ID are consistent, it will always evaluate to the same variation. When test configuration settings change, adding a new variation or changing traffic allocation can change a user’s variation and alter the user experience.

Learn more about bucketing behavior in Full Stack.

A UPS solves this by persisting information about the user in a datastore. At a minimum, it should create a mapping of user ID to variation assignment. Implementing a UPS requires exposing a lookup and save function that either returns or persists a user profile dictionary. Our documentation includes the JSON schema for this dictionary. This service also assumes all user IDs are consistent across all use cases and sessions.

📘

Note

Optimizely recommends caching user information after first lookup to speed future lookups.

Let’s walk through an example. Using Redis or Cassandra for the cache, you can store user profiles in a key-value pair mapping. You can use a hashed email address mapping to a variation assignment. To keep sticky bucketing for six hours at a time, set a time to live (TTL) on each record. As Optimizely buckets each user, the UPS will interface with this cache and make reads/writes to check assignment before bucketing normally.

Build an SDK wrapper

Many developers prefer to use wrappers to both encapsulate the functionality of an SDK and simplify maintenance. This can be done for all the configuration options described above. Our documentation includes a few examples; see Demo apps and SDK wrappers.

Environments

Optimizely's environments feature enables you to confirm behavior and run tests in isolated environments, like development or staging. This makes it easier to safely deploy tests in production. Environments are customizable and should mimic your team's workflow. Most customers use two environments: development and production. This allows engineering and QA teams to safely inspect tests in an isolated setting, while site visitors are exposed to tests running in the production environment.

View production as your real-world workload. A staging environment should mimic all aspects of production so you can test before deployment. In these environments, all aspects of the SDK—including dispatcher and logger—should be production-grade. In local environments like test or development, it is okay to use out-of-the-box implementations instead.

By default, each Optimizely project contains a production environment. We recommend that you create a secondary environment to expose tests to internal teams before users see them. Environments are kept separate and isolated from each other with their own datafiles.

User IDs and attributes

User IDs identify the unique users in your tests. It is especially important in a production setting to both carefully choose the type of user ID and set a broader strategy of maintaining consistent IDs across channels. Our documentation explores different approaches and best practices for choosing a user ID.

Attributes allow you to target users based on specific properties. In Optimizely, you can define which attributes should be included in a test. Then, in the code itself, you can pass an attribute dictionary on a per-user basis to the SDK, which will determine which variation a user sees.

📘

Note

Attribute fields and user IDs are always sent to Optimizely’s backend through impression and conversion events. It is up to you to responsibly handle fields (for example, email addresses) that may contain personally identifiable information (PII). Many customers use standard hash functions to obfuscate PII.

Integrations

Build custom integrations with Full Stack using a notification listener. Use notification listeners to programmatically observe and act on various events that occur within the SDK and enable integrations by passing data to external services.

Here are a few examples:

• Send data to an analytics services and report that user_123 was assigned to variation A.
• Send alerts to data monitoring tools like New Relic and Datadog with SDK events to better visualize and understand how A/B tests can affect service-level metrics.
• Pass all events to an external data tier, like a data warehouse, for additional processing and to leverage business intelligence tools.

QA and testing

Before you go live with your test, we have a few final tips:

• Consider your QA options. To manually test different experiences, force yourself into a variation using forced bucketing or whitelisting.
• Ensure everything is working smoothly in a test or staging environment paired with the corresponding datafile generated from a test environment within Optimizely. This will confirm the datafile is accurate and can be verified by checking your SDK logs.
• Run an A/A test to double-check that data is being captured correctly. This helps ensure there are no differences in conversions between the control and variation treatments. Read more about A/A testing.

If you have questions, contact support. If you think you have found a bug, please file an issue in the SDK’s GitHub repo, and Optimizely will investigate as soon as possible.