Implementation checklist

When preparing to implement Full Stack in a production environment, it is a good idea to thoroughly familiarize yourself with the configuration details and best practices that will streamline the entire process.

📘

If you use SDK versions released in April 2018 or earlier, see a previous version of this topic.

Architectural diagrams

These diagrams give high-level context for several items in this checklist.

The system context diagram shows a top-level view of Optimizely and your app:

The SDK & app container diagrams show more detail:

Datafile management

The datafile is a JSON representation of OptimizelyConfig. It contains all the data needed to deliver and track your features and experiments for an environment in your Full Stack project. Learn more about the datafile.

You have the following options for synchronizing the datafile between your Optimizely project and your application:

• (Recommended) "Pull" method: Our SDKs provide a polling-based datafile management implementation, handling the fetching of the latest version at whatever frequency you set when you instantiate the SDK.
• “Push” method: use webhooks to fetch and manage datafiles based on application changes.
• Custom method: If you need to customize or extend the way you access the datafile for your implementation, you can access the datafile using the Optimzely CDN link. For more information, see Get the datafile.

Other important considerations for datafile management include:

• Caching and persistence.
• Synchronization between SDK instances
• Network availability

📘

Tip

To ensure webhook requests originate from Optimizely, secure your webhook using a token in the request header.

SDK configuration

The Full Stack SDKs are highly configurable and can meet the needs of any production environment, but adequate scaling may require overriding some default behavior to best meet the needs of your application.

Logger

Verbose logs are critical. The default no-operation SDK logger gives you the scaffolding to create a customer logger. It’s fully customizable and can support use cases like writing logs to an internal logging service or vendor. However, it is intentionally non-functional out-of-the-box. Create a logger that suits your needs and pass it to the Optimizely client.

For more information, see the documentation for the logger and the SimpleLogger reference implementation.

Error handler

In a production environment, errors must be handled consistently across the application. The Full Stack SDKs allow you to provide a custom error handler to catch configuration issues like an unknown experiment key or unknown event key. This handler should cause the application to fail gracefully to deliver a normal user experience. It should also ping an external service, like Sentry, to alert the team of an issue.

🚧

Important

If you do not provide a handler, errors will not surface in your application.

User profile service

Building a User Profile Service (UPS) helps maintain consistent variation assignments between users when test configuration settings change.

The Full Stack SDKs bucket users via a deterministic hashing function, so as long as the datafile and user ID are consistent, it will always evaluate to the same variation. When test configuration settings change, adding a new variation or changing traffic allocation can change a user’s variation and alter the user experience.

Learn more about bucketing behavior in Full Stack.

A UPS solves this by persisting information about the user in a datastore. At a minimum, it should create a mapping of user ID to variation assignment. Implementing a UPS requires exposing a lookup and save function that either returns or persists a user profile dictionary. Our documentation includes the JSON schema for this dictionary. This service also assumes all user IDs are consistent across all use cases and sessions.

📘

Tip

We recommend caching user information after first lookup to speed future lookups.

Let us walk through an example. Using Redis or Cassandra for the cache, you can store user profiles in a key-value pair mapping. You can use a hashed email address mapping to a variation assignment. To keep sticky bucketing for six hours at a time, set a time to live (TTL) on each record. As Optimizely buckets each user, the UPS will interface with this cache and make reads/writes to check assignments before bucketing normally.

Build an SDK wrapper

Many developers prefer to use wrappers to both encapsulate the functionality of an SDK and simplify maintenance. This can be done for all the configuration options described above. Our documentation includes a few examples; see Demo apps and SDK wrappers.

Environments

Optimizely's environments feature enables you to confirm behavior and run tests in isolated environments, like development or staging. This makes it easier to safely deploy tests in production. Environments are customizable and should mimic your team’s workflow. Most customers use two environments: development and production. This allows engineering and QA teams to safely inspect tests in an isolated setting, while site visitors are exposed to tests running in the production environment.

View production as your real-world workload. A staging environment should mimic all aspects of production so you can test before deployment. In these environments, all aspects of the SDK—including dispatcher and logger—should be production-grade. In local environments like test or development, it is okay to use the out-of-the-box implementations instead.

Environments are kept separate and isolated from each other with their own datafiles. For more security, Optimizely allows you to create secure environments, which require authentication for datafile requests. Our server-side SDKs support initialization with these authenticated datafiles. We recommend using this feature only in projects that exclusively use server-side SDKs and implementation. If you fetch the datafiles in a client-side environment, they may become accessible to end-users

User IDs and attributes

User IDs identify the unique users in your tests. It is especially important in a production setting to both carefully choose the type of user ID and set a broader strategy of maintaining consistent IDs across channels. Our documentation explores different approaches and best practices for choosing a user ID.

Attributes allow you to target users based on specific properties. In Optimizely, you can define which attributes should be included in a test. Then, in the code itself, you can pass an attribute dictionary on a per-user basis to the SDK, which will determine which variation a user sees.

📘

Note

Attribute fields and user IDs are always sent to Optimizely’s backend through impression and conversion events. It is up to you to responsibly handle fields (for example, email addresses) that may contain personally identifiable information (PII). Many customers use standard hash functions to obfuscate PII.

Integrations

Build custom integrations with Full Stack using a notification listener. Use notification listeners to programmatically observe and act on various events that occur within the SDK and enable integrations by passing data to external services.

Here are a few examples:

• Send data to an analytics services and report that user_123 was assigned to variation A.
• Send alerts to data monitoring tools like New Relic and Datadog with SDK events to better visualize and understand how A/B tests can affect service-level metrics.
• Pass all events to an external data tier, like a data warehouse, for additional processing and to leverage business intelligence tools.

QA and testing

Before you go live with your test, we have a few final tips:

• Consider your QA options. To manually test different experiences, force yourself into a variation using forced bucketing or whitelisting.
• Ensure everything is working smoothly in a test or staging environment paired with the corresponding datafile generated from a test environment within Optimizely. This will confirm the datafile is accurate and can be verified by checking your SDK logs.
• Run an A/A test to double-check that data is being captured correctly. This helps ensure there are no differences in conversions between the control and variation treatments. Read more about A/A testing.

If you have questions, please contact support. If you think you have found a bug, please file an issue in the SDK’s GitHub repo, and we will investigate as soon as possible.