Two-stage and three-stage testing

Add a staging environment to your project to use for QA testing before running experiments in your Production environment; see Create and manage environments for more information.

Remember that each environment has a different datafile, with which you will initialize our SDKs.

📘

Note

You do not necessarily need separate environments to make use of Optimizely’s Environments feature. You can, for example, create an environment for users with a test cookie. You will just need to implement the logic that would initialize Optimizely using the test cookie environment when the user has your test cookie.

Two-stage and three-stage test environments

Here are two examples of environment use cases. You can have an unlimited number of environments, and there are many more possibilities for using them than we have described here. Use as many environments as your workflow requires.

In general, the fewer environments you use, the simpler the workflow. If you use two or more environments, they’ll be grouped together on your experiment list as “Other Environments.”

Two-stage “sandbox” QA

The two-stage “sandbox” QA use case is a common setup in web development and involves these separate, sequential environments:

• Local: code is running only for the development team and QA
• Production: code is deployed and running for all visitors

You can use the "sandbox" approach even if you don’t have separate deployment environments for your application. One common use case is to toggle between the local and production files for debugging. For example, you could load the local file when a certain “test” cookie is enabled, when a certain URL parameter is attached (like ?debug=true), or based on some other check that matches your internal workflow.

Three-stage testing

The three-stage testing use case involves these separate, sequential environments:

• Local: code is running only for the developer who is writing it. You don’t need to create a separate Local environment for each developer, just one environment where the experiment will run during development.
• Staging: code is shared with the larger development team and QA.
• Production: code is deployed and running for all visitors.

Example test environment scenario

Imagine you have three experiments: sort-by-name, show-subtotal, and warn-empty-cart. You set up a new environment named Testing so you can test your experiment code before you deploy in the Production environment. Suppose that:

• You completed testing for the sort-by-name experiment and deployed it.
• You are currently testing the show-subtotal experiment.
• Your warn-empty-cart experiment has been tested but is not yet deployed.

Here is an overview of the statuses of your three experiments in your two environments:

You can keep experiments running in your testing environment that are also running in production if you prefer.

Now, let us imagine that the project owner creates a third environment: Development. Optimizely will generate an additional datafile named Development. Every experiment will be in the “Not started” activity state in Development, and the Development datafile will show experiment status as “Not started.”

When your experiment code is fully functional and ready to test, switch it to the Testing environment by pointing your code to the Testing datafile.