HomeGuidesAPI Reference
Submit Documentation FeedbackJoin Developer CommunityOptimizely GitHubOptimizely NuGetLog In

Two- and three-stage testing

This topic describes best practices for QA testing in a staging environment before deploying to a production environment.

Optimizely Full Stack Flags version comes with two environments automatically: Production and Development. When selecting a flag, you are directed to the Development environment rules.

You can add additional environments and change which environment is set as primary. See Manage environments for more information.

Remember that each environment has a different datafile, with which you will initialize the Optimizely SDKs.

📘

Note

You do not necessarily need separate environments to make use of Optimizely’s Environments feature. You can, for example, create an environment for users with a test cookie. You will just need to implement the logic that would initialize Optimizely using the test cookie environment when the user has your test cookie.

Two-stage and three-stage test environments

Here are two examples of environment use cases. You can have an unlimited number of environments, and there are many more possibilities for using them than we have described here. Use as many environments as your workflow requires.

In general, the fewer environments you use, the simpler the workflow.

Two-stage “sandbox” QA

The two-stage “sandbox” QA use case is a common setup in web development and involves these separate, sequential environments:

  • Local: code is running only for the development team and QA
  • Production: code is deployed and running for all visitors

You can use the "sandbox" approach even if you do not have separate deployment environments for your application. One common use case is to toggle between the local and production files for debugging. For example, you could load the local file when a certain “test” cookie is enabled, when a certain URL parameter is attached (like ?debug=true) or based on some other check that matches your internal workflow.

Three-stage testing

The three-stage testing use case involves these separate, sequential environments:

  • Local: code is running only for the developer who is writing it. You do not need to create a separate Local environment for each developer, just one environment where the experiment will run during development.
  • Staging: code is shared with the larger development team and QA.
  • Production: code is deployed and running for all visitors.

Example test environment scenario

Imagine you have three experiments: sort-by-name, show-subtotal and warn-empty-cart. You set up a new environment named Testing so you can test your experiment code before you deploy it in the Production environment. Suppose that:

  • You completed testing for the sort-by-name experiment and deployed it.
  • You are currently testing the show-subtotal experiment.
  • Your warn-empty-cart experiment has been tested but is not yet deployed.

Here is an overview of the statuses of your three experiments in your two environments:

ExperimentTesting environment activity state and datafile statusProduction environment activity state and datafile status
sort-by-namePaused*Running
show-subtotalRunningNot started
warn-empty-cartPausedNot started

You can keep experiments running in your testing environment that are also running in production if you prefer.

Now, let us imagine that the project owner creates a third environment: Staging. Optimizely will generate an additional datafile named Staging. Every experiment will be in the “Not started” activity state in Staging, and the Staging datafile will show the experiment status as “Not started.”

When your experiment code is fully functional and ready to test, switch it to the Testing environment by pointing your code to the Testingdatafile.