Two- and three-stage testing
This topic describes best practices for QA testing in a staging environment before deploying to a production environment.
Optimizely Feature Experimentation comes with two environments automatically:
Development. When selecting a flag, you are directed to the
Development environment rules.
You can add additional environments and change which environment is set as primary. See Manage environments for more information.
Remember that each environment has a different datafile, with which you will initialize the Optimizely Feature Experimentation SDKs.
You do not necessarily need separate environments to make use of Optimizely Feature Experimentation's Environments feature. You can, for example, create an environment for users with a test cookie. You will just need to implement the logic that would initialize Optimizely using the test cookie environment when the user has your test cookie.
Two-stage and three-stage test environments
Here are two examples of environment use cases. You can have an unlimited number of environments, and there are many more possibilities for using them than we have described here. Use as many environments as your workflow requires.
In general, the fewer environments you use, the simpler the workflow.
Two-stage “sandbox” QA
The two-stage “sandbox” QA use case is a common setup in web development and involves these separate, sequential environments:
- Local: code is running only for the development team and QA
- Production: code is deployed and running for all visitors
You can use the "sandbox" approach even if you do not have separate deployment environments for your application. One common use case is to toggle between the local and production files for debugging. For example, you could load the local file when a certain “test” cookie is enabled, when a certain URL parameter is attached (like ?debug=true) or based on some other check that matches your internal workflow.
The three-stage testing use case involves these separate, sequential environments:
- Local: code is running only for the developer who is writing it. You do not need to create a separate Local environment for each developer, just one environment where the experiment will run during development.
- Staging: code is shared with the larger development team and QA.
- Production: code is deployed and running for all visitors.
Example test environment scenario
Imagine you have three experiments: sort-by-name, show-subtotal and warn-empty-cart. You set up a new environment named
Testing so you can test your experiment code before you deploy it in the Production environment. Suppose that:
- You completed testing for the sort-by-name experiment and deployed it.
- You are currently testing the show-subtotal experiment.
- Your warn-empty-cart experiment has been tested but is not yet deployed.
Here is an overview of the statuses of your three experiments in your two environments:
|Experiment||Testing environment activity state and datafile status||Production environment activity state and datafile status|
You can keep experiments running in your testing environment that are also running in production if you prefer.
Now, let us imagine that the project owner creates a third environment:
Staging. Optimizely Feature Experimentation will generate an additional datafile named
Staging. Every experiment will be in the “Not started” activity state in
Staging, and the
Staging datafile will show the experiment status as “Not started.”
When your experiment code is fully functional and ready to test, switch it to the
Testing environment by pointing your code to the
Updated 3 months ago