Tests are always able to be conducted with more than just two variants. This is known as a A/B/n test. These tests allow you to measure the performance of three or more variations rather than just testing against one controlled variant.
The more traffic to a website, for example, the larger the size of the data pool. The larger the size of the data pool, the more accurate the results will be.
When looking at most A/B/n tests, there is always going to be a recommended limit as to how many changes or variants there are within a test. It’s always best to make a few critical changes in the way of determining the best element to run for the majority of the environment going forward.
However, when multiple changes within a test are needed, one can run what is known as a multivariate test.
Multivariate testing is a technique for testing a hypothesis in which multiple variables are modified, compared to the traditional 2-3 in an A/B or A/Bn test.
The overarching goal of multivariate testing is to ascertain which combination of variants performs the best.
This is best done on websites or mobile apps where elements are dynamic and can be easily modified. This might be changing something like a headline, while also combining these with changes in the variations of content. These tests are then analysed and the best performing element is then put in use on a larger scale.
V1 - A (image) + A (widget)
V2 - B (image) + A (widget)
V3 - A (image) + B (widget)
V4 - A (image) + B (widget)
The process of running a multivariate test is similar to A/B testing, but a key difference being that A/B testing only tests 1-2 variants against a control. A minimum of one variant is tested to determine the effect of a change to one variable. When looking at a multivariate test, multiple variables are tested together, as seen in the example above.