Why AB Test If You Can ABCD Test?

If your education was anything like mine you’ll be well versed in the Scientific Method: coming up with a hypothesis, an experiment to test it, and making sure to control the variable’s one at a time against a control so you can isolate cause and effect. But the real world is messy with lots of different variables to test all with lots of different values to test for each one.

Take the classic A/B test example of optimizing a purchase button on a website. There’s at least 4 easy variables to tweak: color, size, position in the container, and the font. Each of those variables has at least 3 different values to test like small, medium, or large for size and regular, bold, or italics for font. I’m sure you can think of a ton more things to tweak even in this very simple example but just with 4 variables and 3 settings that’s already 3^4 = 81 different tests you’d need to run to test everything in isolation! What if I told you that in just 9 tests you could get almost all the information you need?

Enter the orthogonal array (really a table), sometimes called a Taguchi Array after its creator Dr. Genichi Taguchi of Japan.

I was not taught about these at school despite their extreme usefulness but rather recently learned about them watching this YouTube video that does a great job walking you through what they are and how they’re used.

If you didn’t watch the video I’ll attempt to summarize here. Let’s go back to our button testing as an example but limiting it to 3 variables each with 2 values (settings):

Traditionally in order to measure what the best combination is you would have to test each of the combinations and then measure the results, say how many clicks the button got:

It already takes a moment to work out which variables are most responsible for the change, but what if we add back the 3rd setting to test? We’d have to then run 27 experiments! Instead of doing that, let’s take our simple example with just 2 settings per variable and change the experiment list so we’re changing 2 variables at a time which means we can cut the number of experiments we need to run in half:

Now we can sum up the scores of each variable setting independently and since the other two variables have different settings they effectively cancel each other out as if we were just testing one variable at a time.

So Red scores 5, Blue 7, Small: 3, Large 9, Regular 8, Bold 4. So straight away you can see there’s not much of a difference between Red and Blue, but there’s a significant change for both the Size and Font. Once you have that information you can then iterate on the experiment starting with your best results and seeing if you can do even better.

In a nutshell an orthogonal (Taguchi) array is one where:

• All variables should be tested an equal number of times
• All settings should be tested against all other settings equally
• Multiple variable are changed at once
• The effects of a variable can be determined across experiments by the canceled out changes in other variables

As you’ve seen, the biggest advantage of Taguchi tables are the ability to find the parameters with the highest effect across many variables and settings. If you had 6 variables each with 6 different possibilities you’d need 6^6 = 46,656 different experiments for the full combination space compared to just 36 tests in the Taguchi method! Think about if you had to run each test multiple times to average out experimental uncertainty/noise 😅

The tradeoff there is that the variables must be as independent (hence orthogonal) to each other as possible for this method to work. If two variables are dependent on each other and only produce results when both are set to specific values, your experiments will miss this. You also only can analyze the results as relative changes since you don’t have the full set of data to see the values change truly independently to the other variables.

If your test fits the bill though the tradeoffs are certainly worth the savings though!

Orthogonal (Taguchi) Array Generator and Analyzer

With that, here’s a web based Orthogonal Arrays generator with the button example filled out and some random numbers for the test results. Play around with the values and think how you might use this technique for your own experiments. If you’d like to start from scratch with your own variables and settings hit the “Reset Table” button to clear everything. Thanks to this generator for the logic in generating the experiment values.