Website A/B Testing and Unknown, Unknowns

Whenever I consider risk, I suspect that unknown unknowns do not get enough of my attention. I step back and think, “what might happen that I haven’t thought of?” And always my intuition responds: good luck answering that question properly.

Known unknowns tend to dominate human risk analysis. For instance, I might know that I don’t know whether some action will cause my friend to initiate a blood-feud against me. In this case, the notion of “blood-feud” passes through my head when I consider a course of action, but I don’t know for certain the likelihood of its outcome. This is a known unknown, uncertainty with respect to a certain chain of causation.

Unknown unknowns have a more slippery quality, and in the general case are intractable. To know something presently unknown to me, I must think of it. Yet I cannot simply “think of” an infinite number of unknown consequences when considering a potential action. The human mind doesn’t work that way, and the domain of daily life is too complex. But are unknown unknowns always so impractical? Not necessarily.

Consider the well-defined domain of A/B testing, which nonetheless falls victim to the same biases which crop up in everyday decision-making. Most forms of website A/B testing conduct a search over the known unknowns. You select some feature which you think may affect a metric, and tweak it to see whether your hypothesis is true. This is empirical, sound, and good. But why not also search over the unknown unknowns? In this domain, at least they are enumerable.

A site likely has features that its creator has not considered, but which when adjusted, may positively affect conversion rates. You might never have suspected that increasing the font-size of your call-to-action by 1% would lead to 5% more sign-ups, or that brightening the hue of your button by 3% would result in 7% more click-throughs. The space of small permutations to an existing website design is very large — too large for a human to consider, but not too large for an algorithm.

So stay tuned. And if this interests you, by all means get in touch.