A/B Testing Won't Make Your Org Data-Driven

When comparing A/B testing products you’ll see claims, usually in marketing material, that A/B testing will enable or enhance "data-driven decision making" in your organization. That phrase refers to some ideal org state where decisions about what goes into a product are made purely via conclusions drawn from hard data.

But be realistic about how decisions get made in your org today. You already have a process, and a new tool won’t change that. Especially when the output of that tool is yet another chart that is subject to interpretation.

Lee Maxey makes a good case for preferring the phrase “data-informed” over “data-driven”:

My core issue with the term ["data-driven"] is that it establishes unrealistic expectations for most businesses. Sure, the corporate giants of the world (oil companies, banks, media conglomerates) have visibility into massive amounts of data and can make precise decisions that are truly data-driven. But the vast majority of organizations [...] draw from much more limited data. At most, data is a tool that can, along with other tools like observation and experience, guide the inferences that lead to better decisions.

“Data-informed” is definitely a more realistic way to think about the impact of tools like A/B testing platforms. Even so, be mindful of where in the decision-making process experiment data comes into play.

Say your team has decided to improve some metric in your product like total purchases.

Based on trends in user behavior you’re seeing today + intuition derived from experience, you probably have a handful of ideas on exactly what should change in your product to reach that goal.

In this situation A/B testing is a great tie-breaker. You have several ways you can go that all seem plausible, and a properly implemented experiment can help predict the relative impact of each.

Experimental methodology means extrapolating something about a generalized population from a randomized subset.

It’s not data-driven decision-making, it’s data-informed prediction. There’s lots of setup needed, but it’s a powerful tool.