Is A/B testing dead?

“Today, A/B testing is thriving — it’s been a huge improvement from non-A/B testing,” said George Khachatryan, CEO of AI company OfferFit, in a recent webinar. “At the same time, the people performing these tests every day recognize that’s it’s a lot more difficult than it may seem.”

Designing A/B tests, determining samples sizes, and deploying them takes up a lot of time and resources, and analyzing the findings requires high levels of precision. All in all, the manual tasks required by A/B testing can place a heavy burden on marketers.

“When you’re running a full experimentation program, it’s never enough to run one A/B test,” Khachatryan said. “When you run one, you gain valuable insights, and inevitably want to gain more. So you end up running more.”

He added, “Those who are doing this hands-on realize it just becomes a rapid explosion in the number of tests they need – it becomes infeasible very early on this exponential curve.”

Marketers need a solution that allows them to test a growing number of campaign variables while simultaneously giving them enough time to analyze the data. Fortunately, A/B testing is evolving.

Expanding the power of A/B testing

In the webinar, Khachatryan highlighted the “multi-armed bandit problem” that’s affecting modern A/B testing. In the traditional version of this scenario, a person at a casino must determine which slot machines (the “one-armed bandits” that steal your money) are going to have the best payouts, then figure out which order will be optimal. With A/B testing, the variables are the multi-armed bandits, and the marketer must discover which are most effective so they can allocate more resources to the areas performing well.

“You can think of [a multi-armed bandit] like a smart A/B test,” he said. “It’ll navigate the exploration-exploitation tradeoff — it’ll start randomly pulling those ten handles, but as it goes, it’ll dynamically reallocate resources so that if something looks bad it’ll stop pulling.”

He added, “These multi-arm bandits are designed to experiment just the right amount so you’re learning but also taking advantage of what you’ve already learned.”

While these multi-armed, or A/B, models have served marketers well over the years, there’s a new iteration of the framework that is more accurate and effective. According to Khachatryan, these are “contextual bandits.”

“It does what a multi-armed bandit does, but it takes into account different contexts,” he said. “So, if you have two different customers, with different characteristics, it’ll know to pull different levers.”

Contextual bandit frameworks are essentially automated experimentation and personalization at scale. It’s a model that can completely automate the process, and it’s what every marketer should be moving toward to improve campaign effectiveness at scale.

Automated experimentation is the future

Many of the tech giants have already adopted contextual bandit frameworks, but marketers should note that this technology is still incredibly new. Brands should allocate enough time and resources to make the transition process easier, because, according to Khachatryan, it’s the “future of experimentation.”

“In the past, manual A/B testing works with one mention at a time,” he said. “With these contextual bandits, you can set it up to simlutantous test multiple dimensions.”

Whether it’s testing email subject line efficacy, call-to-action click-through rates, or optimal article posting times, marketers have a lot of experimental data to keep up with. Automated testing solutions can make these processes more manageable by decreasing the time spent on manual tasks, replacing them with continuous automated experimentation.

“You can think of this as the next iteration of experimentation, or test-and-learn programs,” Khachatryan said. “When a marketer sets up this system of continuous automated experimentation, it creates this interplay where you can see what happens, gain insights, and then use those insights to get new ideas.”

He added, “So you still have the agile test-and-learn cycle, but it’s accelerated.”

Time will tell how quickly marketers adopt these automated experimentation technologies. But, with the high level of marketing technology replacements that took place over the past year, there’s a good chance more brands will sign on sooner rather than later.

Watch this webinar presentation at Digital Marketing Depot.