Thoughts and Insights from 2B

A/B Testing Report. How to Create Change Through Executive Reports

In the years that I have spent in presenting data to C-Level clients, I have learned that data alone does not create change. Nobody cares about data, they care about knowledge. Feeding data and numbers to people is not the same as giving them knowledge. As Sir Francis Bacon said, “Knowledge itself is power” and nothing drives change faster than knowledge. Knowledge empowers people to stand up and speak out. When you give people knowledge you are giving them reasons to listen to your recommendations, as well as the tools to implement them.

A/B Testing Information Data

A/B testing is no different. Testing give a myriad of data that can be overwhelming to the leadership. These data can include:

  • The page being tested
  • The section or element of the page
  • What the exact change is
    • Why is it changed?
    • What is the hypothesis?
  • Sample Size
  • Customer or traffic segment being targeted
  • Percent of traffic seeing the test (50/50, 75/25, etc.)
  • Start Date
  • End Date
  • Relevant KPI’s
    • percent of change
    • level of significance
  • Incremental Revenue gain (or loss)
  • Recommendations

This is a lot of information that is not easily presented in a way that empowers the leadership to create change. Especially if there are multiple tests, it is overwhelming and full of nothing but numbers (remember, we analysts love numbers but not everyone else does).

So how do we present this data in a meaningful way? Many companies are just starting out in testing and have no reporting or tracking in place, so they try to fit testing into existing marketing documents. I have seen marketing spreadsheets used that track all website marketing spots on a daily basis, and the tests stuffed into that same document. It created an unreadable nightmare. At any one point there could be any number of tests running as well as tests that were successful and pushed to 100% until they were implemented by IT. This meant that to fit the information into an Excel spreadsheet the font had to be extremely small, and every activity running had copied to every cell, every day. The daily cells grew exponentially. I knew there had to be a better way.

A/B Testing and Reporting

After doing some research on the web, I found someone who had used Excel to create a roadmap that  would present all the information needed, but that wouldn’t grow so large that it would become unreadable. I used their example, modified it to fit my needs, and had a great way to track tests for myself and the people who needed the data (check back for a post about this spreadsheet). However, it was still too much information to present to the leadership. It was information overload to anyone not involved in testing on a daily basis.

Then I found a post by the personalization software company Monetate on creating better reports. They scaled down the data until it fit on one slide. This is their executive summary of the test.A/B Testing Executive Summary

The slide shows where the test took place (black bar at the top), includes images of both versions (20% and $20), reinforces it at the top of the slide with the hypothesis question, adds a winner badge to the winning variation, includes the two relevant KPI’s with their values, and then adds a next step at the bottom. It is a lot of information, but they did it in such a way that  executives can see at a glance what was different, what changed and which one won.

This is the kind of A/B testing report that will create change within an organization.