geralt / Pixabay
A common refrain from the conversion rate optimization world can be heard after a failed test:
“Well that didn’t work. What’s next?”
It is easy to assume that your testing strategy is flawed when you can’t find wins. Failed tests, the ones in which the control version outperforms the new version, don’t have the same feeling as successful tests.
However, we need to avoid the notion that failed tests are a waste of time. In fact, you can learn just as much from a failed test as a successful one. And in that way, all tests are successful tests.
The Testing Cycle
A typical testing cycle starts with analysis. Your team will look at the data to figure out where customers are getting lost, where they are falling out of the funnel. And you will identify the critical pages that you need to improve in order to improve your conversion rate.
Step two after you have identified where is to try to identify why. Why are customers falling off at this point? What about this page is turning them off? For that, you will want to collect qualitative data, if possible.
Once you have your hypothesis, your team will work to design solutions to the problem. They may be as small as changing a word on the page, or changing the color of a button. Or they could be as big as eliminating a step from the process, or designing a new page from the ground up.
Recommended for You
Webcast, December 18th: Optimizing Customer Engagement with User Intent
At this point you have your test version and you have your control version. Now it’s time to define success. Before the test starts, your team must come to an agreement on how you will measure the results. It’s critical that you do this early in the process, because once you start testing there will be times when you’re tempted to read into the results based on prior assumptions.
Then it’s time to start the test and collect the data.
The End of the Testing Cycle
Once you have collected the data, it’s time to declare a winner. If the new version wins, everyone celebrates. You correctly identified a problem and designed a solution that outperformed the original. Congratulations all around!
If the control version wins, everyone is disappointed. But you have learned something. Perhaps the solution that you designed did not solve the problem. Perhaps the problem identified was not the correct one. Or perhaps the solution did solve the problem, but created a new one.
Failed tests require difficult analysis. You need to ask yourself difficult questions, going back to the hypothesis and the solution and evaluating where to go next.
But you don’t have to start from scratch. You now know for certain that your proposed solution was not the right one, which means you are one step closer to finding out what the right one is.
The end of the testing cycle is actually just another step in the process. We call it a cycle for a reason, the process keeps on going. If you have a winner, you move on to the next problem you can solve. If you have a failed test, you need to move forward with a new solution.
