Using Test and Control Groups to Measure Marketing ROI

It’s much easier to ask what results marketing delivers than it is to find the answer. There are several challenges: from knowing when to measure to dealing with multi-touch attribution and influencers. These factors and others make it difficult to measure the contribution that a given marketing activity has on revenue. As John Wanamaker famously remarked, “Half of the money I spend on advertising is wasted; the trouble is, I don’t know which half.” But just because measuring marketing ROI is hard doesn’t mean it’s impossible. Various methods exist to provide marketers with insight into the effectiveness of their different programs.

Let’s walk through one particularly effective method: using test and control groups. Testing the effectiveness of a new initiative against a well-formed control group is a great way to measure its real impact.

Here are two things to keep in mind:

1. To use this method, you need to design your programs to be testable from the outset.
2. You can measure almost anything with the right design, but it’s prohibitively expensive to test everything.

Designing Your Test

The outcome metric—or what you measure—can be revenue, profit, leads, conversion rates, or just about anything else you might want to measure.  This flexibility is useful for programs that have less of a directly measurable impact on revenue.

Here are some places you can start:

Programs & tactics: See if that webinar made a difference in engagement

Messaging: Examine which message and/or copy resonated most with your target audience

Contact frequency: Determine how often to send an email or other communications

Spending levels: Evaluate the impact of changing your campaign budget

You can measure combinations of touches rather than just single touches as well, which is a great way to test and compare different lead nurturing tracks. Also, remember that you’ll need to make sure that the difference between your control and test group is statistically significant—80% confidence should be enough.

Putting It into Practice

Once you have your test variable and outcome variable selected, it’s time to create your groups. To do this, apply the treatment to one component of your target buyer group. Say, for example, that you want to measure the impact of one of your brand advertising campaigns on target awareness. One potential approach would be to split your market into two equal geographic parts, and double campaign spend for one group but not the other.

Then compare the outcomes in the two market segments to analyze campaign effectiveness—did the increase in spend lead to improved results in the test group over the control group? Assuming all other marketing and sales influences on the two groups are the same, you can credit any difference in search or other awareness-related outcomes to the boost in spending. If there are differences in group influences, however, then you will need to change your criteria for group creation. In this situation, you should create your control group via random sampling to avoid confounding variables. 

Pros and Cons of Test and Control Groups

Final Thoughts

Using test and control groups is not flawless, but it is still a valuable way to measure marketing ROI. Some marketers can be hesitant to deploy them because they don’t want prospects missing out on certain treatments, whether it be increased ad spend, or receiving a series of emails. But think of it this way, would you rather have only some people receive the right message, or everyone receive the wrong one?

Without the use of control groups, your default assumption is likely that more marketing actions lead to increases in desired outcomes when the opposite could be true. For instance, a certain group might react negatively above a certain level of communication, leading to decreased engagement and/or spend. But you wouldn’t know that unless you tested it.

Test and control groups are an important method for marketers to learn more about their customers. Ultimately, being fearless when it comes to the testing and measurement of marketing activities is what will set you apart from the competition.

What testing methods do you use to measure ROI? Tell me about it in the comments to keep the discussion going.