Editor’s Note: It is one to say theoretically that having multiple types of information at one’s disposal can lead to improved decision-making, it is another thing sometimes to actually have to live with the consequences. You may find that different types of information may lead to the same conclusions, or you may find that they conflict with one another. If the latter, what is a humble researcher to do? In this thought-provoking piece, Olivier Tilleuill discusses the implications of contradictory findings, and that there may actually be a real business benefit to finding them. A really interesting read.
Here’s a potential scenario – you did an extensive advertising study and found an interesting pattern in the correlation matrix:
- Behavioral KPIs (Visibility, Attention, and Emotional reach) are correlated among themselves
- Survey questions (Recall, Likability, Brand fit and Social media fit) are also correlated
- BUT behavioral KPIs and survey questions do not correlate as much!
What does this discrepancy mean – how do you make sense of it?
Let’s get back to the basics first.
Back to the Basics
In statistics, correlation is any type of relationship between two variables. A positive correlation means that if one variable goes up, the other variable goes up, too – a negative correlation meaning the opposite.
A strong correlation in your behavioral KPIs for a single ad means that if many people noticed your ad, it was more likely to have high attention and emotional reach as well. On the other hand, your ad might perform well in a survey that measures the conscious attitudes of the respondents – they really liked the ad, found it to fit the brand, and could remember which brand it belongs to afterward.
But when you compare the behavioral and conscious KPIs, you find that their correlation is low – if an ad did great on the subconscious variables, it doesn’t mean it would perform well on the survey, and vice versa.
There are three possible conclusions about this:
- Either that surveys are not relevant
- Or that behavioral research is not relevant
- OR that both are valid, complement each other, and significantly increase the predictive power.
Conventional and Behavioral Methods can Have Conflicting Results
If behavioral and conventional methods have a low correlation, from a statistical point of view, this means that they measure different things; and sometimes (not always), they can have conflicting results.
Behavioral KPIs show the actual measurement of how the ad is perceived in the appropriate context. It relies on the underlying mechanisms of our perception, that have a different set of prompts and cues that help us notice things. That’s why ads need to attract attention and get seen first, even to have a chance of relaying a message to its intended audience.
A survey, on the other hand, lets us take our time, evaluate the ad consciously, giving us an account of the ad’s strong and weak points, and it’s potential if the ad is noticed.
Uncorrelated Data is Actually Useful – it Provides a Complete Picture
What do we mean by saying that both sets of KPIs are relevant? They are useful for the ultimate goal of all research – predicting effectiveness.
If you are conducting research and all variables in a group are 100% correlated, that means you only need to have 1 point of data to predict all of the others – essentially, that all the different variables you are testing do not add value.
To understand and predict what will happen, having uncorrelated variables like this is a blessing in disguise. The complementing measurements will paint a more complete picture in your study, and bring about better predictive value (or a higher R2) – assuming that both variables are important.
For example, if your survey results show that the likeability for the tested ad is high, then the correlated variables (e.g., brand fit, channel fit) might get you an accurate and nuanced opinion about the ad, but each new question will not change the conclusion about which ad is the best – they will all score similarly high. Instead, if you introduce an entirely new uncorrelated dimension, e.g., is your ad seen in the social media environment, you get much more valuable info that might influence your decision – because if your ad is not seen, it will not create an impact.
Too Complex to Explain to Internal Stakeholders – So Let’s Just Skip it?
Doing this would be extremely wrong. Many studies show that combining behavioral and conventional measurements increase predictive power by at least 40%. Not measuring the behavioral side is the same as ignoring essential data. Not obtaining the data does not change that fact – and it will arguably be worse, as you do not know just how bad the stimuli might perform on that particular KPI.
To summarize: behavioral and conscious variables are correlated within each group, but they don’t necessarily correlate with each other. They measure different variables that complement each other and provide a big picture of your ad or product, thus enabling you to predict its effectiveness much better.