Audrey Messer, founder and chief analytics officer of Centricity Analytics, explains how bias can be tackled. She also explains how organisations can implement analytics.
Bias
exists and needs to be addressed. Many practitioners in the field do not
necessarily acknowledge that. They think that since they are working with data
and models, the output is then ‘pure’ or free from bias. These are the
observations of Audrey Messer. “In the past few years more and more instances of bias, from
the benign to the possibly illegal, have been revealed. The model’s biased
output generally starts at the beginning with the training dataset and control
group selection.”
Black box
For many companies a key challenge with AI and machine learning is that it is opaque and ‘black-boxy’. As these complex models are used to drive and impact the business it is crucial that internal stakeholders, like executives, sales teams, and even lawyers, need to be involved. “If they don’t understand it, they will have the power to veto its use. Processes will need to be put in place to test for bias, to review whether data is being used properly, according to GRPD, CCPA and HIPAA guidelines.” In order to tackle bias, Messer recommends the following:
1.
The
new world really needs to accelerate and address the fact that there is bias –
intentional or not.
2.
Selecting sample sources and selecting representative training sets are equally
important. “Select incorrectly and the results can drive incorrect, business-impacting,
and in limited cases illegal outcomes.”
3.
In market research a study can be derailed by other factors than just poor
sample selection. “The way the questions are asked, when/how the survey is
deployed and the types of bias of the human researcher all influence the
outcomes. Similarly, be aware that machine learning and AI are vulnerable to
bias in other areas of the model’s design, development and deployment.”
Fighting data with data
Having
worked for Metlife, one of the largest global insurance providers, Messer has
experienced up-close the insurance industry’s transformation from a product focused
strategy to one of customer centric focus. She describes insurance executives as
data driven decision makers, with many of them coming up through the actuary
ranks or with a sales agent background; professionals who always had clear KPI’s.
“The source of misconceptions was that either they had seen data from five to
twenty years ago, or they just hadn’t any data to suggest an alternative. So we
found success in fighting data with data; setting up experiments that had clear
goals, hypothesis and action standards; and being patient in validating models,
having conversations about concerns or beliefs.”
As
for
CLTV (customer lifetime value) models, a crucial tool for insurers to
understand their clients, Messer identifies the following pitfalls:
1.
Small
historical datasets: “For a new company, when is it a good time to leverage the
customer data to calculate and implement the model?”
2.
Predictions are not set in stone: “This is more about the implementation of the
model. When used incorrectly, CLTV is used to justify a self-perpetuating model
of only caring about high-value customers.”
3.
Later acquired customers generally do not look the same as initial customers
(early adopters): “As customers come in from a range of sources, CLTV can
change in unexpected ways, particularly if it is being calculated as just one
number. Generally, to help uncover the nuances of CLTV, building out segments
and cohorts can assist in overcoming this.”
Keeping it simple
Regardless
of the segment, it’s Messer’s strong conviction that analytics should always
drive business decisions. “Yes absolutely, unequivocally and indisputably. Many
companies and executives think that analytics is just a fancy buzzword. They
either A. steer away from it because it is seen as too expensive, difficult or
complex, or B. tools and processes have been adopted without a strategic
alignment and clear problem definition.” She offers solutions for both
scenarios:
A.
“There are some foundational analytics that can be incorporated using a simple
tool like Excel. You don’t have to start with AI/machine learning. For
instance: start with some simple descriptive analytics that describe what
happened, and then create simple charts that show yearly trends. Good
visualizations will identify areas for further investigation. Gaining trust and
incorporating data into decisions will help advance a stronger analytics team
in the long run.”
B.
“There needs to be a strong relationship – trust, communication – between the
analytics and executive teams. Do the executives truly understand what they
will get from an analytics model; what types of questions it can and cannot
answer? Is everyone even aligned on what AI/machine learning is? Is there
alignment on what decisions the ‘computer’ will make versus the decisions a
person will make? If there are risks in the model, has there been a
conversation about how to mitigate those risks?”
Three challenges
When
it comes to the biggest barriers for using analytics within organisations,
Messer identifies three key areas:
1.
Lack of alignment: “The business goals need to drive the analytics and research
problems. Many times, I have seen solutions go in search of problems. This can
be dangerous; it can create distrust between teams, and for executives it
devalues what analytics can offer. I also talk a lot about alignment of KPI and
metrics definitions. I have seen it create a lot of confusion; do the
executives really understand what a key performance indicator is versus, say, a
vanity metric? Are all team members aligned on key definitions, like retention?
Is the formula consistent across the business? Analytics leaders need to be
communicating, building relationships within the organisation, and be clear
about the problem the analytics model is looking to solve.”
2.
Access to data: “This can range from not doing an analysis because the data
isn’t perfect, to having the data but it isn’t quite in useable form. Issues
with data quality can also be in this bucket. Furthermore, is the data
comprehensive enough to provide a fair result? A solution is to start small;
good analytics work creates more analytics work when done correctly.”
3.
Access to talent: “We need analysts that have both strong technical and
interpersonal skills, which is difficult to find. One solution is to partner
with local colleges and universities. Once you find a great analyst, keep him
or her in your network.”
Finally, as a woman in a male-dominated industry, one area of
bias that Messer talks about frequently is the human bias insights
professionals bring to their work. “Our community can enrich itself through
diversity of thought, background and experience, so that sources of bias can be
identified at the get-go.”
About Audrey Messer
Until recently, Audrey headed up global marketing analytics for MetLife. Now she runs her own analytics consulting company, Centricity Analytics, LLC. “For me running my own firm provides a level of freedom to focus on the type of analytics I am particularly passionate about – customer analytics – as well as be able to work with fast growing companies and firms that need help in building out and unlocking the power of analytics.” While there are no typical engagements, Centricity Analytics focusses on solutions such as customer lifetime value, segmentation modelling, customer journey analytics amongst others in both B2B and B2C business models.