When setting up customer experience (CX) programs our clients often ask, “How big of a sample do we need to get representative data”? Like all good researchers our answer is usually, “It depends”. While technically true, that answer is not always helpful. Below I will argue that for most CX programs you probably don’t want to sample at all. Rather, you want to conduct a “census” that surveys all your customers (or at least all those you can).
A Quick Review of Sampling
First, let’s talk about the purposes behind sampling. Later, we’ll explore why those purposes are often no longer applicable to modern CX programs. The purpose of sampling is to select enough people (but not too many) and the right people to accurately represent a population within a given level of precision. In CX research, the population of interest may be all the customers in a company’s industry, the company’s current customers, a particular store’s customers, or anything else. Interestingly, the absolute number of responses needed to represent a population within the same margin of error doesn’t change in proportion to the size of the population. For instance, to represent a population of 300 (e.g. a small town, a store’s customers, etc.) with a precision level of +/- 3 percentage points, you need 235 responses (78% of the population). However, a town ten times that size (3000) only requires 788 responses (26% of the population) and a city of 3 million only requires 1,067 responses (.036% of the population) to achieve the same precision level. The important point here is that as the population size gets smaller, you have to sample a much much larger percentage of the population to get the same level of precision. We’ll come back to that later.
Another very important aspect of sampling that many people forget about is choosing the right people to survey. If your sampling is biased in any way you will not get a representative sample, no matter how many responses you have. Probably the easiest way to think about this is to use an example. If I am conducting a survey about any political issue to determine what Americans (the population) feel about it, I could sample a million Democrats or a million Republicans but my results would not represent Americans. Instead, I need Democrats, Republicans, Independents, and people of other political persuasions in the sample; and I need them in the sample in their proper proportions. The most common way of accomplishing this is to randomly select people from the population to put into the sample.
Why Sampling Might Not Be Appropriate for Modern CX Programs
As mentioned above, sampling is usually done to select enough, but not too many, respondents. The reason most programs want to limit the number of responses they get are two-fold. First, they want to limit costs and second, the additional responses beyond a certain level provide very little incremental value in terms of representing the population. Let’s look at each of these points in CX terms.
Recommended for You
First, cost containment as it applies to surveying used to be much more of an issue when survey methods were expensive. Many programs now rely on email and other very inexpensive electronic survey methods, so sampling for the sake of keeping down data collection costs is not much of an issue. However, if you are in an industry or part of the world where email penetration is low, you might need to rely on more costly methods of data collection including telephone, mail, and in-person interviews. In these cases, sampling is certainly appropriate.
Second, it is also no longer true that additional responses beyond a certain level provides very little incremental value. This is because the goals of many, if not most, CX programs go beyond trying to derive scores for a population. Many CX programs focus on customer recovery. They try to identify at-risk or upset customers and resolve whatever problem they are having. This individual approach to CX requires that as many customers as possible are contacted to determine if they are at-risk. In statistical terms, each and every customer is now “the population of interest.”
Even if your program is not focused on customer recovery, a final issue that often makes sampling inappropriate is that most CX practitioners want their programs to produce reliable results down to lower levels of the organization (e.g., the retail outlet, bank branch, perhaps even the employee). In these cases, the population of interest becomes the lowest level of the organization. As mentioned above, when population sizes get small you need to sample a very large percentage of that population to get representative results. Therefore, to obtain sufficient responses at these low levels of the organization, you need to survey as many customers as possible. It is also important to note that you need to take into account response rates when considering whether or not to sample. In many cases with small populations, you might not be able to obtain your needed number of responses for a certain precision level even if you conduct a census because of the response rate.
A final note just to avoid any confusion. Sampling is different from sample cleaning. Even if you decide not to sample your data, you will still probably want to clean the sample. Not only should the sample be cleaned for things such as duplicate or incorrect email addresses, sample rules should be put in place to avoid contacting the same customer too often.
Not Dead Yet
Monty Python fans might recognize the quote, “Not dead yet,” from Monty Python and the Holy Grail and from Spamalot. That quote applies pretty well to sampling in CX programs. “Not dead yet” but likely to so be soon.