Statistics is a science, but there’s an art to it, too. We use both to provide a clear picture of what the data really say.
There are two main types of research surveys — probability and nonprobability. Probability surveys, like those AmeriSpeak conducts, provide findings you can trust. That’s because they survey carefully created panels of respondents who accurately represent the population as a whole. The results of nonprobability surveys—which are typically conducted online—are less reliable because their panels aren’t carefully assembled or fully representative. But businesses often use nonprobability surveys for market research because they’re much cheaper. We’ve developed a statistical process that corrects the errors inherent in nonprobability surveys, bringing their results within the same margins of error as the gold standard AmeriSpeak survey.
The Survey-Taker’s Challenge
We have to get things right every step of the way.
How TrueNorth Solves Challenges
Opt-in panels—also called convenience, commercial, internet or nonprobability panels—have been studied intensely in the past decade. Most first-tier, peer-reviewed evaluations of opt-in panels conclude these panels do not produce consistently accurate information and often lead to acutely inaccurate findings. Using such panels is like playing Russian roulette: You will never know for any given benchmark whether it is accurate or not. This might be acceptable for less important research, but not research meant to inform major business or policy decisions. TrueNorth is specifically designed to provide a kind of “research insurance” for this “variance of error” that can make opt-in panel surveys inaccurate.
TrueNorth blends opt-in data with high quality probability data from NORC’s probability-based AmeriSpeak® Panel. It uses highly sophisticated machine learning to combine the samples in a way that minimizes any bias in the opt-in sample, question by question.
TrueNorth takes a bit more time than a simple opt-in sample. This is due to needing to attain a high response rate probability sample and the statistical work needed to combine the samples. Overall a TrueNorth project can take as little as two weeks, but four to five is typical.
The key to all survey representation is randomization. Probability panels essentially stir all addresses together and then sample the mix. Consider a bowl of tomato soup, all the ingredients are mixed together so that every spoonful is the same as the next. Opt-in samples allow anyone to join, and typically reach out to members of partnering organizations. For example, an opt-in survey may invite every member of an airline’s miles program. The thinking is that if you simply add enough people, your sample will start to cover everyone. But in practice, this is never true, and the results show it. For example, questions on travel in particular can be quite skewed because of this leveraging of miles program members to become survey panel members.
Probability sampling is more expensive than opt-in sampling, and our statistical process, while highly automated, still requires skilled people to execute them and evaluate the results. So TrueNorth is more expensive than opt-in sampling, but at the same time, TrueNorth is 25% less expensive than studies using exclusively probability samples.
TrueNorth reduces approximately 2/3 of the bias inherent in opt-in samples, and often as much as eighty percent.
TrueNorth relies on probability data from AmeriSpeak to calibrate the opt-in data. TrueNorth looks at every question in a given survey, assesses the bias in the opt-in data, and creates weights that holistically adjust all data simultaneously. As such, no single metric gets forced to be anything specifically, but rather the opt-in data across all questions gets an overarching best-solution adjustment.