Proof that TrueNorth can fix defective consumer research
Opt-in panels (also called convenience, nonprobability, or commercial panels) are often a popular option for market research due to their low cost. However, their surveys are taken by groups of respondents who don’t necessarily represent the entire population, leading to inaccurate results.
To measure how far off the mark typical opt-in surveys are—and to prove that TrueNorth could adjust deficient findings to bring them in line with results from a probability survey, which are more reliable because they are more representative—we did an experiment. The short answer: opt-in surveys are routinely and sometimes wildly inaccurate, but we can make them better.
Our first step was to create a baseline. We did that by conducting a survey using a probability-based AmeriSpeak® Panel. AmeriSpeak, which was created by NORC at the University of Chicago in 2015, has earned an industry-leading reputation for capturing a true picture of the nation, with a random sample of people who look almost exactly like the U.S. Census.
For our inquiry, we asked 1,000 AmeriSpeak respondents about their purchases of 42 consumer items like beverages, energy bars, and personal-care products. The closed-ended questionnaire took about five minutes to complete.
We also sent the same questionnaire to a total of 2,400 respondents from three online opt-in panels. Two of the panels were proprietary opt-in panels, which solicit participants via ads, pop-ups, and corporate partnerships. The third was an aggregator, which collects responses from panels assembled and maintained by others. Each provider was asked to replicate the breakdown of the latest Census by balancing respondents by age, gender, race/ethnicity, and other demographics.
Not one of the nonprobability surveys came close to getting things right. They consistently overstated how many consumers purchased a product, even though they had weighted their samples for key demographics. For example, 21% of Americans bought Sierra Mist in the previous 12 months, according to the AmeriSpeak research. The less rigorously constructed panels found that 30% did.
The opt-in surveys also misjudged who bought these retail items. Looking at nail-care products, the AmeriSpeak survey reported that 17% of buyers were 18 to 29 years old and 31% were 55 and older. The nonprobability questionnaires exaggerated the size of the younger group, finding they made 27% of purchases, and undercounted the older group, also putting their share at 27%.
Altogether, the gaps between the AmeriSpeak benchmark and the nonprobability samples were statistically significant for 80% of purchases by young adults, rendering the nonprobability data worthless.
Nor could a brand know which survey provider might be thoroughly reliable. In our test, Sample B had the lowest error rate in tracking purchases by the under-30 group. But Sample B was also the worst in measuring how many shoppers had picked up soft drinks.
By calibrating these findings with TrueNorth, we can wring out the built-in bias of these nonprobability surveys. Greatly simplified, the process goes like this:
The contents of the probability and nonprobability samples are entered into a machine learning algorithm that determines, for every answer of every question, how much they differ. We call this spread an “importance metric.” Then the program uses these importance metrics to find a single adjustment algorithm that is best able to minimize the bias across every question.
The weighted results are combined into a single sample.
We were able to salvage the inexact findings of the opt-in surveys and produce credible results from a pool of respondents more than three times as large. We dropped the error for Sierra Mist purchasing to 1.3% from 8.6%, and for all 42 consumer products to 1.3% from 6.3% That’s well within the acceptable margin of error.
We also fixed the findings of the nonprobability surveys by subgroups by, for instance, cutting the error rate for purchases of nail-care products by half.
For nearly two out of three cases where the nonprobability results were statistically different from the probability results, we aligned the results with the AmeriSpeak standard. Our solution proved to be a better and more reliable option than depending solely on the too-often-wrong estimates of opt-in panels.
To use TrueNorth for your next research study, email us at TrueNorth@norc.org