Proof that TrueNorth can fix defective consumer research

Our first step was to create a baseline. We did that by conducting a survey using a probability-based AmeriSpeak® Panel. AmeriSpeak, which was created by NORC at the University of Chicago in 2015, has earned an industry-leading reputation for capturing a true picture of the nation, with a random sample of people who look almost exactly like the U.S. Census.

For our inquiry, we asked 1,000 AmeriSpeak respondents about their purchases of 42 consumer items like beverages, energy bars, and personal-care products. The closed-ended questionnaire took about five minutes to complete.

We also sent the same questionnaire to a total of 2,400 respondents from three online opt-in panels. Two of the panels were proprietary opt-in panels, which solicit participants via ads, pop-ups, and corporate partnerships. The third was an aggregator, which collects responses from panels assembled and maintained by others. Each provider was asked to replicate the breakdown of the latest Census by balancing respondents by age, gender, race/ethnicity, and other demographics.

Not one of the nonprobability surveys came close to getting things right. They consistently overstated how many consumers purchased a product, even though they had weighted their samples for key demographics. For example, 21% of Americans bought Sierra Mist in the previous 12 months, according to the AmeriSpeak research. The less rigorously constructed panels found that 30% did.

The opt-in surveys also misjudged who bought these retail items. Looking at nail-care products, the AmeriSpeak survey reported that 17% of buyers were 18 to 29 years old and 31% were 55 and older. The nonprobability questionnaires exaggerated the size of the younger group, finding they made 27% of purchases, and undercounted the older group, also putting their share at 27%.

Altogether, the gaps between the AmeriSpeak benchmark and the nonprobability samples were statistically significant for 80% of purchases by young adults, rendering the nonprobability data worthless.

Nor could a brand know which survey provider might be thoroughly reliable. In our test, Sample B had the lowest error rate in tracking purchases by the under-30 group. But Sample B was also the worst in measuring how many shoppers had picked up soft drinks.

By calibrating these findings with TrueNorth, we can wring out the built-in bias of these nonprobability surveys. Greatly simplified, the process goes like this:

The contents of the probability and nonprobability samples are entered into a machine learning algorithm that determines, for every answer of every question, how much they differ. We call this spread an “importance metric.” Then the program uses these importance metrics to find a single adjustment algorithm that is best able to minimize the bias across every question.

The weighted results are combined into a single sample.