It’s been all over the news lately: a survey by the Innovation Center for U.S. Dairy suggests that 7 percent of American adults believe chocolate milk comes from brown cows.

The takeaway of much of this reporting is that Americans are science illiterate as well as uninformed about how their food is produced. This interpretation is intuitive: research has suggested that Americans lack understanding of many scientific concepts and the story line of Americans as woefully ignorant of science is perennial. As a society, we are also urbanizing and fewer people work in agriculture, so it’s unsurprising that many don’t know how food is made. These survey results line up with this prevailing wisdom.

But is this what the survey is actually telling us? To us as researchers studying science communication and public understanding of science, factors in the survey itself and in the way the media report on it raise questions about how much to read into these findings.

Survey’s results aren’t publicly available

Researchers are trained to look for the original methods whenever they read a new study, especially if the results are surprising. Learning how the study was done provides information that helps determine whether the science is sound and what to make of it.

The chocolate milk survey is described as a nationally representative survey of 1,000 American adults, but this is impossible to verify without seeing how respondents were selected. Likewise, how the survey was conducted – whether it was a phone or online survey, for instance – can have significant impacts on its accuracy. Research suggests that phone surveys may be less accurate than online surveys because they require people to give their responses out loud to another person instead of quietly clicking away in privacy.

For instance, someone who holds racist views may feel comfortable checking a box about it but might avoid openly professing those opinions on the phone to a stranger. It’s unlikely the chocolate milk survey ran into such problems, but depending on the questions asked, other challenges may have presented themselves.

Likewise, it’s difficult to interpret the results of the chocolate milk question without seeing how it was worded. Poorly phrased or confusing questions abound in survey research and complicate the process of interpreting findings.

An NPR interview with Jean Ragalie-Carr, president of the National Dairy Council, is the closest we can get to the actual wording of potential responses: “there was brown cows, or black-and-white cows, or they didn’t know.” But as Glendora Meikle of the Columbia Journalism Review points out, we don’t know if those were the only options presented to respondents.

This matters. For instance, if respondents associate some color cows with dairy production and other color cows with beef production, it’s easy to see how people could become confused. If this is the case, they’re not confused about where chocolate milk comes from, but about the difference between dairy cows and beef cows.

Social scientists call this a problem with validity: the question doesn’t really measure what it’s supposed to measure. Of course, without seeing how the question was worded, we can’t know whether the chocolate milk question had validity.

Indeed, early media coverage focused on the 7 percent statistic but left out the fact that 48 percent of respondents said they don’t know where chocolate milk comes from. This gives context to the 7 percent number. While it’s conceivable that 7 percent of the population doesn’t know that chocolate milk is just milk with chocolate, the idea that a full 55 percent — over half of adults — don’t know or gave an incorrect response begins to strain credulity. This points toward a confusing survey question.

We reached out to Lisa McComb, the senior vice president of communications for Dairy Management, Inc., about the survey. She confirmed that it’s not publicly available. “The purpose of the survey was to gauge some interesting and fun facts about consumers’ perceptions of dairy, not a scientific or academic study intended to be published,” she told us.

Story feeds a popular narrative — and media missed it

Questions about the original findings aside, there’s reason to explore how the media covered the chocolate milk survey.

The results were instantly shared and republished by a mind-boggling number of outlets (a Google Trends search for “chocolate milk” and “brown cows” shows a spike beginning June 15th). This factoid likely garnered such massive attention because it feeds into a popular narrative about American ignorance and science illiteracy.

Our research suggests that people who are often accused of being “anti-science” are not necessarily as unscientific as one might think. The rapid spread of this story is likely related to the desire, unfortunately prominent among many liberals, to see and label other people as ignorant.

Studies suggest we are more likely to accept new information when it confirms what we already want to believe. In this case, the chocolate milk statistic fits well with the notion that Americans are fools, so it’s accepted and republished widely despite the numerous red flags that should give scientifically minded people pause.

But the fact remains that many reporters and news outlets decided to run the story without having seen the original results, instead citing one another’s reporting. This led to some interesting challenges when trying to fact-check the survey: The Washington Post links to Food & Wine’s coverage, which linked to the Innovation Center’s website, which originally publicized the survey results. The Innovation Center, in turn, links to a story on Today.com, which linked right back to the Food & Wine article. This type of circular reporting without seeking out the original source can lead to the spread of misinformation. Unfortunately, as news stories quickly pop up and go viral online, it’s all too likely that we will continue to see such problems in the future.

Importantly, none of this disproves the notion that some adults believe chocolate milk comes from brown cows. It certainly does nothing to undermine the need for increased science education in the United States or suggests that a better understanding of our food production system wouldn’t be beneficial to society. All of these points are still valid. Likewise, this isn’t necessarily evidence that the survey itself is flawed. As McComb notes, the survey is not a scientific one and isn’t meant to be taken as evidence of Americans’ knowledge (or lack thereof) of dairy products. The problem is that it’s being reported on as though it is.

So this survey did point out a lack of science understanding. Ironically, rather than showing Americans’ ignorance of chocolate milk’s origins, the fact that media coverage of this survey was reported so widely and with so few caveats instead showed that many people are not skeptical of the science they read.

The authors do not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and have disclosed no relevant affiliations beyond the academic appointment above.