:::: MENU ::::

Ads, Public Opinion, News Media, Polarization

Public misunderstanding, “birtherism” and flawed survey wording: Research analysis

Tags: | Last updated: March 20, 2014

Last updated: March 20, 2014

Barack Obama (Wikimedia)
Barack Obama (Wikimedia)

Americans’ responses to surveys can sometimes reveal startling ignorance and a puzzling belief in half-truth, even obvious lies, although a topic’s direct relevance to people’s daily lives matters a great deal in terms of knowledge levels. Among the most egregious recent examples of ignorance is the apparently large number of people who did not believe President Obama was born in the United States, even though it is indisputably true that he was born in Hawaii. For some, this public opinion controversy, silly at one level, seems to embody all that is seriously wrong with the country — a public drifting away from facts, slouching toward some permanently balkanized and politically polarized state. It feeds, perhaps, into a certain public/mass media obsession with conspiracy theories. However, new research analysis reveals that questionable and varying polling methods may have played a role in fanning the flames of this “birther” controversy.

Time and again, correcting misinformation has proven exceedingly difficult, as media corrections serve to perpetuate the original error and rebroadcast the misinformation. Researchers find that people often filter information through an underlying set of cultural beliefs, engaging in “motivated reasoning.” The findings of recent studies suggest some better techniques for correcting misinformation; frequently, having a credible messenger delivering the corrective — the power of particular sources — is key to reaching audiences who may not be susceptible to objective facts.

A 2014 study published in Public Opinion Quarterly, “Public Misunderstanding of Political Facts: How Question Wording Affected Estimates of Partisan Differences in Birtherism,” analyzes the structure of certain key polls in 2010 and 2011 that kept alive the misleading issue of President Obama’s origins. The study also provides results from a survey the researchers conducted in May 2011, after President Obama had released his long-form birth certificate to help settle the matter. Polling groups frequently found that there was a large partisan gap in terms of belief on the issue — that Republicans were much more doubtful of President Obama’s birthplace, sometimes exceeding Democrats by as much as 48 percentage points.

The paper’s lead authors, Jon A. Krosnick and Neil Malhotra of Stanford University, provide insight into the nature of the survey industry that should serve as a loud warning for media members: “We put a spotlight on one instantiation of a story that has played out on the national stage over and over again in recent decades: different survey organizations each craft questions ostensibly measuring the same opinion, and yet the organizations employ different question structures and different choices of words. It is almost as if each organization strives not to ask others’ questions and instead seeks to take a unique approach to measurement.”

The study’s findings include:

  • The evidence suggests that “apparently very different results were partly attributable to the use of leading introductory sentences in one closed-ended question. Removal of those sentences, which were not present in other questions, caused obtained results to be more similar to those produced by the other closed-ended questions.”
  • For example, in April 2011 a CBS/New York Times poll asked a closed-ended question in such a way that produced a high degree of skepticism about President Obama’s origins, particularly among Republicans. The survey question read as follows: “According to the Constitution, American presidents must be ‘natural born citizens.’ Some people say Barack Obama was not born in the United States, but was born in another country. Do you think Barack Obama was born in the United States, or do you think he was born in another country?” The partisan gap revealed by this question (48 percentage points difference between Democrats, 19%, and Republicans, 67%) was larger than that produced by the four other surveys conducted during spring 2011 by other groups.
  • In an experiment in May 2011, the study’s researchers replicated the CBS/New York Times survey wording and then compared it to responses to a simpler question: “Is your best guess that Barack Obama was born in the United States or that he was born in another country?” Among Republicans, this simpler question found diminished skepticism about President Obama’s origins, by 17.6 percentage points. This experiment “illustrates the impact of leading introductory sentences and raises caution about employing them in the future.”
  • One explanation for the variation is that, with the introductory sentences in place, “some respondents may have answered differently after hearing the introductory sentences because those words constituted instructions about which answer was anti-Obama. Viewed in this way, the CBS/New York Times question may have accurately tapped some people’s anti-Obama sentiment but did not necessarily accurately tap their understanding of Mr. Obama’s birthplace.”
  • Overall, open-ended questions, where respondents are not forced to choose but rather must provide their own answer, “yielded the highest rates of apparent correct understanding and the smallest partisan gaps.”

The researchers conclude that “it is useful to note that these results constitute a cautionary tale about political observers’ tendency to critique the American public as lacking correct factual political information….  One can make that case based on the evidence from some of the polls done about Mr. Obama’s birthplace. Yet other polls suggest a much greater prevalence of correct understanding. Therefore, it may be important to carefully attend to question-wording effects before blaming the public for being ignorant.”

Related: For media members, the episode is a reminder that it is well worth understanding valid polling methods and asking hard questions before characterizing a given poll as an objective measure of public sentiment. For another case study in survey errors and controversies, see the 2009 Public Opinion Quarterly study, “Dynamics of Poll Performance during the 2008 Presidential Nomination Contest.”

 

Keywords: polling, Hawaii, Obama, presidency, birther, GOP


Writer: | March 20, 2014

Citation: Krosnick, Jon A.; Malhotra, Neil; Mittal, Urja. “Public Misunderstanding of Political Facts: How Question Wording Affected Estimates of Partisan Differences in Birtherism,” Public Opinion Quarterly, March 2014. doi: 10.1093/poq/nft080.

Analysis assignments

Read and analyze the Public Policy Polling survey "Democrats and Republicans Differ on Conspiracy Theory Beliefs."

  1. What do you make of the entire approach of this survey? Which questions do you find questionable? Are any flawed in your view? How should a responsible reporter handle the findings? What sort of context and expert sources might help a news audience understand these findings?

Read the full study titled “Public Misunderstanding of Political Facts: How Question Wording Affected Estimates of Partisan Differences in Birtherism.”

  1. What are the study's key technical terms? Which ones need to be put into language a lay audience can understand?
  2. Do the study’s authors put the research into context and show how they are advancing the state of knowledge about the subject? If so, what did the previous research indicate?
  3. What is the study’s research method? If there are statistical results, how did the scholars arrive at them?
  4. Evaluate the study's limitations. (For example, are there weaknesses in the study's data or research design?)
  5. How could the findings be misreported or misinterpreted by a reporter? In other words, what are the difficulties in conveying the data accurately? Give an example of a faulty headline or story lead.

Newswriting and digital reporting assignments

  1. Write a lead, headline or nut graph based on the study.
  2. Spend 60 minutes exploring the issue by accessing sources of information other than the study. Write a lead (or headline or nut graph) based on the study but informed by the new information. Does the new information significantly change what one would write based on the study alone?
  3. Compose two Twitter messages of 140 characters or fewer accurately conveying the study’s findings to a general audience. Make sure to use appropriate hashtags.
  4. Choose several key quotations from the study and show how they would be set up and used in a brief blog post.
  5. Map out the structure for a 60-second video segment about the study. What combination of study findings and visual aids could be used?
  6. Find pictures and graphics that might run with a story about the study. If appropriate, also find two related videos to embed in an online posting. Be sure to evaluate the credibility and appropriateness of any materials you would aggregate and repurpose.

Class discussion questions

  1. What is the study’s most important finding?
  2. Would members of the public intuitively understand the study’s findings? If not, what would be the most effective way to relate them?
  3. What kinds of knowledgeable sources you would interview to report the study in context?
  4. How could the study be “localized” and shown to have community implications?
  5. How might the study be explained through the stories of representative individuals? What kinds of people might a reporter feature to make such a story about the study come alive?
  6. What sorts of stories might be generated out of secondary information or ideas discussed in the study?

 

Leave a comment