{"id":12074,"date":"2017-05-04T13:01:21","date_gmt":"2017-05-04T18:01:21","guid":{"rendered":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/short-reads\/%year%\/%monthnum%\/%day%\/qa-political-polls-and-the-2016-election\/"},"modified":"2024-04-14T03:16:39","modified_gmt":"2024-04-14T08:16:39","slug":"qa-political-polls-and-the-2016-election","status":"publish","type":"short-read","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/short-reads\/2017\/05\/04\/qa-political-polls-and-the-2016-election\/","title":{"rendered":"Q&A: Political polls and the 2016 election"},"content":{"rendered":"
\"Nonvoter<\/a><\/p>\n

\"Voters<\/a>
Voters cast their ballots at a fire station in Alhambra, California, on Nov. 8, 2016. (Ringo Chiu\/AFP\/Getty Images)<\/figcaption><\/figure><\/figure>\n\n

The outcome of the 2016 presidential election surprised a lot of people \u2013 not least the many political pollsters and analysts covering it. Today the American Association for Public Opinion Research<\/a> (AAPOR), the nation’s leading organization of survey researchers, released a long-awaited report<\/a>\u00a0that examines polling during last year’s long primary and general election campaigns.<\/p>\n\n

\"Nonvoter<\/a><\/p>\n

\"Courtney<\/a>
Courtney Kennedy, Pew Research Center director of survey research<\/figcaption><\/figure><\/figure>\n\n

Courtney Kennedy, Pew Research Center’s director of survey research, chaired the AAPOR task force that produced the report. We sat down with Kennedy recently to discuss its findings and recommendations. The conversation has been condensed and edited for clarity and conciseness.<\/p>\n\n

Ever since Donald Trump’s victory over Hillary Clinton last year, there’s been plenty of criticism of the performance and trustworthiness of polls. Was that the impetus for this report?<\/strong><\/p>\n\n

Actually, this committee was organized back in May 2016, months before any of us had the slightest inkling that last year would be a particularly unusual year for polling. The original intent was pretty straightforward: to evaluate the performance of polls, both in the primary season and the general election; to compare how they did relative to past years; and, to the extent the data would support it, assess whether certain types of polls \u2013 online versus telephone, live versus automated \u2013 did better or worse than others.<\/p>\n\n

But as of midnight or so on Nov. 8,\u00a0<\/span>it was clear that what the committee needed to do had changed. We couldn\u2019t just do this very technical, \u201cwhat was the average deviation\u201d type of report. We needed to, in addition, consider another question: \u201cWhy did the polls seem to systematically underestimate support for Donald Trump?\u201d There already were a number of hypotheses floating around \u2013 such as the so-called \u201cshy Trump\u201d effect (Trump supporters being less willing than others to disclose their support to an interviewer), \u00a0differential nonresponse (Trump supporters being less likely than others to participate in surveys), things of that nature \u2013 and we felt obligated to take on that additional piece.<\/p>\n\n

<\/p>\n\n

The report notes that, while the national polls generally came pretty close to the actual nationwide popular vote (which Clinton won by 2.1 percentage points over Trump), the performance of polls at the state level \u2013 where presidential elections actually are decided \u2013 was a lot spottier. What reasons did you find for that?<\/strong><\/p>\n\n

We found evidence for multiple potential causes. One factor that I think affected everybody who was polling in the battleground states, is the legitimate late change in voter preference in the last week before Election Day. The data on this has its limitations, but the best source is the National Election Pool’s exit poll<\/a>, which has a question about when voters made up their minds about who to vote for in the presidential race. That showed several roughly 20-point swings in favor of Trump among voters making their mind up in the final week. You didn\u2019t really see that nationally, but in Pennsylvania, Michigan, Wisconsin and even Florida, you saw what looks like dramatic movement.<\/p>\n\n

That\u2019s sort of a good news\/bad news finding for pollsters. The good news is, if you interviewed people at a certain point in time and they changed their mind several days later, the poll wouldn’t have detected that. That\u2019s not a flaw in the poll, other than perhaps with the field period in which the pollster decided to do the data collection. But there\u2019s fundamentally nothing that was necessarily off if what was generating most of the error was just honest-to-goodness changes of opinion.<\/p>\n\n

What else did you find at the state level?<\/strong><\/p>\n\n

Another interesting finding had to do with poll respondents’ level of education. A number of studies have shown that in general, people with higher levels of formal education are more likely to take surveys \u2013 it\u2019s a very robust finding. Places like Pew Research Center and others have known that for years, and we address that with our statistical weighting \u2013 that is, we ask people what their education level is and align our survey data so that it matches the U.S. population on education. And I think a lot of us assumed that was common practice in the industry \u2013 that roughly speaking, everybody was doing it. And that\u2019s not what we found. At the state level, more often than not, the polls were not being adjusted for education.<\/p>\n\n

Now in some elections, such as in 2012, that wouldn’t matter, because the very low educated and the very highly educated voted roughly the same way. But 2016 was drastically different \u2013 you had a quite strong linear relationship between education and presidential vote. And that meant that if you had too many college graduates in your poll, which virtually all of us do, and you didn’t weight appropriately, you were almost certainly going to overestimate support for Clinton.<\/p>\n\n

Were there any possible factors for which you didn’t find evidence?<\/strong><\/p>\n\n

Yes. Take the hypothesis that there\u2019s a segment of the Trump support base that does not participate in polls. If that’s true, that\u2019s a huge problem for organizations like ours, and we need to study that and understand it if we\u2019re ever going to fix it. But we looked for evidence of that, and we didn\u2019t find it.<\/p>\n\n

If it\u2019s true that we\u2019re missing a segment of the Trump support base, we would expect to find \u2013 without doing any fancy weighting, just looking at the raw data \u2013\u00a0that people in more rural, deep-red parts of the country would be underrepresented. And we didn\u2019t find that; if anything, they were slightly overrepresented. We did a number of things with a critical eye looking for those types of problems, and did not find them. And so that gave me real reassurance that fundamentally, it\u2019s not that the process of doing polls was broken last year.<\/p>\n\n

What, if anything, can the profession do to address the issues the committee found with state and local polls, especially given that so many of the newspapers and TV stations that historically sponsored them can no longer afford to do so at the same level? <\/strong><\/p>\n\n

There\u2019s lots of evidence to show that the resources that news organizations have for polling seem to be declining over time, and that does two things, I think: There are fewer news organizations doing polling, and those that do \u2013 particularly local news organizations \u2013 are using very low-cost methodology. What the report shows is that there are important design differences among the national polls, which tend to be pretty well resourced, versus the state polls, which tend to be done a lot more quickly using more automated methods with fewer resources. The state polls are half as likely as national polls to have live interviewers, and they\u2019re about half as likely to have adjusted for education in their weighting, which we know to be important. So there are these structural things that seem to have compounded the gap in performance between those state polls and the national polls. We know that on average they\u2019re doing it differently, and in ways that produced greater error in this election. It\u2019s also true that over time, you just see that there\u2019s more error in the state-level polls.<\/p>\n\n

So I could imagine that a professional association like AAPOR might investigate whether this could be addressed, either by professional education or even by trying to organize funding for more rigorous state-level surveys, conducted very close to Election Day, in order to catch people who change their minds late. This would obviously be done by researchers who use very sophisticated, state-of-the-art weighting protocols, so you don\u2019t have things like this education mishap. It\u2019s unclear if that would completely fix the problem, but at least then you\u2019d have an infusion of higher-quality polls into that set of polls that, on average, are done fairly cheaply.<\/p>\n\n

Another piece of the 2016 election cycle was the prominence, even beyond the individual polls themselves, of the data-analysis operations and news sites that aggregated polls and used them not just to predict the final outcome but to give very precise-sounding probabilities that Clinton or Trump would win. How appropriate or useful is it to use polls as predictive tools?<\/strong><\/p>\n\n

Polls aren’t designed to produce precision on the order of “so-and-so has X.X% chance of winning.” There was actually quite a bit of diversity of opinion on the committee on that issue: Some leaned toward being more aggressive in emphasizing that distinction between the predictors and the pollsters; others\u00a0less so.<\/p>\n\n

But there is a distinction. Polling and prognosticating really are two different enterprises. A well-done public opinion survey can tell you what opinion was during the time that interviewing was done, but that really doesn\u2019t speak in a precise way to future behavior. It’s been said before, but it bears repeating: A poll is a snapshot in time, not a way of predicting what will happen.\u00a0As we say in the report, greater caution and humility would seem to be in order for anyone making claims about the likely outcome of an election based in part or in whole on polling data.<\/p>\n\n

Where polls can be useful is in helping answer important questions about what is motivating voters, why people are voting or not voting, how they feel about the policies being debated, how they feel about the candidates themselves. All of those questions are more than deserving of serious answers, and that\u2019s what polls are really best designed to do.<\/p>\n\n

So, can polls still be trusted despite what happened last year?<\/strong><\/p>\n\n

I believe they can. First off, it’s worth pointing out that the performance of election polls isn’t a good indicator of the quality of surveys in general. Election polls differ from other types of surveys in some key ways: Not only do they have to field a representative sample of the public, but they also have to correctly model who among that sample will actually vote. That’s a very difficult task that non-election polls simply don’t have.<\/p>\n\n

It’s important to dispel the notion that polling writ large is broken \u2013 our investigation found that not to be the case. At the same time, we shouldn’t whitewash what happened. There were errors, and the polling industry has taken a reputational hit.\u00a0But the polling community and poll consumers should take some comfort in the fact that we’ve figured out quite a bit about what went wrong and why, and we all can learn from those errors. Some things were outside of pollsters\u2019 control, namely the late shifts in voter preference; other things were in their control and are fixable. The education imbalance, for example, is very fixable.<\/p>\n\n

We as researchers should be talking about the whole story of polling in 2016 \u2013 the differences between the national polls and state polls, the fact that we’ve identified major factors that led to the errors \u2013 in an open, non-defensive way, to dispel the “polling is broken” narrative. That narrative does a disservice to our democracy. Because polling, imperfect as it is, remains the best available tool for measuring the attitudes of all<\/em> Americans. And when it\u2019s done well, it can still produce very useful data. No matter which party is in power, it’s important to have independent, objective researchers measuring how the public feels about major issues of the day.<\/p>","protected":false},"excerpt":{"rendered":"

Courtney Kennedy of Pew Research Center, who chaired survey researchers organization AAPOR’s task force on political polling in the 2016 U.S. elections, discuss the group’s findings and recommendations.<\/p>\n","protected":false},"author":145,"featured_media":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"sub_headline":"","sub_title":"","_crdt_document":"","_prc_public_revisions":[],"_ppp_expiration_hours":0,"_ppp_enabled":false,"ai_generated_summary":"","apple_news_api_created_at":"","apple_news_api_id":"","apple_news_api_modified_at":"","apple_news_api_revision":"","apple_news_api_share_url":"","apple_news_cover_media_provider":"image","apple_news_coverimage":0,"apple_news_coverimage_caption":"","apple_news_cover_video_id":0,"apple_news_cover_video_url":"","apple_news_cover_embedwebvideo_url":"","apple_news_is_hidden":"","apple_news_is_paid":"","apple_news_is_preview":"","apple_news_is_sponsored":"","apple_news_maturity_rating":"","apple_news_metadata":"\"\"","apple_news_pullquote":"","apple_news_pullquote_position":"","apple_news_slug":"","apple_news_sections":[],"apple_news_suppress_video_url":false,"apple_news_use_image_component":false,"relatedPosts":[],"_prc_fork_parent":0,"_prc_fork_status":"","_prc_active_fork":0,"datacite_doi":"","datacite_doi_citation":"","_prc_seo_qr_attachment_id":0,"spoken_article_player_enabled":true,"bylines":[{"key":"d5f984ee-797e-4fd8-b12a-16894e8f5077","termId":842}],"acknowledgements":[],"displayBylines":true,"footnotes":"","prc_watchers":[]},"categories":[68,42,36,25,357,39,360],"bylines":[842],"collection":[],"datasets":[],"_post_visibility":[],"formats":[467],"_fund_pool":[],"languages":[],"regions-countries":[515],"research-teams":[528],"workflow-status":[],"class_list":["post-12074","short-read","type-short-read","status-publish","hentry","category-donald-trump","category-election-2016","category-methodological-research","category-politics-policy","category-survey-methods","category-us-elections-voters","category-voter-files","bylines-drew-desilver","formats-short-read","regions-countries-united-states","research-teams-methods"],"label":"Short Read","post_parent":0,"word_count":1932,"canonical_url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/short-reads\/2017\/05\/04\/qa-political-polls-and-the-2016-election\/","art_direction":{"A1":{"id":22958,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2017\/05\/FT_17.05.02_AAPOR_voters-jpg.webp","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2017\/05\/FT_17.05.02_AAPOR_voters-jpg.webp?w=564&h=317&crop=1","width":564,"height":317,"chartArt":false},"A2":{"id":22958,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2017\/05\/FT_17.05.02_AAPOR_voters-jpg.webp","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2017\/05\/FT_17.05.02_AAPOR_voters-jpg.webp?w=268&h=151&crop=1","width":268,"height":151,"chartArt":false},"A3":{"id":22958,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2017\/05\/FT_17.05.02_AAPOR_voters-jpg.webp","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2017\/05\/FT_17.05.02_AAPOR_voters-jpg.webp?w=194&h=110&crop=1","width":194,"height":110,"chartArt":false},"A4":{"id":22958,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2017\/05\/FT_17.05.02_AAPOR_voters-jpg.webp","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2017\/05\/FT_17.05.02_AAPOR_voters-jpg.webp?w=268&h=151&crop=1","width":268,"height":151,"chartArt":false},"XL":{"id":22958,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2017\/05\/FT_17.05.02_AAPOR_voters-jpg.webp","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2017\/05\/FT_17.05.02_AAPOR_voters-jpg.webp?w=640&h=360&crop=1","width":640,"height":360,"chartArt":false},"social":{"id":22958,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2017\/05\/FT_17.05.02_AAPOR_voters-jpg.webp","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2017\/05\/FT_17.05.02_AAPOR_voters-jpg.webp?w=640&h=360&crop=1","width":640,"height":360,"chartArt":false}},"_embeds":[],"watchers":[],"table_of_contents":[],"datacite_doi":"","prc_seo_data":{"title":"What political pollsters learned from the 2016 election","description":"Courtney Kennedy, who chaired survey association AAPOR's task force on polling in the 2016 U.S. elections, discusses their findings and recommendations.","og_title":"Q&A: Political polls and the 2016 election","og_description":"Courtney Kennedy of Pew Research Center, who chaired survey researchers organization AAPOR's task force on political polling in the 2016 U.S. elections, discuss the group's findings and recommendations.","schema_type":"Article","noindex":false,"canonical_url":"","primary_terms":[],"custom_schema":[],"og_image":22958,"indexnow_submitted_at":null,"gsc_index_status":null},"prepublish_checks":{},"apple_news_notices":[],"jetpack_sharing_enabled":true,"relatedPostsOrdered":[],"bylinesOrdered":[{"key":"d5f984ee-797e-4fd8-b12a-16894e8f5077","termId":842}],"acknowledgementsOrdered":[],"_links":{"self":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/short-read\/12074","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/short-read"}],"about":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/types\/short-read"}],"author":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/users\/145"}],"replies":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/comments?post=12074"}],"version-history":[{"count":2,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/short-read\/12074\/revisions"}],"predecessor-version":[{"id":102506,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/short-read\/12074\/revisions\/102506"}],"wp:attachment":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/media?parent=12074"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/categories?post=12074"},{"taxonomy":"bylines","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/bylines?post=12074"},{"taxonomy":"collection","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/collection?post=12074"},{"taxonomy":"datasets","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/datasets?post=12074"},{"taxonomy":"_post_visibility","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_post_visibility?post=12074"},{"taxonomy":"formats","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/formats?post=12074"},{"taxonomy":"_fund_pool","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_fund_pool?post=12074"},{"taxonomy":"languages","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/languages?post=12074"},{"taxonomy":"regions-countries","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/regions-countries?post=12074"},{"taxonomy":"research-teams","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/research-teams?post=12074"},{"taxonomy":"workflow-status","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/workflow-status?post=12074"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}