{"id":9292,"date":"2020-11-13T12:00:24","date_gmt":"2020-11-13T17:00:24","guid":{"rendered":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/short-reads\/%year%\/%monthnum%\/%day%\/understanding-how-2020s-election-polls-performed-and-what-it-might-mean-for-other-kinds-of-survey-work\/"},"modified":"2024-04-13T23:43:59","modified_gmt":"2024-04-14T04:43:59","slug":"understanding-how-2020s-election-polls-performed-and-what-it-might-mean-for-other-kinds-of-survey-work","status":"publish","type":"short-read","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/short-reads\/2020\/11\/13\/understanding-how-2020s-election-polls-performed-and-what-it-might-mean-for-other-kinds-of-survey-work\/","title":{"rendered":"Understanding how 2020 election polls performed and what it might mean for other kinds of survey work"},"content":{"rendered":"\n
\"Voting
(Brianna Soukup\/Portland Press Herald via Getty Images)<\/figcaption><\/figure><\/div>\n\n\n\n

Taken in the aggregate, preelection polls in the United States pointed to the strong likelihood that Democrat Joe Biden would pick up several states that Hillary Clinton lost in 2016 and, in the process, win a popular and electoral vote majority over Republican President Donald Trump. That indeed came to pass. But the election was much closer than polls suggested in several battleground states (e.g., Wisconsin) and more decisive for Trump elsewhere (e.g., Ohio). Democrats also were disappointed at failing to pick up outright control of the U.S. Senate \u2013 though it remains a possibility \u2013 and at losing seats in the U.S. House and several state legislatures.<\/p>\n\n\n\n

Many who follow public opinion polls are understandably asking how these outcomes could happen, especially after the fairly aggressive steps<\/a> the polling community took to understand and address problems that surfaced in 2016. We are asking ourselves the same thing. In this post, we\u2019ll take a preliminary shot at answering that question, characterizing the nature and scope of the 2020 polling errors and suggesting some possible causes. We\u2019ll also consider what this year\u2019s errors might mean for issue-focused<\/em> surveys, though it will be many months<\/a> before the industry will be able to collect all the data necessary to come to any solid conclusions.  <\/p>\n\n\n\n\n\n\n\n

Preelection polls in the United States pointed to the likelihood that Joe Biden would win a popular and electoral vote majority over Donald Trump. That indeed came to pass. But the election was much closer than polls suggested in several battleground states and more decisive for Trump elsewhere. <\/p><\/blockquote><\/figure>\n\n\n\n

Before talking about what went wrong, there are a couple of important caveats worth noting. First, given the Democratic-leaning tendency to vote by mail this year<\/a> and the fact that mail votes are counted later in many places, the size of the polling errors \u2013 especially at the national level \u2013 will likely end up being smaller than they appeared on election night. Even this week, vote counting continues and estimates of polling errors have shrunk somewhat in many battleground states. It\u2019s also important to recognize that not all states suffered a polling misfire. In many important states that Biden won (at least based on current vote totals), including Arizona, Colorado, Georgia, Minnesota, New Mexico, Nevada and Virginia, polls gave a solid read of the contest.<\/p>\n\n\n\n

All that said, it\u2019s clear that national and many state estimates were not just off, but off in the same direction: They favored the Democratic candidate. To measure by how much, we compared the actual  vote margins between Republicans and Democrats \u2013 both nationally and at the state level \u2013 with the margins in a weighted average of polls from FiveThirtyEight.com<\/a>. Looking across the 12 battleground states from the upper Midwest (where many polls missed the mark) to the Sun Belt and Southwest (where many were stronger), polls overestimated the Democratic advantage by an average of about 4 percentage points. When looking at national polls, the Democratic overstatement will end up being similar, about 4 points, depending on the final vote count<\/a>. That means state polling errors are about the same as in 2016, while the national polling error is slightly larger, at least as of today. Even so, the national polling error of 2020 appears to be similar to the average errors for election polls<\/a> over the past 12 presidential elections.<\/p>\n\n\n\n

The fact that the polling errors were not random, and that they almost uniformly involved underestimates of Republican rather than Democratic performance, points to a systematic cause or set of causes. At this early point in the post-election period, the theories about what went wrong fall roughly into four categories, each of which has different ramifications for the polling industry.<\/p>\n\n\n\n

Partisan nonresponse<\/h2>\n\n\n\n

The suggested problem<\/h4>\n\n\n\n

According to this theory, Democratic voters were more easily reachable and\/or just more willing than Republican voters to respond to surveys, and routine statistical adjustments fell short in correcting for the problem. A variant of this: The overall share of Republicans in survey samples was roughly correct, but the samples underrepresented the most hard-core Trump supporters in the party. One possible corollary of this theory is that Republicans\u2019 widespread lack of trust in institutions like the news media<\/a> \u2013 which sponsors a great deal of polling \u2013 led some people to not want to participate in polls.<\/p>\n\n\n\n

Is this mainly an election polling problem, or would this be of wider concern to issue pollsters as well?<\/h4>\n\n\n\n

Sadly, the latter. If polls are systematically underrepresenting some types of conservatives or Republicans, it has ramifications for surveys that measure all kinds of behaviors and issues, from views on the coronavirus pandemic<\/a> to attitudes toward climate change. Issue polling doesn\u2019t require the kind of 51%-49% precision of modern presidential election polling, of course, but no pollster wants a systematic skew to their data, even if it\u2019s \u201conly\u201d 5 percentage points. <\/p>\n\n\n\n

What could we do to fix it?<\/h4>\n\n\n\n

A straightforward fix to the problem of underrepresenting Trump supporters would be to increase efforts to recruit conservatives and Republicans to polls; increase the statistical weight of those already in the survey to match their share of the population (a process known as \u201cweighting\u201d); or both. Many polls this year weighted on party registration, 2016 vote or self-identified partisanship, but still underestimated GOP support.<\/p>\n\n\n\n

The challenge here is twofold. The first is in estimating the correct share of conservatives and Republicans in the population, since, unlike age, gender and other demographic characteristics, there are no timely, authoritative benchmarks<\/a> on political orientation. Second, just getting the overall share of Republicans in the poll correct may be insufficient if those who are willing to be interviewed are bad proxies for those who are not willing (e.g., more strongly conservative) \u2013 in which case a weighting adjustment within<\/em> partisan groups may be needed.<\/p>\n\n\n\n

\u2018Shy Trump\u2019 voters<\/h2>\n\n\n\n

The suggested problem<\/h4>\n\n\n\n

According to this theory, not all poll respondents who supported Trump may have been honest about their support for him, either out of some sort of concern about being criticized for backing the president or simply a desire to mislead. Considerable research, including by Pew Research Center<\/a>, has failed to turn up much evidence for this idea, but it remains plausible.<\/p>\n\n\n\n

The fact that polls this year underestimated support for other, less controversial Republican candidates \u2013 sometimes by more than they underestimated support for Trump \u2013 suggests that the \u201cshy Trump\u201d hypothesis may not explain very much of the problem.<\/p><\/blockquote><\/figure>\n\n\n\n

Is this mainly an election polling problem, or would this be of wider concern to issue pollsters as well?<\/h4>\n\n\n\n

This would pose a challenge for measuring attitudes about the president in any venue. But if it was limited to the current<\/em> president, it would not have lasting impact. Polls on issues that are less sensitive might be less affected.<\/p>\n\n\n\n

What could we do to fix it?<\/h4>\n\n\n\n

In the electoral context, this is a difficult problem to fix. Pollsters have experimented with approaches to doing so, such as asking respondents how their friends and neighbors planned to vote (in addition to asking respondents how they themselves planned to vote) and then using answers to these questions to adjust their forecasts. But the efficacy of these methods is still uncertain.<\/p>\n\n\n\n

Still, the fact that polls this year underestimated support for other, less controversial Republican candidates \u2013 sometimes by more than they underestimated support for Trump \u2013 suggests that the \u201cshy Trump\u201d hypothesis may not explain very much of the problem.<\/p>\n\n\n\n

Turnout error A: Underestimating enthusiasm for Trump<\/strong><\/h2>\n\n\n\n

The suggested problem<\/h4>\n\n\n\n

Election polls, as opposed to issue polling, have an extra hurdle to clear in their attempt to be accurate: They have to predict which respondents are actually going to cast a ballot and then measure the race only among this subset of \u201clikely voters.\u201d Under this theory, it\u2019s possible that the traditional \u201clikely voter screens\u201d that pollsters use just didn\u2019t work as a way to measure Trump voters\u2019 enthusiasm to turn out for their candidate. In this case, surveys may have had enough Trump voters in their samples, but not counted enough of them as likely voters.<\/p>\n\n\n\n

Is this mainly an election polling problem, or would this be of wider concern to issue pollsters as well?<\/h4>\n\n\n\n

If the main problem this year was a failure to anticipate the size of Republican turnout, the accuracy of issue polls would be much less affected. It would suggest that survey samples may already adequately represent Americans of all political persuasions but still struggle to properly anticipate who will actually turn out to vote, which we know is quite difficult<\/a>. Fortunately, the eventual availability of state voter records<\/a> matched to many election surveys will make it possible to assess the extent to which turnout differences between Trump and Biden supporters explain the errors.<\/p>\n\n\n\n

What could we do to fix it?<\/h4>\n\n\n\n

Back to the mines on reinventing likely voter scales<\/a>.<\/p>\n\n\n\n

Turnout error B: The pandemic effect<\/h2>\n\n\n\n

The suggested problem<\/h4>\n\n\n\n

The once-in-a-generation coronavirus pandemic dramatically altered how people intended to vote, with Democrats disproportionately concerned about the virus and using early voting (either by mail or in person) and Republicans more likely to vote in person on Election Day itself. In such an unusual year \u2013 with so many people voting early for the first time and some states changing their procedures \u2013 it\u2019s possible that some Democrats who thought<\/em> they had, or would, cast a ballot did not successfully do so. A related point is that Trump and the Republican Party conducted a more traditional get-out-the-vote effort in the campaign\u2019s final weeks, with large rallies<\/a> and door-to-door canvassing<\/a>. These may have further confounded likely voter models.<\/p>\n\n\n\n

Is this mainly an election polling problem, or would this be of wider concern to issue pollsters as well? <\/h4>\n\n\n\n

To the extent that polls were distorted by the pandemic, the problems may be confined to this moment in time and this specific election. Issue polling would be unaffected. <\/p><\/blockquote><\/figure>\n\n\n\n

To the extent that polls were distorted by the pandemic, the problems may be confined to this moment in time and this specific election. Issue polling would be unaffected. The pandemic may have created greater obstacles to voting for Democrats than Republicans, a possibility that polls would have a hard time assessing. These are not problems we typically confront with issue polling.<\/p>\n\n\n\n

What could we do to fix it?<\/h4>\n\n\n\n

It\u2019s possible that researchers could develop questions, such as on knowledge of the voting process, that could help predict whether the drop-off between intention to vote and having successfully cast a ballot is higher for some voters than others \u2013 for instance, whether a voter\u2019s mailed ballot is successfully counted or may be rejected for some reason. Treating all early voters as definitely having voted and all Election Day voters as only possible<\/em> voters is a potential mistake that can be avoided.<\/p>\n\n\n\n

Conclusion<\/h2>\n\n\n\n

As we begin to study the performance of 2020 election polling in more detail, it\u2019s also entirely possible that all <\/em>of these factors contributed in some way  \u2013 a \u201cperfect storm\u201d that blew the polls off course.<\/p>\n\n\n\n

Pew Research Center and other polling organizations will devote a great deal of effort to understanding what happened. Indeed, we have already begun to do so. We\u2019ll conduct a review of our own polling, as well as a broader analysis of the polls, and we\u2019ll participate in a task force<\/a> established at the beginning of this year by the American Association for Public Opinion Research (AAPOR) to review election poll performance, as happened in 2016<\/a>. This effort will take time. Relevant data on voter turnout will take months to compile. But make no mistake: We are committed to understanding the sources of the problem, fixing them and being transparent along the way.<\/p>\n","protected":false},"excerpt":{"rendered":"

Many who follow polls are asking how these errors could happen. Here, we\u2019ll take a preliminary shot at answering that question.<\/p>\n","protected":false},"author":584,"featured_media":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_crdt_document":"","sub_headline":"","sub_title":"","_prc_public_revisions":[],"_ppp_expiration_hours":0,"_ppp_enabled":false,"ai_generated_summary":"","apple_news_api_created_at":"2020-11-13T18:28:52Z","apple_news_api_id":"57a7c5a6-729b-49c5-a882-986110ac0ebe","apple_news_api_modified_at":"2020-11-17T15:36:25Z","apple_news_api_revision":"AAAAAAAAAAAAAAAAAAAAAA==","apple_news_api_share_url":"https:\/\/apple.news\/AV6fFpnKbScWogphhEKwOvg","apple_news_cover_media_provider":"image","apple_news_coverimage":0,"apple_news_coverimage_caption":"","apple_news_cover_video_id":0,"apple_news_cover_video_url":"","apple_news_cover_embedwebvideo_url":"","apple_news_is_hidden":"","apple_news_is_paid":"","apple_news_is_preview":"","apple_news_is_sponsored":"","apple_news_maturity_rating":"","apple_news_metadata":"\"\"","apple_news_pullquote":"","apple_news_pullquote_position":"","apple_news_slug":"","apple_news_sections":[],"apple_news_suppress_video_url":false,"apple_news_use_image_component":false,"relatedPosts":[],"_prc_fork_parent":0,"_prc_fork_status":"","_prc_active_fork":0,"datacite_doi":"","datacite_doi_citation":"","_prc_seo_qr_attachment_id":0,"spoken_article_player_enabled":true,"footnotes":""},"categories":[40,36,357],"bylines":[944,631,967],"collection":[],"datasets":[],"_post_visibility":[],"formats":[467],"_fund_pool":[],"languages":[],"regions-countries":[515],"research-teams":[528,520],"class_list":["post-9292","short-read","type-short-read","status-publish","hentry","category-election-2020","category-methodological-research","category-survey-methods","bylines-claudia-deane","bylines-courtney-kennedy","bylines-scott-keeter","formats-short-read","regions-countries-united-states","research-teams-methods","research-teams-politics"],"label":"Short Read","post_parent":0,"word_count":1962,"canonical_url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/short-reads\/2020\/11\/13\/understanding-how-2020s-election-polls-performed-and-what-it-might-mean-for-other-kinds-of-survey-work\/","art_direction":{"A2":{"id":16343,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2020\/11\/FT_20.11.12_Polling_feature-jpg.webp","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2020\/11\/FT_20.11.12_Polling_feature-jpg.webp?w=268&h=151&crop=1","width":268,"height":151,"chartArt":false},"XL":{"id":16343,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2020\/11\/FT_20.11.12_Polling_feature-jpg.webp","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2020\/11\/FT_20.11.12_Polling_feature-jpg.webp?w=720&h=405&crop=1","width":720,"height":405,"chartArt":false},"A3":{"id":16343,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2020\/11\/FT_20.11.12_Polling_feature-jpg.webp","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2020\/11\/FT_20.11.12_Polling_feature-jpg.webp?w=194&h=110&crop=1","width":194,"height":110,"chartArt":false},"A4":{"id":16343,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2020\/11\/FT_20.11.12_Polling_feature-jpg.webp","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2020\/11\/FT_20.11.12_Polling_feature-jpg.webp?w=268&h=151&crop=1","width":268,"height":151,"chartArt":false},"A1":{"id":16343,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2020\/11\/FT_20.11.12_Polling_feature-jpg.webp","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2020\/11\/FT_20.11.12_Polling_feature-jpg.webp?w=564&h=317&crop=1","width":564,"height":317,"chartArt":false},"social":{"id":16343,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2020\/11\/FT_20.11.12_Polling_feature-jpg.webp","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2020\/11\/FT_20.11.12_Polling_feature-jpg.webp?w=1200&h=628&crop=1","width":1200,"height":628,"chartArt":false}},"_embeds":[],"table_of_contents":[],"datacite_doi":"","prc_seo_data":{"title":"What 2020 election poll performance might mean for other survey work","description":"Many who follow polls are asking how these errors could happen. Here, we\u2019ll take a preliminary shot at answering that question.","og_title":"Understanding how 2020 election polls performed and what it might mean for other kinds of survey work","og_description":"Many who follow polls are asking how these errors could happen. Here, we\u2019ll take a preliminary shot at answering that question.","schema_type":"Article","noindex":false,"canonical_url":"","primary_terms":[],"custom_schema":[],"twitter_title":"Understanding how 2020 election polls performed and what it might mean for other kinds of survey work","twitter_description":"Many who follow polls are asking how these errors could happen. Here, we\u2019ll take a preliminary shot at answering that question.","og_image":16343,"twitter_image":379929,"indexnow_submitted_at":null,"gsc_index_status":null},"apple_news_notices":[],"jetpack_sharing_enabled":true,"relatedPostsOrdered":[],"bylinesOrdered":[{"key":"2eebcfee-7e07-44d5-bdeb-0118180dc20d","termId":967},{"key":"9a18d89d-fd61-4c3e-8f7f-ed05dda6eab0","termId":631},{"key":"d7f83e86-5123-4075-8043-3045d504dff9","termId":944}],"acknowledgementsOrdered":[],"_links":{"self":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/short-read\/9292","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/short-read"}],"about":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/types\/short-read"}],"author":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/users\/584"}],"replies":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/comments?post=9292"}],"version-history":[{"count":1,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/short-read\/9292\/revisions"}],"predecessor-version":[{"id":30096,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/short-read\/9292\/revisions\/30096"}],"wp:attachment":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/media?parent=9292"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/categories?post=9292"},{"taxonomy":"bylines","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/bylines?post=9292"},{"taxonomy":"collection","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/collection?post=9292"},{"taxonomy":"datasets","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/datasets?post=9292"},{"taxonomy":"_post_visibility","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_post_visibility?post=9292"},{"taxonomy":"formats","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/formats?post=9292"},{"taxonomy":"_fund_pool","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_fund_pool?post=9292"},{"taxonomy":"languages","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/languages?post=9292"},{"taxonomy":"regions-countries","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/regions-countries?post=9292"},{"taxonomy":"research-teams","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/research-teams?post=9292"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}