The 2016 and 2020 elections raised questions about the state of public opinion polling. Some of the criticism was premature or overheated, considering that polling ultimately got key contours of the 2020 election correct (e.g., the Electoral College and national popular vote winner; Democrats taking control of the Senate). But the consistency with which most poll results differed from those election outcomes is undeniable. Looking at final estimates of the outcome of the 2020 U.S. presidential race, 93% of national polls overstated the Democratic candidate’s support among voters, while nearly as many (88%) did so in 2016.¹

A forthcoming report from the American Association for Public Opinion Research (AAPOR) will offer a comprehensive, industry-wide look at the performance of preelection polls in 2020. But individual polling organizations are also working to understand why polls have underestimated GOP support and what adjustments may be in order. 

Pew Research Center is among the organizations examining its survey processes. The Center does not predict election results, nor does it apply the likely voter modeling needed to facilitate such predictions. Instead our focus is public opinion broadly defined, among nonvoters and voters alike and mostly on topics other than elections. Even so, presidential elections and how polls fare in covering them can be informative. As an analysis discussed, if recent election polling problems stem from flawed likely voter models, then non-election polls may be fine. If, however, the problem is fewer Republicans (or certain types of Republicans) participating in surveys, that could have implications for the field more broadly. 

This report summarizes new research into the data quality of Pew Research Center’s U.S. polling. It builds on prior studies that have benchmarked the Center’s data against authoritative estimates for nonelectoral topics like smoking rates, employment rates or health care coverage. As context, the Center conducts surveys using its online panel, the American Trends Panel (ATP). The ATP is recruited offline via random national sampling of residential addresses. Each survey is statistically adjusted to match national estimates for political party identification and registered voter status in addition to demographics and other benchmarks.² The analysis in this report probes whether the ATP is in any way underrepresenting Republicans, either by recruiting too few into the panel or by losing Republicans at a higher rate. Among the key findings:

Adults joining the ATP in recent years are less Republican than those joining in earlier years. The raw, unweighted share of new ATP recruits identifying as Republican or leaning Republican was 45% in 2015, 40% in the 2018 and 38% in the 2020. This trend could reflect real-world change in participation (i.e., Republicans are increasingly resistant to polling) or real-world change in party affiliation (i.e., that there is a decline in the share of the public identifying as Republican), but it might also reflect methodological changes over time in how the ATP is recruited. Switching from telephone-based recruitment to address-based recruitment in 2018 may have been a factor. Regardless of the cause(s), more weighting correction was needed in 2020 than 2014 (when the panel was created) to make sure that Republicans and Democrats were represented proportional to their estimated share of the population.³

Donald Trump voters were somewhat more likely than others to leave the panel (stop taking surveys) since 2016, though this is explained by their demographics. The overall retention rate of panelists on the ATP is quite high, as 78% of respondents in 2016 were still taking surveys in 2020. But a higher share of 2016 Trump voters (22%) than Hillary Clinton or third-party voters (19%) stopped participating in the ATP during the subsequent four years. The demographic make-up of 2016 Trump voters basically explains this difference. In analysis controlling for voters’ age, race and education level, presidential vote preference does not help predict whether later they decided to leave the panel.

People living in the country’s most and least pro-Trump areas were somewhat less likely than others to join the panel in 2020. Researchers cannot know for sure whether someone is a Republican or Democrat based simply on their address, but election results in their voting precinct provide some insight. Analysis of addresses sampled for panel recruitment in 2020 found that households in the country’s most pro- and most anti-Trump areas were somewhat less likely to join the ATP than households in more politically balanced areas. The share of sampled households joining the ATP was 9% in the country’s most pro-Trump precincts, 8% in the country’s most anti-Trump precincts, and 11% in the remainder of the country. While these differences are not large, they are statistically significant.

Taken together, these findings suggest that achieving proper representation of Republicans is more difficult than it used to be. Survey participation has long been linked to individuals’ levels of education and social trust. Now that the GOP is doing better attracting voters with lower levels of education and, according to some analysts, doing better than in the past attracting low trust adults, Republican participation in surveys is waning, increasing reliance on weighting as a corrective. 

One silver lining is that these effects do not appear to be particularly large, at least at present on the ATP. The differences between Republicans’ and Democrats’ rates of ATP participation tend to be a percentage point or two, only marginally significant in statistical testing. It seems possible for pollsters to close the gap – to increase Republicans’ participation to be more on par with Democrats – by modifying the way surveys are conducted. Based on this research, Pew Research Center is implementing a number of new strategies to improve the representation of its survey panel. 

• Retiring overrepresented panelists. Researchers identified a set of panelists who are demographically overrepresented on the ATP and who, because of their demographic characteristics, contributed to the overstatement of Democratic support in the 2016 and 2020 elections. Later in spring 2021, the Center is retiring a subset of these panelists, removing them from the panel (about 2,500 panelists out of about 13,500 total will be retired). More details about the retirement process can be found in Appendix B.  
  
• Calibrating the political balance of the ATP using a relatively high response rate survey offering mail and online response. Effective January 2021, each ATP survey is being weighted to the partisan distribution from the Center’s National Public Opinion Reference Survey (NPORS), which is a new annual survey using address-based sampling and offering mail or online response. The inaugural NPORS in 2020 had a 29% response rate and over 4,000 completions, most of which were by mail.⁴

• Testing an offline response mode. Part of the challenge with achieving robust representation of certain groups (e.g., from older, rural conservatives) on the ATP is that panelists must take surveys online. While the Center provides tablets and data plans to those without home internet, not everyone wants to be online. This spring the Center is fielding an experiment to determine whether it may be viable to allow panelists to respond over the phone by calling in to a toll-free number and taking a recorded survey (known as in-bound interactive voice response). Respondents receive $10 for completing the call-in survey. Those preferring to answer online can still do so.
  
• Empaneling adults who prefer mail to online surveys. Prior Center work has found that people who respond to an initial survey by mail (instead of online) are very difficult to recruit to the ATP, which is all done online. While such adults are hard to empanel, their inclusion would help with representation of older, less wealthy and less educated Americans. In early 2021 the Center fielded a special recruitment of adults who responded to a Center survey by mail in 2020. The recruitment used priority mailing and a $10 pre-incentive to motivate joining. The recruitment yielded several hundred new panelists.
  
• Developing new recruitment materials. Researchers are retooling the ATP recruitment materials with an eye toward using more accessible language and more compelling arguments for why people should join. Starting in 2021, the Center is sending sampled households a color, trifold brochure about the ATP in addition to the normal cover letter and $2 pre-incentive. The Center is also creating a short video explaining the ATP and why those who have been selected to participate should join. 

One question raised by this multifaceted strategy is whether it might overcorrect for the initial challenge and result in an overrepresentation of Republicans. While that is a possibility, we feel that the risks from too little action are greater. The Center’s analysis pointed to two issues: partisan differences in willingness to join the ATP and in likelihood of dropping out of the panel. In turn, the panel weighting was needing to do an inordinate amount of work to compensate for differences between the panel and the U.S. adult population. The action plan described above speaks to both issues, but only with an eye toward truing things up, not blindly going beyond. Several of the steps are exploratory, determining if and how a design change might improve the panel. Depending on the testing results, such steps (e.g., offering inbound IVR as a supplemental mode) may or may not ultimately be implemented on the ATP. Moreover, steps such as exploring an offline response mode or modifying recruitment materials are expected to improve representation among several harder to reach segments of society, not simply supporters of one candidate. 

A final question is whether such actions are necessary. Indeed, a recent Center analysis found that errors in election estimates of the magnitude seen in the 2020 election have very minor consequences for attitudinal, mass opinion estimates (e.g., views on a government guarantee of health care or perceptions of the impact of immigrants on the country). That simulation-based analysis, while helpful for scoping the scale of the issue, does not speak to the erosion of trust in polling and certainly doesn’t negate pollsters’ obligation to make their surveys as accurate as possible. Even if the steps outlined above yield relatively small effects, we expect that they will improve the data quality in Center surveys. 

Testing for partisan differences in survey panel recruitment

The first step in selecting adults for Center surveys is drawing a random, national sample of residential addresses. We mail these addresses and ask a randomly selected adult to join our survey panel. One way that a partisan imbalance could emerge is if Republicans are less likely than Democrats to agree to join, or vice versa. Determining whether this is happening is difficult because the ideal data do not exist. Our surveys sample from all U.S. adults, and there is no database to tell us whether the adults we asked to join favor one party or another.

We can, however, answer this question for the people we asked to join who are registered to vote and live a state that records party registration. Researchers took the 16,001 addresses sampled in 2020 for recruitment and matched them to a national registered voter file. This matching yielded 23,503 registered voter records. Some 42% of those voter records were registered with a political party. This analysis finds no clear indication that people’s likelihood of joining the panel is related to partisanship. The share of registered Republicans at addresses we sampled who agreed to join the ATP (12%) was not statistically different from the share of registered Democrats who agreed to join (13%).  

A different approach yielded a more discernable pattern. The alternate approach involved looking at the community in which people live – specifically whether it is a pro-Trump area or not – to make inferences about the people asked to join the panel. Researchers did this by looking at precinct-level voting data. At the time of this analysis, only data from the 2016 election was available. Researchers analyzed what share of the precinct’s voters backed Donald Trump in 2016 and looked to see if there was a relationship with willingness to join the ATP. 

Overall, the relationship is fairly noisy. Willingness to join the ATP does not consistently increase or decrease as precincts get progressively more supportive of Trump. That said, there is some indication that willingness to join the panel is slightly lower at both extremes. In the most pro-Trump areas – precincts across the U.S. with the highest Trump vote share – 9% of sampled households agreed to join the panel. In the most anti-Trump areas – precincts with the lowest Trump vote share – 8% of sampled households agreed to join the panel. In the rest of the country 11% of sampled households agreed to join the panel.⁵

In analysis controlling for local levels of wealth, education, and racial composition, the electoral support for Trump remains a negative predictor (albeit a modest one) of a household’s likelihood of joining the ATP.⁶ On balance, these analyses suggest that Trump supporters may be slightly less likely than others to join the ATP. 

Testing for partisan differences in dropping out of the panel

Another way that a partisan imbalance could emerge is if Republicans are more likely to drop out of the panel than Democrats, or vice versa. There are several reasons why people drop out of survey panels, including becoming too busy or disinterested, changing contact information and losing touch, incapacitation, death, or removal by panel management. On the ATP most dropout is from panelist inactivity (i.e., not responding to several consecutive surveys) eventually leading to their removal.

As a starting point, researchers examined panelists who completed the 2016 post-election survey, which attempted to interview the entire panel. Researchers determined which of these panelists from 2016 were still taking surveys fours year later in 2020. The majority of the 2016 panelists (78%) remained active in 2020, while 22% had dropped out. Panelists who said they voted for Trump in 2016 were somewhat more likely to drop out of the panel than those who voted for another candidate (22% versus 19%, respectively). This result, while based on just one panel, lends some support to the notion that Trump supporters have become slightly less willing to participate in surveys in recent years.

Dropout rates varied across other dimensions as well. For example, panelists who in 2016 were younger and had lower levels of formal education were more likely to drop out of the panel than others. In fact, when controlling for a panelist’s age and education level, Trump voters were not significantly more likely to leave the panel than other voters. In other words, the higher dropout rate among Trump supporters is likely explained by their demographic characteristics. 

Evaluating the composition of the panel

Another way researchers evaluated the ATP was to look at the overall shares of Republicans and Democrats and determine if those shares were correct. This simple question is extremely difficult to answer, for several reasons:

• No timely, definitive data exist to provide an answer. Polling data are timely but often limited by sampling as well as other potential errors. Gold-standard surveys like the General Social Survey (GSS) are quite accurate but less timely and exclude the 8% of adults who are not citizens. Presidential elections are authoritative but exclude the roughly 40% of adults who cannot or do not vote. 
  
• Surveys require statistical adjustments called “weighting.” It is debatable whether investigators should focus only on the weighted (adjusted) partisan estimates or whether they should also consider a panel’s unweighted (raw) partisan estimates. 

Keeping these limitations in mind, researchers analyzed the partisan composition of the ATP over time, by recruitment cohort. Since the ATP was created in 2014, the Center has usually, though not always, fielded an annual recruitment to add new panelists. The size and design of the recruitment has changed over time. Notably, starting in 2018, the recruitment switched to address-based sampling (ABS) instead telephone random digit dial (RDD).

The analysis found that the recruitment cohorts generally have been getting less Republican over time. The raw, unweighted share of new ATP recruits identifying as Republican or leaning Republican was 45% in 2015, 40% in 2018 and 38% in 2020.  The forces behind that trend are not entirely clear, as there are at least three potential explanations. 

The methodological change in 2018 from using RDD to ABS to recruit panelists may have played a role. The RDD-recruited cohorts both had proportionally more Republicans than the ABS-recruited cohorts. Another possible explanation for the trend is that the GOP has been losing adherents gradually over time. In other words, the unweighted ATP recruitments may reflect a real decline in the share of adults identifying as Republican nationally. While national demographic changes suggest that is plausible, this idea is not supported by the Center’s or other survey organizations’ research. For example, Pew Research Center, the General Social Survey and Gallup all show the share of U.S. adults identifying as Republican or leaning Republican being generally stable since 2016. Since there is no compelling evidence that there was a significant decline in Republican affiliation from 2014 to 2020, this explanation seems unlikely. 

Another explanation could be that Republicans are increasingly unwilling to participate in surveys. This idea would suggest that the unweighted ATP recruitments reflect a real decline in the share of Republicans responding to surveys or joining survey panels. This idea is not supported by GSS or Gallup poll trends. However, it is consistent with one prominent interpretation of polling errors in the 2016 and 2020 elections: that they stem from certain types of Republicans not participating in polls.

Whatever the cause, the newer cohorts are less Republican. This trend has not, however, had much effect on ATP survey estimates. This is because every ATP survey features a weighting adjustment for political affiliation. This means that the surveys are weighted to align with the share of U.S. adults who identify as a Democrat or Republican, based on an external source.⁷ The weighted partisan balance on the ATP has been rather stable. The weighted share of adults in ATP surveys who are Republican or lean to the Republican Party has stayed in the 42% to 45% range for six years. 

While weighting helps to make ATP estimates nationally representative, the trend in the unweighted data is a concern. The recent increases in the size of the weighting correction on partisanship suggests that the panel would benefit from shoring up participation among harder to reach groups. If that is successful, then reliance on weighting will lessen. 

As mentioned earlier, Pew Research Center is pursuing several measures. The first step is adjusting the composition of the existing panel. Researchers identified panelists belonging to overrepresented groups (e.g., panelists who get weighted down rather than up). A subset of this group of panelists, which skews highly educated and collectively leans Democratic, is being retired from the panel, meaning they will no longer be surveyed. Details of the retirement plan are in Appendix B. Researchers are also exploring new and potentially more effective ways to recruit adults who have historically been difficult to empanel (which includes lower socioeconomic status adults of all races and political views). This includes recruitment of adults who are resistant to taking surveys online, developing new ATP recruitment materials, and exploring an offline response mode. 

The impact from these modifications on ATP estimates will generally be subtle because panel surveys have long been weighting on key dimensions like partisanship, education, and civic and political engagement. But even small improvements in accuracy are worth pursuing and relying less on weighting as a corrective will make estimates more precise. 

This report was made possible by The Pew Charitable Trusts. Pew Research Center is a subsidiary of The Pew Charitable Trusts, its primary funder.

This report is a collaborative effort based on the input and analysis of the following individuals:

Research team

Courtney Kennedy, Director, Survey Research
Jesse Lopez,  Associate Panel Manager
Scott Keeter, Senior Survey Advisor
Arnold Lau, Research Analyst
Nick Hatley, Research Analyst
Nick Bertoni, Senior Panel Manager

Methodology

Andrew Mercer, Senior Research Methodologist
Ashley Amaya, Senior Research Methodologist
Dorene Asare-Marfo, Research Methodologist

Communications and editorial

Rachel Weisel, Senior Communications Manager
Calvin Jordan, Communications Associate
Travis Mitchell, Copy Editor

Graphic design and web publishing

Bill Webster, Information Graphics Designer
Travis Mitchell, Digital Producer

Other colleagues at Pew Research Center provided helpful comments on this study, including Carroll Doherty, Jocelyn Kiley, Claudia Deane and Michael Dimock.

The American Trends Panel survey methodology

The American Trends Panel (ATP), created by Pew Research Center, is a nationally representative panel of randomly selected U.S. adults. Panelists participate via self-administered web surveys. Panelists who do not have internet access are provided with a tablet and wireless internet connection. Interviews are conducted in both English and Spanish. The overall target population for ATP surveys is non-institutionalized persons ages 18 and older, living in the U.S., including Alaska and Hawaii. The panel is managed by Ipsos.

Panel recruitment

The ATP was created in 2014, with the first cohort of panelists invited to join the panel at the end of a large, national, landline and cellphone random-digit-dial survey that was conducted in both English and Spanish. Two additional recruitments were conducted using the same method in 2015 and 2017, respectively. Across these three surveys, a total of 19,718 adults were invited to join the ATP, of whom 9,942 (50%) agreed to participate.

In August 2018, the ATP switched from telephone to address-based recruitment. Invitations were sent to a random, address-based sample of households selected from the U.S. Postal Service’s Delivery Sequence File. Two additional recruitments were conducted using the same method in 2019 and 2020, respectively. Across these three address-based recruitments, a total of 17,161 adults were invited to join the ATP, of whom 15,134 (88%) agreed to join the panel and completed an initial profile survey. In each household, the adult with the next birthday was asked to go online to complete a survey, at the end of which they were invited to join the panel. Of the 25,076 individuals who have ever joined the ATP, 13,568 remained active panelists and continued to receive survey invitations at the time the most recent panel wave used in this report was conducted.

The U.S. Postal Service’s Delivery Sequence File has been estimated to cover as much as 98% of the population, although some studies suggest that the coverage could be in the low 90% range.⁸ The American Trends Panel never uses breakout routers or chains that direct respondents to additional surveys.

Incentives

All respondents are offered a post-paid incentive for their participation in ATP surveys. Respondents can choose to receive the post-paid incentive in the form of a check or a gift code to Amazon.com or could choose to decline the incentive. Incentive amounts range from $5 to $20 depending on whether the respondent belongs to a part of the population that is harder or easier to reach. Differential incentive amounts were designed to increase panel survey participation among groups that traditionally have low survey response propensities.

Data quality checks

To ensure high-quality data, the Center’s researchers perform data quality checks to identify any respondents showing clear patterns of satisficing. This includes checking for very high rates of leaving questions blank, as well as always selecting the first or last answer presented. A small number of individuals were removed from each wave as a result of these checks. 

The 2020 ATP recruitment

One section of this report focuses specifically on the 2020 ATP recruitment. That sample was drawn from the U.S. Postal Service Computerized Delivery Sequence File (DSF) and was provided by MSG (Marketing Systems Group). Occupied residential addresses (including “drop points”) in all U.S. states (including Alaska and Hawaii) and the District of Columbia had a nonzero chance of selection. The draw was a national, stratified random sample, with differential probabilities of selection across the mutually exclusive strata. Strata are defined using address-level flags for the likely age, race, and ethnicity of adults living in the household.

Ipsos sent initial mailings in a 9-by-12-inch envelope via first class mail to the sampled households. These packets included two $1 bills and a letter asking a member of the household to complete an online survey using the website and password provided. If two or more adults were in the household, the letter asked the adult with the next birthday to complete the survey. Sampled households were later sent a reminder postcard and then a reminder letter via first class mail. 

Households in Hispanic strata received all materials in English and Spanish. All other households received materials in English only. Adults who completed the survey were sent a $10 post-incentive.

Weighting

A detailed description of how ATP surveys are weighted is provided in Appendix A. 

© Pew Research Center, 2021

Protocols for weighting ATP surveys have evolved over the history of the ATP. The modifications have been gradual changes introduced over time, as opposed to sudden, dramatic shifts in the weighting approach. This appendix describes the standard weighting protocol in place for ATP surveys in the latter half of 2020. 

The ATP data was weighted in a multistep process that accounts for multiple stages of sampling and nonresponse that occur at different points in the survey process. First, each panelist begins with a base weight that reflects their probability of selection for their initial recruitment survey (and the probability of being invited to participate in the panel in cases where only a subsample of respondents were invited). The base weights for panelists recruited in different years are scaled to be proportionate to the effective sample size for all active panelists in their cohort. To correct for nonresponse to the initial recruitment surveys and gradual panel attrition, the base weights for all active panelists are calibrated to align with the population benchmarks identified in the accompanying table to create a full-panel weight. 

For ATP waves in which only a subsample of panelists are invited to participate, a wave-specific base weight is created by adjusting the full-panel weights for subsampled panelists to account for any differential probabilities of selection for the particular panel wave. For waves in which all active panelists are invited to participate, the wave-specific base weight is identical to the full-panel weight.

In the final weighting step, the wave-specific base weights for panelists who completed the survey are again calibrated to match the population benchmarks specified above. These weights are trimmed (typically at about the 1st and 99th percentiles) to reduce the loss in precision stemming from variance in the weights. Sampling errors and testing of statistical significance take into account the effect of weighting. 

Center researchers developed a plan to retire a subset of panelists who are overrepresented both demographically and politically. In raw, unadjusted numbers, the ATP contains proportionately too many college graduates, registered voters and Democratic-leaning adults. This is neither intentional nor unique to the ATP; instead it stems from such adults being more amenable to taking surveys and is a common challenge in modern polling. Each ATP survey is weighting to correct for these patterns. 

The retirement strategy is built around two key metrics: how the unweighted vote preference on the ATP aligns with the 2020 election, and panelists’ frame weights. The retirement focuses on unweighted 2020 vote estimates on the ATP as opposed to weighted figures because the goal is to stop relying so heavily on weighting. A “frame weight” is a number assigned to each member of the ATP. It represents how much a panelist needs to get weighted up or weighted down to make the entire panel representative of all U.S. adults. Panelists who are overrepresented on the panel have low frame weight values (between 0 and 1). Panelists who are underrepresented have frame weight values greater than 1. 

By retiring about 2,500 of our roughly 13,500 active panelists, we can align the unweighted ATP with the picture of the nation revealed by the 2020 election. The retirement strategy entails the following steps:

1. Identify panelists who are overrepresented based on their frame weight. Panelists with frame weight values 0.5 or less (i.e., people substantially overrepresented) were deemed eligible for retirement, while those with larger weight values were deemed ineligible. 

2. Identify panelists without hard to reach characteristics. There are some specific subgroups that are underrepresented, extremely expensive to recruit, and important to data quality. It is not productive to retire such people. Researchers created a protective flag for panelists with any of the following characteristics: responds in Spanish, uses a Center-provided tablet, age 18-24, or has high school education level or less. Panelists with this protective flag were deemed ineligible for retirement. 

3. Addressing the partisan balance. Researchers created a 2020 presidential vote preference variable for all panelists, including nonvoters. The data came from the 2020 post-election survey, which measured vote choice among voters and candidate preference among nonvoters. Based on their answers, panelists were categorized as: 

• Voted/Preferred Trump 
• Voted/Preferred Biden 
• Voted/Preferred another candidate 
• Noncitizen 
• Refused question or did not respond to post-election survey

Researchers then used information associated with vote choice (party affiliation, race/ethnicity, education, age, gender and metro status) to impute vote/preference for panelists in the last category. Panelists in the first category were deemed ineligible for retirement. 

4. Subsample from the retirement-eligible panelists.In total, about 3,800 panelists had a frame weight value of 0.5 or less and have none of the protected characteristics (Spanish language, tablet, age 18-24, high school or less, Trump supporter). These panelists were eligible for retirement. The next step was to subsample from them such that the entire panel aligns with the portrait of the country revealed by the election without relying on weighting. Researchers subsampled from the retirement-eligible panelists with probability proportional to the inverse of their frame weight. In other words, the most overrepresented panelists (those weighted down the most) were the most likely to be retired. For this process only, the frame weight was modified so that the panel aligns not just with demographics of the U.S. population but also with the vote outcome and voter turnout rate in the 2020 presidential election.

Researchers simulated subsampling varying fractions of those eligible for retirement. For each simulation, researchers computed the unweighted 2020 vote using post-election survey respondents who would not be retired under the simulation. The simulations are unweighted because the goal is to stop relying on weighting to fix these biases. The analysis indicated that retiring about 2,500 of the eligible panelists (scenario 3) would align the panel to the 2020 vote outcome of Biden receiving 51% of the vote and Trump receiving 47%. This is the scenario being implemented this year.

Fortunately, retiring those roughly 2,500 panelists does not meaningfully impair surveys conducted on the ATP. After the retirement, the panel will still have over 11,000 active panelists, which is more than enough for measuring U.S. public opinion. 

This retirement strategy is not a perfect, permanent solution for ensuring a proper partisan balance on the ATP, but it is the most effective immediate step we can take to eliminate the imbalance that we currently rely on weighting to correct. 

Implications for ATP estimates

Researchers used two recent surveys to examine the effects from implementing the retirement. This was done through simulation. Researchers removed the retired panelists from two survey datasets and then reweighted each survey using only the remaining respondents. Researchers compared these simulated post-retirement estimates to the pre-retirement estimates released from the surveys. Any differences in the estimates are attributable to the retirement strategy. 

Overall, the effects from the retirement are subtle, slightly increasing estimates for some conservative attitudes. For example, a survey in late November 2020 found that 31% of adults said that allegations of voter fraud had been getting too little attention. After applying the retirement simulation and reweighting the data, this estimate was 32%. Other estimates, such as confidence in various institutions or whether people intend to get the COVID-19 vaccine, did not change at all. On average, the retirement changes survey figures based on all U.S. adults by less than 1 percentage point. In instances where the retirement does move estimates, the change is typically a small increase in support for a Republican-leaning viewpoint.

Researchers also examined how the retirement affects estimates based on key subgroups (e.g., race, ethnicity, political party, sex). For nearly all the groups examined, the retirement moves estimates very slightly or not at all. For example, estimates for Black, Latino, or Asian adults moved by just 0.2, 0.4, and 0.3 percentage points on average, respectively. The effect from retirement was, however, more pronounced for White college graduates. Adults in this group are among the most overrepresented on the ATP and, thus, the most likely to be retired. Estimates for White college graduates moved by 2.2 percentage points on average after applying the retirement. For example, in the late November 2020 survey, 42% of White college graduates reported feeling comfortable eating out in a restaurant, given the situation with the coronavirus outbreak. After applying the retirement, this rate increases to 45%. While these changes to ATP estimates are not big and dramatic, they do subtly increase measured support for Republican-leaning viewpoints. 

One downside of the retirement strategy is that it results in a slight understatement of standard errors for ATP estimates. This is because the ATP weights are not modified to account for the retirement. In testing where researchers did modify the weights, all the overrepresentation in the ATP that the retirement is designed to eliminate was reintroduced, and the improvements to estimates disappeared. After the retirement is implemented later this spring, ATP weighting and standard errors will continue to account for differential probabilities of selection in recruitment and differential nonresponse, but they won’t include an additional adjustment for the retirement. 

In sum, the retirement of overrepresented panelists has only a subtle effect on estimates. This comports with Center research finding that modest differences in a survey’s partisan balance have little effect on public opinion estimates. However, when the retirement does have an effect, there tends to be a slight increase in support for Republican-leaning viewpoints. To be sure, the “truing up” from the retirement will only last so long. Over time, people’s decisions about joining the panel and continuing to participate will take their effect. The greater willingness of certain adults to participate in surveys is a strong societal force that will be hard to fix precisely. Therefore, the panelist retirement is one of several strategies the Center is pursuing.