Introduction

Most probability-based political and social surveys of the general public in the United States continue to be conducted by telephone. In most of these surveys, respondents are interviewed only once. This model of conducting surveys is facing serious challenges as response rates drop, costs increase and newer alternative methods present appealing alternatives.

One alternative that is hardly new, but is attracting renewed attention, is the panel survey. Panel surveys involve repeated interviews with the same pool of respondents over time. Panels have been around for decades, and high-quality, probability-based panels (those recruited through random samples of the population) are an integral part of the federal statistical system. Panels recruited through non-probability methods or “convenience” samples have found widespread applicability in the market research world. But with a very small number of notable exceptions, panels have not been widely used by organizations committed to conducting probability-based political and social surveys on a regular basis. This is changing.

Several features of panels make them an appealing alternative to one-time, “cross-sectional” surveys. The initial costs of recruiting a panel, while potentially large, can be amortized over time because the panel can yield multiple individual surveys at relatively lower marginal costs per survey. Over time, far more can be learned about the panelists’ social, demographic and political characteristics than is feasible in a single survey. Related to this, external databases of information about panelists, such as voter and consumer files, can be integrated with survey responses to yield additional insights. Panels allow for the measurement of individual-level change over time, something that is not possible with cross-sectional surveys. Moreover, panels make it relatively easy to use multiple modes of interviewing. In particular, self-administration using Web or a paper questionnaire has desirable characteristics for the measurement of many kinds of attitudes and behaviors. And perhaps most obvious, follow-up studies are ideally facilitated in a panel design.

With these advantages in mind, Pew Research Center set out in early 2014 to build a probability-based panel — the American Trends Panel — to supplement our traditional method of data collection in the U.S. — the random digit dial (RDD) telephone survey. This report describes the steps taken to build and manage the panel and our experiences with it in 2014.

Panel Models

Probability-based survey panels come in many varieties. Some attempt to be representative of the entire population, while others focus on subgroups such as teens or young adults. Some involve nearly continuous interviewing, while others may collect data once a year or even less frequently. We envisioned creating a panel of randomly selected adults that would represent the U.S. adult population and could be used for social and political surveys similar to those we conduct with cross-sectional samples. Our goal was to collect data from panelists about once per month. We wanted most surveys to be self-administered, rather than interviewer-administered.

One well-known model for this type of panel is GfK’s KnowledgePanel (formerly known as Knowledge Networks’ panel of the same name).  Originally begun in the late 1990s, KnowledgePanel is a large nationally-representative survey panel of over 55,000 panelists.  Panelists are now recruited using address-based sampling (ABS), though some existing panelists were previously recruited through landline RDD surveys.  Interviewing occurs in both English and Spanish. Some Latino panelists continue to be recruited by telephone in high-density, Latino areas. All panelists are surveyed online. Those who did not have a computer and/or access to the internet at the time of their recruitment were provided with the necessary equipment and access.  In addition, GfK has built and managed several smaller custom versions of its KnowledgePanel for specific clients.

Two other multi-purpose, non-governmental panels in operation are the RAND American Life Panel at the RAND Corporation and the Understanding America Study at the University of Southern California. The RAND panel has more than 6,000 participants who have been recruited using a variety of sampling methods and sources; the majority of the panel was recruited using ABS and RDD (landline and cell). Panelists are surveyed via the internet and are provided a computer and/or internet access if they need it.

The Understanding America Study has approximately 2,000 panelists who have been recruited using ABS. Panelists are surveyed via the internet and are provided a tablet computer with internet access if they need it.

Gallup maintains a nationally-representative panel of approximately 60,000 adults recruited via RDD and ABS. Most panelists participate via the internet, while some are interviewed by mail or phone.

One other panel currently under construction is the AmeriSpeak panel by NORC at the University of Chicago. The sample frame is the area probability NORC National Sample Frame. This panel, with an initial size expected to be 10,000 members, is being recruited using a variety of methods including email, U.S. mail, telephone, and face-to-face recruitment. Panelists without internet access will usually be interviewed by telephone.

The American Trends Panel Blueprint

After considering various models, we opted to recruit the American Trends Panel via a large RDD telephone survey conducted in early 2014 on the subject of political polarization. The study had a total sample size of about 10,000, providing a large base for the panel recruitment. All respondents in the telephone survey received a common core of questions about their political values and engagement, along with a comprehensive set of demographic questions, ensuring a good baseline of information about respondents who agreed to join the panel as well about those who refused. The telephone survey and panel recruitment was funded in part by grants from the William and Flora Hewlett Foundation and the John D. and Catherine T. MacArthur Foundation and the generosity of Don C. and Jeane M. Bertsch.

We decided that the standard mode of interview for panelists with access to the internet would be self-administration on a desktop, laptop, tablet or smartphone. To ensure coverage of individuals who do not have access to the internet or did not want to use the internet for taking surveys, we decided to build the capability to survey them by mail with a paper questionnaire. Providing computers and internet access was not, in our view, a cost-effective approach to cover this portion of the population. We also have the option to interview this group by telephone, and did so in the first wave of interviewing and on another wave that incorporated a test of interview modes. But concerns about mode-of-interview effects led us to prefer the option of a mail survey for this relatively small portion (12%) of the panel.

Like most of the other national panels, we provided a small incentive for joining the panel ($10 in cash) and for completing each panel survey (detailed below). During 2014, surveys were conducted approximately once per month, and will be conducted approximately every two-three months in 2015. A second recruitment effort is planned for later in 2015 to dilute the effects of panel conditioning – the possibility that panelists become acclimated to the interview process and survey content and no longer respond in the same ways that they did when first interviewed – and to replenish panel membership because of inevitable attrition over time.

Although we built this panel with the explicit goal of having it serve the research needs of our yearlong study of political polarization, it was also very much an experiment, given our lack of past experience with panel research. Among the questions we wanted to answer were the following:

• What percentage of RDD respondents who were offered panel membership would join and participate?
• Does the language used to recruit respondents to the panel affect the proportion of respondents who agree to participate?
• How well would the demographic and political composition of the panelists match the composition of those who were recruited? How well would the composition of the panelists match the overall U.S. population?
• How engaged would panelists be? Would most of them take most surveys, or would participation be more intermittent?
• How serious would attrition be over the course of the year and how would it affect the representativeness of the panel?
• For the Web portion of the panel, how important would mobile devices be for completing surveys?
• Would this method of data collection be cost effective for our purposes?

The American Trends Panel was designed by Pew Research Center staff in collaboration with staff at Abt SRBI. Overall direction of the panel is the responsibility of Pew Research Center. Ongoing data collection is conducted and managed by Abt SRBI.

Recruitment

A 54% majority of those who were invited to join the panel agreed to do so, but not all of these individuals actually took part in panel surveys.  Of those invited, 43% joined and responded to at least one wave in 2014, 33% responded to at least five waves and 20% responded to all waves.

As noted earlier, an experiment was conducted regarding how to ask respondents to join the panel. Half were invited to “participate in future surveys,” while the other half were invited to “join our American Trends Panel” to test whether a vague or specific request would perform better. Beyond how this invitation was phrased, all other characteristics of the panel and responsibilities of panelists were explained in exactly the same way.

A somewhat higher portion of the “future surveys” group (45%) than the “join the panel” group (42%) agreed to join the panel and responded to at least one survey. However, there was no difference between the two groups in joining and responding to five or more waves, or responding to all waves.

Panel Composition

Nearly all surveys are subject to some degree of non-response bias, which is typically addressed using weighting. The telephone survey used to recruit the panelists had biases that are common to similar telephone surveys: compared with all adults, respondents were more likely to be white, older, better educated, married and politically engaged. The characteristics of those who agreed to join the panel and who took part in a typical panel wave were very similar to all telephone survey respondents, with a couple of notable exceptions: Those who joined and participated in the panel are even more likely than the phone respondents to be college graduates (e.g., 52% in Wave 9 vs. 41% in the phone survey) and to be active in politics (e.g., 44% have contacted an elected official in the past two years vs. 34% among the recruitment survey sample). We also observed a bias in the racial composition of the participants; 78% of those who took part in Wave 9 are non-Hispanic white, compared with 72% in the unweighted phone sample and 66% in the Census parameter.

All of the demographic biases noted here are corrected with post-stratification weighting, though doing so reduces the effective sample size of the study. Weighting also reduces the bias in political engagement. For example, 44% of the unweighted respondents to Wave 9 have contacted an elected official in the past two years; weighting reduces this to 30%, not much different from the weighted figure in the telephone survey (28%). Except for one wave that dealt specifically with the upcoming 2014 congressional election (Wave 7, Sept.-Oct. 2014), no effort was made to correct for the bias in political engagement through weighting. Weighting to correct such bias is controversial, given the absence of generally accepted parameters for this characteristic. Accordingly, future recruitment to the panel will attempt to mitigate this and other kinds of biases through differential rates of recruitment for certain groups (such as those with low levels of political interest), as well as higher incentives for hard-to-reach individuals.

Because much of the work of Pew Research Center focuses on public opinion about politics and public policy, it is especially important that the panel be unbiased with respect to the partisanship and ideological orientation of the panelists. Across the eight waves of the panel conducted in 2014, the unweighted share of those who identify as Democratic varied between 33% and 34%, while the percent Republican varied between 22% and 24%. Compared with the unweighted telephone survey, Democrats were slightly more numerous in the unweighted panel waves (by 3-4 percentage points), while Republican numbers in the panel were nearly identical to those in the telephone survey. People who refused to lean toward either party were slightly underrepresented in the unweighted panel data (12% in the telephone survey, 7%-8% in the panel).

These differences are corrected by the weighting, which includes a parameter for party affiliation based on a rolling average of the three most recent political telephone surveys conducted by Pew Research Center.

Beyond partisan affiliation, the panel includes a measure of the ideological orientation of each panelist, based on a 10-item scale of ideological consistency developed as a part of Pew Research Center’s study of political polarization. On this measure using unweighted data, individuals classified as “consistently liberal” (having a score of +7 to +10 on a scale that varies between -10 and +10) are overrepresented in the panel, relative to the telephone survey: 20%-21% in the panel waves were in the consistently liberal category, compared with 15% in the phone survey. Conservatives are not underrepresented in the panel, relative to the telephone survey; instead, those in the “mixed” ideological group – with scores between -2 and +2 on the ideological consistency scale – are slightly less numerous in the panel (30%-31% in the panel, vs. 35% in the phone survey).

The standard panel weighting has the effect of nearly eliminating these ideological biases in a typical wave (across the eight waves, between 13%-14% of the weighted samples are classified as consistently liberal, vs. 12% in the weighted phone survey; 37%-39% are mixed, vs. 39% in the phone survey).

Attrition

The American Trends Panel has suffered very little attrition throughout its first year. Response rates to individual surveys were quite stable across eight waves in 2014. Based on those who agreed to join the panel (including those who never took part in a wave), the Web response rate varied between 60% and 64%. The mail/phone group’s response rate varied more, but there was no consistent increase or decrease over time. The cumulative response rate, which accounts for the response rate to the recruitment telephone survey, agreement to join the panel, attrition from the panel and the response rate to each wave remained almost constant over time at approximately 3.5%.

A survey’s design effect is a statistical measure of the extent to which the sampling design and the weighting affect the overall precision of the survey’s estimates. Larger design effects generally indicate lower precision, meaning that the effective sample size of the study is smaller than the actual sample size. The design effect for each wave of the panel was approximated as one plus the squared coefficient of variation of the weights. One way to interpret the design effect is as a measure of how much the data had to be weighted in order to look like the target population. On an unweighted basis, there was no meaningful change in the demographics of the panel over time (details about the demographic composition of the panel over time are presented in the appendix). On a weighted basis, the only differences were between wave 7 and all other waves on voter registration status due to a special pre-election weighting protocol that matched voter registration and U.S. House vote intention to the results from a September telephone survey.

Mobile Devices

Great care was taken to ensure that the Web portion of the panel was optimized for completion on a mobile device, specifically a mobile phone or a tablet. The combined share of respondents completing on a mobile device rose from 24% in wave 1 to 34% in wave 9. This was due to an increase in the use of smartphones to complete the survey, which rose from 16% to 26% while tablets remained between 8% and 9% of the total.

There are many demographic differences between those who chose to complete their survey on a mobile phone, tablet or PC. Both mobile phone and tablet respondents are more likely to be female than PC respondents. Compared with PC or tablet respondents, mobile phone respondents are younger, more likely to be non-white, never married, not registered to vote and to hold mixed, rather than consistent, ideological views. Tablet and PC respondents are more likely to be Republican or lean Republican, and to have contacted an elected official, than mobile respondents. Mobile phone respondents have the lowest average income, followed by PC respondents, with tablet respondents at the top. Mobile phone respondents have lower education than PC respondents.

Given that the kinds of people who took the surveys on a mobile device were also the kinds of people underrepresented in the panel (relative to the benchmarks), it is critically important that Web surveys be optimized for these kinds of devices. It may also be beneficial in future panel recruitment to stress that the surveys can be taken anytime, anywhere, on a mobile phone or tablet.

Javascript

One reason Web surveys have become popular is the ability to add arresting visuals and interactivity to the questionnaire. But most content of this nature may require respondents to have JavaScript, a computer programming language, enabled on their devices. JavaScript makes it possible to use drag-and-drop features, scale sliders and other widgets in a questionnaire. But among our Web panelists, approximately 15% do not have Javascript-enabled devices and thus cannot see such features. These respondents tend to be older (33% aged 65+, compared with 19% among those with JavaScript) and more likely to be non-Hispanic whites (86% among non-JavaScript panelists, vs. 78% among those with JavaScript).

Fortunately, previous research indicates that the flashier tools JavaScript makes possible do not improve and may in fact degrade data quality. There are legitimate and useful applications for JavaScript in Web surveys (e.g., the ability to compute a running sum of percentages that must add to 100%). Our approach has been to program non-Javascript versions of questions to be automatically substituted for respondents whose computers are flagged as not having JavaScript.

Cost

Although precise costs for the panel are confidential, we are able to share some general information for those researchers who might be contemplating undertaking a similar research effort.

The first point to note is that the model used by the American Trends Panel leverages an existing telephone survey as the principal vehicle for recruitment to the panel. The additional interview time required to offer the invitation and collect the respondent’s contact information averaged about three minutes. The Gallup panel uses its ongoing tracking survey to recruit participants, but most of the other probability-based panels are using address-based sampling with a mail or phone interview to recruit.

A second observation is that the infrastructure to house and manage the panel requires a substantial investment of staff time and expertise, but once that infrastructure is in place, the panel can be managed and even expanded with considerably less effort.

Third, because the bulk of the interviewing is done online, the data-collection costs on a per-case basis are significantly smaller than for a telephone survey. Offsetting this somewhat, however, are the costs associated with interviewing the non-Web portion of the panel. In order to minimize mode of interview differences, we have tried to use mail surveys with this part of the panel whenever possible. But producing a mail questionnaire is time consuming, and there are costs associated with mailing and processing.

In summary, a probability-based panel can be a relatively cost-effective means of collecting high quality survey data, but a significant up-front investment in time and expertise is required. Panels require ongoing management from staff dedicated to the task. Over time, panels require refreshment to replace members who drop out and to offset the effects of panel conditioning.

Recruitment

All current members of the American Trends Panel (ATP) were originally recruited from the 2014 Political Polarization and Typology Survey, a large (n=10,013) national landline and cellphone RDD survey conducted Jan. 23- March 16, 2014, in English and Spanish. At the end of that survey, respondents were invited to join the panel. The invitation was extended to all respondents who use the internet (from any location) and also to most respondents who do not use the internet.¹

Of the 10,013 adults interviewed, 9,809 were invited to take part in the panel. Half were invited to “participate in future surveys,” while the other half were invited to “join our American Trends Panel” as part of an experiment to test the best way to frame panel participation. All other characteristics of the panel were explained to respondents in exactly the same way.

Respondents were told they would be paid $10 for agreeing to participate in monthly surveys on different topics, and if they were an internet user they were told the surveys would be taken online. Additionally, they would be given $5 for each survey they completed. Hispanic respondents and young adults ages 18-25 were offered $10 per survey because these groups historically have been less likely than others to take part in panels. They were then asked for their email and mailing addresses. The 5,338 respondents who initially agreed to join the panel were then sent a welcome packet in either English or Spanish as appropriate. The 4,741 panelists who provided a mailing address were mailed a packet which contained a cover letter, a $10 bill for agreeing to join the panel and a brochure about the panel. Both the letter and the brochure contained the Web address for the ATP website (with English and Spanish webpage content). The 4,777 panelists who provided an email address were also emailed the cover letter that had a link to the ATP website and a digital copy of the ATP brochure. Included in the 4,777 are the 597 panelists who provided only an email address and no mailing address. They were separately emailed a $10 Amazon gift card for joining.

Monthly Data Collection Protocol

Once empaneled, panelists were sent approximately one survey each month on a variety of topics.  These were in English or Spanish based on each panelist’s preference. The length of the surveys varied between 15 and 20 minutes. Topics covered included politics, media consumption, personality traits, religion, the internet, use of technology and knowledge about science and other subjects. Panelists with internet access (“Web panelists”) took the surveys online, while the panelists without internet access, and a small number of panelists who had the internet but didn’t have or wouldn’t provide an email address (“non-Web panelists”), took the surveys via another mode. This second mode was typically mail, but telephone mode was used in two instances (once in the first panel wave, and again in Wave 5 for an experiment in testing the effects of mode of administration).

For each survey the Web panelists were sent an email invite with a unique link to the survey questionnaire. They were also mailed a postcard concurrently alerting them to check their email inbox for the next survey invitation. Up to four email reminders were sent to Web non-respondents. Once they had taken the survey they were sent their contingent incentive. At the end of the first panel survey, we asked respondents if they would prefer the money as an Amazon gift card or as a check, and this payment method was used for the subsequent panel waves.

Non-Web panelists received up to three mailings for each survey. The first full mailing was sent via first class mail for English-speaking panelists and via priority mail for Spanish-speaking panelists to convey importance and to make up time lost due to translation. The mailing included a cover letter with a prepaid incentive of a $5 (or $10) bill affixed with a glue dot. The mailing also contained a questionnaire in English or Spanish and a self-addressed, stamped return envelope. All respondents were sent a reminder postcard. Then a second full mailing was sent to all mail panelists with a note to disregard if the questionnaire had already been returned. This second mailing, sent via first class mail to English-speaking respondents and priority mail to Spanish speakers, contained a cover letter, questionnaire and self-addressed, stamped return envelope but no new incentive.

Special Non-Respondent Follow-up Wave

During the latter part of data collection for Wave 1, a special non-respondent follow-up survey was conducted to attempt to verify contact information and to determine whether persons recruited to the panel who had not yet responded to Wave 1 intended to participate in that wave or future waves. A total of 2,228 panelists who had not yet responded to Wave 1 were called between April 17-27, yielding 1,124 completed interviews. Only 34 respondents indicated a desire to be removed from the panel. Many respondents indicated that they had not received a welcome packet, or had changed their mailing address or email address. A total of 332 individuals subsequently completed Wave 1, and another 54 started the Wave 1 interview but did not complete it by the time the survey closed. For operational purposes, this non-response, follow-up study is considered to be Wave 2 of the panel, though it was not designed to represent the general population and has no substantive questions in it. It is not included in any tables in this report.

Weighting the Panel Data

Once the data collection ends, typically four weeks from the start, the mail and Web data are combined and weighted. The monthly panel weighting protocol uses the original base weights from the recruitment telephone survey, which account for each panelist’s probability of selection. The base weight uses a single frame estimation to adjust for the probability that the respondent’s phone number was selected from the sampling frame, the overlap in the landline and cellphone frames, and the within-household selection in the landline sample. For a subset of the panelists, an additional adjustment is included in the base weight to account for the fact that they belong to a group that was subsampled for invitation to the panel.

The next step in the weighting process is a propensity adjustment for nonresponse to the panel invitation, which in later waves was updated to account for other forms of attrition. Of 9,809 telephone respondents who were invited to join the panel, 5,338 (54.4%) accepted. A propensity score adjustment was computed to correct for differential nonresponse to the panel invitation. A logistic regression model was estimated in which accepting the panel invitation was regressed on sampling frame (landline vs. cell), incentive amount ($5/$10 per survey), internet user, race, marital status, child in the household, age, education, religious service attendance, household income, frequency of voting, opinion of the Tea Party movement (agree with/disagree with), whether or not they contacted an elected official in the last two years, political ideology, and statistically significant 2-way interactions (p<.05).  Hispanic ethnicity was excluded from the model because Hispanics were offered a different incentive than most non-Hispanics. Gender and the number of adults in the household were not predictive and excluded from the model.  The set of predictors considered for the model are variables that are routinely measured in surveys conducted for Pew Research Center. The estimated propensities were used to divide cases into approximately equal size groups using the quintiles of the estimated propensity score.  The propensity score adjustment was computed as the inverse of the response rate in each quintile. This quintile approach helps to protect against model misspecification, relative to using the inverse of the response propensities.

The next step in the weighting process is post-stratification to target population parameters. The propensity-adjusted base weights for the panelists responding to a particular panel survey are calibrated to population benchmarks using raking, or iterative proportional fitting. This adjustment is designed to reduce the risk of bias stemming from nonresponse at the various stages of the panel design (the RDD survey used for recruitment, the invitation to join the panel and the panel survey).  The raking dimensions include age, gender, education, race, ethnicity and Census region from the most recently available American Community Survey. Population density is based on data from the 2010 Decennial Census. Telephone service parameters (landline only, cellphone only and both landline and cellphone) are projections based on the most recent National Health Interview Survey Wireless Substitution Early Release Report. Internet access (user vs. non-user) as measured in the 2014 Political Polarization and Typology Survey (the recruitment survey) is also included as a parameter. This is because most panelists take the surveys online, so there is a concern that internet users could be over-represented in the survey estimates if this dimension is not controlled in the raking. Finally, party affiliation (Republican, Lean Republican, No Lean, Lean Democrat, Democrat) is included using as a parameter the average from the three most recent dual-frame RDD monthly surveys conducted for Pew Research Center. The rationale for this was a concern that panelists with a particular political affiliation might respond to panel surveys at a higher rate than other groups; however this concern proved to be unfounded, as the unweighted distribution of cases on party affiliation typically tracks the average used as a parameter. All raking targets are based on the non-institutionalized U.S. adult (age 18+) population.

The final step in the weighting process is trimming. The distribution of the raked weights is trimmed to reduce extreme values. Panel waves are typically trimmed at the 1^st and 99^th or the 2^nd and 98^th percentiles.