{"id":101635,"date":"2016-08-01T13:49:26","date_gmt":"2016-08-01T18:49:26","guid":{"rendered":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/2016\/08\/01\/flashpoints-in-polling\/"},"modified":"2024-04-14T04:18:09","modified_gmt":"2024-04-14T09:18:09","slug":"flashpoints-in-polling","status":"publish","type":"post","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2016\/08\/01\/flashpoints-in-polling\/","title":{"rendered":"Flashpoints in Polling"},"content":{"rendered":"<p class=\"wp-block-paragraph\"><em>By Claudia Deane, Courtney Kennedy, Scott Keeter and Kyley McGeeney<\/em><\/p>\n\n<p class=\"wp-block-paragraph\"><em>Updated Oct. 24, 2016<\/em><\/p>\n\n<p class=\"wp-block-paragraph\">Can polls be trusted? This question is on the minds of seemingly everyone who follows the 2016 campaign, though it is hardly unique to this election cycle. The answer is complicated, thanks to myriad challenges facing polling and the fact that pollsters have reacted to these challenges in disparate ways.<\/p>\n\n<p class=\"wp-block-paragraph\">Some polls are conducted literally overnight with convenience samples and undergo little or no adjustment. Others are painstakingly fielded for days or even weeks with robust designs and may be adjusted using cutting-edge techniques. These dramatic differences, which have been shown to affect accuracy, are often opaque to news consumers. What follows is a big-picture review of the state of polling, organized around a number of key flashpoints with links to references and research for those who want to better understand the field.<\/p>\n\n<h2 id=\"the-fast-fade-of-the-landline-telephone\" class=\"wp-block-heading\">The fast fade of the landline telephone<\/h2>\n\n<p class=\"wp-block-paragraph\">Landline telephones served as the workhorses of survey research for many years. Found in nearly every home, immovably anchored in one geographic location and providing an implicit promise that a respondent was somewhere relatively private, they dominated the survey landscape for four decades.<\/p>\n\n<p class=\"wp-block-paragraph\"><strong>The challenge: <\/strong>Landline phones are going the way of the dinosaur, rapidly being replaced by cellphones. According to the National Center for Health Statistics, which tracks their use, fewer than one-in-ten American adults were wireless only 10 years ago. <a href=\"http:\/\/www.cdc.gov\/nchs\/data\/nhis\/earlyrelease\/wireless201605.pdf\">In 2015<\/a>, nearly half were.<\/p>\n\n<p class=\"wp-block-paragraph\">This means that if a pollster only dials landline numbers, they would systematically exclude half the population \u2013 and it\u2019s far from a random half. Government <a href=\"http:\/\/www.cdc.gov\/nchs\/data\/nhis\/earlyrelease\/wireless201605.pdf\">estimates<\/a> show that Americans who have a cellphone but not a landline are disproportionately young and nonwhite, among other differences.<\/p>\n\n<p class=\"wp-block-paragraph\">By and large, telephone pollsters reacted to this evolution in one of three ways: by dialing cellphones in addition to landlines; by supplementing landline samples with opt-in samples conducted online; or by continuing to argue that all is well. (See the 2016 Michigan Democratic <a href=\"http:\/\/fivethirtyeight.com\/features\/why-the-polls-missed-bernie-sanders-michigan-upset\/\">primary<\/a> for how that last option can turn out.) Organizations that dial both landlines and cellphones typically use live interviewers, while those mixing landline and online samples tend to use automated telephone surveys known as interactive voice response (IVR).<\/p>\n\n<p class=\"wp-block-paragraph\"><strong>The research: <\/strong>While independent research on hybrid \u201clandline IVR + internet\u201d designs is scarce, there is a sizable body of knowledge on live interviewer polls that dial landline and cellphone numbers. The results are encouraging. Contrary to claims that the decline of landlines has polling \u201c<a href=\"http:\/\/www.newyorker.com\/magazine\/2015\/11\/16\/politics-and-the-new-machine\">teetering on the edge of disaster<\/a>,\u201d there is <a href=\"https:\/\/www.washingtonpost.com\/news\/the-fix\/wp\/2016\/01\/28\/reports-of-the-demise-of-polls-have-been-greatly-exaggerated\/\">solid evidence<\/a> that well-designed telephone polls are as accurate, if not more so, today than they were a generation ago.<\/p>\n\n<p class=\"wp-block-paragraph\">A new <a href=\"https:\/\/www.survoxinc.com\/wp-content\/uploads\/2016\/04\/MRAConference040516.pdf\">study<\/a> demonstrates that as pollsters have been dialing proportionately more cellphones and fewer landlines, data quality has been improving. And Pew Research Center\u2019s <a href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/2016\/08\/01\/the-twilight-of-landline-interviewing\">latest work on this topic<\/a> shows that public opinion surveys of U.S. adults could go 100% cellphone and not see a diminution in data quality. That\u2019s a far cry from the mobile phone being the death of survey research! And as for the lament that cellphone interviews cost more than landline, <a href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/2016\/10\/24\/cellphone-activity-flags\">research has explored ways<\/a> to increase the efficiency of interviewing on cellphones.\n<a name=\"rise-of-smartphones\"><\/a><\/p>\n\n<h2 id=\"the-rise-of-smartphones\" class=\"wp-block-heading\">The rise of smartphones<\/h2>\n\n<p class=\"wp-block-paragraph\">These days, it\u2019s common to see people glued to their smartphones. So what are survey researchers doing to take advantage of this powerful technology?<\/p>\n\n<p class=\"wp-block-paragraph\"><strong>The challenge: <\/strong>Smartphones offer a multitude of ways to invite or remind people to take a survey, as well as a way to collect actual survey responses. Not everybody has a smartphone, however (only 68% of U.S. adults do), \u00a0meaning that they must be used in conjunction with other contact and data collection methods to ensure everyone can participate in a survey. And even when they can be used they are not without limitations. Obstacles like small screens and technical difficulties abound.<\/p>\n\n<p class=\"wp-block-paragraph\"><strong>The research:<\/strong> Many studies have been conducted that look at best practices for using smartphones for survey research. Some focus on how to design web surveys to be completed on a mobile device, which is important given that 43% of web panelists in our <a href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methodology\/u-s-survey-research\/american-trends-panel\/\">American Trends Panel<\/a> complete their surveys on a smartphone. Pew Research Center published a <a href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/2015\/06\/11\/tips-for-creating-web-surveys-for-completion-on-a-mobile-device\/\">summary of some of these<\/a> last year that specifically addresses some of the limitations of mobile devices and how to overcome them.<\/p>\n\n<p class=\"wp-block-paragraph\">The Center has also <a href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/2015\/04\/01\/app-vs-web-for-surveys-of-smartphone-users\/\">experimented<\/a> with having respondents use a smartphone application to complete a web survey. It was here that technical difficulties proved the most problematic, but in the end the app data held its own compared with data from a web browser survey. The Center has also published a report looking at an <a href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/2016\/09\/07\/text-message-notification-for-web-surveys\">experiment using text messages<\/a> to send survey invitations and reminders. Academic researchers have also experimented with text messages as a contact method for surveys (see <a href=\"http:\/\/jssam.oxfordjournals.org\/content\/early\/2014\/08\/27\/jssam.smu015\">one<\/a> of these <a href=\"http:\/\/poq.oxfordjournals.org\/content\/78\/4\/951.abstract\">two<\/a> pieces for an example). <a href=\"http:\/\/journals.plos.org\/plosone\/article?id=10.1371\/journal.pone.0128337\">Another group<\/a> looked at using text messages for data collection purposes, while <a href=\"http:\/\/www.aapor.org\/AAPOR_Main\/media\/AnnualMeetingProceedings\/2014\/Session-J-1-1-Marlar-Y.pdf\">still others<\/a> compared the effect of using text messages for contact versus using them for data collection. The research in this area continues to evolve, but it\u2019s clear that the power of the smartphone has the potential to be harnessed to the benefit of survey research.<\/p>\n\n<h2 id=\"still-have-the-same-cellphone-number-you-did-in-2006\" class=\"wp-block-heading\">Still have the same cellphone number you did in 2006?<\/h2>\n\n<p class=\"wp-block-paragraph\">One of the major differences between cellphones and landlines is that when people move, they can take their cellphone numbers with them. In many ways, this is a convenience. But it can be a huge headache for pollsters trying to survey at the state or local level.<\/p>\n\n<p class=\"wp-block-paragraph\"><strong>The challenge: <\/strong>Telephone researchers using random-digit dial (RDD) typically draw a sample of phone numbers from the geographic area being surveyed. For example, if they are polling in Ohio, they draw random samples of Ohio landlines and Ohio cellphone numbers. If people living there have a cellphone number from a different state, they won\u2019t be included in that sample. Similarly, people with numbers associated with Ohio but who are living elsewhere <em>will <\/em>be called, leading to wasted effort on the part of the pollster. Excluding a group of people such as those newly arrived to a state, especially one that\u2019s shown to be demographically distinct, has the potential to bias survey estimates.<\/p>\n\n<p class=\"wp-block-paragraph\">How big of a problem are out-of-area cellphone numbers for pollsters? The answer depends entirely on the geographic population they are trying to target. For national polls, it is not a problem. By dialing nationwide, pollsters reach a target population of \u201cU.S. adults\u201d regardless of any internal mismatch across cities and states. If they then want to report survey findings by geography, researchers can use respondents\u2019 self-reported location rather than the location associated with their cellphone. But for pollsters conducting surveys at the state or local level, especially in areas where the mismatch rate is high, out-of-area cellphone numbers are a serious concern. That\u2019s \u00a0because self-reported location is not available when selecting the sample of numbers to call.<\/p>\n\n<p class=\"wp-block-paragraph\"><strong>The research: <\/strong>A <a href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/2016\/08\/01\/moving-without-changing-your-cellphone-number-a-predicament-for-pollsters\">new study by Pew Research Center<\/a> reports that 10% of U.S. adults have a cellphone number that doesn\u2019t match the state where they actually live. For urban dwellers, some 41% have a number that doesn\u2019t match the <em>city<\/em> where they live. The trend is most pronounced in Washington, D.C., where 55% of residents with a cellphone have a number from outside the District. In some parts of the country, by contrast, this is almost a nonissue. In Michigan, for example, just 5% of adults with cellphones have an out-of-state number. While <a href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/2015\/11\/18\/advances-in-telephone-survey-sampling\/\">some technical solutions<\/a> are in the works, presently there is no surefire way to capture people with out-of-state or out-of-area numbers in subnational telephone polls of adults.<\/p>\n\n<p class=\"wp-block-paragraph\">Some practitioners, especially partisan pollsters, sidestep this problem by sampling from a voter file rather than using RDD. Voter files contain whatever telephone number people listed when they registered to vote. In some cases, that is a cellphone number people held on to during a move. Practitioners sticking with RDD, by contrast, typically either just ignore the problem and hope that it does not lead to bias or supplement their survey with cellphone numbers known to have a billing address in the state or city being surveyed.<\/p>\n\n<h2 id=\"election-polls\" class=\"wp-block-heading\">Election polls<\/h2>\n\n<p class=\"wp-block-paragraph\">In an election year, it\u2019s impossible to ignore the most prominent member of the polling family: \u201chorse race\u201d polling, so named because its primary use is to determine which candidate is ahead in a particular race at a particular point in time. As has been true in prior cycles, polling was used this year to determine which candidates would appear on stage in debates, as well as to predict the outcome of state party primaries and of course to predict the ultimate prize, the general election. Horse race polling is prone to most of the issues outlined in this report, but it also has an additional challenge that is unique to its purpose: the hunt for the elusive \u201clikely voter.\u201d<\/p>\n\n<p class=\"wp-block-paragraph\"><strong>The challenge:<\/strong> What sounds obvious is actually notoriously difficult to pull off. To figure out which candidate will win, you need to speak only to registered voters who will actually take the time to vote. And with general election turnout hovering around 60% of the adult population, that means you need to sort out a lot of people. Add to this the fact that admitting that they don\u2019t plan to vote makes some respondents feel bad enough that they indulge in a little wishful answering and you have a real problem. The challenge of predicting how many people will vote and who they will be adds an additional layer of potential error to election polling.<\/p>\n\n<p class=\"wp-block-paragraph\"><strong>The research: <\/strong>To solve this problem, researchers have developed a hoard of possible \u201c<a href=\"https:\/\/assets.pewresearch.org\/wp-content\/uploads\/sites\/5\/2014\/10\/Pew-Research-2012-Election-Questions.pdf\">likely voter models<\/a>\u201d that they can apply to their survey to try to identify those people who will actually turn out on Election Day. These models might, for example, ask respondents how much thought they have given to the election, whether they vote regularly and whether they know where people in their area go to cast their ballots.<\/p>\n\n<p class=\"wp-block-paragraph\">One interesting twist is that the modern era of access to actual \u201cvoter files\u201d \u2013 lists compiled by states and aggregated by commercial vendors that denote which individuals actually voted in which elections \u2013 means that researchers no longer have to rely solely on your word as to whether you will vote. They can also check past behavior. <a href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/2016\/01\/07\/can-likely-voter-models-be-improved\/\">Recent research from Pew Research Center<\/a> finds that models that include information about people\u2019s past voting behavior outperform others.<\/p>\n\n<p class=\"wp-block-paragraph\">Other researchers have come at the problem from a different angle. Rather than trying to perfect a likely voter model, they focus instead on fielding large samples and weighting them to very detailed models of the electorate. Specifically, some researchers have had <a href=\"https:\/\/5harad.com\/papers\/forecasting-with-nonrepresentative-polls.pdf\">success<\/a> using advanced statistical techniques, which have catchy acronyms like MrP (multilevel regression and post-stratification) and BART (Bayesian additive regression trees). These techniques can be used to leverage information about the profile of voters in past elections to adjust polling data for an upcoming election.<\/p>\n\n<p class=\"wp-block-paragraph\">Researchers applying these methods are less concerned with fielding a nationally representative sample and more concerned with producing heavily modeled estimates that reflect all available information about the electorate. They concede that their samples\u00a0 may be horribly biased but argue that that can be overcome through statistical modeling. Poll consumers interested in the cutting edge of opinion polling would do well to follow firms testing out these types of approaches.<\/p>\n\n<h2 id=\"response-rates\" class=\"wp-block-heading\">Response rates<\/h2>\n\n<p class=\"wp-block-paragraph\">Response rates measure how well pollsters are doing in reaching the people they set out to reach. They generally reflect two different but related phenomena: the extent to which researchers can contact their sampled respondent and the extent to which they can convince the respondent to do the interview.<\/p>\n\n<p class=\"wp-block-paragraph\"><strong>The challenge: <\/strong>Response rates, once seen as a primary indicator of a survey\u2019s quality, have been declining for <a href=\"http:\/\/www.nap.edu\/read\/18293\/chapter\/2\">decades<\/a>. They are in the single digits for many public polls conducted by nonprofit or commercial organizations. Though this certainly doesn\u2019t look good, nonresponse is only a problem for the poll\u2019s results if the people who don\u2019t talk to pollsters are markedly different than the people who do. Pew Research Center did its <a href=\"http:\/\/poq.oxfordjournals.org\/content\/64\/2\/125.short?rss=1&amp;ssource=mfc\">first major study of the impact of survey nonresponse in 1997<\/a>, when the Center\u2019s telephone response rate was 36%. When we <a href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/politics\/2012\/05\/15\/assessing-the-representativeness-of-public-opinion-surveys\/\">updated the study in 2012<\/a>, the average response rate was just 9%.<\/p>\n\n<p class=\"wp-block-paragraph\"><strong>The research:<\/strong> Declining response rates have been a fixation for pollsters, who track their ebb like hawks. While few would claim the single-digit rates are something to boast about, research thus far \u2013 including the <a href=\"https:\/\/www.washingtonpost.com\/news\/the-fix\/wp\/2016\/01\/28\/reports-of-the-demise-of-polls-have-been-greatly-exaggerated\/\">15-year track compiled by SSRS\u2019s David Dutwin,<\/a> below \u2013 has not suggested that decreasing response rates are leading to increasing error rates. On the other hand, one of the few types of nonresponse bias that has been consistently noted by researchers is the fact that those who participate in surveys are also more likely to participate in civic activities \u2013 they are more likely to be \u201cjoiners,\u201d in other words. Our own Scott Keeter and Drew DeSilver have a wrap-up on <a href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/short-reads\/2015\/07\/21\/the-challenges-of-polling-when-fewer-people-are-available-to-be-polled\/\">the challenges of polling when fewer people are available to be polled<\/a>, that covers this issue and others.<\/p>\n\n<figure class=\"wp-block-image aligncenter\"><a class=\"image-box\" href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/2016\/08\/01\/flashpoints-in-polling\/pm_2016-08-01_overview-00-00\/\" rel=\"attachment wp-att-281991\"><img data-dominant-color=\"ececec\" data-has-transparency=\"false\" style=\"--dominant-color: #ececec;\" decoding=\"async\" sizes=\"(max-width: 620px) 100vw, 620px\" class=\"wp-image-281991 not-transparent\" src=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/2016\/07\/PM_2016.08.01_overview-00-00.png\" alt=\"\" data-attachid=\"281991\"><\/a><\/figure>\n\n<h2 id=\"online-polls\" class=\"wp-block-heading\">Online polls<\/h2>\n\n<p class=\"wp-block-paragraph\">Online surveys have become a staple of the survey landscape. Enabled by the rise and reach of the internet, they allow researchers to recruit large numbers of Americans into a survey quickly and at low cost.<\/p>\n\n<p class=\"wp-block-paragraph\">While some online polls are recruited offline and based on probability samples (see our <a href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methodology\/u-s-survey-research\/american-trends-panel\/\">American Trends Panel<\/a>), the lion\u2019s share \u2013 and the focus of this section \u2013\u00a0 are nonprobability polls. These are polls for which respondents are not selected at random, with a known probability of selection, from a known population. Instead, these are mostly convenience samples, with people being recruited via techniques such as online banner ads. Some of these online polls are done with panels \u2013 groups of people who have agreed to respond to an ongoing series of surveys from the same company or organization, while others are one-off polls conducted with people intercepted in the course of some other online activity such as trying to access a website, checking a social media account or using a search engine.<\/p>\n\n<p class=\"wp-block-paragraph\"><strong>The challenge:<\/strong> Most online polls do not start with a random sample, so researchers must rely heavily on demographic quotas, weighting or other modeling techniques to produce their estimates. Generally speaking, this requires the pollster to adjust for all of the relevant ways that their online sample is different from the U.S. population.<\/p>\n\n<p class=\"wp-block-paragraph\">Of course, even well-designed telephone RDD polls require weighting. To date, though, most research has shown that online polls tend to have more error than RDD polls and, thus, require more modeling.<\/p>\n\n<p class=\"wp-block-paragraph\"><strong>The research: <\/strong>In 2016, online nonprobability surveys play a major role in the survey universe, and here at the Center we are eager to better understand their strengths and weaknesses. A <a href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/2016\/05\/02\/evaluating-online-nonprobability-surveys\/\">recent study by our methods team<\/a> looked at nine online surveys conducted by eight different vendors and found their results varied widely. The results from the top performer looked relatively good when compared with high quality government-sponsored benchmark surveys. Others performed less well. One key finding was that extreme caution should be exercised when using online polls to look at data among key racial and ethnic groups, as estimates for blacks and Hispanics based on the online sources tested were particularly inaccurate.<\/p>\n\n<p class=\"wp-block-paragraph\">As to the Center\u2019s current approach, our director of survey research recently explained <a href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/short-reads\/2016\/05\/02\/q-a-online-nonprobability-polls\/\">in an online Q&amp;A<\/a> that \u201cwe are planning more work on this topic, but for now it remains experimental. We are still using telephone random-digit-dialing as our primary design, though we employ nonprobability samples for questionnaire testing and for\u00a0<a href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/social-trends\/2015\/08\/19\/exploring-racial-bias-among-biracial-and-single-race-adults-the-iat\/\">certain specialized studies<\/a>.\u201d<\/p>\n\n<p class=\"wp-block-paragraph\">The leading trade association for survey researchers, the American Association for Public Opinion Research (AAPOR), also issued an in-depth <a href=\"https:\/\/www.aapor.org\/AAPOR_Main\/media\/MainSiteFiles\/NPS_TF_Report_Final_7_revised_FNL_6_22_13.pdf\">report<\/a> on nonprobability research in 2013.<\/p>\n\n<h2 id=\"nutrition-labels-for-polls\" class=\"wp-block-heading\">\u2018Nutrition labels\u2019 for polls<\/h2>\n\n<p class=\"wp-block-paragraph\">The differences in how polls are conducted can be vast. Yet to the average news consumer, such differences are nearly always opaque. Polls are treated as a commodity. Today, poll watchers can be as lost as grocery shoppers before nutrition labeling became mandatory in 1990. Some pollsters provide extensive information on their websites about how they conduct their work, but many do not.<\/p>\n\n<p class=\"wp-block-paragraph\"><strong>The problem:<\/strong> Standard reporting \u2013 which is typically just a poll\u2019s sample size and margin of error \u2013 masks important differences between polls in how they are collected and statistically adjusted. In some cases, the differences that are ignored explain why some polls are more accurate than others. Reporters need some way of differentiating polls that are reliable \u2013 or at least open about their methodology \u2013 from polls that are not.<\/p>\n\n<p class=\"wp-block-paragraph\"><strong>The research:<\/strong> Enter the <a href=\"http:\/\/www.aapor.org\/transparency.aspx\">Transparency Initiative<\/a> \u2013 the polling community\u2019s major push to encourage pollsters to be open about how they conduct their public polls. This initiative, which has polling organizations pledge to disclose certain essential details about their methods, was launched by AAPOR in 2014 and currently has more than 75 members, <a href=\"http:\/\/www.aapor.org\/Standards-Ethics\/Transparency-Initiative\/Current-Members.aspx\">including Pew Research Center<\/a>.<\/p>\n\n<p class=\"wp-block-paragraph\">The initiative makes no judgment about the quality or rigor of the methods being disclosed. It simply acknowledges members\u2019 willingness to publicly disclose their procedures. Much as food producers report things like calories and sodium, pollsters belonging to the Transparency Initiative are required to report how their sample was selected, the frame from which it\u00a0was drawn if there was one, and information about weighting adjustment, among other details. Belonging to the <a href=\"http:\/\/ncpp.org\/\">National Council on Public Polls<\/a> (NCPP) and depositing survey data with the <a href=\"https:\/\/ropercenter.cornell.edu\/\">Roper Center for Public Opinion Research<\/a> are similar, though perhaps less elaborate, indicators that a polling firm is committed to promoting high standards in polling.<\/p>\n\n<p class=\"wp-block-paragraph\">But is it doing any good? For data journalists and other reporters looking for a way to know whether to trust a poll, the answer is yes. According to FiveThirtyEight\u2019s <a href=\"http:\/\/fivethirtyeight.com\/features\/the-state-of-the-polls-2016\/\">Nate Silver<\/a>, \u201cParticipation in the AAPOR Transparency Initiative, NCPP or the Roper Center archive continues to be a strong predictor of polling accuracy.\u201d For poll watchers looking to find out what went into that poll they just consumed, it\u2019s a step in the right direction.<\/p>\n\n<h2 id=\"survey-research-meets-big-data\" class=\"wp-block-heading\">Survey research meets big data<\/h2>\n\n<p class=\"wp-block-paragraph\">There are now more ways than ever to find out what large numbers of Americans think, value and do. That\u2019s because the digital revolution has made it possible to see what they do via tracking of online behavior; read what they think on Facebook, Twitter and blogs; and determine what they value by where they spend their time online and what they purchase.<\/p>\n\n<p class=\"wp-block-paragraph\"><strong>The challenge:\u00a0 <\/strong>This kind of \u201cbig data\u201d is often in commercial or private hands, unavailable to researchers. It was often created for purposes other than research and so is not designed to answer the questions scholars want to ask of it. And it is most often not fully representative of the populations of interest. (A case in point: not everyone is on Twitter.)<\/p>\n\n<p class=\"wp-block-paragraph\">According to Robert Groves, a former Census Bureau director and current Georgetown University provost and Pew Research Center board member, survey researchers are in for 10 years of hard work as they try to grapple with the challenge of bringing this big data to bear on the field of public opinion. In his <a href=\"https:\/\/blog.provost.georgetown.edu\/during-the-incubation-of-a-blended-data-world\/\">blog<\/a>, Groves notes, \u201cThe days when a single data source deserves our complete confidence are limited in much social and economic measurement. The future will be won by those deeply attuned to the error properties of the data being blended.\u201d<\/p>\n\n<p class=\"wp-block-paragraph\"><strong>The research:<\/strong> AAPOR recently produced a comprehensive <a href=\"http:\/\/www.aapor.org\/AAPOR_Main\/media\/MainSiteFiles\/images\/BigDataTaskForceReport_FINAL_2_12_15_b.pdf\">report<\/a> on how survey researchers are trying to make use of big data. Pew Research Center launched Data Labs in the fall of 2015 to begin experimenting with new sources of data and new analysis techniques including machine learning, natural language processing and web scraping in order to supplement our more traditional lines of work in survey and demographic research. Watch this space for results over the upcoming months.<\/p>\n\n<p class=\"wp-block-paragraph\">\u00a0<\/p>","protected":false},"excerpt":{"rendered":"<p>Many people wonder: Can polls be trusted?  The following essay contains a big-picture review of the state of polling, organized around a number of key areas.<\/p>\n","protected":false},"author":370,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"sub_headline":"","sub_title":"","_crdt_document":"","_prc_public_revisions":[],"_ppp_expiration_hours":0,"_ppp_enabled":false,"ai_generated_summary":"","relatedPosts":[],"reportMaterials":[],"multiSectionReport":[{"key":"_migrate_0","postId":101646}],"package_parts__enabled":false,"package_parts":[],"_prc_fork_parent":0,"_prc_fork_status":"","_prc_active_fork":0,"datacite_doi":"","datacite_doi_citation":"","_prc_seo_qr_attachment_id":0,"spoken_article_player_enabled":true,"bylines":[],"acknowledgements":[],"displayBylines":true,"footnotes":"","prc_watchers":[]},"categories":[42,359,450,401,357,358],"tags":[],"bylines":[],"collection":[],"datasets":[],"level_of_effort":[],"primary_audience":[],"information_type":[],"_post_visibility":[],"formats":[458],"_fund_pool":[],"languages":[],"regions-countries":[],"research-teams":[528],"workflow-status":[],"class_list":["post-101635","post","type-post","status-publish","format-standard","hentry","category-election-2016","category-nonprobability-surveys","category-online-surveys","category-survey-basics","category-survey-methods","category-telephone-surveys","formats-report","research-teams-methods"],"label":false,"post_parent":0,"word_count":3315,"canonical_url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2016\/08\/01\/flashpoints-in-polling\/","art_direction":false,"_embeds":[],"watchers":[],"table_of_contents":[{"id":101635,"title":"Flashpoints in Polling","slug":"flashpoints-in-polling","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2016\/08\/01\/flashpoints-in-polling\/","is_active":true},{"id":101646,"title":"Acknowledgements","slug":"acknowledgements-4-4","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2016\/08\/01\/acknowledgements-4-4\/","is_active":false}],"report_materials":"","report_pagination":{"current_post":{"id":101635,"title":"Flashpoints in Polling","slug":"flashpoints-in-polling","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2016\/08\/01\/flashpoints-in-polling\/","is_active":true,"page_num":1},"next_post":{"id":101646,"title":"Acknowledgements","slug":"acknowledgements-4-4","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2016\/08\/01\/acknowledgements-4-4\/","is_active":false,"page_num":2},"previous_post":null,"pagination_items":[{"id":101635,"title":"Flashpoints in Polling","slug":"flashpoints-in-polling","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2016\/08\/01\/flashpoints-in-polling\/","is_active":true,"page_num":1},{"id":101646,"title":"Acknowledgements","slug":"acknowledgements-4-4","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2016\/08\/01\/acknowledgements-4-4\/","is_active":false,"page_num":2}]},"parent_info":{"parent_title":"Flashpoints in Polling","parent_id":101635},"materialsOrdered":[],"chaptersOrdered":[{"key":"_migrate_0","postId":101646}],"partsOrdered":[],"partsEnabled":false,"datacite_doi":"","prc_seo_data":{"title":"Flashpoints in Polling","description":"Many people wonder: Can polls be trusted?  The following essay contains a big-picture review of the state of polling, organized around a number of key areas.","og_title":"Flashpoints in Polling","og_description":"Many people wonder: Can polls be trusted?  The following essay contains a big-picture review of the state of polling, organized around a number of key areas.","schema_type":"Article","noindex":false,"canonical_url":"","primary_terms":[],"custom_schema":[],"og_image":0,"indexnow_submitted_at":null,"gsc_index_status":null},"prepublish_checks":{"prc-image-alt-text":{"status":"incomplete","message":"1 image is missing alt text.","data":{"count":1}},"prc-about-this-research":{"status":"incomplete","message":"Add an \"About this research\" details block.","data":null},"prc-paragraph-count":{"status":"complete","message":"Found 44 paragraphs.","data":{"count":44}},"prc-internal-link":{"status":"complete","message":"Found 16 internal links.","data":{"count":16}}},"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"relatedPostsOrdered":[],"bylinesOrdered":[],"acknowledgementsOrdered":[],"_links":{"self":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/posts\/101635","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/users\/370"}],"replies":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/comments?post=101635"}],"version-history":[{"count":1,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/posts\/101635\/revisions"}],"predecessor-version":[{"id":137563,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/posts\/101635\/revisions\/137563"}],"wp:attachment":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/media?parent=101635"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/categories?post=101635"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/tags?post=101635"},{"taxonomy":"bylines","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/bylines?post=101635"},{"taxonomy":"collection","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/collection?post=101635"},{"taxonomy":"datasets","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/datasets?post=101635"},{"taxonomy":"level_of_effort","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/level_of_effort?post=101635"},{"taxonomy":"primary_audience","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/primary_audience?post=101635"},{"taxonomy":"information_type","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/information_type?post=101635"},{"taxonomy":"_post_visibility","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_post_visibility?post=101635"},{"taxonomy":"formats","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/formats?post=101635"},{"taxonomy":"_fund_pool","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_fund_pool?post=101635"},{"taxonomy":"languages","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/languages?post=101635"},{"taxonomy":"regions-countries","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/regions-countries?post=101635"},{"taxonomy":"research-teams","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/research-teams?post=101635"},{"taxonomy":"workflow-status","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/workflow-status?post=101635"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}