{"id":14290,"date":"2014-07-28T16:08:59","date_gmt":"2014-07-28T21:08:59","guid":{"rendered":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/short-reads\/%year%\/%monthnum%\/%day%\/qa-what-the-new-york-times-polling-decision-means\/"},"modified":"2024-04-14T03:22:10","modified_gmt":"2024-04-14T08:22:10","slug":"qa-what-the-new-york-times-polling-decision-means","status":"publish","type":"short-read","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/short-reads\/2014\/07\/28\/qa-what-the-new-york-times-polling-decision-means\/","title":{"rendered":"Q&A: What the New York Times\u2019 polling decision means"},"content":{"rendered":"

The New York Times<\/em> and CBS News made big news in the polling world this weekend when they announced<\/a> that they will begin using online survey panels from YouGov as part of their election coverage. YouGov, a U.K.-based research firm founded in 2000, uses such panels rather than traditional telephone surveys; the panel the Times <\/em>and CBS are using has more than 100,000 members. The Times, <\/em>citing concerns about the dearth of high-quality, non-partisan survey data, particularly at the state level, says it plans to include YouGov results as part of \u201ca diverse suite of surveys employing diverse methodologies.\u201d<\/p>\n\n

\"\"<\/a><\/p>\n

\"Keeter\"
Scott Keeter, the Pew Research Center’s director of survey research<\/figcaption><\/figure><\/figure>\n\n

While panels have long been used by market researchers, they\u2019re relatively new in the opinion-research field, and views on them are sharply divided. We asked Scott Keeter, the Pew Research Center\u2019s director of survey research, to explain the issues at stake and give us his preliminary thoughts.<\/p>\n\n

What\u2019s different about the YouGov panel surveys from the surveys previously used by the Times, <\/em>CBS and Pew Research?<\/strong><\/p>\n\n

There are two big differences. One is that these are conducted entirely online, among internet users. People who don\u2019t use the internet aren\u2019t included (more on this below). The other \u2013 and arguably the biggest — difference is that the samples for these surveys are selected using so-called non-probability sampling methods. For decades, only probability \u2013 or random \u2013 samples have been generally accepted as a scientific way to produce accurate, representative samples for surveys.<\/p>\n\n

Explain the difference between a probability and a non-probability sample.<\/strong><\/p>\n\n

The American Association for Public Opinion Research, the leading association of survey research professionals, has explained it well. Here\u2019s how their Task Force on Non-Probability Sampling<\/a> put it in their major report last year: \u201cIn a probability sample, everyone in the population of interest (e.g., all registered voters in a political poll) has a chance of being selected for an interview. Knowing those chances is critical to creating valid statistical estimates.\u201d Non-probability samples, in contrast, \u201care those in which the participants are chosen or choose themselves so that the chance of being selected is not known.\u201d<\/p>\n\n

[es]<\/p>\n\n\n

The biggest benefits of probability sampling are (1) the fact that the researchers are selecting respondents, rather than allowing respondents to volunteer to participate, and (2) that knowing the chances of selecting an individual allows us to estimate how much any given sample is likely to differ from what we would have found had we interviewed everyone in the population. That shows up as a survey\u2019s \u201cmargin of error.\u201d There is no comparable margin of error for non-probability samples.<\/p>\n\n

So why would the Times and CBS look to non-probability methods? Do probability-based methods such as random-digit dialing (RDD) telephone surveys have their own problems?<\/strong><\/p>\n\n

Indeed they do. As our previous research<\/a> on survey non-response has shown, it\u2019s increasingly difficult to reach and interview people. But there are two important points to make. First, telephone polls \u2013 including ours<\/a> — did a generally good job predicting elections in 2012 and in previous years. Our samples continue to look very much like the population<\/a> on most indicators for which we have reliable data from the Census and other high-quality sources.<\/p>\n\n

Second, the troubles faced by one method of research cannot, by themselves, be used to justify the adoption of an alternative. The alternative has to prove itself to be accurate enough and precise enough for the purposes to which we currently apply RDD telephone surveys. The field knows a lot about the biases and problems with our current methods. There is a much less complete understanding of the biases and problems with non-probability methods, but I\u2019m hopeful that we\u2019ll gain that understanding through a process of careful experimentation<\/p>\n\n

What does it mean for journalism, and for the survey world more generally, that the Times and CBS, of all news organizations, have decided to use YouGov\u2019s panel? Would you expect other news organizations to follow now that those two have broken the ice?<\/strong><\/p>\n\n

This is a very big deal in the survey world. Until now, no major news organization has put its brand on using surveys based on non-probability methods. The move has set off a very lively debate on Twitter among journalists and pollsters. There are strong opinions about the issue of non-probability samples. Here\u2019s one view, from Washington Post<\/em> pollster Scott Clement:<\/p>\n\n

News of the weekend: @nytimes<\/a> and @CBSNewsPoll<\/a> abandon decades of quality research methods.<\/p>— Scott Clement (@sfcpoll) July 27, 2014<\/a><\/blockquote>