{"id":13477,"date":"2015-05-14T13:41:16","date_gmt":"2015-05-14T18:41:16","guid":{"rendered":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/short-reads\/%year%\/%monthnum%\/%day%\/where-web-surveys-produce-different-results-than-phone-interviews\/"},"modified":"2024-04-14T03:19:44","modified_gmt":"2024-04-14T08:19:44","slug":"where-web-surveys-produce-different-results-than-phone-interviews","status":"publish","type":"short-read","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/short-reads\/2015\/05\/14\/where-web-surveys-produce-different-results-than-phone-interviews\/","title":{"rendered":"Methods can matter: Where Web surveys produce different results than phone interviews"},"content":{"rendered":"

Over the past year, Pew Research Center\u00a0conducted an experiment<\/a>\u00a0to see if\u00a0the mode by which someone was surveyed \u2013 in this case, a telephone survey with an interviewer versus a self-administered survey on the Web \u2013 would have any effect on the answers people gave. We used two randomly selected groups from our\u00a0American Trends Panel<\/a>\u00a0to do this, asking both groups the same set of 60 questions.<\/p>\n\n

\"Respondents<\/a><\/figure>\n\n

The result?\u00a0Overall, our study found that it was fairly common to see differences in responses between those who took the survey with an interviewer by phone and those who took the survey on their own (self-administered) online, but typically the differences were not large. There was a mean difference of 5.5 percentage points and a median difference of 5\u00a0points across the 60 questions.<\/p>\n\n

But there were three broad types of questions that produced larger differences (known as mode effects) between the responses of those interviewed\u00a0by phone vs. Web. These differences are noteworthy given that many pollsters, market research firms and political organizations are increasingly turning to online surveys which, compared with phone surveys, are generally less expensive to produce and faster in\u00a0yielding\u00a0results.<\/p>\n\n

Here are three of the areas that showed the biggest mode gaps in responses from the phone and Web groups in our study:<\/p>\n\n

<\/p>\n\n

<\/strong><\/p>\n\n

\"People<\/a><\/figure>\n\n

People expressed more negative views of politicians in Web surveys than in phone surveys.\u00a0Those\u00a0who took\u00a0Web surveys were far more likely than those interviewed on the phone to give various political figures a \u201cvery unfavorable\u201d rating. This tendency\u00a0was especially concentrated among members of the opposite party of each figure rated.\u00a0Hillary Clinton\u2019s ratings are a good example of this pattern. When asked on the phone, 36% of Republicans and those who lean Republican told interviewers they had a \u201cvery unfavorable\u201d opinion of Clinton, but that number jumped to 53% on the Web. However, as with most of the political figures, Clinton\u2019s positive<\/em> ratings varied only modestly by mode \u2013\u00a053% rated her positively on the Web, compared with 57% on the phone.<\/p>\n\n

The same patterns seen with Clinton were also\u00a0evident for Republican political figures. Web respondents were 13 points more likely than phone respondents to have a \u201cvery unfavorable” view of Sarah Palin. Among Democrats and Democratic leaners, 63% expressed a very unfavorable view of Palin on the Web, compared with only 44% on the phone.<\/p>\n\n

People who took phone surveys were more likely than those who took Web surveys to say that certain groups of people \u2013\u00a0such as gays and lesbians, Hispanics, and blacks \u2013\u00a0faced \u201ca lot\u201d of discrimination<\/strong>. Here, the impact of the mode of interview varied by the race and ethnicity of the respondents.<\/p>\n\n

\"Sizeable<\/a><\/figure>\n\n

When asked about discrimination against gays and lesbians, 62% of respondents on the phone said they faced \u201ca lot\u201d of discrimination, but only 48% gave the same answer on the Web. The mode effect on this question appeared among both Democrats and Republicans.<\/p>\n\n

Telephone survey respondents were also more likely than Web respondents to say that Hispanics faced \u201ca lot\u201d of discrimination (54% in the phone survey, 42% in the Web survey). There was also a difference in response by mode among Hispanics questioned for this study: 41% on the Web said they faced discrimination, while 61% on the phone said this. The mode difference for white respondents was 14 points. But among black respondents, there was no significant mode effect: 66% of blacks interviewed by phone said Hispanics face a lot of discrimination, while 61% of those interviewed on the Web said the same.<\/p>\n\n

When asked about discrimination against blacks, more phone respondents (54%) than Web respondents (44%) said they faced a lot of discrimination. This pattern was clear among whites, where 50% on the phone and just 37% on the Web said blacks faced a lot of discrimination. But among blacks, the pattern was reversed: 71% of black respondents interviewed by phone say they faced \u201ca lot\u201d of discrimination, while on the Web 86% gave this answer.<\/p>\n\n

People were more likely to say they are happy with their family and social life when asked by a person over the phone\u00a0than\u00a0when answering questions on the Web<\/strong>. Among phone survey respondents, 62% said they were \u201cvery satisfied\u201d with their family life, while just 44% of Web respondents said this. Asked about their social life, 43% of phone respondents said they were very satisfied, while just 29% of Web respondents gave that answer. These sizable differences are evident among most social and demographic groups in the survey.<\/p>\n\n

Respondents were also asked how satisfied they were with their local community as a place to live. Phone respondents were again more positive, with 37% rating their community as an excellent place to live, compared with 30% of Web respondents. But there was no significant difference by mode in the percentage who gave a negative rating of\u00a0their community (\u201conly fair\u201d or \u201cpoor\u201d).<\/p>\n\n

So, what\u2019s going on here?<\/strong> These examples are consistent with the theory that when people are interacting with an interviewer, they are more likely to give answers that paint themselves or their communities in a positive light, and less likely to portray themselves negatively. This appears to be the case with the questions presenting the largest differences in the study \u2013\u00a0satisfaction with family and social life, as well as questions about the ability to pay for food and medical care. These findings are consistent with other research that has found that when there is a human interviewer, respondents tend to give answers that\u00a0would\u00a0be\u00a0considered\u00a0more socially desirable \u2013\u00a0a phenomenon known as the \u201csocial desirability bias.\u201d<\/p>\n\n

On the political questions, however, other recent research<\/a> has suggested that when interviewers are presenting the questions, respondents may choose answers that are less likely to produce an uncomfortable interaction with the interviewer.\u00a0This dynamic may also be in effect among black respondents on the phone who \u2013 compared with those surveyed on the Web \u2013 are less likely to tell an interviewer that blacks face a lot of discrimination. In the interest of maintaining rapport with an interviewer, respondents may self-censor or moderate their views in ways that they would not online.<\/p>\n\n

Altogether, our\u00a0findings suggest that there may be advantages to online surveys, particularly if the survey seeks to measure topics that are sensitive or subject to social desirability\u00a0because of the willingness of respondents to express more negative attitudes about their personal lives or toward political figures on the Web.<\/p>\n\n

That said, researchers need to carefully consider the trade-offs between the two survey modes.\u00a0This study only looked at differences in how people answer questions differently online and over the phone. But the survey mode can affect what kinds of people are included in the survey as well.\u00a0Telephone surveys continue to provide access to survey samples that are broadly representative of the general public, even in the face of declining response rates. Many Americans still lack reliable access to the internet, and traditional phone surveys have been found to perform better than many probability-based Web surveys among some respondents, including financially struggling individuals, those with low levels of education, and minorities with low language proficiency.\u00a0Given these trade-offs, we are continuously working\u00a0to make our traditional methods even more robust<\/a>\u00a0in addition to exploring new methods for understanding public opinion.<\/p>","protected":false},"excerpt":{"rendered":"

A Pew Research Center experiment found several key areas where Web surveys produced different results than those conducted by phone.<\/p>\n","protected":false},"author":74,"featured_media":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"sub_headline":null,"sub_title":"","_prc_public_revisions":[],"_ppp_expiration_hours":0,"_ppp_enabled":false,"ai_generated_summary":"","apple_news_api_created_at":"","apple_news_api_id":"","apple_news_api_modified_at":"","apple_news_api_revision":"","apple_news_api_share_url":"","apple_news_cover_media_provider":"image","apple_news_coverimage":0,"apple_news_coverimage_caption":"","apple_news_cover_video_id":0,"apple_news_cover_video_url":"","apple_news_cover_embedwebvideo_url":"","apple_news_is_hidden":"","apple_news_is_paid":"","apple_news_is_preview":"","apple_news_is_sponsored":"","apple_news_maturity_rating":"","apple_news_metadata":"\"\"","apple_news_pullquote":"","apple_news_pullquote_position":"","apple_news_slug":"","apple_news_sections":[],"apple_news_suppress_video_url":false,"apple_news_use_image_component":false,"relatedPosts":[],"_prc_fork_parent":0,"_prc_fork_status":"","_prc_active_fork":0,"datacite_doi":"","datacite_doi_citation":"","_prc_seo_qr_attachment_id":0,"spoken_article_player_enabled":true,"displayBylines":true,"footnotes":"","prc_watchers":[]},"categories":[101,88,36,450,28,357,358],"bylines":[967],"collection":[],"datasets":[],"_post_visibility":[],"formats":[467],"_fund_pool":[],"languages":[],"regions-countries":[515],"research-teams":[528],"workflow-status":[],"class_list":["post-13477","short-read","type-short-read","status-publish","hentry","category-discrimination-prejudice","category-leaders","category-methodological-research","category-online-surveys","category-race-ethnicity","category-survey-methods","category-telephone-surveys","bylines-scott-keeter","formats-short-read","regions-countries-united-states","research-teams-methods"],"label":"Short Read","post_parent":0,"word_count":1189,"canonical_url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/short-reads\/2015\/05\/14\/where-web-surveys-produce-different-results-than-phone-interviews\/","art_direction":{"A1":{"id":24778,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2015\/05\/mode-study-01.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2015\/05\/mode-study-01.png?w=312&h=317&crop=1","width":312,"height":317,"chartArt":false},"A2":{"id":24778,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2015\/05\/mode-study-01.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2015\/05\/mode-study-01.png?w=268&h=151&crop=1","width":268,"height":151,"chartArt":false},"A3":{"id":24778,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2015\/05\/mode-study-01.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2015\/05\/mode-study-01.png?w=194&h=110&crop=1","width":194,"height":110,"chartArt":false},"A4":{"id":24778,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2015\/05\/mode-study-01.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2015\/05\/mode-study-01.png?w=268&h=151&crop=1","width":268,"height":151,"chartArt":false},"XL":{"id":24778,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2015\/05\/mode-study-01.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2015\/05\/mode-study-01.png?w=312&h=405&crop=1","width":312,"height":405,"chartArt":false},"social":{"id":24778,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2015\/05\/mode-study-01.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2015\/05\/mode-study-01.png?w=312&h=449&crop=1","width":312,"height":449,"chartArt":false}},"_embeds":[],"watchers":[],"table_of_contents":[],"datacite_doi":"","prc_seo_data":{"title":"Methods can matter: Where Web surveys produce different results than phone interviews","description":"A Pew Research Center experiment found several key areas where Web surveys produced different results than those conducted by phone.","og_title":"Methods can matter: Where Web surveys produce different results than phone interviews","og_description":"A Pew Research Center experiment found several key areas where Web surveys produced different results than those conducted by phone.","schema_type":"Article","noindex":false,"canonical_url":"","primary_terms":[],"custom_schema":[],"og_image":24778,"indexnow_submitted_at":null,"gsc_index_status":null},"prepublish_checks":{},"apple_news_notices":[],"jetpack_sharing_enabled":true,"relatedPostsOrdered":[],"bylinesOrdered":[{"key":"d64c7b9e1c378c2d7000ebeee6338c8e","termId":967}],"acknowledgementsOrdered":[],"_links":{"self":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/short-read\/13477","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/short-read"}],"about":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/types\/short-read"}],"author":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/users\/74"}],"replies":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/comments?post=13477"}],"version-history":[{"count":2,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/short-read\/13477\/revisions"}],"predecessor-version":[{"id":103253,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/short-read\/13477\/revisions\/103253"}],"wp:attachment":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/media?parent=13477"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/categories?post=13477"},{"taxonomy":"bylines","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/bylines?post=13477"},{"taxonomy":"collection","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/collection?post=13477"},{"taxonomy":"datasets","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/datasets?post=13477"},{"taxonomy":"_post_visibility","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_post_visibility?post=13477"},{"taxonomy":"formats","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/formats?post=13477"},{"taxonomy":"_fund_pool","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_fund_pool?post=13477"},{"taxonomy":"languages","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/languages?post=13477"},{"taxonomy":"regions-countries","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/regions-countries?post=13477"},{"taxonomy":"research-teams","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/research-teams?post=13477"},{"taxonomy":"workflow-status","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/workflow-status?post=13477"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}