{"id":111428,"date":"2021-04-21T16:23:00","date_gmt":"2021-04-21T21:23:00","guid":{"rendered":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/\/\/examining-how-survey-mode-affects-americans-views-of-international-affairs\/"},"modified":"2024-04-14T04:10:35","modified_gmt":"2024-04-14T09:10:35","slug":"examining-how-survey-mode-affects-americans-views-of-international-affairs","status":"publish","type":"decoded","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/2021\/04\/21\/examining-how-survey-mode-affects-americans-views-of-international-affairs\/","title":{"rendered":"Examining how survey mode affects Americans\u2019 views of international affairs"},"content":{"rendered":"\n
\"\"<\/a>
Pew Research Center illustration<\/figcaption><\/figure>\n\n\n\n

Over the past several years, Pew Research Center has migrated much of its U.S. polling<\/a> from phone to its online American Trends Panel<\/a> (ATP). Data collection for the U.S. component of the annual Global Attitudes Survey<\/a> \u2014 a major, cross-national study typically fielded in dozens of countries via face-to-face and phone interviews \u2014 followed suit this year<\/a>.<\/p>\n\n\n\n

Mindful that mode differences can affect response patterns as well as analysis of findings, we conducted a \u201cbridge\u201d survey in 2020, fielding the same Global Attitudes questionnaire online and by phone. While the Center has previously examined the implications of migrating online<\/a> for a variety of mostly domestically oriented survey items, this was our first extensive test of how mode might affect how Americans answer foreign policy and internationally focused questions.<\/p>\n\n\n\n

Our bridge survey included a total of 78 substantive opinion questions. In this post, we\u2019ll summarize the range of percentage-point differences across all items but concentrate our analysis on response patterns to four-point scales \u2014 the most common question format on the Global Attitudes Survey. Specifically, we\u2019ll examine how key four-point scale questions performed in terms of direction and intensity of responses, as well as rates of item non-response. Our findings underscore the importance of examining all three elements when interpreting questions and trends that have migrated from phone to online.<\/p>\n\n\n\n

We should note at the outset that we confirmed that the samples for the phone and online components of the bridge survey were comparable and that the online sample matched the characteristics of a typical ATP survey wave.<\/p>\n\n\n\n

Differences in attitudes by phone vs. online<\/h3>\n\n\n\n

The chart below summarizes the percentage-point difference between phone and online for each of the 78 survey items included in the bridge survey. Only the single largest difference by response categories for any given question is reported. For example, let\u2019s say 20% of our phone respondents said \u201cyes\u201d to a particular question, 15% said \u201cno\u201d and 65% said \u201cmaybe.\u201d And let\u2019s say 30% of our online respondents said \u201cyes\u201d to the same question, 10% said \u201cno\u201d and 60% said \u201cmaybe.\u201d In this hypothetical, we would report the 10-point difference for the \u201cyes\u201d category since it\u2019s the largest of the three.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Generally, to achieve statistical significance at the 95% confidence level, a difference of 4 points or higher is needed. When we look at the largest differences by response category, 73% of questions (57 of 78) met this threshold, including 19 survey items in which the margin of difference was 10 points or more.<\/p>\n\n\n\n

Two-thirds of these double-digit differences were associated with items that employed a four-point scale. These differences by mode will factor in how we analyze and report changes in Global Attitudes findings in the U.S. To inform that analysis, we took a closer look at response dynamics across<\/em> categories, focusing on key trend items that employ four-point scales.<\/p>\n\n\n\n

Trend questions, four-point scales and mode effects<\/h3>\n\n\n\n

As mentioned above, four-point scale questions were the most common question type on the Global Attitudes bridge survey (34 items, or 44% of all substantive opinion questions) and also the question type associated with some of the largest percentage-point differences across modes. While these overall findings are instructive, we sought to better understand the response dynamics within individual questions. This meant examining responses across all categories and assessing the implications for analysis and interpretation.<\/p>\n\n\n\n

Four-point scale items allow respondents to express their opinion in terms of direction (e.g., positive versus negative) and intensity (e.g., very, somewhat, not very much or none at all). Some of the Global Attitude Survey\u2019s most consistent trend questions employ four-point scales. For example, confidence in global leaders to do the right thing regarding world affairs is measured using a four-point scale: a lot of confidence, some confidence, not too much confidence or no confidence at all.<\/p>\n\n\n\n

The bridge survey posed this confidence question about six world leaders: German Chancellor Angela Merkel, French President Emanuel Macron, British Prime Minister Boris Johnson, Chinese President Xi Jinping, Russian President Vladimir Putin and then-U.S. President Donald Trump. For all of these leaders except Trump, there were double-digit differences across modes for one response category.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

As the chart reveals, for non-U.S. leaders, the response category \u201cnot too much confidence\u201d consistently exhibited the largest percentage-point differences across modes, ranging from 10 points for Putin to 14 points for Macron and Xi. The direction of difference was also consistent: online ATP respondents were more likely than phone respondents to answer \u201cnot too much confidence.\u201d Phone respondents, in turn, were consistently more likely to give the extreme response \u201cno confidence at all\u201d when asked about foreign leaders, though the mode differences were smaller, ranging from 2 points for Macron up to 7 points for Xi. (Part of these differences is attributable to higher item nonresponse on the phone, which is described below.)<\/p>\n\n\n\n

The response pattern diverged when Americans were asked about confidence in their own president. No single response category for Trump had a mode difference that exceeded 10 points, and the largest observed difference was in the share of respondents who answered \u201ca lot of confidence.\u201d<\/p>\n\n\n\n

The response patterns by mode for the questions about foreign leaders were fairly common across four-point scale items on the bridge survey. For instance, there were 12 four-point questions with differences of 5 points or more in the extreme negative response category, with 10 of them showing higher rates for phone respondents than ATP panelists.<\/p>\n\n\n\n

To get a better sense for what such mode differences might mean for the analysis of Global Attitudes findings, we took a closer look at response dynamics across all categories of key four-point scale trend items.<\/p>\n\n\n\n

One example was the question about confidence in Chinese president Xi. As noted, ATP panelists were 14 points more likely to say they had \u201cnot too much confidence\u201d in Xi compared to those on the phone, while those on the phone were 7 points more likely than online respondents to say they had \u201cno confidence at all.\u201d These are differences in intensity rather than direction of opinion. If we focus only on direction, by recoding the four categories into two (confidence versus no confidence), the single largest category difference by mode is reduced to 7 points (lack of confidence in Xi). The collapse of four-point scales to bi-modal response categories is a standard feature of Global Attitudes reporting. In this case, it highlights a meaningful consistency in the overall level of Americans\u2019 confidence in China\u2019s leader. Of course, the collapse of categories does not eliminate the mode effect, and the interpretation of the trend should take into account possible shifts in intensity of opinion, too.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Item nonresponse for four-point scale items is lower online than phone<\/h3>\n\n\n\n

The rate of item nonresponse is a frequent concern when migrating from interviewer-administered phone surveys to self-administered online polls. The ATP, like many web-based surveys, does not explicitly offer respondents the option of answering \u201cdon\u2019t know\u201d or \u201crefuse.\u201d In fact, the way ATP panelists would not respond to a question is by skipping it completely. And while \u201cdon\u2019t know\u201d or \u201crefuse\u201d options are not explicitly offered on traditional Global Attitudes phone surveys, respondents nonetheless give these answers with substantial frequency in interviewer-administered environments. The bridge survey confirmed this pattern: Phone respondents were more likely than their ATP counterparts to offer \u201cdon\u2019t know\u201d or \u201crefuse\u201d responses.<\/p>\n\n\n\n

Some of the four-point scale questions that had the largest nonresponse differences across modes were the items on confidence in world leaders to do the right thing in world affairs, with four of six items having higher nonresponse by phone than online (ranging from 5 to 12 points). Another set of four-point scale questions with relatively large item nonresponse differences were those about favorability toward other countries and international organizations. Of seven such questions, six had higher nonresponse rates by phone.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Looking at the example of American attitudes toward NATO, nonresponse rates were 16 points lower on ATP than by phone. This had a substantial impact when coupled with mode differences across the item\u2019s other four response categories, especially \u201csomewhat favorable\u201d and \u201csomewhat unfavorable.\u201d When we collapsed all four categories down to two, this lower item nonresponse rate led to both a more positive and<\/em> more negative opinion of NATO among ATP respondents (5 percentage points more positive and 10 points more negative). Thus, in the case of the NATO question, the migration from phone to online was associated with a change in two <\/em>directions, as well as differences in opinion intensity.<\/p>\n\n\n\n

Item nonresponse across modes differed less for the question measuring U.S. favorability than for parallel items about foreign countries and international organizations. The same was true for the question about confidence in Trump, compared with other international leaders. In each case, differences in intensity, more than direction, were evident across modes.<\/p>\n\n\n\n

Conclusions and further considerations<\/h3>\n\n\n\n

Insights from the Global Attitudes bridge survey in the U.S. suggest that issue salience \u2014 that is, how cognitively engaged the respondent is with a particular topic or issue \u2014 may play a role in response differences. For instance, we noted more striking mode effects when it came to relatively low-salience questions about foreign leaders as well as other countries and international organizations. By contrast, questions about the U.S. and its president showed more subdued mode differences with lower nonresponse. Managing how best to reliably measure low-salience issues across modes will be an ongoing challenge \u2014 especially as engagement with topics, as well as response biases, can vary across countries.<\/p>\n\n\n\n

Another takeaway from this study is that the analysis and interpretation of questions and trends spanning modes benefits from paying attention to both the direction and intensity of opinion, as reflected across response categories. A single question can reveal both consistencies and differences in response patterns as we migrate the Global Attitudes Survey from phone to online. The survey\u2019s tradition of collapsing four-point scales into dichotomous categories in its reporting may prove helpful in tracking high-level trends. At the same time, attention to how the intensity of opinion may be changing will be key to interpreting the strength and momentum of core trends going forward.<\/p>\n","protected":false},"excerpt":{"rendered":"

Responses to survey questions asked online can differ from those asked on the phone.<\/p>\n","protected":false},"author":377,"featured_media":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"sub_headline":"","sub_title":"","_crdt_document":"","_prc_public_revisions":[],"_ppp_expiration_hours":0,"_ppp_enabled":false,"ai_generated_summary":"","relatedPosts":[],"_prc_fork_parent":0,"_prc_fork_status":"","_prc_active_fork":0,"datacite_doi":"","datacite_doi_citation":"","_prc_seo_qr_attachment_id":0,"spoken_article_player_enabled":true,"bylines":[{"key":"_sfdqzwd9s","termId":789}],"acknowledgements":[],"displayBylines":true,"footnotes":"","prc_watchers":[]},"categories":[357],"bylines":[789],"collection":[],"_post_visibility":[],"decoded-category":[532],"formats":[454],"_fund_pool":[],"languages":[],"regions-countries":[],"research-teams":[524],"workflow-status":[],"class_list":["post-111428","decoded","type-decoded","status-publish","hentry","category-survey-methods","bylines-martha-mcroy","decoded-category-survey-methods","formats-decoded","research-teams-decoded"],"label":"Decoded","post_parent":0,"word_count":1680,"canonical_url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/2021\/04\/21\/examining-how-survey-mode-affects-americans-views-of-international-affairs\/","art_direction":{"A1":{"id":125870,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/2021.04.21_decoded_featured.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/2021.04.21_decoded_featured.png?w=564&h=317&crop=1","width":564,"height":317,"caption":"","chartArt":false},"A2":{"id":125870,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/2021.04.21_decoded_featured.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/2021.04.21_decoded_featured.png?w=268&h=151&crop=1","width":268,"height":151,"caption":"","chartArt":false},"A3":{"id":125870,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/2021.04.21_decoded_featured.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/2021.04.21_decoded_featured.png?w=194&h=110&crop=1","width":194,"height":110,"caption":"","chartArt":false},"A4":{"id":125870,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/2021.04.21_decoded_featured.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/2021.04.21_decoded_featured.png?w=268&h=151&crop=1","width":268,"height":151,"caption":"","chartArt":false},"XL":{"id":125870,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/2021.04.21_decoded_featured.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/2021.04.21_decoded_featured.png?w=720&h=405&crop=1","width":720,"height":405,"caption":"","chartArt":false},"social":{"id":125870,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/2021.04.21_decoded_featured.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/2021.04.21_decoded_featured.png?w=1200&h=628&crop=1","width":1200,"height":628,"caption":"","chartArt":false}},"_embeds":[],"watchers":[],"table_of_contents":[],"datacite_doi":"","prc_seo_data":{"title":"Examining how survey mode affects Americans\u2019 views of international affairs","description":"Responses to survey questions asked online can differ from those asked on the phone.","og_title":"Examining how survey mode affects Americans\u2019 views of international affairs","og_description":"Responses to survey questions asked online can differ from those asked on the phone.","schema_type":"Article","noindex":false,"canonical_url":"","primary_terms":{"category":43},"custom_schema":[],"og_image":125870,"indexnow_submitted_at":null,"gsc_index_status":null},"prepublish_checks":{},"jetpack_sharing_enabled":true,"relatedPostsOrdered":[],"bylinesOrdered":[{"key":"_sfdqzwd9s","termId":789}],"acknowledgementsOrdered":[],"_links":{"self":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111428","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded"}],"about":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/types\/decoded"}],"author":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/users\/377"}],"replies":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/comments?post=111428"}],"version-history":[{"count":2,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111428\/revisions"}],"predecessor-version":[{"id":138518,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111428\/revisions\/138518"}],"wp:attachment":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/media?parent=111428"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/categories?post=111428"},{"taxonomy":"bylines","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/bylines?post=111428"},{"taxonomy":"collection","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/collection?post=111428"},{"taxonomy":"_post_visibility","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_post_visibility?post=111428"},{"taxonomy":"decoded-category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded-category?post=111428"},{"taxonomy":"formats","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/formats?post=111428"},{"taxonomy":"_fund_pool","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_fund_pool?post=111428"},{"taxonomy":"languages","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/languages?post=111428"},{"taxonomy":"regions-countries","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/regions-countries?post=111428"},{"taxonomy":"research-teams","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/research-teams?post=111428"},{"taxonomy":"workflow-status","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/workflow-status?post=111428"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}