{"id":111165,"date":"2023-09-06T18:58:59","date_gmt":"2023-09-06T23:58:59","guid":{"rendered":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/\/\/how-adding-a-dont-know-response-option-can-affect-cross-national-survey-results\/"},"modified":"2024-04-14T04:10:23","modified_gmt":"2024-04-14T09:10:23","slug":"how-adding-a-dont-know-response-option-can-affect-cross-national-survey-results","status":"publish","type":"decoded","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/2023\/09\/06\/how-adding-a-dont-know-response-option-can-affect-cross-national-survey-results\/","title":{"rendered":"How adding a \u2018Don\u2019t know\u2019 response option can affect cross-national survey results"},"content":{"rendered":"\n
\"\"
Pew Research Center illustration<\/figcaption><\/figure>\n\n\n\n

Pew Research Center\u2019s international surveys have typically not given respondents the explicit option to say that they don\u2019t know the answer to a particular question. In surveys conducted face-to-face or by phone, we\u2019ve instead allowed respondents to voluntarily skip questions as they see fit. Similarly, in self-administered surveys conducted online, respondents can skip questions by selecting \u201cNext page\u201d and moving on. But the results of many survey experiments<\/a> show that people are much less likely to skip questions online than when speaking to interviewers in person or on the phone.<\/p>\n\n\n\n

This skipping effect raises a question for cross-national surveys that use different modes of interviewing people: Would it be better to give respondents an explicit \u201cDon\u2019t know\u201d option in online surveys for potentially low-salience questions even if the same is not<\/em> done in face-to-face or phone surveys?<\/p>\n\n\n\n

To explore this question, we conducted multiple split-form experiments on nationally representative online survey panels in the United States and Australia.1<\/a><\/sup> In both countries, we asked people about international political leaders and gave half of the sample the option to choose \u201cNever heard of this person\u201d while withholding this option for the other half of respondents.  <\/p>\n\n\n\n

Similarly, we asked questions in Australia assessing various elements of Chinese soft power such as the country\u2019s universities, military, standard of living, technological achievements and entertainment. We offered \u201cNot sure\u201d as a response option to half the sample, while the other half was not shown this option.<\/p>\n\n\n\n

In the sections that follow, we\u2019ll evaluate the impact of the \u201cNever heard of this person\u201d and \u201cNot sure\u201d options (which we\u2019ll jointly refer to as \u201cDon\u2019t know\u201d options) and assess whether adding such options can improve comparability across survey modes.<\/p>\n\n\n\n

How does adding a “Don’t know” option affect views of world leaders?<\/h3>\n\n\n\n

In both the U.S. and Australia, giving respondents the option to say they had never heard of certain world leaders resulted in more of them choosing that option. Specifically, we tend to see a shift of topline results from the more moderate response options \u2013 \u201cSome confidence\u201d and \u201cNot too much confidence\u201d \u2013 to the \u201cDon\u2019t know\u201d option. This was especially the case for leaders who received the largest shares of \u201cDon\u2019t know\u201d responses such as French President Emmanuel Macron, German Chancellor Olaf Scholz and Indian Prime Minister Narendra Modi.<\/p>\n\n\n\n

For example, in the U.S., when respondents were asked about their confidence in Modi without<\/em> a \u201cDon\u2019t know\u201d option, 34% said they had some confidence in the Indian leader and 42% said they had not too much confidence in him. But when we offered respondents the option to say they had never heard of Modi, the share saying they had some confidence in him dropped 14 percentage points and the share saying they had not too much confidence in him dropped 17 points. On the other hand, the share of respondents who expressed either of the two more forceful response options \u2013 that they had a lot of confidence in Modi or no confidence at all in him \u2013 were comparable whether the \u201cDon\u2019t know\u201d option was presented or not.<\/p>\n\n\n\n

\"A<\/a><\/figure>\n\n\n\n

While Australians were generally less likely than Americans to say they had never heard of the leaders in question, the response patterns were broadly similar between the two countries. For these lesser-known leaders, the share of respondents who simply refused to answer the question also decreased when respondents were able to say they did not know the leader.<\/p>\n\n\n\n

The experiment in the U.S. and Australia allowed us to observe other patterns. For example, in both countries, the groups most likely to choose the \u201cDon\u2019t know\u201d answer option<\/a> included women, people without a college degree and the youngest respondents.<\/p>\n\n\n\n

How does adding a \u201cDon\u2019t know\u201d option affect views of China\u2019s soft power?<\/h3>\n\n\n\n

To assess how a \u201cDon\u2019t know\u201d option might affect a different type of question, we ran the same experiment on a battery of questions in Australia that asked respondents to rate China\u2019s universities, standard of living, military, technological achievements and entertainment (including movies, music and television). Respondents could rate each of these aspects of Chinese soft power as the best, above average, average, below average or the worst compared with other wealthy nations.<\/p>\n\n\n\n

Here, too, adding a \u201cDon\u2019t know\u201d option resulted in a shift away from the middle answer options and toward the \u201cDon\u2019t know\u201d option. For example, when asked about Chinese universities without<\/em> the \u201cDon\u2019t know\u201d option, 47% of Australians rated them as average. But when we included the \u201cDon\u2019t know\u201d option, the share of Australians saying this dropped to 33%. Similarly, the share of Australians who rated Chinese entertainment as average fell from 40% without the \u201cDon\u2019t know\u201d option to 27% with it. There were few notable differences in the other response categories.<\/p>\n\n\n\n

How does adding a \u201cDon\u2019t know\u201d option change how we interpret cross-national findings?<\/h3>\n\n\n\n

We almost always analyze our international survey data in the context of how countries compare with one another. For example, we\u2019d like to know if people in a certain country are the most or least likely to have confidence in a given leader. However, as shown above, including an explicit \u201cDon\u2019t know\u201d option can meaningfully change our results and make it challenging to compare countries against one another in this way.<\/p>\n\n\n\n

For the questions used in this experiment, adding a \u201cDon\u2019t know\u201d option generally did not<\/em> result in major shifts in terms of how countries compare with one another. The main exception was Americans\u2019 confidence in Germany’s leader, Scholz.<\/p>\n\n\n\n

Yet even in his case, the inclusion of the \u201cDon\u2019t know\u201d option did not substantially change the major conclusions we might draw from cross-national comparisons: Americans\u2019 confidence in Scholz is relatively low, especially compared with the broad confidence he inspires in the Netherlands, Sweden and Germany itself<\/a>.<\/p>\n\n\n\n

\"A<\/a><\/figure>\n\n\n\n

We see a few potential reasons as to why there is a limited effect on the overall pattern of results. For one, there is a wide range of opinions across the countries surveyed: The share expressing confidence in Scholz ranges from 76% in the Netherlands to just 16% in Argentina. The 11-point difference between responses with and without a \u201cDon\u2019t know\u201d option in the U.S., therefore, is relatively small when compared with the overall range across the other 23 countries surveyed.<\/p>\n\n\n\n

Also, including the \u201cDon\u2019t know\u201d response did not seem to affect just positive or negative responses. Instead, there was similar attrition from the \u201cSome confidence\u201d and \u201cNot too much confidence\u201d categories into the \u201cDon\u2019t know\u201d category.<\/p>\n\n\n\n

Conclusion<\/h3>\n\n\n\n

We believe including an explicit \u201cDon\u2019t know\u201d option allows respondents to reflect their opinions more precisely in online surveys. While there may be significant effects on our topline survey results, our experiment does not indicate that adding a \u201cDon\u2019t know\u201d option would greatly change the conclusions we can draw from cross-national analysis. Nor does it show that demographic groups treated question formats differently. Therefore, we\u2019re inclined to offer a \u201cDon\u2019t know\u201d option on future online surveys that are part of our cross-national survey research efforts.<\/p>\n\n\n\n

While previous research shows that respondents are more likely to refuse to answer questions in face-to-face and phone surveys<\/a> than in online surveys, we do see potential for further inquiry as to how phone and face-to-face respondents would treat an explicit \u201cDon\u2019t know\u201d on low-salience questions.<\/p>\n\n\n\n

\n\n\n
  1. While the U.S. panel is fully online, the Australia panel is mixed-mode, and a small percentage (3%) of respondents elected to take the survey over the phone. They too were presented with an explicit \u201cDon\u2019t know\u201d option in one condition and their responses are included in this analysis. \u21a9\ufe0e<\/a><\/li><\/ol>","protected":false},"excerpt":{"rendered":"

    In our surveys, people are much less likely to skip questions online than when speaking to interviewers in person or on the phone; we explore how offering a “Don’t know” option in online surveys affects results. <\/p>\n","protected":false},"author":648,"featured_media":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"sub_headline":"","sub_title":"","_crdt_document":"","_prc_public_revisions":[],"_ppp_expiration_hours":0,"_ppp_enabled":false,"ai_generated_summary":"","relatedPosts":[],"_prc_fork_parent":0,"_prc_fork_status":"","_prc_active_fork":0,"datacite_doi":"","datacite_doi_citation":"","_prc_seo_qr_attachment_id":0,"spoken_article_player_enabled":true,"bylines":[{"key":"_txmp7t5f1","termId":591},{"key":"_jblwobuu2","termId":826},{"key":"_ymjzugrc6","termId":954}],"acknowledgements":[],"displayBylines":true,"footnotes":"[{\"id\":\"68bca8df-1410-48bf-9130-f47cb3b179d0\",\"content\":\"While the U.S. panel is fully online, the Australia panel is mixed-mode, and a small percentage (3%) of respondents elected to take the survey over the phone. They too were presented with an explicit \\u201cDon\\u2019t know\\u201d option in one condition and their responses are included in this analysis.\"}]","prc_watchers":[]},"categories":[357],"bylines":[826,591,954],"collection":[],"_post_visibility":[],"decoded-category":[532],"formats":[454],"_fund_pool":[],"languages":[],"regions-countries":[],"research-teams":[524],"workflow-status":[],"class_list":["post-111165","decoded","type-decoded","status-publish","hentry","category-survey-methods","bylines-janell-fetterolf","bylines-jordan-lippert","bylines-laura-silver","decoded-category-survey-methods","formats-decoded","research-teams-decoded"],"label":"Decoded","post_parent":0,"word_count":1288,"canonical_url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/2023\/09\/06\/how-adding-a-dont-know-response-option-can-affect-cross-national-survey-results\/","art_direction":{"A1":{"id":125612,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2023\/09\/d_2023.09.06_dk-responses_featured.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2023\/09\/d_2023.09.06_dk-responses_featured.png?w=564&h=317&crop=1","width":564,"height":317,"caption":"Pew Research Center illustration","chartArt":false},"A2":{"id":125612,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2023\/09\/d_2023.09.06_dk-responses_featured.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2023\/09\/d_2023.09.06_dk-responses_featured.png?w=268&h=151&crop=1","width":268,"height":151,"caption":"Pew Research Center illustration","chartArt":false},"A3":{"id":125612,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2023\/09\/d_2023.09.06_dk-responses_featured.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2023\/09\/d_2023.09.06_dk-responses_featured.png?w=194&h=110&crop=1","width":194,"height":110,"caption":"Pew Research Center illustration","chartArt":false},"A4":{"id":125612,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2023\/09\/d_2023.09.06_dk-responses_featured.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2023\/09\/d_2023.09.06_dk-responses_featured.png?w=268&h=151&crop=1","width":268,"height":151,"caption":"Pew Research Center illustration","chartArt":false},"XL":{"id":125612,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2023\/09\/d_2023.09.06_dk-responses_featured.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2023\/09\/d_2023.09.06_dk-responses_featured.png?w=720&h=405&crop=1","width":720,"height":405,"caption":"Pew Research Center illustration","chartArt":false},"social":{"id":125612,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2023\/09\/d_2023.09.06_dk-responses_featured.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2023\/09\/d_2023.09.06_dk-responses_featured.png?w=1200&h=628&crop=1","width":1200,"height":628,"caption":"Pew Research Center illustration","chartArt":false}},"_embeds":[],"watchers":[],"table_of_contents":[],"datacite_doi":"","prc_seo_data":{"title":"How \u2018Don\u2019t know\u2019 response options affect cross-national surveys | Decoded","description":"In our surveys, people are much less likely to skip questions online than when speaking to interviewers in person or on the phone; we explore how offering a \"Don't know\" option in online surveys affects results.","og_title":"How adding a \u2018Don\u2019t know\u2019 response option can affect cross-national survey results","og_description":"In our surveys, people are much less likely to skip questions online than when speaking to interviewers in person or on the phone; we explore how offering a \"Don't know\" option in online surveys affects results.","schema_type":"Article","noindex":false,"canonical_url":"","primary_terms":{"category":43,"research-teams":597},"custom_schema":[],"twitter_description":"In our surveys, people are much less likely to skip questions online than when speaking to interviewers in person or on the phone; we explore how offering a \"Don't know\" option in online surveys affects results.","og_image":125612,"indexnow_submitted_at":null,"gsc_index_status":null},"prepublish_checks":{},"jetpack_sharing_enabled":true,"relatedPostsOrdered":[],"bylinesOrdered":[{"key":"_txmp7t5f1","termId":591},{"key":"_jblwobuu2","termId":826},{"key":"_ymjzugrc6","termId":954}],"acknowledgementsOrdered":[],"_links":{"self":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111165","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded"}],"about":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/types\/decoded"}],"author":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/users\/648"}],"replies":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/comments?post=111165"}],"version-history":[{"count":2,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111165\/revisions"}],"predecessor-version":[{"id":138488,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111165\/revisions\/138488"}],"wp:attachment":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/media?parent=111165"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/categories?post=111165"},{"taxonomy":"bylines","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/bylines?post=111165"},{"taxonomy":"collection","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/collection?post=111165"},{"taxonomy":"_post_visibility","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_post_visibility?post=111165"},{"taxonomy":"decoded-category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded-category?post=111165"},{"taxonomy":"formats","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/formats?post=111165"},{"taxonomy":"_fund_pool","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_fund_pool?post=111165"},{"taxonomy":"languages","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/languages?post=111165"},{"taxonomy":"regions-countries","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/regions-countries?post=111165"},{"taxonomy":"research-teams","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/research-teams?post=111165"},{"taxonomy":"workflow-status","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/workflow-status?post=111165"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}