This study measured the effect of survey mode of administration on respondents\u2019 answer choices across 27 different questions about political figures, policy preferences and core political values. Looking at such a broad range of questions provides a great deal of insight into the kinds of items that could be adversely affected by the choice of survey mode, but it also introduces complexities that need to be addressed in the analysis. Specifically, the more comparisons one makes, the greater the likelihood of finding statistically significant results not because of any real underlying difference but simply because of random variability. If the threshold for statistical significance is set at the conventional p=0.05 level, on average we would expect to see one in 20 comparisons show up as significant, even if there were no real underlying effect. A common approach to addressing this problem of multiple comparisons is to increase the p-value of each test to account for the number of tests being performed, reducing the number of significant findings.<\/p>\n\n
For this report, researchers chose not to adjust for multiple comparisons when discussing the experimental findings. However, when reserchers perform such an adjustment in this study using a technique known as the Benjamini-Hochberg procedure[12. numoffset=12″ Benjamini, Yoav and Yosef Hochberg. 1995. \u201cControlling the false discovery rate: a practical and powerful approach to multiple testing.\u201d Journal of the Royal Statistical Society Series B 57, 289\u2013300], the number of significant differences among the 27 primary mode comparisons drops from four to one[13. Only the difference in the share of the U.S. adults who favor a national effort to deport all immigrants who are now living in the U.S. illegally remains statistically significant after adjusting for multiple comparisons.].<\/p>\n\n
Why then discuss unadjusted significance tests throughout this report? Although there are technical reasons to believe that such corrections are too conservative, the decision primarily has to do with the consequences of being wrong. In this case, it is preferable to err on the side of detecting potential sources of bias so that they can be subjected to additional research and scrutiny, even if some turn out to be overstated, than it is for problems to slip by undetected.<\/p>\n\n
To this end, it is important not to confuse \u201csignificant\u201d with \u201ctrue\u201d and \u201cnonsignificant\u201d with \u201cfalse.\u201d Nonsignificant differences may represent effects that are real but small, and would require a larger sample size to measure with sufficient precision. Likewise, some results with p-values just under the 0.05 threshold could have easily been on the other side if random mode assignment or response had turned out slightly differently.<\/p>","protected":false},"excerpt":{"rendered":"
This study measured the effect of survey mode of administration on respondents\u2019 answer choices across 27 different questions about political figures, policy preferences and core political values. Looking at such a broad range of questions provides a great deal of insight into the kinds of items that could be adversely affected by the choice of […]<\/p>\n","protected":false},"author":367,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"sub_headline":"","sub_title":"","_prc_public_revisions":[],"_ppp_expiration_hours":0,"_ppp_enabled":false,"ai_generated_summary":"","_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"relatedPosts":[],"reportMaterials":[],"multiSectionReport":[],"package_parts__enabled":false,"package_parts":[],"_prc_fork_parent":0,"_prc_fork_status":"","_prc_active_fork":0,"datacite_doi":"","datacite_doi_citation":"","_prc_seo_qr_attachment_id":0,"spoken_article_player_enabled":true,"displayBylines":true,"footnotes":"","prc_watchers":[],"jetpack_post_was_ever_published":false},"categories":[68,359,357,358],"tags":[],"bylines":[],"collection":[],"datasets":[],"level_of_effort":[],"primary_audience":[],"information_type":[],"_post_visibility":[],"formats":[458],"_fund_pool":[],"languages":[],"regions-countries":[],"research-teams":[528],"workflow-status":[],"class_list":["post-101684","post","type-post","status-publish","format-standard","hentry","category-donald-trump","category-nonprobability-surveys","category-survey-methods","category-telephone-surveys","formats-report","research-teams-methods"],"label":false,"post_parent":101715,"word_count":423,"canonical_url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2017\/03\/31\/appendix-c-multiple-comparisons-adjustment\/","art_direction":{"A1":{"id":121153,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2017\/03\/PM_17.03.21_teamMode_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2017\/03\/PM_17.03.21_teamMode_feature.png?w=564&h=317&crop=1","width":564,"height":317,"chartArt":false},"A2":{"id":121154,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2017\/03\/PM_17.03.21_teamMode_crop.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2017\/03\/PM_17.03.21_teamMode_crop.png?w=268&h=151&crop=1","width":268,"height":151,"chartArt":false},"A3":{"id":121154,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2017\/03\/PM_17.03.21_teamMode_crop.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2017\/03\/PM_17.03.21_teamMode_crop.png?w=194&h=110&crop=1","width":194,"height":110,"chartArt":false},"A4":{"id":121154,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2017\/03\/PM_17.03.21_teamMode_crop.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2017\/03\/PM_17.03.21_teamMode_crop.png?w=268&h=151&crop=1","width":268,"height":151,"chartArt":false},"XL":{"id":121153,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2017\/03\/PM_17.03.21_teamMode_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2017\/03\/PM_17.03.21_teamMode_feature.png?w=640&h=320&crop=1","width":640,"height":320,"chartArt":false},"social":{"id":121153,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2017\/03\/PM_17.03.21_teamMode_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2017\/03\/PM_17.03.21_teamMode_feature.png?w=640&h=320&crop=1","width":640,"height":320,"chartArt":false}},"_embeds":[],"watchers":[],"table_of_contents":[{"id":101715,"title":"Are Telephone Polls Understating Support for Trump?","slug":"are-telephone-polls-understating-support-for-trump-2","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2017\/03\/31\/are-telephone-polls-understating-support-for-trump-2\/","is_active":false},{"id":101665,"title":"Acknowledgements","slug":"acknowledgements-7-3","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2017\/03\/31\/acknowledgements-7-3\/","is_active":false},{"id":101695,"title":"Methodology","slug":"methodology-5-8","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2017\/03\/31\/methodology-5-8\/","is_active":false},{"id":101706,"title":"Appendix A: Summary of mode differences by question","slug":"appendix-a-summary-of-mode-differences-by-question","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2017\/03\/31\/appendix-a-summary-of-mode-differences-by-question\/","is_active":false},{"id":101676,"title":"Appendix B: Mode effects as a source of error in political surveys","slug":"appendix-b-mode-effects-as-a-source-of-error-in-political-surveys","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2017\/03\/31\/appendix-b-mode-effects-as-a-source-of-error-in-political-surveys\/","is_active":false},{"id":101684,"title":"Appendix C: Multiple comparisons adjustment","slug":"appendix-c-multiple-comparisons-adjustment","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2017\/03\/31\/appendix-c-multiple-comparisons-adjustment\/","is_active":true}],"report_materials":"","report_pagination":{"current_post":{"id":101684,"title":"Appendix C: Multiple comparisons adjustment","slug":"appendix-c-multiple-comparisons-adjustment","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2017\/03\/31\/appendix-c-multiple-comparisons-adjustment\/","is_active":true,"page_num":6},"next_post":null,"previous_post":{"id":101676,"title":"Appendix B: Mode effects as a source of error in political surveys","slug":"appendix-b-mode-effects-as-a-source-of-error-in-political-surveys","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2017\/03\/31\/appendix-b-mode-effects-as-a-source-of-error-in-political-surveys\/","is_active":false,"page_num":5},"pagination_items":[{"id":101715,"title":"Are Telephone Polls Understating Support for Trump?","slug":"are-telephone-polls-understating-support-for-trump-2","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2017\/03\/31\/are-telephone-polls-understating-support-for-trump-2\/","is_active":false,"page_num":1},{"id":101665,"title":"Acknowledgements","slug":"acknowledgements-7-3","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2017\/03\/31\/acknowledgements-7-3\/","is_active":false,"page_num":2},{"id":101695,"title":"Methodology","slug":"methodology-5-8","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2017\/03\/31\/methodology-5-8\/","is_active":false,"page_num":3},{"id":101706,"title":"Appendix A: Summary of mode differences by question","slug":"appendix-a-summary-of-mode-differences-by-question","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2017\/03\/31\/appendix-a-summary-of-mode-differences-by-question\/","is_active":false,"page_num":4},{"id":101676,"title":"Appendix B: Mode effects as a source of error in political surveys","slug":"appendix-b-mode-effects-as-a-source-of-error-in-political-surveys","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2017\/03\/31\/appendix-b-mode-effects-as-a-source-of-error-in-political-surveys\/","is_active":false,"page_num":5},{"id":101684,"title":"Appendix C: Multiple comparisons adjustment","slug":"appendix-c-multiple-comparisons-adjustment","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2017\/03\/31\/appendix-c-multiple-comparisons-adjustment\/","is_active":true,"page_num":6}]},"parent_info":{"parent_title":"Are Telephone Polls Understating Support for Trump?","parent_id":101715},"materialsOrdered":[],"chaptersOrdered":[],"partsOrdered":[],"partsEnabled":false,"datacite_doi":"","prc_seo_data":{"title":"Are Telephone Polls Understating Support for Trump?","description":"This study measured the effect of survey mode of administration on respondents\u2019 answer choices across 27 different questions about political figures, policy preferences and core political values. Looking at such…","og_title":"Appendix C: Multiple comparisons adjustment","og_description":"","schema_type":"Article","noindex":false,"canonical_url":"","primary_terms":[],"custom_schema":[],"og_image":121153,"indexnow_submitted_at":null,"gsc_index_status":null},"prepublish_checks":{"prc-image-alt-text":{"status":"complete","message":"No image blocks in content.","data":null},"prc-about-this-research":{"status":"incomplete","message":"Add an \"About this research\" details block.","data":null},"prc-paragraph-count":{"status":"complete","message":"Found 4 paragraphs.","data":{"count":4}},"prc-internal-link":{"status":"incomplete","message":"Add at least one internal link.","data":{"count":0}}},"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"relatedPostsOrdered":[],"bylinesOrdered":[],"acknowledgementsOrdered":[],"_links":{"self":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/posts\/101684","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/users\/367"}],"replies":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/comments?post=101684"}],"version-history":[{"count":2,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/posts\/101684\/revisions"}],"predecessor-version":[{"id":134391,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/posts\/101684\/revisions\/134391"}],"wp:attachment":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/media?parent=101684"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/categories?post=101684"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/tags?post=101684"},{"taxonomy":"bylines","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/bylines?post=101684"},{"taxonomy":"collection","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/collection?post=101684"},{"taxonomy":"datasets","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/datasets?post=101684"},{"taxonomy":"level_of_effort","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/level_of_effort?post=101684"},{"taxonomy":"primary_audience","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/primary_audience?post=101684"},{"taxonomy":"information_type","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/information_type?post=101684"},{"taxonomy":"_post_visibility","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_post_visibility?post=101684"},{"taxonomy":"formats","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/formats?post=101684"},{"taxonomy":"_fund_pool","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_fund_pool?post=101684"},{"taxonomy":"languages","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/languages?post=101684"},{"taxonomy":"regions-countries","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/regions-countries?post=101684"},{"taxonomy":"research-teams","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/research-teams?post=101684"},{"taxonomy":"workflow-status","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/workflow-status?post=101684"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}