{"id":101758,"date":"2016-05-02T09:50:38","date_gmt":"2016-05-02T14:50:38","guid":{"rendered":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/2016\/05\/02\/overall-variability-in-estimates-across-samples\/"},"modified":"2024-04-14T04:14:39","modified_gmt":"2024-04-14T09:14:39","slug":"overall-variability-in-estimates-across-samples","status":"publish","type":"post","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2016\/05\/02\/overall-variability-in-estimates-across-samples\/","title":{"rendered":"4. Overall variability in estimates across samples"},"content":{"rendered":"<p class=\"wp-block-paragraph\">Generally, we expect that surveys representing the same target population should produce estimates that are consistent with one another; that is, measures taken at roughly the same time should not vary dramatically from survey to survey.<\/p>\n\n<figure class=\"wp-block-image alignright\"><a href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/2016\/05\/02\/overall-variability-in-estimates-across-samples\/pm_2016-05-02_nonprobability-ch4-00\/\" rel=\"attachment wp-att-279696\"><img decoding=\"async\" class=\"wp-image-279696\" src=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/2016\/04\/PM_2016.05.02_nonprobability-ch4-00.png\" alt=\"Estimates from different online samples run the gamut from highly variable to highly consistent\" ><\/a><\/figure>\n\n<p class=\"wp-block-paragraph\">For some of the items in this study, this is the case. For example, the highest estimate for the proportion never married is only 4.4 percentage points higher than the lowest. The proportion living at their residence for less than a year and the proportion with a driver\u2019s license are similarly consistent across samples. At the other end of the spectrum, estimates of the proportion who knew that the Republican Party controlled both the House of Representatives and Senate span over a range of 26 percentage points, from 52% (sample I) to 78% (American Trends Panel) with the others dispersed in between.<\/p>\n\n<p class=\"wp-block-paragraph\">Other items exhibit more idiosyncratic patterns. The estimates of the share of adults who worry that computers and technology are being used to\u00a0invade their privacy is densely clustered around 27% except for sample E, the low outlier at 21% and the ATP, the high outlier at 34%.<\/p>\n\n<p class=\"wp-block-paragraph\">As we might expect, items closer to zero or 100% tend to display somewhat less variability between samples than items closer to 50%, though several items that fall near 50% exhibit low variability as well. For instance, the grand mean (average across all 10 samples) for the percentage who rate their health as excellent or very good is 47%, with samples differing from that by an average of 2.1 percentage points. The grand mean for the share that believes the government should do more to solve problems is 54% with an average deviation of 1.9 points.<\/p>\n\n<p class=\"wp-block-paragraph\">Other than these ceiling and floor effects, there are no clear patterns that explain why some items vary across samples and others do not. Behavioral and attitudinal items are both well represented at both ends of the spectrum, as are potentially sensitive or socially desirable items. We also do not know if these patterns would persist if we were to repeat this exercise, as we lack measures of consistency over repeated surveys from the same sample sources.<\/p>","protected":false},"excerpt":{"rendered":"<p>Generally, we expect that surveys representing the same target population should produce estimates that are consistent with one another; that is, measures taken at roughly the same time should not vary dramatically from survey to survey. For some of the items in this study, this is the case. For example, the highest estimate for the [&hellip;]<\/p>\n","protected":false},"author":346,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"sub_headline":"","sub_title":"","_prc_public_revisions":[],"_ppp_expiration_hours":0,"_ppp_enabled":false,"ai_generated_summary":"","relatedPosts":[],"reportMaterials":[],"multiSectionReport":[],"package_parts__enabled":false,"package_parts":[],"_prc_fork_parent":0,"_prc_fork_status":"","_prc_active_fork":0,"datacite_doi":"","datacite_doi_citation":"","_prc_seo_qr_attachment_id":0,"spoken_article_player_enabled":true,"displayBylines":true,"footnotes":"","prc_watchers":[]},"categories":[],"tags":[],"bylines":[],"collection":[],"datasets":[],"level_of_effort":[],"primary_audience":[],"information_type":[],"_post_visibility":[],"formats":[458],"_fund_pool":[],"languages":[],"regions-countries":[],"research-teams":[528],"workflow-status":[],"class_list":["post-101758","post","type-post","status-publish","format-standard","hentry","formats-report","research-teams-methods"],"label":false,"post_parent":101714,"word_count":345,"canonical_url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2016\/05\/02\/overall-variability-in-estimates-across-samples\/","art_direction":false,"_embeds":[],"watchers":[],"table_of_contents":[{"id":101714,"title":"Evaluating Online Nonprobability Surveys","slug":"evaluating-online-nonprobability-surveys","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2016\/05\/02\/evaluating-online-nonprobability-surveys\/","is_active":false},{"id":101729,"title":"1. Assessing the accuracy of online nonprobability surveys","slug":"assessing-the-accuracy-of-online-nonprobability-surveys","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2016\/05\/02\/assessing-the-accuracy-of-online-nonprobability-surveys\/","is_active":false},{"id":101740,"title":"2. Accuracy in estimating multivariate relationships","slug":"accuracy-in-estimating-multivariate-relationships","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2016\/05\/02\/accuracy-in-estimating-multivariate-relationships\/","is_active":false},{"id":101750,"title":"3. Demographic, political and interest profiles","slug":"demographic-political-and-interest-profiles","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2016\/05\/02\/demographic-political-and-interest-profiles\/","is_active":false},{"id":101758,"title":"4. Overall variability in estimates across samples","slug":"overall-variability-in-estimates-across-samples","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2016\/05\/02\/overall-variability-in-estimates-across-samples\/","is_active":true},{"id":101768,"title":"5. Variation in online nonprobability survey design","slug":"variation-in-online-nonprobability-survey-design","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2016\/05\/02\/variation-in-online-nonprobability-survey-design\/","is_active":false},{"id":101779,"title":"Acknowledgements","slug":"acknowledgements-2-5","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2016\/05\/02\/acknowledgements-2-5\/","is_active":false}],"report_materials":"","report_pagination":{"current_post":{"id":101758,"title":"4. Overall variability in estimates across samples","slug":"overall-variability-in-estimates-across-samples","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2016\/05\/02\/overall-variability-in-estimates-across-samples\/","is_active":true,"page_num":5},"next_post":{"id":101768,"title":"5. Variation in online nonprobability survey design","slug":"variation-in-online-nonprobability-survey-design","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2016\/05\/02\/variation-in-online-nonprobability-survey-design\/","is_active":false,"page_num":6},"previous_post":{"id":101750,"title":"3. Demographic, political and interest profiles","slug":"demographic-political-and-interest-profiles","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2016\/05\/02\/demographic-political-and-interest-profiles\/","is_active":false,"page_num":4},"pagination_items":[{"id":101714,"title":"Evaluating Online Nonprobability Surveys","slug":"evaluating-online-nonprobability-surveys","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2016\/05\/02\/evaluating-online-nonprobability-surveys\/","is_active":false,"page_num":1},{"id":101729,"title":"1. Assessing the accuracy of online nonprobability surveys","slug":"assessing-the-accuracy-of-online-nonprobability-surveys","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2016\/05\/02\/assessing-the-accuracy-of-online-nonprobability-surveys\/","is_active":false,"page_num":2},{"id":101740,"title":"2. Accuracy in estimating multivariate relationships","slug":"accuracy-in-estimating-multivariate-relationships","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2016\/05\/02\/accuracy-in-estimating-multivariate-relationships\/","is_active":false,"page_num":3},{"id":101750,"title":"3. Demographic, political and interest profiles","slug":"demographic-political-and-interest-profiles","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2016\/05\/02\/demographic-political-and-interest-profiles\/","is_active":false,"page_num":4},{"id":101758,"title":"4. Overall variability in estimates across samples","slug":"overall-variability-in-estimates-across-samples","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2016\/05\/02\/overall-variability-in-estimates-across-samples\/","is_active":true,"page_num":5},{"id":101768,"title":"5. Variation in online nonprobability survey design","slug":"variation-in-online-nonprobability-survey-design","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2016\/05\/02\/variation-in-online-nonprobability-survey-design\/","is_active":false,"page_num":6},{"id":101779,"title":"Acknowledgements","slug":"acknowledgements-2-5","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2016\/05\/02\/acknowledgements-2-5\/","is_active":false,"page_num":7}]},"parent_info":{"parent_title":"Evaluating Online Nonprobability Surveys","parent_id":101714},"materialsOrdered":[],"chaptersOrdered":[],"partsOrdered":[],"partsEnabled":false,"datacite_doi":"","prc_seo_data":{"title":"4. Overall variability in estimates across samples","description":"As the costs and nonresponse rates of traditional, probability-based surveys seem to grow each year, the advantages of online surveys are obvious \u2013 they are fast and cheap, and the technology is pervasive. There is, however, one fundamental problem: There is no comprehensive sampling frame for the internet, no way to draw a national sample for which virtually everyone has a chance of being selected.","og_title":"4. Overall variability in estimates across samples","og_description":"","schema_type":"Article","noindex":false,"canonical_url":"","primary_terms":[],"custom_schema":[],"og_image":0,"indexnow_submitted_at":null,"gsc_index_status":null},"prepublish_checks":{"prc-image-alt-text":{"status":"complete","message":"All images have alt text.","data":null},"prc-about-this-research":{"status":"incomplete","message":"Add an \"About this research\" details block.","data":null},"prc-paragraph-count":{"status":"complete","message":"Found 5 paragraphs.","data":{"count":5}},"prc-internal-link":{"status":"complete","message":"Found 1 internal link.","data":{"count":1}}},"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"relatedPostsOrdered":[],"bylinesOrdered":[],"acknowledgementsOrdered":[],"_links":{"self":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/posts\/101758","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/users\/346"}],"replies":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/comments?post=101758"}],"version-history":[{"count":2,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/posts\/101758\/revisions"}],"predecessor-version":[{"id":134381,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/posts\/101758\/revisions\/134381"}],"wp:attachment":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/media?parent=101758"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/categories?post=101758"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/tags?post=101758"},{"taxonomy":"bylines","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/bylines?post=101758"},{"taxonomy":"collection","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/collection?post=101758"},{"taxonomy":"datasets","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/datasets?post=101758"},{"taxonomy":"level_of_effort","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/level_of_effort?post=101758"},{"taxonomy":"primary_audience","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/primary_audience?post=101758"},{"taxonomy":"information_type","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/information_type?post=101758"},{"taxonomy":"_post_visibility","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_post_visibility?post=101758"},{"taxonomy":"formats","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/formats?post=101758"},{"taxonomy":"_fund_pool","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_fund_pool?post=101758"},{"taxonomy":"languages","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/languages?post=101758"},{"taxonomy":"regions-countries","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/regions-countries?post=101758"},{"taxonomy":"research-teams","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/research-teams?post=101758"},{"taxonomy":"workflow-status","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/workflow-status?post=101758"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}