{"id":101111,"date":"2023-09-07T17:53:21","date_gmt":"2023-09-07T22:53:21","guid":{"rendered":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/2023\/09\/07\/comparing-two-types-of-online-survey-samples\/"},"modified":"2025-04-23T23:59:53","modified_gmt":"2025-04-24T03:59:53","slug":"comparing-two-types-of-online-survey-samples","status":"publish","type":"post","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2023\/09\/07\/comparing-two-types-of-online-survey-samples\/","title":{"rendered":"Comparing Two Types of Online Survey Samples"},"content":{"rendered":"<h2 class=\"has-text-align-left wp-block-prc-block-subtitle\" aria-level=\"2\">Opt-in samples are about half as accurate as probability-based panels<\/h2>\n\n<div style=\"border-width:1px;padding-bottom:var(--wp--preset--spacing--20);padding-left:var(--wp--preset--spacing--30);padding-right:var(--wp--preset--spacing--30);padding-top:var(--wp--preset--spacing--20);--block-gap: inherit\" class=\"wp-block-prc-block-collapsible has-background has-ui-beige-very-light-background-color has-border-color has-ui-beige-dark-border-color\" id=\"how-we-did-this\" data-wp-interactive=\"{&quot;namespace&quot;:&quot;prc-block\\\/collapsible&quot;}\" data-wp-context=\"{&quot;collapsibleId&quot;:&quot;how-we-did-this&quot;,&quot;isOpen&quot;:false}\" data-wp-class--is-open=\"context.isOpen\" data-wp-init--scroll-into-view=\"callbacks.onInitScrollIntoView\"><div class=\"wp-block-prc-block-collapsible__title\" data-wp-on--click=\"actions.onClick\"><div>How we did this<\/div><button class=\"wp-block-prc-block-collapsible__icon\"><span data-wp-bind--hidden=\"context.isOpen\"><i class=\"icon icon-library__light icon__circle-plus\"><svg style=\"width: 1em; height: 1em;\"><use xlink:href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/plugins\/prc-icon-library\/build\/icons\/sprites\/light.svg#circle-plus\"><\/use><\/svg><\/i><\/span><span data-wp-bind--hidden=\"!context.isOpen\" hidden><i class=\"icon icon-library__light icon__circle-minus\"><svg style=\"width: 1em; height: 1em;\"><use xlink:href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/plugins\/prc-icon-library\/build\/icons\/sprites\/light.svg#circle-minus\"><\/use><\/svg><\/i><\/span><\/button><\/div><div class=\"wp-block-prc-block-collapsible__content\">\n\n<p class=\"wp-block-paragraph\">Pew Research Center designed this study to assess the current state of online survey sampling methods \u2013 both probability-based and opt-in \u2013 and determine their accuracy on general population estimates for all U.S. adults and on estimates for key demographic subgroups. To do this, we administered a common questionnaire to samples obtained from three probability-based online panels, one of which was the Center\u2019s American Trends Panel (ATP), and three online opt-in sample providers. The surveys were administered between June 14 and July 21, 2021, and included interviews with a total of 29,937 U.S. adults.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The target size for each sample was 5,000 adults to minimize the impact of sampling error on estimates for demographic subgroups that comprise only a small percentage of the total U.S. population. Because the purpose of this study is methodological, the names of the vendors are masked. Instead, we refer to the samples as probability panels 1, 2 and 3 and opt-in samples 1, 2, and 3 throughout this report.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The probability-based panels use traditional probability-based methods for recruiting a random sample of U.S. adults. Specifically, all three panels use address-based sampling (ABS) for panel recruitment. ABS begins with a random sample of households from the U.S. Post Office\u2019s Delivery Sequence File, a near complete list of all residential addresses in the United States. Individuals in sampled households are contacted via mail and invited to join the panel and continue taking surveys periodically online. Although all three panels use similar methods for recruitment, differences in the timing and design of recruitments, the use of incentives, sampling procedures for individual panel waves, and panel maintenance practices could result in samples that are not altogether comparable to one another.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The three opt-in samples in this study are based on different but common approaches to online opt-in sampling. Opt-in sample 1 comes from a panel aggregator, or marketplace, in which individual respondents are drawn from many opt-in sample sources that have agreed to make their sample available to the aggregator. Opt-in sample 2 is sourced entirely from a single opt-in panel. Opt-in sample 3 is a blend, with about three-fifths sourced from a single opt-in panel and the remainder sourced from three sample aggregators. All three opt-in samples use a common set of quotas on age by gender, race and ethnicity, and educational attainment.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The same weighting scheme was applied to all six samples following Pew Research Center\u2019s standard procedure for weighting the ATP. Complete details of the weighting procedure and the design of the individual samples can be found in <a href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2023\/09\/07\/benchmarking-methodology\/\">the methodology<\/a>.<\/p>\n\n<\/div><\/div>\n\n<div style=\"border-width:1px;padding-bottom:var(--wp--preset--spacing--20);padding-left:var(--wp--preset--spacing--30);padding-right:var(--wp--preset--spacing--30);padding-top:var(--wp--preset--spacing--20);--block-gap: inherit\" class=\"wp-block-prc-block-collapsible has-background has-ui-beige-very-light-background-color has-border-color has-ui-beige-dark-border-color\" id=\"terminology\" data-wp-interactive=\"{&quot;namespace&quot;:&quot;prc-block\\\/collapsible&quot;}\" data-wp-context=\"{&quot;collapsibleId&quot;:&quot;terminology&quot;,&quot;isOpen&quot;:false}\" data-wp-class--is-open=\"context.isOpen\" data-wp-init--scroll-into-view=\"callbacks.onInitScrollIntoView\"><div class=\"wp-block-prc-block-collapsible__title\" data-wp-on--click=\"actions.onClick\"><div>Terminology<\/div><button class=\"wp-block-prc-block-collapsible__icon\"><span data-wp-bind--hidden=\"context.isOpen\"><i class=\"icon icon-library__light icon__circle-plus\"><svg style=\"width: 1em; height: 1em;\"><use xlink:href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/plugins\/prc-icon-library\/build\/icons\/sprites\/light.svg#circle-plus\"><\/use><\/svg><\/i><\/span><span data-wp-bind--hidden=\"!context.isOpen\" hidden><i class=\"icon icon-library__light icon__circle-minus\"><svg style=\"width: 1em; height: 1em;\"><use xlink:href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/plugins\/prc-icon-library\/build\/icons\/sprites\/light.svg#circle-minus\"><\/use><\/svg><\/i><\/span><\/button><\/div><div class=\"wp-block-prc-block-collapsible__content\">\n\n<p class=\"wp-block-paragraph\"><strong>Probability-based panel:<\/strong> This refers to a national survey panel recruited using random sampling from a database that includes most people in the population. Today, most such panels in the United States recruit by drawing random samples of residential addresses or telephone numbers. Typically, data collection with these panels is done online. However, some of these panels interview a small fraction of respondents (usually about 5% or fewer) using an offline mode such as live telephone. These panels are \u201cprobability-based\u201d because the chance that each address or phone number is selected is known. However, the chance that each selected person will join the panel or take surveys after joining is not known.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Online opt-in sample:<\/strong> These samples are recruited using a variety of methods that are sometimes referred to as \u201cconvenience sampling.\u201d Respondents are not selected randomly from the population but are recruited from a variety of online sources such as ads on social media or search engines, websites offering rewards in exchange for survey participation, or self-enrollment in an opt-in panel. Some opt-in samples are sourced from a panel (or multiple panels), while others rely on intercept techniques where respondents are invited to take a one-off survey.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Benchmark:<\/strong> These are \u201cground-truth\u201d measures used to assess the <a href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/short-reads\/2018\/12\/06\/how-asking-about-your-sleep-smoking-or-yoga-habits-can-help-pollsters-verify-their-findings\/\">accuracy of survey estimates<\/a>. For example, survey-based estimates for the share of voters who voted for each candidate in the 2020 presidential election are compared to a benchmark based on an official tally by the Federal Election Commission (FEC). Survey estimates are deemed more accurate the closer they are to the benchmark value. In this study, the benchmarks come from high-quality federal surveys such as the American Community Survey or administrative records like the FEC vote tally. Although these benchmarks come from \u201cgold-standard\u201d data sources, they are not entirely free from error. As such, they are not \u201ctrue\u201d population values but rather the best available approximations.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Error:<\/strong> This is the difference between an individual survey estimate and its corresponding benchmark value. Error can be either positive or negative depending on whether the survey estimate is higher or lower than the benchmark. For example, the FEC benchmark for the share of voters who voted for Donald Trump in the 2020 presidential election is 47%. If a survey estimated that share to be 42%, the error would be -5 percentage points because it came in 5 points below the benchmark. If the estimate were 49%, the error would be +2 points.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Absolute error:<\/strong> This is the absolute value of the error for a survey estimate. It describes the <em>size<\/em> of the error irrespective of its direction (positive or negative). For example, two estimates that have error of +5 and -5 percentage points, respectively, both have an absolute error of 5 points.<\/p>\n\n\n\n<figure class=\"wp-block-image alignright size-640-wide is-resized\"><a rel=\"attachment wp-att-120891\" href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2023\/09\/07\/comparing-two-types-of-online-survey-samples\/pm_09-07-23_benchmarking-00-02-png\/\"><img data-dominant-color=\"ececec\" data-has-transparency=\"false\" decoding=\"async\"  sizes=\"(max-width: 480px) 480px, (max-width: 782px) 782px, 640px\" srcset=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2023\/08\/pm_09.07.23_benchmarking-00-02.png?resize=480,350 480w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2023\/08\/pm_09.07.23_benchmarking-00-02.png?resize=620,452 620w\" src=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2023\/08\/pm_09.07.23_benchmarking-00-02.png?w=620\" alt=\"A table showing an example of measuring error for a benchmark variable.\" class=\"wp-image-120891 not-transparent\" style=\"--dominant-color: #ececec; width:310px\" width=\"310\" \/><\/a><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Average absolute error (for a benchmark variable):<\/strong> This is a measure that summarizes the average size of errors across all the categories within a single benchmark variable. For example, the smoking status variable has four categories: 1) Smoke every day, 2) Smoke some days, 3) Do not now smoke and 4) Never smoked 100 cigarettes. A survey\u2019s estimates for each category will have different levels of error, both positive and negative. For a given survey, the absolute error for the smoking status variable is equal to the sum of the absolute errors for each category divided by the number of categories.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Average absolute error (for a sample):<\/strong> Average absolute error can also be used to summarize the overall level of error across many different benchmarks within a single sample. When used in this context, the average absolute error for a sample is equal to the sum of the average absolute errors for each benchmark variable divided by the total number of benchmark variables.<\/p>\n\n<\/div><\/div>\n\n\n<p class=\"wp-block-paragraph\">As the field of public opinion research <a href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2023\/04\/19\/how-public-polling-has-changed-in-the-21st-century\/\">continues its steady movement<\/a> toward online data collection, probability-based panels and opt-in samples have emerged as the two most common approaches to surveying individuals online. At the same time, the methodologies and industry practices for both kinds of samples are evolving.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To shed light on the current state of online probability-based and opt-in samples, Pew Research Center conducted a study to compare the accuracy of six online surveys of U.S. adults \u2013 three from probability-based panels and three from opt-in sources. This is the first such study to include samples from multiple probability-based panels, allowing for their side-by-side comparison.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The surveys in this study were administered between June 14 and July 21, 2021, and included interviews with a total of 29,937 U.S. adults, approximately 5,000 in each sample. Because this is a methodological study, the names of the sample providers are masked.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The study compared each sample\u2019s accuracy on <a href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/short-reads\/2018\/12\/06\/how-asking-about-your-sleep-smoking-or-yoga-habits-can-help-pollsters-verify-their-findings\/\">28 benchmark variables<\/a> drawn from high-quality government data sources. These benchmarks included a variety of measures on topics such as voting, health, and respondents\u2019 work, family and living situations. (Refer to <a href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/wp-content\/uploads\/sites\/10\/2023\/09\/pm_09.07.23_benchmarking-appendix.pdf\">the appendix<\/a> for the full list of benchmarks and their sources.)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The study\u2019s key findings include:<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"on-average-error-on-opt-in-samples-was-about-twice-that-of-probability-based-panels\">On average, error on opt-in samples was about twice that of probability-based panels<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">For estimates among U.S. adults on 28 benchmark variables, opt-in samples 1, 2 and 3 had average absolute errors of 6.4, 6.1 and 5.0, respectively, for an overall average of 5.8 percentage points. This was about twice that of the probability-based online panels, for which average absolute error was 2.6 points overall (2.3, 3.0 and 2.5 on probability panels 1, 2 and 3, respectively).<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"online-opt-in-samples-had-especially-large-errors-for-18-to-29-year-olds-and-hispanic-adults\">Online opt-in samples had especially large errors for 18- to 29-year-olds and Hispanic adults<\/h4>\n\n\n\n<figure class=\"wp-block-image alignright size-640-wide is-resized\"><a rel=\"attachment wp-att-120888\" href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2023\/09\/07\/comparing-two-types-of-online-survey-samples\/pm_09-07-23_benchmarking-00-01-png\/\"><img data-dominant-color=\"eff0ef\" data-has-transparency=\"false\" loading=\"lazy\" decoding=\"async\"  srcset=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2023\/08\/pm_09.07.23_benchmarking-00-01.png?resize=480,391 480w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2023\/08\/pm_09.07.23_benchmarking-00-01.png?resize=782,637 782w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2023\/08\/pm_09.07.23_benchmarking-00-01.png?resize=840,684 840w\" sizes=\"(max-width: 480px) 480px, (max-width: 782px) 782px, 640px\" height=\"521\" src=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2023\/08\/pm_09.07.23_benchmarking-00-01.png?w=640\" alt=\"A dot plot showing larger errors on online opt in samples, especially for 18- to 29-year-olds and Hispanic adults.\" class=\"wp-image-120888 not-transparent\" style=\"--dominant-color: #eff0ef; width:420px\" width=\"640\" \/><\/a><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">On 25 variables for which subgroup-level benchmarks were available, the online opt-in samples averaged 11.2 percentage points of error for 18- to 29-year-olds and 10.8 points for Hispanic adults\u2013 each about 5 points higher than for U.S. adults overall (6.4&nbsp;points on the same 25 variables). By comparison, the average absolute error on the probability-based panels was 3.6 points for both young adults and Hispanic adults, less than 2 points higher than the error for all adults. A similar level of error was seen on the probability-based panels for other traditionally hard-to-survey subgroups such as those with no more than a high school education (3.6 points) and non-Hispanic Black adults (3.8 points).<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"error-was-concentrated-in-a-handful-of-variables-on-the-probability-based-panels-but-widespread-on-the-opt-in-samples\">Error was concentrated in a handful of variables on the probability-based panels but widespread on the opt-in samples<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">On each of the probability-based panels, the number of benchmarks for which average absolute error was greater than 5 percentage points ranged from two to five out of 28. About half (between 14 and 15 benchmarks) had under 2 points of average absolute error. Large errors were more widespread on the opt-in samples, which had between 11 and 17 benchmarks with error greater than 5 points. Fewer benchmarks (between three and seven) on the opt-in samples had average absolute error below 2 points.<\/p>\n\n\n\n<h4 id=\"probability-based-panels-consistently-overestimated-2020-voter-turnout\" class=\"wp-block-heading\">Probability-based panels consistently overestimated 2020 voter turnout<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">The only benchmark that had consistently high error on all three probability-based panels was voter turnout in the 2020 presidential election, which they each overestimated by +8 or +9 percentage points. This suggests that despite the inclusion of volunteerism and voter registration in weighting adjustments, the overrepresentation of politically and civically engaged individuals remains an important challenge for probability-based panels. By contrast, turnout was one of the most accurate variables on the opt-in samples, two of which came within 1 point of the benchmark while the third exceeded the benchmark by +3 points.<\/p>\n\n\n\n<h4 id=\"much-of-the-error-on-the-opt-in-samples-appears-to-be-due-to-bogus-respondents-who-usually-say-yes-regardless-of-the-question\" class=\"wp-block-heading\">Much of the error on the opt-in samples appears to be due to \u2018bogus respondents,\u2019 who usually say \u2018Yes\u2019 regardless of the question<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">In the online opt-in samples, an average of 8% of all adults, 15% of 18- to 29-year-olds and 19% of Hispanic adults answered \u201cYes\u201d on at least 10 out of 16 Yes\/No questions that were asked of every respondent. The corresponding shares on the probability-based panels were between 1% and 2% for each group. Similarly large shares reported having received at least three of four types of government benefits (Social Security, food stamps, unemployment compensation or workers\u2019 compensation) even though such individuals are virtually nonexistent in the true population. It is highly unlikely that the few individuals who do fit that description are massively overrepresented in online opt-in samples. Instead, this pattern suggests that much of the error on the online opt-in samples is due to the presence of <a href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2020\/02\/18\/assessing-the-risks-to-online-polls-from-bogus-respondents\/\">\u201cbogus respondents,\u201d<\/a> who make little or no effort to answer questions truthfully.<\/p>\n\n\n\n<h3 data-is-section=\"true\" data-wp-context=\"{&quot;id&quot;:&quot;what-is-new-about-this-benchmarking-study&quot;}\" data-wp-interactive=\"{&quot;namespace&quot;:&quot;prc-block\\\/table-of-contents&quot;}\" class=\"wp-block-heading\" id=\"what-is-new-about-this-benchmarking-study\">What is new about this benchmarking study?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Polling insiders may wonder why we conducted this study. Other research teams (<a href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2016\/05\/02\/evaluating-online-nonprobability-surveys\/\">including our own several years ago<\/a>) have done <a href=\"https:\/\/academic.oup.com\/poq\/article-abstract\/82\/4\/707\/5151369\">similar comparisons<\/a> and arrived at the <a href=\"https:\/\/academic.oup.com\/poq\/article\/81\/S1\/213\/3749202\">same result<\/a>: Probability-based samples tend to be more accurate, even if <a href=\"https:\/\/www.washingtonpost.com\/politics\/2020\/11\/25\/which-2020-election-polls-were-most-least-accurate\/\">recent election polls are an exception<\/a>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">One major reason we conducted this study is because it offers something new. This benchmarking study is the first to estimate the accuracy of <em>multiple<\/em> online probability-based panels in the United States. This allows us to answer a previously unaddressed question: Do different probability-based panels tend to offer similar data quality or not? The three probability-based panels tested in this study performed about the same. The average absolute error ranged from just 2.3 to 3.0 percentage points \u2013 good news for researchers seeking a reliable polling method. It\u2019s worth noting that this study considered fairly general topics (e.g., employment, marital status), and that the results might differ if a survey focused on a more niche topic, like poverty.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The second major reason we conducted this study was that in the time since <a href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2016\/05\/02\/evaluating-online-nonprobability-surveys\/\">our 2016 study<\/a>, there have been a number of major changes to the ATP\u2019s methodology made in response to that study\u2019s findings, <a href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/religion\/2021\/01\/14\/measuring-religion-in-pew-research-centers-american-trends-panel\/\">other challenges<\/a> that have <a href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2021\/04\/08\/confronting-2016-and-2020-polling-limitations\/\">arisen in the intervening years<\/a>, and the Center\u2019s evolving research needs:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>We changed how we recruit, moving from telephone to an address-based approach.<\/li>\n\n\n\n<li>We began subsampling the panel rather than routinely interviewing everyone.<\/li>\n\n\n\n<li>We changed how we weight the panel, adding in adjustments for volunteerism, religion and other factors.<\/li>\n\n\n\n<li>We began retiring panelists from overrepresented segments of the population.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">The study reported here is, in part, our effort to measure whether those improvements made a difference, allowing us to determine how the new, improved ATP stacks up against opt-in samples and against other probability-based panels.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Pew Research Center conducted a study to compare the accuracy of six online surveys of U.S. adults \u2013 three from probability-based panels and three from opt-in sources. On average, the absolute error on opt-in samples was about twice that of probability-based panels. <\/p>\n","protected":false},"author":658,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"sub_headline":"Opt-in samples are about half as accurate as probability-based panels","sub_title":"Opt-in samples are about half as accurate as probability-based panels","_prc_public_revisions":[],"_ppp_expiration_hours":0,"_ppp_enabled":false,"ai_generated_summary":"","_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"apple_news_api_id":"","apple_news_api_revision":"","apple_news_api_created_at":"","apple_news_api_modified_at":"","apple_news_api_share_url":"","apple_news_api_pending":"","apple_news_is_preview":false,"apple_news_is_hidden":false,"relatedPosts":[],"reportMaterials":[{"key":"_3lajd9bm9","type":"report","attachmentId":2446,"url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/wp-content\/uploads\/sites\/10\/2023\/09\/pm_09.07.23_benchmarking-report.pdf"},{"key":"_38uhyofal","type":"questionnaire","attachmentId":2406,"url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/wp-content\/uploads\/sites\/10\/2023\/09\/pm_09.07.23_benchmarking_questionnaire.pdf"},{"key":"_37uovu6cc","type":"link","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/wp-content\/uploads\/sites\/10\/2023\/09\/pm_09.07.23_benchmarking-appendix.pdf","label":"Appendix: Benchmarks and measures of error by sample","icon":"supplemental"}],"multiSectionReport":[{"key":"_wmwnnclwf","postId":101127},{"key":"_uw5nsjjw4","postId":101128},{"key":"_bdc9ny7sk","postId":101126},{"key":"_dyn4opp20","postId":101122}],"package_parts__enabled":false,"package_parts":[],"datacite_doi":"","datacite_doi_citation":"","_prc_seo_qr_attachment_id":0,"spoken_article_player_enabled":true,"displayBylines":true,"footnotes":"","prc_watchers":[],"_prc_fork_parent":0,"_prc_fork_status":"","_prc_active_fork":0,"jetpack_post_was_ever_published":false},"categories":[36,450,357],"tags":[],"bylines":[968,719],"collection":[],"datasets":[1992],"level_of_effort":[],"primary_audience":[],"information_type":[],"_post_visibility":[],"formats":[458],"_fund_pool":[],"languages":[],"regions-countries":[515],"research-teams":[528],"workflow-status":[],"class_list":["post-101111","post","type-post","status-publish","format-standard","hentry","category-methodological-research","category-online-surveys","category-survey-methods","bylines-andrew-mercer","bylines-arnold-lau","datasets-2021-benchmarking-study","formats-report","regions-countries-united-states","research-teams-methods"],"label":false,"post_parent":0,"word_count":2064,"canonical_url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2023\/09\/07\/comparing-two-types-of-online-survey-samples\/","art_direction":{"A1":{"id":120894,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2023\/09\/pm_09.07.23_benchmarking-topic.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2023\/09\/pm_09.07.23_benchmarking-topic.png?w=564&h=317&crop=1","width":564,"height":317,"caption":"","chartArt":false},"A2":{"id":120898,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2023\/09\/pm_09.07.23_benchmarking-crop.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2023\/09\/pm_09.07.23_benchmarking-crop.png?w=268&h=151&crop=1","width":268,"height":151,"caption":"","chartArt":true},"A3":{"id":120898,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2023\/09\/pm_09.07.23_benchmarking-crop.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2023\/09\/pm_09.07.23_benchmarking-crop.png?w=194&h=110&crop=1","width":194,"height":110,"caption":"","chartArt":true},"A4":{"id":120898,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2023\/09\/pm_09.07.23_benchmarking-crop.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2023\/09\/pm_09.07.23_benchmarking-crop.png?w=268&h=151&crop=1","width":268,"height":151,"caption":"","chartArt":true},"XL":{"id":120894,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2023\/09\/pm_09.07.23_benchmarking-topic.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2023\/09\/pm_09.07.23_benchmarking-topic.png?w=720&h=405&crop=1","width":720,"height":405,"caption":"","chartArt":false},"social":{"id":120901,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2023\/09\/pm_09.07.23_benchmarking-featured.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2023\/09\/pm_09.07.23_benchmarking-featured.png?w=1200&h=628&crop=1","width":1200,"height":628,"caption":"","chartArt":false}},"_embeds":[],"watchers":[],"table_of_contents":[{"id":101111,"title":"Comparing Two Types of Online Survey Samples","slug":"comparing-two-types-of-online-survey-samples","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2023\/09\/07\/comparing-two-types-of-online-survey-samples\/","is_active":true},{"id":101127,"title":"1. Assessing the accuracy of estimates for U.S. adults","slug":"assessing-the-accuracy-of-estimates-for-u-s-adults","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2023\/09\/07\/assessing-the-accuracy-of-estimates-for-u-s-adults\/","is_active":false},{"id":101128,"title":"2. Assessing the accuracy of estimates among demographic subgroups","slug":"assessing-the-accuracy-of-estimates-among-demographic-subgroups","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2023\/09\/07\/assessing-the-accuracy-of-estimates-among-demographic-subgroups\/","is_active":false},{"id":101126,"title":"Acknowledgments","slug":"benchmarking-acknowledgments","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2023\/09\/07\/benchmarking-acknowledgments\/","is_active":false},{"id":101122,"title":"Methodology","slug":"benchmarking-methodology","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2023\/09\/07\/benchmarking-methodology\/","is_active":false}],"report_materials":[{"key":"_3lajd9bm9","type":"report","attachmentId":2446,"url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/wp-content\/uploads\/sites\/10\/2023\/09\/pm_09.07.23_benchmarking-report.pdf","label":"","icon":""},{"key":"_38uhyofal","type":"questionnaire","attachmentId":2406,"url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/wp-content\/uploads\/sites\/10\/2023\/09\/pm_09.07.23_benchmarking_questionnaire.pdf","label":"","icon":""},{"key":"_37uovu6cc","type":"link","attachmentId":0,"url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/wp-content\/uploads\/sites\/10\/2023\/09\/pm_09.07.23_benchmarking-appendix.pdf","label":"Appendix: Benchmarks and measures of error by sample","icon":"supplemental"},{"type":"dataset","id":1992,"label":"2021 Benchmarking Study","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/dataset\/2021-benchmarking-study\/"}],"report_pagination":{"current_post":{"id":101111,"title":"Comparing Two Types of Online Survey Samples","slug":"comparing-two-types-of-online-survey-samples","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2023\/09\/07\/comparing-two-types-of-online-survey-samples\/","is_active":true,"page_num":1},"next_post":{"id":101127,"title":"1. Assessing the accuracy of estimates for U.S. adults","slug":"assessing-the-accuracy-of-estimates-for-u-s-adults","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2023\/09\/07\/assessing-the-accuracy-of-estimates-for-u-s-adults\/","is_active":false,"page_num":2},"previous_post":null,"pagination_items":[{"id":101111,"title":"Comparing Two Types of Online Survey Samples","slug":"comparing-two-types-of-online-survey-samples","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2023\/09\/07\/comparing-two-types-of-online-survey-samples\/","is_active":true,"page_num":1},{"id":101127,"title":"1. Assessing the accuracy of estimates for U.S. adults","slug":"assessing-the-accuracy-of-estimates-for-u-s-adults","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2023\/09\/07\/assessing-the-accuracy-of-estimates-for-u-s-adults\/","is_active":false,"page_num":2},{"id":101128,"title":"2. Assessing the accuracy of estimates among demographic subgroups","slug":"assessing-the-accuracy-of-estimates-among-demographic-subgroups","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2023\/09\/07\/assessing-the-accuracy-of-estimates-among-demographic-subgroups\/","is_active":false,"page_num":3},{"id":101126,"title":"Acknowledgments","slug":"benchmarking-acknowledgments","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2023\/09\/07\/benchmarking-acknowledgments\/","is_active":false,"page_num":4},{"id":101122,"title":"Methodology","slug":"benchmarking-methodology","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/2023\/09\/07\/benchmarking-methodology\/","is_active":false,"page_num":5}]},"parent_info":{"parent_title":"Comparing Two Types of Online Survey Samples","parent_id":101111},"materialsOrdered":[{"key":"_3lajd9bm9","type":"report","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/wp-content\/uploads\/sites\/10\/2023\/09\/pm_09.07.23_benchmarking-report.pdf","label":"","attachmentId":2446,"icon":""},{"key":"_38uhyofal","type":"questionnaire","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/wp-content\/uploads\/sites\/10\/2023\/09\/pm_09.07.23_benchmarking_questionnaire.pdf","label":"","attachmentId":2406,"icon":""},{"key":"_37uovu6cc","type":"link","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/methods\/wp-content\/uploads\/sites\/10\/2023\/09\/pm_09.07.23_benchmarking-appendix.pdf","label":"Appendix: Benchmarks and measures of error by sample","attachmentId":0,"icon":"supplemental"}],"chaptersOrdered":[{"key":"_wmwnnclwf","postId":101127},{"key":"_uw5nsjjw4","postId":101128},{"key":"_bdc9ny7sk","postId":101126},{"key":"_dyn4opp20","postId":101122}],"partsOrdered":[],"partsEnabled":false,"datacite_doi":"","prc_seo_data":{"title":"Comparing Accuracy of 2 Types of Online Survey Samples %page%","description":"Pew Research Center conducted a study to compare the accuracy of six online surveys of U.S. adults \u2013 three from probability-based panels and three from opt-in sources. On average, the absolute error on opt-in samples was about twice that of probability-based panels.","og_title":"Comparing Two Types of Online Survey Samples","og_description":"Pew Research Center conducted a study to compare the accuracy of six online surveys of U.S. adults \u2013 three from probability-based panels and three from opt-in sources. On average, the absolute error on opt-in samples was about twice that of probability-based panels.","schema_type":"Article","noindex":false,"canonical_url":"","primary_terms":{"regions-countries":10814863,"research-teams":10813973},"custom_schema":[],"twitter_title":"Comparing Two Types of Online Survey Samples","twitter_description":"Pew Research Center conducted a study to compare the accuracy of six online surveys of U.S. adults \u2013 three from probability-based panels and three from opt-in sources. On average, the absolute error on opt-in samples was about twice that of probability-based panels.","og_image":120901,"twitter_image":2449,"indexnow_submitted_at":null,"gsc_index_status":null},"prepublish_checks":{"prc-image-alt-text":{"status":"complete","message":"All images have alt text.","data":null},"prc-about-this-research":{"status":"incomplete","message":"Add an \"About this research\" details block.","data":null},"prc-paragraph-count":{"status":"complete","message":"Found 26 paragraphs.","data":{"count":26}},"prc-internal-link":{"status":"complete","message":"Found 12 internal links.","data":{"count":12}}},"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"relatedPostsOrdered":[],"bylinesOrdered":[{"key":"_1v8zhjdl6","termId":968},{"key":"_p0rnhdlry","termId":719}],"acknowledgementsOrdered":[],"_links":{"self":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/posts\/101111","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/users\/658"}],"replies":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/comments?post=101111"}],"version-history":[{"count":2,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/posts\/101111\/revisions"}],"predecessor-version":[{"id":223841,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/posts\/101111\/revisions\/223841"}],"wp:attachment":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/media?parent=101111"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/categories?post=101111"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/tags?post=101111"},{"taxonomy":"bylines","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/bylines?post=101111"},{"taxonomy":"collection","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/collection?post=101111"},{"taxonomy":"datasets","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/datasets?post=101111"},{"taxonomy":"level_of_effort","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/level_of_effort?post=101111"},{"taxonomy":"primary_audience","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/primary_audience?post=101111"},{"taxonomy":"information_type","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/information_type?post=101111"},{"taxonomy":"_post_visibility","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_post_visibility?post=101111"},{"taxonomy":"formats","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/formats?post=101111"},{"taxonomy":"_fund_pool","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_fund_pool?post=101111"},{"taxonomy":"languages","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/languages?post=101111"},{"taxonomy":"regions-countries","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/regions-countries?post=101111"},{"taxonomy":"research-teams","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/research-teams?post=101111"},{"taxonomy":"workflow-status","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/workflow-status?post=101111"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}