{"id":574097,"date":"2026-09-30T13:55:19","date_gmt":"2026-09-30T17:55:19","guid":{"rendered":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/?p=574097"},"modified":"2026-09-30T13:55:19","modified_gmt":"2026-09-30T17:55:19","slug":"synthetic-surveys-and-timely-and-topical-questions","status":"publish","type":"post","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/data-labs\/2026\/09\/30\/synthetic-surveys-and-timely-and-topical-questions\/","title":{"rendered":"Synthetic surveys and \u2018timely and topical\u2019 questions"},"content":{"rendered":"\n
About this research<\/summary>\n

This Pew Research Center report examines whether AI-generated survey data can accurately duplicate the results of high-quality public opinion polls.<\/p>\n\n\n\n

We conducted this methodological experiment to better understand an emerging method in survey research. The Center believes that speaking to the public is essential to measuring public opinion and has no current or future plans to use AI models to generate survey results. Read more about our AI policy<\/a>.<\/p>\n\n\n\n

For this study, we had an AI model take on the role of real respondents from our American Trends Panel<\/a> (ATP) and answer the same surveys administered to human panelists. These surveys included many different question formats and asked about a wide range of topics, attitudes and behaviors.<\/p>\n\n\n\n

Why did we do this?<\/h4>\n\n\n\n

The Center regularly studies and evaluates advances and trends in polling research<\/a>. As AI-based polling becomes more widely used in the industry<\/a>, we wanted to understand the data these \u201csynthetic samples\u201d produce and how it compares with high-quality surveys of real humans on topics of broad public interest. That question is the focus of this study; it does not address other uses of AI models in polling.<\/p>\n\n\n\n

How did we do this?<\/h4>\n\n\n\n

We used a \u201cdigital twins\u201d approach for this study, asking an AI model to adopt the personas of real humans who are members of the ATP. We gave the model a wide range of information about each panelist<\/a>, including their self-reported demographic information and their responses to questions on our 2025 political typology survey<\/a>. We then showed the model questions from three different ATP surveys conducted in the first half of 2026 and asked it to respond to those same questions as the assigned persona. The model was given each question on the survey in order from start to finish, with exactly the same instructions and survey programming that the real humans who took it received.<\/p>\n\n\n\n

To make a direct comparison between our AI and human polls, we only replicated surveys for the ATP panelists from each wave who had also completed the political typology survey. As a result, figures for U.S. adults in this report may differ slightly from those previously published by the Center.<\/p>\n\n\n\n

We tested multiple models, but all synthetic results unless noted otherwise come from Anthropic\u2019s Claude Opus 4.6. At the time we conducted this study, Opus 4.6 was the most recent Anthropic model available for commercial use and had the best performance of several we evaluated.<\/p>\n\n\n\n

Here are the survey questions from Wave 185<\/a>, Wave 190<\/a> and Wave 192<\/a> used for this analysis, the detailed responses<\/a> from the ATP and AI surveys, and the methodologies<\/a> for the original ATP survey waves and our AI surveys.<\/p>\n<\/details>\n\n\n\n

Public opinion often seems slow to change, but the opinions and attitudes of the public can sometimes shift quickly and dramatically in response to current events. This is part of what\u2019s interesting and challenging about studying public opinion. Sometimes people change their minds relatively rapidly, as happened with former President George W. Bush\u2019s increased approval rating<\/a> in the wake of 9\/11. At other times, segments of the population can switch places on issues, as happened with Democrats\u2019 and Republicans\u2019 perceptions of the social media platform X (formerly Twitter)<\/a> after Elon Musk purchased the company.<\/p>\n\n\n\n

One appeal of AI polling is its speed. At least in theory, these polls can be fielded nearly instantly without the need to assemble a representative sample of humans who are willing to take the survey. At the same time, other ways of inferring public opinion without directly measuring it through polling (like statistical modeling) can struggle to identify changes in sentiment when they happen quickly, such as in response to breaking news.<\/p>\n\n\n\n

We found that polling with AI models is no different. In our analysis comparing surveys of AI respondents with surveys of real humans, there were numerous instances in which the model failed to accurately reflect public opinion in unique cultural or economic circumstances. That is especially \u2013 but not always \u2013 true when those circumstances developed after the model\u2019s knowledge cutoff.[2. numoffset=”2″ LLMs all have a knowledge cutoff \u2013 essentially the date on which the model stopped ingesting new data and training actually began. The model we used for most of this analysis, Claude Opus 4.6, has a knowledge cutoff in May 2025<\/a>.]<\/p>\n\n\n\n

This analysis is part of a larger evaluation of AI-generated synthetic samples in public opinion research. Read<\/em> a summary of the main findings<\/a><\/em> and refer to the<\/em> methodology<\/a><\/em> for more details on how we conducted our synthetic poll and compared it with real survey results.<\/em><\/p>\n\n\n\n