{"id":111464,"date":"2020-10-14T14:52:00","date_gmt":"2020-10-14T19:52:00","guid":{"rendered":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/\/\/testing-survey-questions-ahead-of-time-can-help-sharpen-a-polls-focus\/"},"modified":"2024-04-14T04:10:35","modified_gmt":"2024-04-14T09:10:35","slug":"testing-survey-questions-ahead-of-time-can-help-sharpen-a-polls-focus","status":"publish","type":"decoded","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/2020\/10\/14\/testing-survey-questions-ahead-of-time-can-help-sharpen-a-polls-focus\/","title":{"rendered":"Testing survey questions ahead of time can help sharpen a poll\u2019s focus"},"content":{"rendered":"\n
\"\"<\/a><\/figure>\n\n\n\n

Survey researchers often try to determine how respondents might react to different kinds of questions. This can help ensure that proposed survey questions are as clear as possible and measuring what researchers set out to measure.<\/p>\n\n\n\n

Pew Research Center frequently tests survey questions ahead of time through qualitative research methods such as focus groups<\/a>, cognitive interviews, pretesting (often using an online, nonprobability sample<\/a>), or a combination of these approaches.<\/p>\n\n\n\n

In this post, I\u2019ll walk through our preparations for a comparative, cross-national survey we\u2019re planning to field in the United States, United Kingdom, France and Germany later this fall. In this case, we used focus groups and a short survey experiment to test some questions for the upcoming poll. I\u2019ll discuss two of those survey questions here.<\/p>\n\n\n\n

Measuring pride \u2014 or shame \u2014 in a nation\u2019s history<\/h3>\n\n\n\n

One of the topics in the focus groups \u2014 conducted in the U.S. and UK in fall 2019<\/a> \u2014 was a discussion on what makes people proud to be American or British, as well as what makes them embarrassed.<\/p>\n\n\n\n

One interesting observation is that participants pointed to each country\u2019s history as either a source of pride or<\/em> shame, depending on the political orientation of the focus group<\/a>. For example, in British focus groups consisting of those who voted to leave the European Union (\u201cLeavers\u201d), participants emphasized their pride in the former British empire and the role that the UK played in spreading democracy and the English language worldwide. But in British focus groups composed of those who voted to remain<\/em> in the EU (\u201cRemainers\u201d), participants expounded on how the country\u2019s history of colonialism and imperialism made them feel embarrassed, citing examples such as the \u201cmess\u201d in Hong Kong and how the UK bore responsibility for \u201cdestabilizing regions\u201d and \u201cmaking one tribe fight the other.\u201d These respondents also emphasized that Britain\u2019s involvement in the slave trade was problematic, an issue that has since boiled over as protesters toppled a statue of slave trader Robert Colston<\/a> in Bristol, England.<\/p>\n\n\n\n

In the U.S., too, the focus groups were separated by issues of history, slavery and racism. Those composed of Republicans and Republican-leaning independents emphasized their pride in American history and the ways in which the country has been a beacon of hope and a \u201cland of opportunity,\u201d while those comprised of Democrats and Democratic leaners highlighted past injustices and unequal opportunity for Black people. Democratic focus groups also singled out the perceived hypocrisy of America as a \u201cmelting pot\u201d despite its legacy of slavery and mistreatment of Native Americans, among other injustices.<\/p>\n\n\n\n

Based on these divides, we were interested in drafting a survey question that could measure the relative tendency to be proud or ashamed of the history of one\u2019s country. But we wanted to resolve two issues first. One was whether we should we provide a middle answer option so people could report being both proud and <\/em>ashamed in their country\u2019s history. The other was whether the question might inadvertently elicit answers related to each country\u2019s handling of the current COVID-19 pandemic, which was not<\/em> our intention. (We know from a previous survey that people in the U.S. and UK are divided about their government\u2019s handling of COVID-19<\/a>.)<\/p>\n\n\n\n

To study the first issue, we used an online nonprobability sample to randomly assign people to receive one of two survey questions. We used a nonprobability sample because our pretesting typically focuses on understanding how different question wordings perform \u2014 something that can be explored through experiments on nonprobability samples<\/a> \u2014 rather than understanding what percentage of the U.S. population has a particular view. For this experiment, half of the respondents in the online nonprobability sample were given the following binary, forced choice question:<\/p>\n\n\n\n

\u201cWhich comes closer to your view even if neither is exactly right? There are some times when I am not proud of the U.S.\/UK <\/em>OR<\/em><\/strong> I am always proud of the U.S.\/UK, no matter what.\u201d<\/em><\/p>\n\n\n\n

The other half were given the following three-part question instead:<\/p>\n\n\n\n

\u201cWhich comes closer to your view, even if neither is exactly right? I\u2019m almost always proud of this country <\/em>OR<\/em><\/strong> I\u2019m almost always ashamed of this country <\/em>OR<\/em><\/strong> I\u2019m often proud of this country, but I\u2019m often ashamed of it as well.\u201d<\/em><\/p>\n\n\n\n

Since people were randomly assigned to one question or the other, we know that the only differences between the two groups should have been which question they received and not other factors that might affect results.<\/p>\n\n\n\n

Results indicated that half or more in each country that got the second version of the question chose the middle option of both pride and<\/em> shame. In the U.S., similar shares chose \u201calways proud\u201d in both question formats, whereas in the UK, fewer people chose \u201calways proud\u201d when the middle option was present. Thus, while the two versions of the question produced somewhat similar results for the \u201calways proud\u201d group, the three-part question provided more information, allowing us to more clearly parse those who were \u201calways ashamed\u201d from those who fell into the middle category of being often proud and often ashamed. Regardless of the formulation of the question, Republicans in the U.S. were more likely than Democrats to say they were \u201calways proud,\u201d as were Leavers in the UK when compared to Remainers.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

To gain some insight into the second issue flagged \u2014 whether respondents might be thinking of their government\u2019s response to the coronavirus outbreak \u2014 we decided to explore the cognitive processes behind individual answers by asking respondents to explain what they were thinking in an open-ended question following whichever of the two survey questions they answered.<\/p>\n\n\n\n

Examining these open-ended responses, we saw that while COVID-19 did come up, respondents\u2019 feelings of pride and shame often had a longer time horizon. For example, people mentioned the history of the two countries; general freedoms and principles that they attributed to the countries; characteristics of the publics; and more. While some mentioned politics, more emphasis was often placed on the current administration in each country than on the specific handling of the COVID-19 pandemic. In fact, responses felt qualitatively similar to what we heard in the focus groups, suggesting that we were capturing our concept of interest, rather than a short-term sentiment about the pandemic. For example, in focus groups, we heard things like \u201cAll you have to do is travel outside the U.S. to realize how good we have it here.\u201d In open-ended responses online, one American participant similarly voiced, \u201cI\u2019ve been to 45 countries \u2014 none better.\u201d<\/p>\n\n\n\n

In addition to the valuable information we gained from the three-part question in terms of analytically separating out the \u201calways ashamed\u201d group from those who feel more mixed, we also learned from the open-ended answers that some respondents who received the two-part question mentioned that they often felt both proud and<\/em> ashamed of their country\u2019s history and had difficulty responding. As a result, we decided to field the three-part question instead.<\/p>\n\n\n\n

Measuring nostalgia<\/h3>\n\n\n\n

Another question we asked in the focus groups was about when<\/em> each country was at its best. Participants named different dates or eras, but four key themes emerged.<\/p>\n\n\n\n

First, both Britons and Americans highlighted \u201csimpler\u201d times \u2013before the internet era, Facebook and generally when people had a slower pace of life and didn\u2019t have to be constantly responsive to email or texts. Second, people emphasized times when economic opportunity seemed plentiful and they felt like the ordinary people could make a good living and improve their quality of life compared to their parents. Third, people touted times when the countries had come together and been less polarized, whether because of a natural disaster (e.g. Hurricane Harvey in the U.S.), a terrorist attack (e.g. 9\/11 in the U.S.), cultural events (e.g. royal weddings in the UK) or the Olympics, among others.<\/p>\n\n\n\n

A fourth theme also emerged: the sense that each country has never<\/em> been at its best. People in both countries \u2014 and especially those in groups composed of Remainers in the UK or Democrats in the U.S. \u2014 highlighted how the \u201cbest\u201d times can depend on one\u2019s vantage point. This was underscored by a U.S. respondent who noted that even after World War II, when Americans could celebrate beating the Nazis, civil rights remained limited for some. This sentiment was echoed by many who said that while both countries had made progress, historically there were always people disenfranchised based on race, sexuality or gender.<\/p>\n\n\n\n

To get at this concept in our survey, we drafted the following question:<\/p>\n\n\n\n

\u201cThinking about the U.S.\/UK, which of the following best reflects your view? We were better in the past <\/em>OR<\/em><\/strong> The best years are still ahead.\u201d<\/em><\/p>\n\n\n\n

We also drafted a second, similar version, to explore whether slightly different wording would change the results:<\/p>\n\n\n\n

\u201cThinking about the U.S.\/UK, which of the following best reflects your view? Our country\u2019s best years are already behind us <\/em>OR<\/em><\/strong> Our country\u2019s best years are still to come.\u201d<\/em><\/p>\n\n\n\n

In the UK, people answered the question similarly, regardless of the wording. Around half generally felt the country was better in the past, while half said it will be better in the future. In the U.S., however, question wording appears to have mattered. While 45% said in the first question option that \u201cwe were better in the past,\u201d a considerably smaller share (34%) chose the corollary in the second question option: \u201cOur country\u2019s best years are already behind us.\u201d In addition, we noticed that the ideological differences we observed in the focus groups \u2014 with Leavers in the UK and Republicans in the U.S. more likely to be nostalgic for the past \u2014 were either slight, not present, or reversed. In the UK, we saw that on both versions of the question, Remainers were actually more<\/em> likely to say their country was better in the past than Leavers, though those divides were more stark in the first version of the question.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

On this question, too, we were concerned that some respondents might think of the COVID-19 pandemic when evaluating whether their country\u2019s best years were in the past or in the future, so we asked the same open-ended question that we asked about national pride or shame. As was the case in the earlier question, while some people did bring up the outbreak, we didn\u2019t find any major cause for concern on this front. But the open-ended answers did<\/em> help us understand why the ideological differences we observed in the focus groups were more muted in the test survey.<\/p>\n\n\n\n

Most notably, people often chose different survey responses while giving the same explanation for their choice \u2014 e.g. political polarization, the election of Boris Johnson or Donald Trump or COVID-19. For example, in the U.S., two respondents who were both dissatisfied with the Trump presidency answered the closed-ended question differently. One who said \u201cwe were better in the past\u201d explained his reasoning as, \u201cEverything has gone bad since we had Trump as president.\u201d The other said \u201cthe best years are still ahead,\u201d but described a similar rationale: \u201cVery many things will change for the positive once we\u2019re unshackled from Trump.\u201d We found many examples like this in the open-ended answers, suggesting that while the question may have captured people\u2019s general predilections, it was not<\/em> adequately capturing the sense of nostalgia that we were trying to measure. Given this, we decided not to run this question on the survey.<\/p>\n\n\n\n

The importance of testing<\/h3>\n\n\n\n

While organizing focus groups and online pretesting adds quite a bit of time \u2014 and expense \u2014 to our survey research process, we find it incredibly valuable. For this project, focus groups were particularly valuable for highlighting themes and topics that were related to our key areas of interest but that we had never asked survey questions about before. And the pretesting showed us which of those topics we were successfully asking about in a closed-ended format and which may have slightly missed the mark, allowing us to recalibrate, adjust and create the best possible survey instrument for our larger, nationally representative survey.<\/p>\n","protected":false},"excerpt":{"rendered":"

Testing questions in advance can help ensure they are as clear as possible\u200a\u2014\u200aand that they measure what researchers set out to measure.<\/p>\n","protected":false},"author":377,"featured_media":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"sub_headline":"","sub_title":"","_prc_public_revisions":[],"_ppp_expiration_hours":0,"_ppp_enabled":false,"ai_generated_summary":"","relatedPosts":[],"datacite_doi":"","datacite_doi_citation":"","_prc_seo_qr_attachment_id":0,"spoken_article_player_enabled":true,"displayBylines":true,"footnotes":"","prc_watchers":[],"_prc_fork_parent":0,"_prc_fork_status":"","_prc_active_fork":0},"categories":[357],"bylines":[954],"collection":[],"_post_visibility":[],"decoded-category":[532],"formats":[454],"_fund_pool":[],"languages":[],"regions-countries":[],"research-teams":[524],"workflow-status":[],"class_list":["post-111464","decoded","type-decoded","status-publish","hentry","category-survey-methods","bylines-laura-silver","decoded-category-survey-methods","formats-decoded","research-teams-decoded"],"label":"Decoded","post_parent":0,"word_count":2068,"canonical_url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/2020\/10\/14\/testing-survey-questions-ahead-of-time-can-help-sharpen-a-polls-focus\/","art_direction":{"A1":{"id":125887,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/2020.10.14_decoded_featured.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/2020.10.14_decoded_featured.png?w=564&h=317&crop=1","width":564,"height":317,"caption":"","chartArt":false},"A2":{"id":125887,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/2020.10.14_decoded_featured.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/2020.10.14_decoded_featured.png?w=268&h=151&crop=1","width":268,"height":151,"caption":"","chartArt":false},"A3":{"id":125887,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/2020.10.14_decoded_featured.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/2020.10.14_decoded_featured.png?w=194&h=110&crop=1","width":194,"height":110,"caption":"","chartArt":false},"A4":{"id":125887,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/2020.10.14_decoded_featured.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/2020.10.14_decoded_featured.png?w=268&h=151&crop=1","width":268,"height":151,"caption":"","chartArt":false},"XL":{"id":125887,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/2020.10.14_decoded_featured.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/2020.10.14_decoded_featured.png?w=720&h=405&crop=1","width":720,"height":405,"caption":"","chartArt":false},"social":{"id":125887,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/2020.10.14_decoded_featured.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/2020.10.14_decoded_featured.png?w=1200&h=628&crop=1","width":1200,"height":628,"caption":"","chartArt":false}},"_embeds":[],"watchers":[],"table_of_contents":[],"datacite_doi":"","prc_seo_data":{"title":"Testing survey questions ahead of time can help sharpen a poll\u2019s focus","description":"Testing questions in advance can help ensure they are as clear as possible\u200a\u2014\u200aand that they measure what researchers set out to measure.","og_title":"Testing survey questions ahead of time can help sharpen a poll\u2019s focus","og_description":"Testing questions in advance can help ensure they are as clear as possible\u200a\u2014\u200aand that they measure what researchers set out to measure.","schema_type":"Article","noindex":false,"canonical_url":"","primary_terms":{"category":43},"custom_schema":[],"og_image":125887,"indexnow_submitted_at":null,"gsc_index_status":null},"prepublish_checks":{},"jetpack_sharing_enabled":true,"relatedPostsOrdered":[],"bylinesOrdered":[{"key":"_cfz9rqn9m","termId":954}],"acknowledgementsOrdered":[],"_links":{"self":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111464","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded"}],"about":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/types\/decoded"}],"author":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/users\/377"}],"replies":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/comments?post=111464"}],"version-history":[{"count":2,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111464\/revisions"}],"predecessor-version":[{"id":138519,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111464\/revisions\/138519"}],"wp:attachment":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/media?parent=111464"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/categories?post=111464"},{"taxonomy":"bylines","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/bylines?post=111464"},{"taxonomy":"collection","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/collection?post=111464"},{"taxonomy":"_post_visibility","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_post_visibility?post=111464"},{"taxonomy":"decoded-category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded-category?post=111464"},{"taxonomy":"formats","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/formats?post=111464"},{"taxonomy":"_fund_pool","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_fund_pool?post=111464"},{"taxonomy":"languages","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/languages?post=111464"},{"taxonomy":"regions-countries","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/regions-countries?post=111464"},{"taxonomy":"research-teams","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/research-teams?post=111464"},{"taxonomy":"workflow-status","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/workflow-status?post=111464"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}