{"id":111245,"date":"2022-12-08T17:59:55","date_gmt":"2022-12-08T22:59:55","guid":{"rendered":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/\/\/how-we-keep-our-online-surveys-from-running-too-long\/"},"modified":"2024-04-14T04:10:23","modified_gmt":"2024-04-14T09:10:23","slug":"how-we-keep-our-online-surveys-from-running-too-long","status":"publish","type":"decoded","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/2022\/12\/08\/how-we-keep-our-online-surveys-from-running-too-long\/","title":{"rendered":"How we keep our online surveys from running too long"},"content":{"rendered":"\n
\"\"<\/a>
Pew Research Center illustration<\/figcaption><\/figure>\n\n\n\n

The longer a survey is, the fewer people are willing to complete it<\/a>. This is especially true for surveys conducted online, where attention spans can be short. While there is no magic length that an online survey should be, Pew Research Center caps the length of its online American Trends Panel<\/a> (ATP) surveys at 15 minutes, based on prior research<\/a>.<\/p>\n\n\n\n

\"Questions<\/a><\/figure>\n\n\n\n

But how can we know how long a survey will take to complete before<\/em> it\u2019s been taken? This is where question counting rules come in. In this post, we\u2019ll explain how Pew Research Center developed and now uses these rules to keep our online surveys from running too long.<\/p>\n\n\n\n

In a nutshell, we classify survey questions based on their format. Each question format has a \u201cpoint\u201d value reflecting how long it usually takes people to answer. Formats that people tend to answer quickly (e.g., a single item in a larger battery) have a lower point value than questions that require more time (e.g., an open-ended question where respondents are asked to write in their own answers). A 15-minute ATP survey is budgeted at 85 points, so before the survey begins, researchers sum up all of the question point values to make sure the total is 85 or less.<\/p>\n\n\n\n

The Center\u2019s point system was developed using historical ATP response times. When administering an online survey, researchers can see how long it takes each respondent to answer the questions on each screen. This information has allowed us to determine, for example, that more challenging open-ended questions can take people minutes<\/em> to answer, while an ordinary stand-alone question takes only about 10 seconds<\/em> or so.<\/p>\n\n\n\n

Question types<\/h3>\n\n\n\n

Stand-alone questions<\/h4>\n\n\n\n
\"Stand-alone<\/a><\/figure>\n\n\n\n

Stand-alone questions are the most common type on the ATP. They are likely what comes to mind when someone imagines a classic survey experience \u2013 a straightforward question followed by a set of answer choices. In our counting scheme, these questions count as one point.<\/p>\n\n\n\n

Battery items<\/h4>\n\n\n\n
\"Each<\/a><\/figure>\n\n\n\n

Battery items are another popular form of survey question. Batteries refer to a series of questions with the same stem (e.g., \u201cWhat is your overall opinion of\u2026\u201d), followed by a variety of answer options. Each answer option is assigned two-thirds of a point (0.67 points) in our question counting rules. For example, a battery of five items would count as 5 x 0.67, or 3.35 points in total.<\/p>\n\n\n\n

Open-ended questions<\/h4>\n\n\n\n

Open-ended questions are among the most difficult and time-consuming for respondents to answer. They are also the most likely type for respondents to skip<\/a> without answering. Open-ended questions require respondents to form answers themselves rather than selecting from the options that are provided. While a standard stand-alone question can be answered in a matter of seconds, open-ended questions often take a minute or more to answer. As a result, these questions have markedly higher points assigned to them in the question counting guidelines.<\/p>\n\n\n\n

\"Some<\/a><\/figure>\n\n\n\n

But not all open-ended questions are created equal. Over the years, we\u2019ve noticed that some take much longer to answer than others. For example, questions like, \u201cIn a sentence or two, please describe why you think Americans’ level of confidence in the federal government is a very big problem\u201d would take respondents much longer to answer than a question like, \u201cWho would be your choice for the Republican nomination for president?\u201d Questions that ask for more detail are assigned eight points, while questions that are more straightforward are assigned five points.<\/p>\n\n\n\n

Check-all questions<\/h4>\n\n\n\n
\"Check-all<\/a><\/figure>\n\n\n\n

Check-all questions ask the respondent to select all response options that apply to them. These questions can yield less accurate data than stand-alone questions<\/a>, so they are rarely used on the ATP. Check-all questions count for two points.<\/p>\n\n\n\n

Vignettes<\/h4>\n\n\n\n

Vignettes present respondents with a hypothetical scenario and then ask questions about that scenario. Typically, at least one detail of the scenario is changed for a random subset of the respondents, allowing researchers to determine how that detail affects people\u2019s answers.<\/p>\n\n\n\n

From a survey length standpoint, vignettes are notable for showing respondents a block of text describing a particular scenario. They entail more reading than the other question types. Vignettes are rare on the ATP and are budgeted one point for every 50 words in length.<\/p>\n\n\n\n

\"Vignettes<\/a><\/figure>\n\n\n\n

Thermometer ratings<\/h4>\n\n\n\n

Feeling thermometers ask respondents to rate something on a scale from 0 to 100, in which 0 represents the coldest, most negative view and 100 represents the warmest and most positive.<\/p>\n\n\n\n

\"Thermometer<\/a><\/figure>\n\n\n\n

They are budgeted for 1.5 points because they are more difficult than a question with discrete answer options. It can sometimes be challenging for respondents to map their opinion of someone or something onto a 0-100 scale.<\/p>\n\n\n\n

Conclusion<\/strong><\/h4>\n\n\n\n

Thanks to these question counting rules, the incentives we offer for survey completion and the goodwill and patience of our panelists, about 80% or more of the respondents selected for each survey complete the questionnaire. In other words, the survey-level response rate is usually 80% or higher. Among the narrower set of people who log on to the survey and complete at least one item, about 99% typically complete the entire questionnaire. That is, the break-off rate is usually about 1%.<\/p>\n","protected":false},"excerpt":{"rendered":"

While there is no magic length that an online survey should be, Pew Research Center caps the length of its online American Trends Panel surveys at 15 minutes.<\/p>\n","protected":false},"author":377,"featured_media":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"sub_headline":"","sub_title":"","_prc_public_revisions":[],"_ppp_expiration_hours":0,"_ppp_enabled":false,"ai_generated_summary":"","relatedPosts":[],"_prc_fork_parent":0,"_prc_fork_status":"","_prc_active_fork":0,"datacite_doi":"","datacite_doi_citation":"","_prc_seo_qr_attachment_id":0,"spoken_article_player_enabled":true,"displayBylines":true,"footnotes":"","prc_watchers":[]},"categories":[357],"bylines":[631,779],"collection":[],"_post_visibility":[],"decoded-category":[532],"formats":[454],"_fund_pool":[],"languages":[],"regions-countries":[],"research-teams":[524],"workflow-status":[],"class_list":["post-111245","decoded","type-decoded","status-publish","hentry","category-survey-methods","bylines-courtney-kennedy","bylines-nick-hatley","decoded-category-survey-methods","formats-decoded","research-teams-decoded"],"label":"Decoded","post_parent":0,"word_count":839,"canonical_url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/2022\/12\/08\/how-we-keep-our-online-surveys-from-running-too-long\/","art_direction":{"A1":{"id":125667,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/12\/MB_22.12.06_surveyLength_featured-1.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/12\/MB_22.12.06_surveyLength_featured-1.png?w=564&h=317&crop=1","width":564,"height":317,"caption":"","chartArt":false},"A2":{"id":125667,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/12\/MB_22.12.06_surveyLength_featured-1.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/12\/MB_22.12.06_surveyLength_featured-1.png?w=268&h=151&crop=1","width":268,"height":151,"caption":"","chartArt":false},"A3":{"id":125667,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/12\/MB_22.12.06_surveyLength_featured-1.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/12\/MB_22.12.06_surveyLength_featured-1.png?w=194&h=110&crop=1","width":194,"height":110,"caption":"","chartArt":false},"A4":{"id":125667,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/12\/MB_22.12.06_surveyLength_featured-1.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/12\/MB_22.12.06_surveyLength_featured-1.png?w=268&h=151&crop=1","width":268,"height":151,"caption":"","chartArt":false},"XL":{"id":125667,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/12\/MB_22.12.06_surveyLength_featured-1.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/12\/MB_22.12.06_surveyLength_featured-1.png?w=720&h=405&crop=1","width":720,"height":405,"caption":"","chartArt":false},"social":{"id":125667,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/12\/MB_22.12.06_surveyLength_featured-1.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/12\/MB_22.12.06_surveyLength_featured-1.png?w=1200&h=628&crop=1","width":1200,"height":628,"caption":"","chartArt":false}},"_embeds":[],"watchers":[],"table_of_contents":[],"datacite_doi":"","prc_seo_data":{"title":"How we keep our online surveys from running too long","description":"While there is no magic length that an online survey should be, Pew Research Center caps the length of its online American Trends Panel surveys at 15 minutes.","og_title":"How we keep our online surveys from running too long","og_description":"While there is no magic length that an online survey should be, Pew Research Center caps the length of its online American Trends Panel surveys at 15 minutes.","schema_type":"Article","noindex":false,"canonical_url":"","primary_terms":{"category":43},"custom_schema":[],"og_image":125667,"indexnow_submitted_at":null,"gsc_index_status":null},"prepublish_checks":{},"jetpack_sharing_enabled":true,"relatedPostsOrdered":[],"bylinesOrdered":[{"key":"_4f02s29mf","termId":779},{"key":"_4lbrn5e4w","termId":631}],"acknowledgementsOrdered":[],"_links":{"self":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111245","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded"}],"about":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/types\/decoded"}],"author":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/users\/377"}],"replies":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/comments?post=111245"}],"version-history":[{"count":2,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111245\/revisions"}],"predecessor-version":[{"id":138489,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111245\/revisions\/138489"}],"wp:attachment":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/media?parent=111245"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/categories?post=111245"},{"taxonomy":"bylines","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/bylines?post=111245"},{"taxonomy":"collection","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/collection?post=111245"},{"taxonomy":"_post_visibility","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_post_visibility?post=111245"},{"taxonomy":"decoded-category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded-category?post=111245"},{"taxonomy":"formats","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/formats?post=111245"},{"taxonomy":"_fund_pool","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_fund_pool?post=111245"},{"taxonomy":"languages","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/languages?post=111245"},{"taxonomy":"regions-countries","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/regions-countries?post=111245"},{"taxonomy":"research-teams","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/research-teams?post=111245"},{"taxonomy":"workflow-status","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/workflow-status?post=111245"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}