{"id":111631,"date":"2019-03-06T15:45:00","date_gmt":"2019-03-06T20:45:00","guid":{"rendered":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/\/\/on-a-scale-from-1-to-10-how-much-do-the-numbers-used-in-survey-scales-really-matter\/"},"modified":"2024-04-14T04:10:41","modified_gmt":"2024-04-14T09:10:41","slug":"on-a-scale-from-1-to-10-how-much-do-the-numbers-used-in-survey-scales-really-matter","status":"publish","type":"decoded","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/2019\/03\/06\/on-a-scale-from-1-to-10-how-much-do-the-numbers-used-in-survey-scales-really-matter\/","title":{"rendered":"On a scale from 1 to 10, how much do the numbers used in survey scales really matter?"},"content":{"rendered":"\n
\"\"<\/a>
(Related post: <\/em>Small changes in survey scales can matter when measuring political ideology in Europe<\/em><\/a>)<\/em><\/figcaption><\/figure>\n\n\n\n

You\u2019re probably familiar with surveys that ask you to rate yourself on a scale from 1 to 10, whether they ask about your happiness, job satisfaction or even political leanings. Sometimes, these questions use a 0\u201310 scale instead. But do these numbers really matter, apart from their rank order? Is it possible, for instance, that some survey respondents might be more likely to describe themselves with the number 1 than the number 0 because 1 is a positive number and 0 is not?<\/p>\n\n\n\n

Understanding how the numeric labels on scales might influence survey responses<\/a> is an area of ongoing investigation<\/a> for researchers. At Pew Research Center, we\u2019ve spent some time thinking about this, too.<\/p>\n\n\n\n

We recently ran an experiment testing whether the numeric labels used in one of our scales influenced how respondents reported their political ideology. We compared the results from a survey question that used 0 as the minimum value on a scale with one that used 1 instead. (Other researchers have considered how people respond to political ideology questions<\/a> and the limitations of asking about political ideology<\/a> more generally.)<\/p>\n\n\n\n

We conducted this experiment as part of a 2017 telephone survey<\/a> about religion and pluralism in Western Europe. (For the survey methodology, see here<\/a>.) To carry it out, we randomly assigned respondents in France, Germany and the United Kingdom our political ideology question with one of two 7-point scales: either 0\u20136 or 1\u20137. The full question wording using our traditional 0\u20136 scale was this: \u201cSome people talk about politics in terms of left, center and right. On a left-right scale from 0 to 6, with 0 indicating extreme left and 6 indicating extreme right, where would you place yourself?\u201d<\/em><\/p>\n\n\n\n

Our goal was to address several questions:<\/p>\n\n\n\n

1. Do different scale endpoints result in different frequency distributions across the ideological spectrum<\/a> or in different rates of item nonresponse?<\/p>\n\n\n\n

2. Do differently numbered scales affect the broader left-right lean of the scale<\/a>?<\/p>\n\n\n\n

3. Do the two scales result in different demographic profiles<\/a> for groups on the ideological spectrum (for example, comparing the demographics of the ideological center on each scale)?<\/p>\n\n\n\n

4. Is our understanding of how ideological groups feel about major political parties and important societal issues<\/a> influenced by which response scale we use to measure ideology?<\/p>\n\n\n\n

Below, you\u2019ll find the results of our experiment. For ease of discussion, I\u2019ve labeled the interior points of the scales even though the question itself only assigned labels to the endpoints (that is, Extreme Left and Extreme Right). The resulting scale (in ascending \u201cnumeric order\u201d) is as follows: Extreme Left, Left, Center-Left, Center, Center-Right, Right and Extreme Right.<\/p>\n\n\n\n

Distributions across the ideological spectrum<\/h4>\n\n\n\n

The concern that fewer people would be willing to self-report as Extreme Left when that response is associated with 0 rather than 1 is unfounded. In all three countries, respondents were equally likely to pick the Extreme Left answer category regardless of which numeric scale was used. Similarly, in each country, item nonresponse rates (the rates at which people chose not to answer the ideology questions) were consistent between the two scales.<\/p>\n\n\n\n

\"\"<\/a><\/figure>\n\n\n\n

Furthermore, while Left self-reports differed across the scales in all three countries, it was not in the direction that would support the hypothesis that people would be unwilling to choose the Extreme Left category when it\u2019s assigned a value of 0. If this hypothesis were true, we\u2019d expect a spike of 0\u20136 scale respondents choosing Left (numeric value 1) as compared to the 1\u20137 scale respondents choosing Left (numeric value 2). In fact, more respondents who were assigned the 1\u20137 scale chose Left in each country.<\/p>\n\n\n\n

Across all six scale distributions we studied, there was a peak near the middle that tapered toward the scale\u2019s ends. In the UK, both scale versions peaked at the ideological Center, regardless of the numeric value associated with it. But in France and Germany, the modal category was associated with the numeric value 3, regardless of the ideological label: The number 3 represents Center on the 0\u20136 scale, but <\/em>Center-Left on the 1\u20137 scale. There were also significant differences between the scales at the Center positions in France and Germany, with the 0\u20136 scales having more Center responses.<\/p>\n\n\n\n

These outcomes suggest that when a scale is easily divided in half \u2014 for example, when the maximum value is 6 rather than 7 \u2014 it\u2019s more likely for respondents to select the midpoint. Previous research<\/a> has found that respondents are likely to assume that half the top endpoint is a scale\u2019s midpoint, so when half the top endpoint is not an answer option (e.g., 3.5 on the 1\u20137 scale), respondents seeking the central point on the scale may sometime choose 3 (not the scale midpoint) and sometimes choose 4 (the actual scale midpoint). Those who received the 0\u20136 scale could more easily find the midpoint (3) by halving the top endpoint.<\/p>\n\n\n\n

See <\/em>here<\/em><\/a> for supplementary tables and graphics.<\/em><\/p>\n\n\n\n

Left-right lean of the scales<\/h4>\n\n\n\n

To determine if the differently numbered scales affected the broader left-right lean of the scale, I condensed each seven-point scale into three general categories: Left-c, Center and Right-c. (The \u201c-c\u201d differentiates these categories from the similarly named categories in the seven-point scales.) Left-c is a combination of the Extreme Left, Left and Center-Left categories, Right-c is a combination of the corresponding categories at the other end of the scales, and Center is simply the Center category in both scales.<\/p>\n\n\n\n

In all three countries, using the 1\u20137 scale increased self-identification as Left-c in comparison to the 0\u20136 scale. Scholars have noted that the meaning of the center<\/a>in political ideology scales can vary based on context and respondent interpretation. The use of 0 vs. 1 as the lower endpoint could have altered the respondent perception of the question and its answer options, leading to different distributions.<\/p>\n\n\n\n

\"\"<\/a><\/figure>\n\n\n\n

See <\/em>here<\/em><\/a> for supplementary tables and graphics.<\/em><\/p>\n\n\n\n

Demographic differences<\/h4>\n\n\n\n

In all three countries, when using the condensed distribution of Left-c, Center and Right-c, there are no statistically significant differences between the two scales when it comes to demographic structure (age, gender and education). For example, within each country, the Left-c is similarly divided between old and young, male and female, and more and less educated \u2014 no matter which scale is used.<\/p>\n\n\n\n

When looking at the full 7-point ideology scales, only four categories have sufficient sample sizes (roughly 100 cases or more) to make comparisons between the two scales. These are the Left, Center-Left, Center and Center-Right categories. (Center is the same as in the condensed distribution so is not considered again). Looking at the demographic makeup of these categories, there is only one statistically significant difference: Using France\u2019s 0\u20136 scale, a majority of the Left is male (56%), while on the 1\u20137 scale, a minority of the Left is male (41%).<\/p>\n\n\n\n

See <\/em>here<\/em><\/a> for supplementary tables.<\/em><\/p>\n\n\n\n

Political party favorability and other societal issues<\/h4>\n\n\n\n

We also asked respondents in each country if they had a very favorable, somewhat favorable, somewhat unfavorable or very unfavorable opinion of several national political parties. For this analysis, I selected two traditional parties<\/a> from each country (one party each from the left and the right): France\u2019s Socialist Party and the Republicans; Germany\u2019s Social Democratic Party (SPD) and Christian Democratic Union (CDU); and the UK\u2019s Labour Party and Conservative Party. I compared \u201cnet\u201d favorability (combining very favorable and somewhat favorable) toward the parties for each ideological category between the scales.<\/p>\n\n\n\n

Overall, political ideology corresponds consistently to political party favorability, no matter which scale is used. There are a few differences across the three countries, but nothing suggests a systematic difference between the two ideology scales.<\/p>\n\n\n\n

Meanwhile, to get an idea of how the two scales interact with other topics, I compared the different measures of ideology across a variety of substantive questions. The items selected were about religion (the share who say religion is very important in their life), nationalism (the share who completely\/mostly agree that \u201cour people are not perfect, but our culture is superior to others\u201d), minority religions (the share who say Islam is fundamentally incompatible with their national culture and values), immigration (the share who think the number of immigrants to their country should be reduced) and social issues (the share who think abortion should be legal in all or most cases). The survey did not ask questions about economic policy, which could also relate to an individual\u2019s political ideology.<\/p>\n\n\n\n

By and large, respondents in the corresponding ideology categories from the different scales give the same answers to these questions. This is true for both the consolidated left-center-right categorization and the categories from the seven-point scales for which we have sufficient sample sizes to do an analysis (roughly 100 cases or more). Across all these points of comparison, nothing suggests systematic differences between the ideology scales. And the individual differences \u2014 even when fairly wide \u2014 tend not to change the overarching story of a given group\u2019s stance on a particular topic. For example, using France\u2019s 0\u20136 scale, 59% of the Left thinks abortion should be legal, while on France\u2019s 1\u20137 scale, 87% of the Left supports legalized abortion. This is a statistically significant difference, to be sure, but it still shows broad support among the Left.<\/p>\n\n\n\n

See <\/em>here<\/em><\/a> for supplementary tables.<\/em><\/p>\n\n\n\n

Conclusions<\/h4>\n\n\n\n

If a researcher\u2019s goal is to get a sense of the relative size of the political left, center or right within a country, there can be meaningful differences depending on whether one uses the 0\u20136 scale or the 1\u20137 scale \u2014 especially when it comes to the end of the scale closest to 0. Further research is required to know which scale presents a more valid measure of political ideology.<\/p>\n\n\n\n

However, an ideological group\u2019s demographic profile, favorability toward specific political parties and attitudes on other important issues do not drastically change when different numeric values are used in a seven-point scale. The overall impact of using a 0\u20136 scale or a 1\u20137 scale as an independent variable or control for social research appears minimal.<\/p>\n","protected":false},"excerpt":{"rendered":"

Understanding how the numeric labels on scales might\u00a0influence survey responses\u00a0is an\u00a0area of ongoing investigation\u00a0for researchers. <\/p>\n","protected":false},"author":655,"featured_media":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"sub_headline":"","sub_title":"","_prc_public_revisions":[],"_ppp_expiration_hours":0,"_ppp_enabled":false,"ai_generated_summary":"","relatedPosts":[],"datacite_doi":"","datacite_doi_citation":"","_prc_seo_qr_attachment_id":0,"spoken_article_player_enabled":true,"displayBylines":true,"footnotes":"","prc_watchers":[],"_prc_fork_parent":0,"_prc_fork_status":"","_prc_active_fork":0},"categories":[357],"bylines":[866],"_post_visibility":[],"decoded-category":[532],"formats":[],"_fund_pool":[],"languages":[],"regions-countries":[],"research-teams":[524],"workflow-status":[],"collection":[],"class_list":["post-111631","decoded","type-decoded","status-publish","hentry","category-survey-methods","bylines-jonathan-evans","decoded-category-survey-methods","research-teams-decoded"],"label":"Decoded","post_parent":0,"word_count":1661,"canonical_url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/2019\/03\/06\/on-a-scale-from-1-to-10-how-much-do-the-numbers-used-in-survey-scales-really-matter\/","art_direction":{"A1":{"id":125997,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/03.06.2019_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/03.06.2019_feature.png?w=564&h=317&crop=1","width":564,"height":317,"caption":"","chartArt":false},"A2":{"id":125997,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/03.06.2019_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/03.06.2019_feature.png?w=268&h=151&crop=1","width":268,"height":151,"caption":"","chartArt":false},"A3":{"id":125997,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/03.06.2019_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/03.06.2019_feature.png?w=194&h=110&crop=1","width":194,"height":110,"caption":"","chartArt":false},"A4":{"id":125997,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/03.06.2019_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/03.06.2019_feature.png?w=268&h=151&crop=1","width":268,"height":151,"caption":"","chartArt":false},"XL":{"id":125997,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/03.06.2019_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/03.06.2019_feature.png?w=720&h=405&crop=1","width":720,"height":405,"caption":"","chartArt":false},"social":{"id":125997,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/03.06.2019_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/03.06.2019_feature.png?w=1200&h=628&crop=1","width":1200,"height":628,"caption":"","chartArt":false}},"_embeds":[],"prc_checklist":{},"watchChildPosts":false,"watchers":[],"table_of_contents":[],"datacite_doi":"","prc_seo_data":{"title":"On a scale from 1 to 10, how much do the numbers used in survey scales really matter?","description":"Understanding how the numeric labels on scales might\u00a0influence survey responses\u00a0is an\u00a0area of ongoing investigation\u00a0for researchers. ","og_title":"On a scale from 1 to 10, how much do the numbers used in survey scales really matter?","og_description":"Understanding how the numeric labels on scales might\u00a0influence survey responses\u00a0is an\u00a0area of ongoing investigation\u00a0for researchers. ","schema_type":"Article","noindex":false,"canonical_url":"","primary_terms":{"category":43},"custom_schema":[],"og_image":125997,"indexnow_submitted_at":null,"gsc_index_status":null},"jetpack_sharing_enabled":true,"relatedPostsOrdered":[],"bylinesOrdered":[{"key":"_l2f17emp3","termId":866}],"acknowledgementsOrdered":[],"_links":{"self":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111631","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded"}],"about":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/types\/decoded"}],"author":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/users\/655"}],"replies":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/comments?post=111631"}],"version-history":[{"count":2,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111631\/revisions"}],"predecessor-version":[{"id":138543,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111631\/revisions\/138543"}],"wp:attachment":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/media?parent=111631"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/categories?post=111631"},{"taxonomy":"bylines","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/bylines?post=111631"},{"taxonomy":"_post_visibility","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_post_visibility?post=111631"},{"taxonomy":"decoded-category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded-category?post=111631"},{"taxonomy":"formats","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/formats?post=111631"},{"taxonomy":"_fund_pool","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_fund_pool?post=111631"},{"taxonomy":"languages","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/languages?post=111631"},{"taxonomy":"regions-countries","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/regions-countries?post=111631"},{"taxonomy":"research-teams","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/research-teams?post=111631"},{"taxonomy":"workflow-status","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/workflow-status?post=111631"},{"taxonomy":"collection","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/collection?post=111631"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}