{"id":111567,"date":"2019-10-14T14:22:00","date_gmt":"2019-10-14T19:22:00","guid":{"rendered":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/\/\/how-we-check-numbers-and-facts-at-pew-research-center\/"},"modified":"2024-04-14T04:10:38","modified_gmt":"2024-04-14T09:10:38","slug":"how-we-check-numbers-and-facts-at-pew-research-center","status":"publish","type":"decoded","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/2019\/10\/14\/how-we-check-numbers-and-facts-at-pew-research-center\/","title":{"rendered":"How we check numbers and facts at Pew Research Center"},"content":{"rendered":"\n
\"\"<\/a><\/figure>\n\n\n\n

Pew Research Center is a fact tank<\/a> \u2014 a nonpartisan research organization committed to creating a foundation of facts to help inform policymakers and the public about important issues, without taking policy or advocacy positions. To that end, all of the material we release, from reports to blog posts to interactive graphics and even tweets, goes through a rigorous verification process we call a \u201cnumber check.\u201d<\/p>\n\n\n\n

It\u2019s called<\/em> a number check, but we check much more than the numbers. Apart from verifying that the numbers cited in our writing or associated charts are correct, we ask ourselves other questions, too: Are we describing the numbers in a way that\u2019s accurate and fits with the narrative? Are the claims we make, such as calling something a \u201cmajority,\u201d<\/a> statistically significant? Are we leaving out any keys to understanding the data, such as a definition of the survey \u201cuniverse\u201d<\/a>? Is our writing biased or misleading in any way? Will it be understood by our audience?<\/p>\n\n\n\n

Although there are slight differences in the process depending on the nature of the data used in a given report (as well as the needs of the specific team within the Center that\u2019s publishing it), the process is rigorous no matter what we\u2019re checking. To illustrate how we go about it, here\u2019s a step-by-step walkthrough of how I number check a piece of domestically focused survey research:<\/p>\n\n\n\n

Getting to know the data<\/h4>\n\n\n\n

Ideally, a number check is done by someone who is somewhat familiar with the project or subject matter but who didn\u2019t contribute to the writing or graphics at all. This reduces the chances that the number checker will make any of the same errors as the author and also allows for a fresh perspective.<\/p>\n\n\n\n

First, I talk to the author or authors of the content to familiarize myself with the data source, discuss any concerns they may have about sensitive language or possible pitfalls with the data, and assess the number-checking needs for the underlying data. While our survey data goes through a rigorous quality control check before we start analyzing it \u2014 mainly to identify anomalies or programming errors \u2014 any custom tables or statistical programming syntax the author created will likely need to be number checked or replicated. This is to ensure variables are coded correctly and that the syntax is performing the intended statistical analyses. If the research is based on a secondary source, such as U.S. Census Bureau data, the number checker will need to verify that the correct file was downloaded and review or replicate the author\u2019s analysis.<\/p>\n\n\n\n

Checking every word and number<\/h4>\n\n\n\n

Once the underlying data sources are verified, I can dive into the actual text and graphics of the report or blog post. All number checkers have their own methods, but I always number check on paper, crossing off each word and number in pencil as I confirm it. Suggestions and style changes go in blue pen and wrong numbers are pointed out in red. At the end of the process, I do a double check to make sure all changes were put into the final version correctly and check off each note in green pen. Here\u2019s an example of what a number-checked blog post can look like:<\/p>\n\n\n\n

\"\"<\/a><\/figure>\n\n\n\n

When number checking a report or blog post, I usually start by checking the charts and tables and asking myself the following questions:<\/p>\n\n\n\n

\u00b7 Is each number shown in the chart correct? Are numbers rounded correctly?<\/p>\n\n\n\n

\u00b7 Is the underlying<\/em> number that was plotted in the chart correct? For example, does a point on a line chart labeled 50% actually represent 50% on the chart? Is a bar labeled 38% longer than one labeled 30%?<\/p>\n\n\n\n

\"\"<\/a>
What’s wrong with this chart?<\/figcaption><\/figure>\n\n\n\n

\u00b7 Are all the subgroups in a given chart (e.g., Republicans, Democrats, independents) based on large enough sample sizes that we feel confident about our estimates? (Our usual rule of thumb is to require an effective sample size<\/a> of about 100 or more.)<\/p>\n\n\n\n

\u00b7 Does the sub-headline accurately describe the universe<\/a> the chart is based on? Are the sub-headline and chart labels clear in explaining what the chart conveys? If the chart is based on a survey question, do the chart labels and sub-headline wording capture the survey question?<\/p>\n\n\n\n

\u00b7 Are the notes and source in the chart accurate? Are any notes missing? For example, if the chart displays data by race and ethnicity, do we have a note to clarify that whites and blacks include only single-race non-Hispanics, while Hispanics are of any race?<\/p>\n\n\n\n

\u00b7 Is the headline accurate and relevant to the main finding of the chart?<\/p>\n\n\n\n

\u00b7 Can this chart stand alone? If it is shared out of context on social media, does it have all the critical information needed to understand it, or could it be misinterpreted?<\/p>\n\n\n\n

Verifying claims and characterizations<\/h4>\n\n\n\n

Next, I begin to check the text of the report or blog post, including the methodology, terminology and any additional tables or appendices. Much of it can usually be checked against the charts and tables I\u2019ve already verified. For the rest, I will refer back to the data source. Sometimes, the author will reference a previous Pew Research Center report or outside research. In that case, I\u2019ll read enough of the referenced text to determine if they\u2019ve accurately summarized it, pulled a number from it or quoted it. I will also check any references to current events or historical facts by checking dates, claims, names and other basic facts against a reputable source.<\/p>\n\n\n\n

One of the most challenging aspects of number checking is verifying the author\u2019s claims and characterizations of the numbers. I run the appropriate statistical tests to determine whether, for example, 51% is a statistically significant majority (in survey data, it\u2019s usually not<\/a>), whether women are more likely than men to say there are too few women in high political office (they are<\/a>), or whether the long-term decline in unauthorized immigration from Mexico is really a decline (it is<\/a>).<\/p>\n\n\n\n

Since probability-based survey estimates are associated with a margin of error<\/a>, we can\u2019t confidently say that a change over time is statistically significant if the margins of error for the two figures overlap. An increase or decrease of only a couple of percentage points usually does not meet this bar. We take our language seriously, and any use of the words \u201cplurality,\u201d \u201cmajority,\u201d \u201cminority,\u201d \u201cmost,\u201d \u201cincrease\u201d and \u201cdecrease\u201d (along with some others) must be rigorously checked for statistical significance.<\/p>\n\n\n\n

In cases where we don\u2019t<\/em> have a hard-and-fast institutional rule, number checkers sometimes must exercise their own judgment. In a situation where 54% of Americans approve of a policy and 46% disapprove, for instance, should this be characterized as majority approval? Or are Americans about evenly divided on the issue in question, even though the difference may be statistically significant? Context matters, and we try to be as fair as possible.<\/p>\n\n\n\n

The number checker also has to pay close attention to other types of language. While authors try to vary their phrasing a little throughout a report to make their text more readable, there\u2019s a balance between accurately portraying the wording that was used in a survey question and using creative language. Consider a question from a 2016 survey of police officers<\/a>:<\/p>\n\n\n\n

Do you think the deaths of African Americans during encounters with police in recent years are isolated incidents or signs of a broader problem between African Americans and the police?<\/em><\/p>\n\n\n\n

This is a long question, so the authors naturally sought ways to shorten it rather than writing it verbatim dozens of times throughout the report. In an early draft, one of those ways was to refer to the incidents asked about in the survey question as \u201crecent deadly encounters between police and blacks.\u201d While our survey was in the field, however, a black man shot and killed five police officers<\/a> in Dallas. With that incident in mind, I thought \u201crecent deadly encounters between police and blacks\u201d was not specific enough shorthand since it could also refer to the Dallas shooting \u2014 clearly not<\/em> what the survey question was referring to. We settled on \u201cdeaths of blacks during encounters with police in recent years\u201d as an alternative to restating the entire survey question in every reference.<\/p>\n\n\n\n

Checking for consistency<\/h4>\n\n\n\n

After all of the above is done, I look through the material again to check for style and consistency. I\u2019ll catch much of this in my initial number check (and the copy editor will also serve as a backup), but at this stage in the process I like to make sure the whole report is cohesive and matches our internal style guidelines. A lot of this focuses on charts. For example: Are we using the same wording in the sub-headline and labels from chart to chart when they\u2019re describing the same things? In opposing bar charts, do we consistently have the \u201coppose\u201d numbers on the left side and the \u201cfavor\u201d numbers on the right side? Are we using the same color scheme to represent men and women throughout? Is the language for notes and sources the same throughout? Other things we look for in the text include whether numbers are consistently rounded to the same decimal place.<\/p>\n\n\n\n

Discussing changes with the author or authors<\/h4>\n\n\n\n

After I\u2019m done with my number check, I usually take my written comments and translate them into a marked-up version in Microsoft Word. This gives me a second chance to review my comments and think about how best to implement them. Then I send the document to the author or authors so they can review my edits, double check any changes I\u2019ve made, answer any questions that arose, accept or reject my edits and comments (with an explanation for anything rejected), and send me back a clean copy. Then I go through my hard copy and check off that all my edits were made correctly. If anything I felt strongly about didn\u2019t get changed, the author and I will have a conversation to come to a compromise or explain our respective reasoning.<\/p>\n\n\n\n

\"\"<\/a>
A typical blog post returned to the author with my number check comments
<\/figcaption><\/figure>\n\n\n\n

Seeing each piece of research through to publication<\/h4>\n\n\n\n

The number checking doesn\u2019t end there! The number checker will follow along with the publication process until the research they checked is publicly released. After a number check is complete, the report or blog post goes to one of Pew Research Center\u2019s copy editors, who take their own pass through it to improve clarity and ensure all style guidelines are followed. When the copy edit is done, both the number checker and the author review whatever changes the copy editor made to make sure no numbers (or the meaning of particular phrases or sentences) have inadvertently changed.<\/p>\n\n\n\n

Once the report or blog post is loaded into our content management system ahead of publication, I\u2019ll review it once again to make sure no paragraphs were inadvertently dropped, all hyperlinks work, and all charts are placed in the appropriate spots in the text. Sometimes colleagues see the final draft and have feedback for the author, in which case I always review any post-number check changes. Finally, I\u2019ll check any associated materials, such as press releases, interactives and tweets that will be sent from any of our institutional accounts<\/a>, particularly since tweets are highly visible distillations of our research.<\/p>\n","protected":false},"excerpt":{"rendered":"

All of the material we release goes through a rigorous verification process we call a \u201cnumber check.\u201d<\/p>\n","protected":false},"author":655,"featured_media":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"sub_headline":"","sub_title":"","_crdt_document":"","_prc_public_revisions":[],"_ppp_expiration_hours":0,"_ppp_enabled":false,"ai_generated_summary":"","relatedPosts":[],"_prc_fork_parent":0,"_prc_fork_status":"","_prc_active_fork":0,"datacite_doi":"","datacite_doi_citation":"","_prc_seo_qr_attachment_id":0,"spoken_article_player_enabled":true,"bylines":[{"key":"_l7g6gyrwi","termId":924}],"acknowledgements":[],"displayBylines":true,"footnotes":"","prc_watchers":[]},"categories":[],"bylines":[924],"collection":[],"_post_visibility":[],"decoded-category":[535],"formats":[],"_fund_pool":[],"languages":[],"regions-countries":[],"research-teams":[524],"workflow-status":[],"class_list":["post-111567","decoded","type-decoded","status-publish","hentry","bylines-anna-brown","decoded-category-communicating-findings","research-teams-decoded"],"label":"Decoded","post_parent":0,"word_count":1918,"canonical_url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/2019\/10\/14\/how-we-check-numbers-and-facts-at-pew-research-center\/","art_direction":{"A1":{"id":125941,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.14.2019_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.14.2019_feature.png?w=564&h=317&crop=1","width":564,"height":317,"caption":"","chartArt":false},"A2":{"id":125941,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.14.2019_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.14.2019_feature.png?w=268&h=151&crop=1","width":268,"height":151,"caption":"","chartArt":false},"A3":{"id":125941,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.14.2019_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.14.2019_feature.png?w=194&h=110&crop=1","width":194,"height":110,"caption":"","chartArt":false},"A4":{"id":125941,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.14.2019_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.14.2019_feature.png?w=268&h=151&crop=1","width":268,"height":151,"caption":"","chartArt":false},"XL":{"id":125941,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.14.2019_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.14.2019_feature.png?w=700&h=394&crop=1","width":700,"height":394,"caption":"","chartArt":false},"social":{"id":125941,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.14.2019_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.14.2019_feature.png?w=700&h=394&crop=1","width":700,"height":394,"caption":"","chartArt":false}},"_embeds":[],"watchers":[],"table_of_contents":[],"datacite_doi":"","prc_seo_data":{"title":"How we check numbers and facts at Pew Research Center","description":"All of the material we release goes through a rigorous verification process we call a \u201cnumber check.\u201d","og_title":"How we check numbers and facts at Pew Research Center","og_description":"All of the material we release goes through a rigorous verification process we call a \u201cnumber check.\u201d","schema_type":"Article","noindex":false,"canonical_url":"","primary_terms":{"category":46},"custom_schema":[],"og_image":125941,"indexnow_submitted_at":null,"gsc_index_status":null},"prepublish_checks":{},"jetpack_sharing_enabled":true,"relatedPostsOrdered":[],"bylinesOrdered":[{"key":"_l7g6gyrwi","termId":924}],"acknowledgementsOrdered":[],"_links":{"self":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111567","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded"}],"about":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/types\/decoded"}],"author":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/users\/655"}],"replies":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/comments?post=111567"}],"version-history":[{"count":2,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111567\/revisions"}],"predecessor-version":[{"id":138532,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111567\/revisions\/138532"}],"wp:attachment":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/media?parent=111567"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/categories?post=111567"},{"taxonomy":"bylines","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/bylines?post=111567"},{"taxonomy":"collection","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/collection?post=111567"},{"taxonomy":"_post_visibility","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_post_visibility?post=111567"},{"taxonomy":"decoded-category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded-category?post=111567"},{"taxonomy":"formats","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/formats?post=111567"},{"taxonomy":"_fund_pool","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_fund_pool?post=111567"},{"taxonomy":"languages","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/languages?post=111567"},{"taxonomy":"regions-countries","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/regions-countries?post=111567"},{"taxonomy":"research-teams","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/research-teams?post=111567"},{"taxonomy":"workflow-status","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/workflow-status?post=111567"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}