{"id":111113,"date":"2024-03-28T18:53:34","date_gmt":"2024-03-28T23:53:34","guid":{"rendered":""},"modified":"2024-04-26T14:18:35","modified_gmt":"2024-04-26T18:18:35","slug":"reproducibility-as-part-of-code-quality-control","status":"publish","type":"decoded","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/2024\/03\/28\/reproducibility-as-part-of-code-quality-control\/","title":{"rendered":"Reproducibility as part of code quality control"},"content":{"rendered":"\n<figure class=\"wp-block-image size-640-wide is-resized\"><a href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/\/\/reproducibility-as-part-of-code-quality-control\/d_24-03-28_reproducibility_feature-png\/\" rel=\"attachment wp-att-125533\"><img data-dominant-color=\"c9cccb\" data-has-transparency=\"false\" loading=\"lazy\" decoding=\"async\"  srcset=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2024\/03\/d_24.03.28_reproducibility_feature.png?resize=480,270 480w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2024\/03\/d_24.03.28_reproducibility_feature.png?resize=782,440 782w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2024\/03\/d_24.03.28_reproducibility_feature.png?resize=960,540 960w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2024\/03\/d_24.03.28_reproducibility_feature.png?resize=1200,675 1200w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2024\/03\/d_24.03.28_reproducibility_feature.png?resize=1400,788 1400w\" sizes=\"(max-width: 480px) 480px, (max-width: 782px) 782px, 640px\" height=\"360\" width=\"640\" src=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2024\/03\/d_24.03.28_reproducibility_feature.png?w=640\" alt=\"Pew Research Center illustration\" class=\"wp-image-125533 not-transparent\" style=\"--dominant-color: #c9cccb; width:640px\" \/><\/a><figcaption class=\"wp-element-caption\">Pew Research Center illustration<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Computational social scientists rely on code as the main instrument for doing our work. Code is a powerful tool, but even the most seasoned coders can make mistakes in ways that might call the results of their analysis into question.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Having <a href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/2023\/04\/05\/how-we-review-code-at-pew-research-center\/\">a good process for code review<\/a> is critical to ensuring that what we report is correct. In this post, we&#8217;ll discuss a key piece of our code review process at Pew Research Center: reproducibility<em>.<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">When we talk about reproducibility, we often think of it as a framework for improving research collaboration and the broader research process. It gives others the ability to replicate our results using the same data and methods, and to reuse and build on that work.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">But reproducibility can also help facilitate collaboration among researchers on the same team, which is what we\u2019ll focus on here. Our broad philosophy for facilitating collaboration through reproducibility is twofold:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>We want to<\/strong> <strong>make it as easy as possible for our researchers to check each other\u2019s code,&nbsp; even if someone hasn\u2019t worked on that project before<\/strong>. Reviewers should be able to easily build the computational environment that the original researchers used for their analysis \u2013 the core tenet of computational reproducibility. Being able to work in a replicate environment also simplifies things like figuring out how the project is organized and documented.<\/li>\n\n\n\n<li><strong>We want to<\/strong> <strong>provide as much flexibility as possible to individual researchers and minimize the amount of engineering resources needed to support each project.<\/strong> Our team uses R and Python; stores data in files and databases; and works within shared environments and on personal computers. We want to avoid overly specific technical recommendations that may not travel well from project to project. &nbsp;<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"our-reproducibility-toolkit-nbsp\">Our reproducibility toolkit&nbsp;<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">The core piece of our internal reproducibility process is a consistent, shared project structure for all our team\u2019s active projects. This structure lets researchers move between projects as contributors or reviewers because they\u2019re already familiar with where things live and can easily find all the information they need.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Our specific tool is a customized <a href=\"https:\/\/github.com\/cookiecutter\/cookiecutter\"><code>cookiecutter<\/code> template<\/a> that includes a standardized directory structure and guidelines for how to structure code and store data for every project.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">One of our template\u2019s guidelines is that we prefer code to be structured as batch scripts rather than interactive notebooks. To ensure that the order in which scripts are executed is as transparent as possible, <a href=\"https:\/\/snakemake.readthedocs.io\/en\/stable\/\">we use <code>Snakemake<\/code><\/a>, which helps users figure out what will happen when they run a piece of code. It can identify which inputs the code will consume and which outputs, like figures or other datasets, it will produce.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Even a well-documented set of scripts will fail if those scripts are run in a new environment that does not include the same versions of the same packages that the original researchers used. To address that problem, <a href=\"https:\/\/docs.conda.io\/en\/latest\/\">we adopted <code>conda<\/code><\/a>, a package and environment management system that helps ensure each project has a way of declaring the computational environment used to produce its results. We chose <code>conda<\/code> because it can manage R and Python installations and handle the integrity of the environment down to system-level libraries. For R-only projects, <a href=\"https:\/\/cran.r-project.org\/web\/packages\/renv\/index.html\">we sometimes use <code>renv<\/code><\/a> to either replace or complement <code>conda<\/code>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We often want to ensure our code can run on any machine, but we think it\u2019s useful to have a shared platform that everyone can access. Our <a href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/2021\/12\/09\/how-we-built-our-data-science-infrastructure-at-pew-research-center\/\">data science platform<\/a>, a JupyterHub instance, is a shared environment that gives researchers flexible computational resources and which we use as the reference architecture on which projects need to be replicable. &nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Finally, we standardize the locations where our project assets \u2013 raw data, as well as the code to work with it \u2013 are stored. We do this so they are accessible to all team members. We use <a href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/2022\/08\/01\/how-pew-research-center-uses-git-and-github-for-version-control\/\">GitHub for our code<\/a>, and we typically store data in our AWS infrastructure.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We classify data assets into two categories: those that need to be preserved for the future because they cannot be reproduced, such as original raw data we collected; and intermediate datasets that eventually can be discarded. This separation means that at the end of a project, all relevant code will live in GitHub with references to a fixed, shared, \u201cread-only\u201d location in a private AWS S3 bucket or database. This bucket contains all the data that we cannot or should not reproduce.&nbsp;&nbsp;<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"reproducibility-as-a-workflow-problem-nbsp\">Reproducibility as a workflow problem&nbsp;<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">With the workflow described above, anyone on our team can easily rerun any project on their own \u2013 from accessing the canonical version of the underlying data to recreating the computational environment in which the analysis was run.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The downside is that this adds work for the original researchers. They now need to worry not only about making sure that things work for them, but also about ensuring that things will work for others. Put simply: Following these reproducibility policies takes time and attention from individual researchers in the short term, in exchange for long-term benefits to the team as a whole.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In this way, reproducibility is not all that different from writing documentation. As end users, we all welcome code that has been nicely documented because it makes our lives easier. But in the race to complete a project, we often feel we could be doing something more productive with our time. When developing this workflow, we wanted to balance the end goal of reproducibility with the demands of the process itself.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We\u2019ve also learned that reproducibility tools are not simply a neutral layer on top of code. These tools force researchers to structure their workflow in a particular way. For instance, researchers can find <code>Snakemake<\/code> challenging if they\u2019re accustomed to keeping all of their data in memory throughout a single session. Similarly, researchers working in any virtual environment are expected to keep their environment in sync by deleting packages that are no longer needed or by documenting all new dependencies. That\u2019s hard to do, especially during project phases when researchers are trying out many different packages in rapid succession.&nbsp;&nbsp;&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Ultimately, improving reproducibility means adding new tools and changing the workflows that researchers are used to \u2013 and who likes that? Even beyond the tools themselves, this effort requires a determined, ongoing culture of reproducibility to help overcome these obvious costs.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">At every level of our team, we try to promote a mindset that views research as a process. In this mindset, how a repository is structured and documented is just as important to doing good research as the results that appear in the final publication.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this post, we discuss reproducibility as a part of Pew Research Center&#8217;s code review process.<\/p>\n","protected":false},"author":658,"featured_media":125533,"comment_status":"closed","ping_status":"closed","template":"","meta":{"sub_headline":"","sub_title":"","_prc_public_revisions":[],"_ppp_expiration_hours":0,"_ppp_enabled":false,"ai_generated_summary":"","relatedPosts":[],"_prc_fork_parent":0,"_prc_fork_status":"","_prc_active_fork":0,"datacite_doi":"","datacite_doi_citation":"","_prc_seo_qr_attachment_id":0,"spoken_article_player_enabled":true,"displayBylines":true,"footnotes":"","prc_watchers":[]},"categories":[353],"bylines":[707,656],"collection":[],"_post_visibility":[],"decoded-category":[530],"formats":[454],"_fund_pool":[],"languages":[],"regions-countries":[],"research-teams":[524],"workflow-status":[],"class_list":["post-111113","decoded","type-decoded","status-publish","has-post-thumbnail","hentry","category-data-science","bylines-emma-remy","bylines-gonzalo-rivero","decoded-category-data-science","formats-decoded","research-teams-decoded"],"label":"Decoded","post_parent":0,"word_count":1106,"canonical_url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/2024\/03\/28\/reproducibility-as-part-of-code-quality-control\/","art_direction":{"A1":{"id":125533,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2024\/03\/d_24.03.28_reproducibility_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2024\/03\/d_24.03.28_reproducibility_feature.png?w=564&h=317&crop=1","width":564,"height":317,"chartArt":false,"caption":"Pew Research Center illustration"},"A2":{"id":125535,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2024\/03\/d_24.03.28_reproducibility_crop.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2024\/03\/d_24.03.28_reproducibility_crop.png?w=268&h=151&crop=1","width":268,"height":151,"chartArt":false,"caption":""},"A3":{"id":125535,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2024\/03\/d_24.03.28_reproducibility_crop.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2024\/03\/d_24.03.28_reproducibility_crop.png?w=194&h=110&crop=1","width":194,"height":110,"chartArt":false,"caption":""},"A4":{"id":125535,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2024\/03\/d_24.03.28_reproducibility_crop.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2024\/03\/d_24.03.28_reproducibility_crop.png?w=268&h=151&crop=1","width":268,"height":151,"chartArt":false,"caption":""},"XL":{"id":125533,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2024\/03\/d_24.03.28_reproducibility_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2024\/03\/d_24.03.28_reproducibility_feature.png?w=720&h=405&crop=1","width":720,"height":405,"chartArt":false,"caption":"Pew Research Center illustration"},"social":{"id":125533,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2024\/03\/d_24.03.28_reproducibility_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2024\/03\/d_24.03.28_reproducibility_feature.png?w=1200&h=628&crop=1","width":1200,"height":628,"chartArt":false,"caption":"Pew Research Center illustration"}},"_embeds":[],"watchers":[],"table_of_contents":[],"datacite_doi":"","prc_seo_data":{"title":"Reproducibility as part of code quality control","description":"In this post, we discuss reproducibility as a part of Pew Research Center's code review process.","og_title":"Reproducibility as part of code quality control","og_description":"In this post, we discuss reproducibility as a part of Pew Research Center's code review process.","schema_type":"Article","noindex":false,"canonical_url":"","primary_terms":{"category":1,"research-teams":524},"custom_schema":[],"twitter_description":"In this post, we discuss reproducibility as a part of Pew Research Center's code review process.","og_image":125533,"indexnow_submitted_at":null,"gsc_index_status":null},"prepublish_checks":{},"jetpack_sharing_enabled":true,"relatedPostsOrdered":[],"bylinesOrdered":[{"key":"_i6btrhtig","termId":707},{"key":"_sflr8ihde","termId":656}],"acknowledgementsOrdered":[],"_links":{"self":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111113","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded"}],"about":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/types\/decoded"}],"author":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/users\/658"}],"replies":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/comments?post=111113"}],"version-history":[{"count":3,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111113\/revisions"}],"predecessor-version":[{"id":141435,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111113\/revisions\/141435"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/media\/125533"}],"wp:attachment":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/media?parent=111113"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/categories?post=111113"},{"taxonomy":"bylines","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/bylines?post=111113"},{"taxonomy":"collection","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/collection?post=111113"},{"taxonomy":"_post_visibility","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_post_visibility?post=111113"},{"taxonomy":"decoded-category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded-category?post=111113"},{"taxonomy":"formats","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/formats?post=111113"},{"taxonomy":"_fund_pool","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_fund_pool?post=111113"},{"taxonomy":"languages","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/languages?post=111113"},{"taxonomy":"regions-countries","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/regions-countries?post=111113"},{"taxonomy":"research-teams","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/research-teams?post=111113"},{"taxonomy":"workflow-status","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/workflow-status?post=111113"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}