{"id":24325,"date":"2021-03-05T09:22:10","date_gmt":"2021-03-05T14:22:10","guid":{"rendered":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/2021\/03\/05\/methodology\/"},"modified":"2025-04-23T23:57:16","modified_gmt":"2025-04-24T03:57:16","slug":"methodology","status":"publish","type":"post","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/2021\/03\/05\/methodology\/","title":{"rendered":"Methodology"},"content":{"rendered":"\n<h3 data-is-section=\"true\" data-wp-context=\"{&quot;id&quot;:&quot;the-american-trends-panel-survey-methodology&quot;}\" data-wp-interactive=\"{&quot;namespace&quot;:&quot;prc-block\\\/table-of-contents&quot;}\" class=\"wp-block-heading\" id=\"the-american-trends-panel-survey-methodology\">The American Trends Panel survey methodology<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The American Trends Panel (ATP), created by Pew Research Center, is a nationally representative panel of randomly selected U.S. adults. Panelists participate via self-administered web surveys. Panelists who do not have internet access at home are provided with a tablet and wireless internet connection. Interviews are conducted in both English and Spanish. The panel is being managed by Ipsos.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Data in this report is drawn from the panel wave conducted Aug. 31 to Sept. 7, 2020. A total of 9,220 panelists responded out of 9,810 who were sampled, for a response rate of 94%. This does not include two panelists who were removed from the data due to extremely high rates of refusal or straightlining. The cumulative response rate accounting for nonresponse to the recruitment surveys and attrition is 5%. The break-off rate among panelists who logged on to the survey and completed at least one item is 1%. The margin of sampling error for the full sample of 9,220 respondents is plus or minus 1.7 percentage points. The survey is weighted to be representative of the U.S. adult population by gender, race, ethnicity, partisan affiliation, education and other categories.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">More information about this wave of the American Trends Panel can be found&nbsp;<a href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/journalism\/2020\/10\/07\/following-covid-19-news-methodology\/\">here<\/a>.&nbsp;<\/p>\n\n\n\n<h3 data-is-section=\"true\" data-wp-context=\"{&quot;id&quot;:&quot;content-analysis&quot;}\" data-wp-interactive=\"{&quot;namespace&quot;:&quot;prc-block\\\/table-of-contents&quot;}\" class=\"wp-block-heading\" id=\"content-analysis\">Content analysis<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Researchers conducted their analysis on a collection of 9,220 raw responses to the following open-ended survey questions:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em>Before we finish the survey, we\u2019re interested in getting a deeper understanding of how people have been experiencing the coronavirus outbreak, and how it might have impacted them in both negative and positive ways. Please take a moment to reflect on how the outbreak has affected your own life, and then answer the next two questions in as much detail as you can.<\/em><\/p>\n\n\n\n<ol class=\"wp-block-list\" type=\"1\"><li><em>Thinking about how things have been going in your life since the beginning of the coronavirus outbreak, in what ways, if any, have things been difficult or challenging for you personally?<\/em><\/li><li><em>Thinking about how things have been going in your life since the beginning of the coronavirus outbreak, what, if anything, have been some positive aspects of the situation for you personally?<\/em><\/li><\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">The two questions were asked in a random order. Overall, 84% of respondents (8,143) provided an answer to one or both of the questions (the 16% of respondents that skipped both questions are not included in this analysis). From these responses, researchers drew a half-sample of 4,071 responses to assess and analyze their content. Responses that were written in Spanish \u2013 3% of the sample \u2013 were machine translated and examined by researchers to confirm their interpretability.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">After drawing exploratory samples to identify common themes in the responses, researchers developed a codebook to measure the different types of positive and negative changes. Coders were shown respondents\u2019 answers to both the positive and negative questions side-by-side, prefixed by \u201cPOSITIVE:\u201d and \u201cNEGATIVE:\u201d labels to indicate the prompt that elicited each response. Coders were instructed to first indicate whether each response mentioned any positive and\/or negative changes, and then to mark those that fell into any of six different categories: relationships; free time or activities; work situation; economics; physical or mental health; and society, politics, safety precautions or rules. Additional instructions and rules were provided to help coders make their decisions. Two in-house researchers iteratively drew and coded samples and expanded the codebook rules until acceptable inter-rater reliability was achieved.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"codebook\">Codebook<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\"><em>We asked two questions on a recent survey, one about negative impacts of the pandemic, and another about unexpectedly positive effects. The \u2018POSITIVE\u2019 and \u2018NEGATIVE\u2019 labels below indicate how a respondent answered each question, but they may have mentioned positive things when answering the negative question, and vice versa. Please use the labels as a guide \u2013 if they don\u2019t provide additional context for their response, you can assume that it\u2019s answering the question that was asked.&nbsp;<strong>If they give more details, use that to help make your decision.<\/strong><\/em><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><em>\u201cNEGATIVE: more time at home: \u2013 assume this is a negative thing<\/em><\/li><li><em>\u201cNEGATIVE: work has been challenging, but I have more time at home\u201d \u2013 they mention work as a negative thing, but time at home is a positive thing<\/em><\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><em>Only mark something if it is a&nbsp;<strong>change or difference<\/strong>&nbsp;in their life; if they mention something that has stayed the same (\u201cMy family has stayed healthy,\u201d \u201cI\u2019m still broke and unemployed\u201d) do not count it.<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em>NOTE: the POSITIVE and NEGATIVE prompt markers in the responses are guides. Just because respondents wrote something, doesn\u2019t mean what they wrote matched the prompt \u2013 read the responses carefully when answering this. Any positive changes mentioned in either prompt counts as specifics, even if they aren\u2019t captured by any of the checkbox categories below.<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><br><strong>POSITIVE CHANGES: Does this response mention any specific positive changes during the pandemic?<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">RELATIONSHIPS: quality\/frequency of social interactions (themselves)<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Spending more time with family\/friends\/neighbors<\/li><li>Growing closer to others\/developing better relationships<\/li><li>Enjoying not having to socialize as much<\/li><li>DO NOT COUNT church unless a social aspect is mentioned<\/li><li>DO NOT COUNT spending time with pets<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">FREE TIME\/ACTIVITIES: Free time, activities or hobbies<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Having more free time\/more time to do things around the house<\/li><li>Being able to do more things\/spending more time doing things at home<\/li><li>Picking up new activities or hobbies<\/li><li>Learning new things<\/li><li>Less time spent on unfulfilling activities like driving (WORK SITUATION)<\/li><li>Slowing down\/having a slower pace of life\/being able to relax more (PHYSICAL\/MENTAL HEALTH)<\/li><li>Enjoying spending (more) time at home<\/li><li>Having more time for personal reflection (PHYSICAL\/MENTAL HEALTH)<\/li><li>DO NOT COUNT socializing or seeing\/visiting people unless it mentions a specific activity like \u2018going out\u2019 or \u2018traveling\u2019<\/li><li>USE THE PROMPTS AS CONTEXT: \u2018more time at home\u2019 under the POSITIVE prompt would count, but under the NEGATIVE one it would not<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">WORK SITUATION: quality or availability of job\/work\/career (themselves or people they know)<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Being able to work from home<\/li><li>Having an improved work situation<\/li><li>Having a better work-life balance<\/li><li>Finding a new job<\/li><li>Not having to go into the office<\/li><li>Not having a commute (FREE TIME\/ACTIVITIES)<\/li><li>Not having to work as much<\/li><li>Success maintaining their business<\/li><li>Used the situation as an opportunity to retire or do something different<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">ECONOMICS: Finances, money or economics (in any sense)<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Saving more money\/spending less money<\/li><li>Receiving stimulus or unemployment funds\/having an increase in income<\/li><li>Feeling like the economy is doing well<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">PHYSICAL\/MENTAL HEALTH: mental or physical health outcomes, conditions or behaviors (themselves or people they know)<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Slowing down\/having a slower pace of life\/being able to relax more (FREE TIME\/ ACTIVITIES)<\/li><li>Feeling happier or more relaxed\/less stressed\/more content<\/li><li>Having more time for personal reflection (FREE TIME\/ACTIVITIES)<\/li><li>Improved spiritual life\/greater spirituality<\/li><li>Greater appreciation for things that are important\/having greater clarity<\/li><li>Feeling a renewed sense of gratitude for what they have<\/li><li>Paying more attention to their health\/being in better health\/losing weight<\/li><li>Doing healthy activities like exercising, walking, hiking, eating better, getting more sleep<\/li><li>Doing fewer unhealthy activities\/drinking less<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">SOCIETY\/POLITICS\/SAFETY PRECAUTIONS\/RULES: views on politics, social conditions, safety precautions like wearing masks, and\/or rules and regulations<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Approves of government response to the pandemic\/explicitly expresses support for safety precautions\/rules\/regulations\/restrictions<\/li><li>Expresses satisfaction with or support for certain political groups\/views<\/li><li>Positive developments in society like: people practicing better hygiene, following safety precautions, less pollution,&nbsp;less traffic, places being less crowded, positive environmental developments<\/li><li>Expresses satisfaction with increased awareness of social\/political issues\/the \u2018truth\u2019 being exposed<\/li><li>DO NOT COUNT simple mentions of following safety precautions\/rules; they need to clearly mention it as a positive thing that they support<\/li><li>DO NOT COUNT appreciating essential workers<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><br><strong>NEGATIVE CHANGES: Does this response mention any specific negative changes during the pandemic?<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">RELATIONSHIPS: quality\/frequency of social interactions<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Not being able to socialize, see or visit people\/diminished social life\/fewer social interactions<\/li><li>Deteriorating relationships\/increased conflict<\/li><li>Lack of physical contact with others<\/li><li>The death of family members or friends (PHYSICAL\/MENTAL HEALTH)<\/li><li>The specific feelings of isolation or loneliness (PHYSICAL\/MENTAL HEALTH)<\/li><li>DO NOT COUNT church unless a social aspect is mentioned<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">FREE TIME\/ACTIVITIES: Free time, activities or hobbies<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Not being able to do things they used to do or were planning on doing (e.g. shopping, church, travel, events, vacations, appointments)<\/li><li>Feeling pent up\/having cabin fever\/being stuck at home<\/li><li>Having too much free time\/feeling bored<\/li><li>Not having enough free time<\/li><li>Not being able to go to church\/missing church<\/li><li>DO NOT COUNT not being able to socialize or see\/visit people unless it mentions a specific activity like \u2018going out\u2019 or \u2018traveling\u2019<\/li><li>USE THE PROMPTS AS CONTEXT: \u2018more time at home\u2019 under the NEGATIVE prompt would count, but under the POSITIVE one it would not<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">WORK SITUATION: quality or availability of job\/work\/career (themselves or people they know)<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Having to work from home<\/li><li>Job insecurity\/feeling uncertain about their employment<\/li><li>Losing their job\/complaining about not having a job\/not enough work\/having their hours or pay cut\/having less work available\/getting furloughed (ECONOMIC)<\/li><li>Having to go to work (being an essential worker)<\/li><li>More stress at or about work\/feeling unsafe at work<\/li><li>Being forced to retire<\/li><li>DO NOT COUNT general unemployment or generic references to people losing their jobs or being out of work (that would be ECONOMIC instead)<\/li><li>DO NOT COUNT not having enough money unless it mentions a pay reduction or loss of income<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">ECONOMICS: Finances, money or economics (in any sense)<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Losing their job\/complaining about not having a job\/not enough work\/having their hours or pay cut\/having less work available\/getting furloughed (WORK SITUATION)<\/li><li>General unemployment or generic references to people losing their jobs or being out of work<\/li><li>Negative impacts on the economy or small businesses\/business closing\/general poor health of the economy<\/li><li>Spending more money<\/li><li>Having less income\/paycheck reductions<\/li><li>Running out of money<\/li><li>More financial uncertainty<\/li><li>Product shortages or scarcity<\/li><li>DO NOT COUNT references to \u2018job insecurity\u2019 or other work-related issues unless there is a clear and obvious economic impact (i.e. job loss or one of the other specific topics mentioned above)<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">PHYSICAL\/MENTAL HEALTH: mental or physical health outcomes, conditions or behaviors (themselves or people they know)<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Not being able to get healthcare\/avoiding healthcare or doctors<\/li><li>Being in worse health than they were before\/gaining weight<\/li><li>Doing unhealthy activities like drinking more<\/li><li>Doing fewer healthy activities like exercising, walking or hiking<\/li><li>Eating poorly\/gaining weight<\/li><li>Not being able to exercise or go to the gym<\/li><li>The death of family members or friends (RELATIONSHIPS)<\/li><li>Clearly expresses concern or worry about being at-risk for COVID or getting COVID\/worried about pre-existing condition<\/li><li>Not feeling safe<\/li><li>Someone specific (themselves or someone they know) being or getting sick or dying, COVID or otherwise<\/li><li>The specific feelings of isolation or loneliness (RELATIONSHIPS)<\/li><li>Feeling mentally taxed\/feeling like they can\u2019t enjoy life<\/li><li>Describes feeling stressed, depressed, afraid, angry, sad, frustrated or worried<\/li><li>DO NOT COUNT statements that sound angry or depressed etc; they need to mention a mental\/emotional state or feeling a certain way emotionally<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">DO NOT COUNT statements that describe careful behavior (e.g. \u2018we are being cautious\u2019) or simple mentions of pre-existing conditions (e.g. \u2018we are at risk\u2019) unless they specifically mention being worried, anxious, afraid, etc.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">SOCIETY\/POLITICS\/SAFETY PRECAUTIONS\/RULES: views on politics, social conditions, safety precautions like wearing masks, and\/or rules and regulations<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Feeling like they\u2019re being lied too\/not trusting the \u2018truth\u2019<\/li><li>Expresses disapproval of certain political groups\/views<\/li><li>Expresses frustration with or skepticism of&nbsp;safety precautions\/masks\/rules\/regulations\/restrictions\/lockdowns\/curfews<\/li><li>Disapproves of government response to the pandemic<\/li><li>Negative developments in society like people not following safety precautions or not taking the virus seriously, more partisan conflict, people being rude<\/li><li>Negative views of certain public figures or institutions like politicians, the media, or scientists<\/li><li>Feeling stigmatized or judged for their views<\/li><li>Being sequestered\/being on mandatory lockdown<\/li><li>Thinking the virus is overblown and that things should get back to normal<\/li><li>INCLUDES mentions of safety precautions like masks and social distancing or mention of rules\/restrictions\/lockdowns\/curfews if they appear under the NEGATIVE prompt, unless additional context suggests otherwise<\/li><\/ul>\n\n\n\n<ul class=\"wp-block-list\"><li>DO NOT COUNT generic reference to \u2018quarantine\u2019 if they\u2019re just using it to refer to the general pandemic situation<br><\/li><li>Final inter-rater reliability between the two in-house coders, as measured by Cohen\u2019s Kappa, is reported below for a final sample of 200 responses. The frequency of each category and the size of the sample was sufficient to determine with 95% confidence that the true Kappa values for each category are at least 0.7.<\/li><\/ul>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"alignright size-large is-resized\"><a rel=\"attachment wp-att-31715\" href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/2021\/03\/05\/methodology\/pdl_03-05-21_covid-methodology-0-png\/\"><img data-dominant-color=\"ebebeb\" data-has-transparency=\"false\" style=\"--dominant-color: #ebebeb;\" decoding=\"async\" src=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-0.png?w=620\" alt=\"\" class=\"wp-image-31715 not-transparent\" width=\"310\" srcset=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-0.png 620w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-0.png?resize=300,287 300w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-0.png?resize=423,405 423w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-0.png?resize=200,192 200w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-0.png?resize=260,249 260w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-0.png?resize=310,297 310w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-0.png?resize=420,402 420w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-0.png?resize=160,153 160w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-0.png?resize=320,307 320w\" sizes=\"(max-width: 620px) 100vw, 620px\" \/><\/a><\/figure><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">Final inter-rater reliability between the two in-house coders, as measured by Cohen\u2019s Kappa, is reported below for a final sample of 200 responses. The frequency of each category and the size of the sample was sufficient to determine with 95% confidence that the true Kappa values for each category are at least 0.7.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Before conducting the full content analysis, researchers separated the responses into two groups: those that contained potentially sensitive personal details, and those that did not. In-house researchers then coded 360 responses that did contain such information, and used Amazon\u2019s Mechanical Turk service to code the remaining 3,711 that did not. Three Mechanical Turk workers coded each response, and their decisions were collapsed using a two-out-of-three consensus threshold to produce coding decisions comparable to those of our own in-house researchers.<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large is-resized\"><a rel=\"attachment wp-att-31722\" href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/2021\/03\/05\/methodology\/pdl_03-05-21_covid-methodology-1-png\/\"><img data-dominant-color=\"e9e9e9\" data-has-transparency=\"false\" style=\"--dominant-color: #e9e9e9;\" loading=\"lazy\" decoding=\"async\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" height=\"493\" src=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-1.png?w=1024\" alt=\"\" class=\"wp-image-31722 not-transparent\" width=\"1024\" srcset=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-1.png 1288w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-1.png?resize=300,144 300w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-1.png?resize=768,370 768w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-1.png?resize=1024,493 1024w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-1.png?resize=841,405 841w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-1.png?resize=200,96 200w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-1.png?resize=260,125 260w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-1.png?resize=310,149 310w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-1.png?resize=420,202 420w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-1.png?resize=640,308 640w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-1.png?resize=740,356 740w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-1.png?resize=160,77 160w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-1.png?resize=320,154 320w\" \/><\/a><\/figure><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">To assess this inter-rater reliability, in-house researchers coded a subset of the responses that were sent to Mechanical Turk. A review of initial results showed promising reliability but also indicated that Mechanical Turk workers were often missing some of the more nuanced coding rules. These mistakes were consistent and obvious; for example, Mechanical Turk workers were successfully identifying negative changes related to physical health, but appeared to be commonly missing responses that described negative&nbsp;<em>mental<\/em>&nbsp;health changes such as depression and stress. After noticing these patterns, researchers were able to develop a handful of pattern-matching rules to make corrections to the Mechanical Turk workers\u2019 decisions based on whether specific keywords (e.g., \u201cdepress\u201d or \u201cstress\u201d) were mentioned in the positive or negative section of each response\u2019s text.<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"alignright size-large is-resized\"><a rel=\"attachment wp-att-31730\" href=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/2021\/03\/05\/methodology\/pdl_03-05-21_covid-methodology-2-png\/\"><img data-dominant-color=\"ececec\" data-has-transparency=\"false\" style=\"--dominant-color: #ececec;\" decoding=\"async\" src=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-2.png?w=840\" alt=\"\" class=\"wp-image-31730 not-transparent\" width=\"420\" srcset=\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-2.png 840w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-2.png?resize=300,241 300w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-2.png?resize=768,618 768w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-2.png?resize=503,405 503w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-2.png?resize=200,161 200w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-2.png?resize=260,209 260w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-2.png?resize=310,249 310w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-2.png?resize=420,338 420w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-2.png?resize=640,515 640w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-2.png?resize=740,596 740w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-2.png?resize=160,129 160w, https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/03\/PDL_03.05.21_covid.methodology-2.png?resize=320,258 320w\" sizes=\"(max-width: 840px) 100vw, 840px\" \/><\/a><\/figure><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">After coding a final expanded sample of 400 responses, researchers determined that this combined Mechanical Turk and pattern-matching approach produced results comparable to our own in-house decisions, with the exception of positive changes related to society, politics, safety precautions and rules.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For the final analysis, researchers combined the 360 responses that were coded in-house with the 3,711 responses that were coded by Mechanical Turk workers, for a final sample of 4,071 responses. Fully 95% of these responses mentioned at least one positive or negative change; 67% mentioned at least one of each, 22% mentioned one or more negative changes but no positive changes, and 5% mentioned one or more positive changes but no negative ones.<\/p>\n\n\n\n<h3 data-is-section=\"true\" data-wp-context=\"{&quot;id&quot;:&quot;demographic-differences&quot;}\" data-wp-interactive=\"{&quot;namespace&quot;:&quot;prc-block\\\/table-of-contents&quot;}\" id=\"demographic-differences\" class=\"wp-block-heading\">Demographic differences<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">All reported demographic differences were confirmed to be statistically significant with 95% confidence using logistic regressions that isolated the relationships between demographic factors and a respondent\u2019s likelihood of mentioning a positive or negative change in each category. The regression models included variables for race\/ethnicity, political party identification, education, age, urbanicity, parenthood, income, gender, geographic region and language (English vs. Spanish), and also controlled for the overall length of each responses using log-scaled word counts.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The American Trends Panel survey methodology The American Trends Panel (ATP), created by Pew Research Center, is a nationally representative panel of randomly selected U.S. adults. Panelists participate via self-administered web surveys. Panelists who do not have internet access at home are provided with a tablet and wireless internet connection. Interviews are conducted in both [&hellip;]<\/p>\n","protected":false},"author":367,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_crdt_document":"","sub_headline":"","sub_title":"","_prc_public_revisions":[],"_ppp_expiration_hours":0,"_ppp_enabled":false,"ai_generated_summary":"","bylines":[],"acknowledgements":[],"displayBylines":true,"relatedPosts":[],"reportMaterials":[],"multiSectionReport":[],"package_parts__enabled":false,"package_parts":[],"_prc_fork_parent":0,"_prc_fork_status":"","_prc_active_fork":0,"datacite_doi":"","datacite_doi_citation":"","_prc_seo_qr_attachment_id":0,"spoken_article_player_enabled":true,"footnotes":""},"categories":[],"tags":[],"bylines":[],"collection":[],"datasets":[],"level_of_effort":[],"primary_audience":[],"information_type":[],"_post_visibility":[2643],"formats":[458],"_fund_pool":[],"languages":[],"regions-countries":[515],"research-teams":[],"class_list":["post-24325","post","type-post","status-publish","format-standard","hentry","_post_visibility-hidden-on-index","formats-report","regions-countries-united-states"],"label":false,"post_parent":0,"word_count":2483,"canonical_url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/2021\/03\/05\/methodology\/","art_direction":false,"_embeds":[],"table_of_contents":[],"report_materials":"","report_pagination":{"current_post":null,"next_post":null,"previous_post":null,"pagination_items":[]},"parent_info":{"parent_title":"Methodology","parent_id":24325},"materialsOrdered":[],"chaptersOrdered":[],"partsOrdered":[],"partsEnabled":false,"datacite_doi":"","prc_seo_data":{"title":"Methodology","description":"The American Trends Panel survey methodology The American Trends Panel (ATP), created by Pew Research Center, is a nationally representative panel of randomly selected U.S. adults. Panelists participate via self-administered&hellip;","og_title":"Methodology","og_description":"The American Trends Panel survey methodology The American Trends Panel (ATP), created by Pew Research Center, is a nationally representative panel of randomly selected U.S. adults. Panelists participate via self-administered&hellip;","schema_type":"Article","noindex":false,"canonical_url":"","primary_terms":[],"custom_schema":[],"og_image":0,"indexnow_submitted_at":null,"gsc_index_status":null},"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"relatedPostsOrdered":[],"bylinesOrdered":[],"acknowledgementsOrdered":[],"_links":{"self":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/posts\/24325","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/users\/367"}],"replies":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/comments?post=24325"}],"version-history":[{"count":2,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/posts\/24325\/revisions"}],"predecessor-version":[{"id":224171,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/posts\/24325\/revisions\/224171"}],"wp:attachment":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/media?parent=24325"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/categories?post=24325"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/tags?post=24325"},{"taxonomy":"bylines","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/bylines?post=24325"},{"taxonomy":"collection","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/collection?post=24325"},{"taxonomy":"datasets","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/datasets?post=24325"},{"taxonomy":"level_of_effort","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/level_of_effort?post=24325"},{"taxonomy":"primary_audience","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/primary_audience?post=24325"},{"taxonomy":"information_type","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/information_type?post=24325"},{"taxonomy":"_post_visibility","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_post_visibility?post=24325"},{"taxonomy":"formats","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/formats?post=24325"},{"taxonomy":"_fund_pool","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_fund_pool?post=24325"},{"taxonomy":"languages","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/languages?post=24325"},{"taxonomy":"regions-countries","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/regions-countries?post=24325"},{"taxonomy":"research-teams","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/research-teams?post=24325"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}