Researchers also searched for tweets about the presidential candidates\u2019 running mates, Sen. Kamala Harris and Vice President Mike Pence. In the preelection dataset, the only tweet that mentioned Pence also referenced Donald Trump and was not analyzed separately. Eleven tweets mentioned Kamala Harris but not Joe Biden. Because of this low volume for tweets mentioning Pence and Harris, the preelection set of tweets focuses only on mentions of the presidential candidates. In the post-election dataset, we again saw few mentions of Pence independent from mentions of Trump. Tweets mentioning Harris without Biden, such as mentions of her historic accession to the vice presidency, are included in the post-election analysis. These account for 370 tweets, or 14%, of post-election tweets that mention Biden or Harris.<\/p>\n\n\n\n
To evaluate the performance of the regular expression, researchers took a random sample of 100 tweets about Trump and 100 tweets about Biden created between Aug. 28 and Sept. 30, 2020, and an additional two groups of 100 flagged as not mentioning Trump or Biden from the same time period. A researcher examined these sets and independently determined whether they mentioned the presidential candidates in order to compare human decisions with the results from the regular expression. Overall, the human decisions agreed with the keyword method 98% of the time. Cohen\u2019s Kappa \u2013 a statistic that examines agreement while adjusting for chance agreement \u2013 was 0.96 on average between Trump and Biden mentions for the same comparison. Another researcher also classified the set of 200 tweets for each candidate independently to ensure their decisions were comparable. Cohen\u2019s Kappa for coder-to-code comparisons was 0.98 for mentions of Biden and 1 (perfect agreement) for Trump.<\/p>\n\n\n\n
Coding of legislators\u2019 pre-election tweets<\/h3>\n\n\n\nSentiment coding for tweets about presidential candidates before the election<\/h4>\n\n\n\n
Using the validated tweets about presidential candidates, researchers hand-coded the sentiment of each tweet into one of three categories: positive, neutral or negative.<\/p>\n\n\n\n
Tweets with a positive mention:<\/p>\n\n\n\n
- Include any tweets in which the writer explicitly says the above parties are doing a good job, showing leadership, etc.<\/li>
- Also include general expressions of thanks or gratitude or wishing them luck in the upcoming election.<\/li><\/ul>\n\n\n\n
Tweets with a negative mention:<\/p>\n\n\n\n
- Include any tweets in which the writer explicitly says the above parties are doing a bad job, failing at key tasks or at their jobs more broadly or letting Americans down.<\/li>
- Also include general expressions of shame or embarrassment, or hoping that they will lose in the upcoming election.<\/li>
- Or refer to the candidates in disparaging terms, such as the phrases \u201cSleepy Joe\u201d or calling something negative \u201cTrumpian.\u201d<\/li><\/ul>\n\n\n\n
Tweets that are neutral:<\/p>\n\n\n\n
- Include any tweets that do not express feelings toward any candidates on either party\u2019s presidential ticket or are strictly descriptive in nature regarding the candidate or related polling and news. For example, \u201cLooking forward to joining the panel on the eve of the first Biden-Trump presidential debate. https:\/\/t.co\/LkQo18EgyE<\/a>\u201d is considered a neutral tweet toward both candidates.<\/li><\/ul>\n\n\n\n
Researchers coded tweets about Trump and tweets about Biden separately in order to accurately classify tweets that mention both candidates. In the majority of cases, researchers were able to determine a tweet\u2019s sentiment based solely from the text of the tweet. In some cases, such as links to political cartoons, researchers followed links or images for additional context before making a decision. Through this process, researchers removed 10 false positives. It is important to note that this method may fail to correctly classify tweets that use irony or sarcasm to convey feelings or ideas about the candidates.<\/p>\n\n\n\n
Identifying subtopics in the preelection sentiment coding<\/h4>\n\n\n\n
![\"Categories](\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/01\/PG_2021.01.13_Global-Legislators_M-01.png?w=310\")
<\/a><\/figure><\/div>\n\n\n\nIn addition to classifying the sentiment of these tweets, researchers also coded them into subtopics. Researchers manually examined this sample of tweets and retweets to assess which topics were prevalent when discussing the candidates. Based on manual examination, researchers generated a list of coding categories and separately coded the tweets into agreed-upon categories. The tweets were analyzed in two batches \u2013 those about Biden and those about Trump \u2013 and coded as being about that candidate specifically using the same criteria. Tweets that mentioned both Trump and Biden were coded independently in each set.<\/p>\n\n\n\n
Five researchers then separately coded 200 tweets and achieved high levels of intercoder reliability on the subset that they all reviewed (average Krippendorf\u2019s alpha between the Biden and Trump set ranging from 0.76 to 0.87 for each topic) before each coding part of the remaining sample.<\/p>\n\n\n\n
Researchers coded the tweets primarily by reading the text of each tweet. In some cases, when a tweet contained attachments such as links or images, they were considered for additional context before making a decision. It is important to note that this method may fail to correctly classify tweets that use irony or sarcasm to convey feelings or ideas about the U.S. In other words, this analysis captures only the specific subset of the larger conversation on Twitter among legislators that involves overt or explicit discussion of, or engagement with, these topics.<\/p>\n\n\n\n
Identifying tweets about the United States<\/h4>\n\n\n\n
To identify tweets about the United States broadly, researchers used a case-insensitive regular expression that consists of the terms: \u201cUnited States,\u201d \u201cunitedstates,\u201d \u201cUS,\u201d \u201cUSA,\u201d \u201cU.S.,\u201d etc. This pattern identified 1,905 tweets as mentioning the U.S. explicitly by name. Researchers also used negative lookahead<\/a> to exclude mentions of places contain the word America but do not refer to the country, e.g., Latin America. After accounting for false positives, researchers analyzed 1,794 tweets about the United States in the final dataset.<\/p>\n\n\n\n
To evaluate the performance of the regular expression, researchers took a random sample of 400 tweets created between Aug. 28 and Sept. 30, 2020. Two researchers examined this set to determine whether they mentioned the U.S. by name in order to compare human decisions with the decisions from the regular expression. Overall, the human decisions agreed with the keyword method 96% of the time. Cohen\u2019s Kappa was 0.90 for the same comparison. Another researcher also classified the set of 400 tweets independently to ensure their decisions were comparable. Cohen\u2019s Kappa for coder-to-code comparisons was 0.96.<\/p>\n\n\n\n
Topic coding for tweets related to the U.S.<\/h4>\n\n\n\n
![\"Categories](\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/01\/PG_2021.01.13_Global-Legislators_M-02.png?w=310\")
<\/a><\/figure><\/div>\n\n\n\nResearchers limited their analysis of tweets that explicitly mentioned the U.S. to those created from Aug. 28 through Sept. 30, 2020. Researchers manually examined this sample of tweets and retweets to assess which topics were prevalent when discussing these topics. Based on this manual examination, researchers generated a list of coding categories and separately coded the tweets into agreed-upon categories.<\/p>\n\n\n\n
Four researchers then separately coded 100 tweets and achieved high levels of intercoder reliability on the subset that they all reviewed (Krippendorf\u2019s alpha ranging from 0.75 to 0.96 for each topic) before each coding part of the remaining sample. For 111 tweets, researchers found false positives or ambiguous content and dropped these from the analysis. For example, the \u201cUS\u201d keyword falsely tagged tweets containing currency symbols (e.g., \u201cUS$\u201d) that are not actually about the U.S.<\/p>\n\n\n\n
Researchers coded the tweets mainly through reading the text of each tweet. In some cases, when a tweet\u2019s topic could not be discerned from the text alone, researchers used links or images in the tweet for additional context before making a decision. It is important to note that this method may fail to correctly classify tweets that use irony or sarcasm to convey feelings or ideas about the U.S. In other words, this analysis captures only the specific subset of the larger conversation on Twitter among legislators that involves overt or explicit discussion of, or engagement with, these topics.<\/p>\n\n\n\n
For more information on the Center\u2019s global legislators database research, see \u201cFor Global Legislators on Twitter, an Engaged Minority Creates Outsize Share of Content\u201d<\/a>and its methodology page here<\/a>.<\/p>\n\n\n\n
Coding of legislators\u2019 post-election tweets<\/h3>\n\n\n\n
Sentiment coding for tweets about presidential candidates after the election<\/h4>\n\n\n\n
Using the validated tweets about presidential candidates and their running mates sent from Nov. 7 to 14, researchers hand-coded the sentiment of each tweet into one of three categories: positive, neutral or negative.<\/p>\n\n\n\n
Tweets with a positive mention:<\/p>\n\n\n\n
- Include any tweets in which the writer explicitly says the above parties are doing a good job, showing leadership, etc.<\/li>
- Also include general expressions of thanks, gratitude or satisfaction with respect to the election outcome.<\/li><\/ul>\n\n\n\n
Tweets with a negative mention:<\/p>\n\n\n\n
- Include any tweets in which the writer explicitly says the above parties are doing a bad job, failing at key tasks or at their jobs more broadly or letting Americans down.<\/li>
- Also include general expressions of shame, embarrassment or dissatisfaction with the results of the election. Expressions of satisfaction or gratitude toward certain candidate\u2019s defeat in the election also belong in this category.<\/li>
- Or refer to the candidates in disparaging terms, such as the phrases \u201cSleepy Joe\u201d or calling something negative \u201cTrumpian.\u201d<\/li><\/ul>\n\n\n\n
Tweets that are neutral:<\/p>\n\n\n\n
- Include any tweets that do not express feelings toward any candidate on either party\u2019s presidential ticket or are strictly descriptive in nature regarding the candidate, election or related polling and news.<\/li><\/ul>\n\n\n\n
Researchers coded tweets about Trump and tweets about Biden separately in order to accurately classify tweets that mention both candidates. In the majority of cases, researchers were able to determine a tweet\u2019s sentiment based solely on the text of the tweet. In some cases, such as links to political cartoons, researchers followed links or images for additional context before making a decision. Through this process, researchers removed 10 false positives. It is important to note that this method may fail to correctly classify tweets that use irony or sarcasm to convey feelings or ideas about the candidates.<\/p>\n\n\n\n
Identifying subtopics in the post-election sentiment coding<\/h4>\n\n\n\n
In addition to classifying the sentiment of these tweets, researchers also coded them into subtopics. Researchers manually examined this sample of tweets and retweets to assess which topics were prevalent when discussing the candidates. Based on manual examination, researchers generated a list of coding categories and separately coded the tweets into agreed-upon categories. The tweets were analyzed in two batches \u2013 those about Biden\/Harris and those about Trump\/Pence \u2013 and coded about that candidate pair specifically using the same criteria. Tweets that mentioned both Trump and Biden were coded independently in each set.<\/p>\n\n\n\n
- Include any tweets that do not express feelings toward any candidate on either party\u2019s presidential ticket or are strictly descriptive in nature regarding the candidate, election or related polling and news.<\/li><\/ul>\n\n\n\n
- Include any tweets that do not express feelings toward any candidates on either party\u2019s presidential ticket or are strictly descriptive in nature regarding the candidate or related polling and news. For example, \u201cLooking forward to joining the panel on the eve of the first Biden-Trump presidential debate. https:\/\/t.co\/LkQo18EgyE<\/a>\u201d is considered a neutral tweet toward both candidates.<\/li><\/ul>\n\n\n\n
![\"Categories](\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2021\/01\/PG_2021.01.13_Global-Legislators_M-03.png?w=310\")
<\/a><\/figure><\/div>\n\n\n\n