{"id":111675,"date":"2018-08-28T18:00:00","date_gmt":"2018-08-28T23:00:00","guid":{"rendered":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/\/\/using-supervised-machine-learning-to-quantify-political-rhetoric\/"},"modified":"2024-04-14T04:10:45","modified_gmt":"2024-04-14T09:10:45","slug":"using-supervised-machine-learning-to-quantify-political-rhetoric","status":"publish","type":"decoded","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/2018\/08\/28\/using-supervised-machine-learning-to-quantify-political-rhetoric\/","title":{"rendered":"Using supervised machine learning to quantify political rhetoric"},"content":{"rendered":"\n
\"\"<\/a>
(Illustration by Pew Research Center and Sira Anamwong\/Shutterstock.com)<\/figcaption><\/figure>\n\n\n\n

Several posts<\/a> on this blog have examined<\/a> unsupervised<\/em> methods of natural language processing. These algorithms and models can help researchers explore and analyze large amounts of text. But as those posts point out<\/a>, a supervised<\/em>approach \u2014 one in which researchers train a classification model to make new predictions based on of a set of human-labeled documents \u2014 is often a better way to measure interesting concepts in text.<\/p>\n\n\n\n

A recent Pew Research Center report<\/a> used supervised methods to examine over 700,000 Facebook posts published by members of Congress between Jan. 1, 2015, and Dec. 31, 2017. The project\u2019s goal was to classify whether lawmakers in Congress expressed political support or opposition in those posts, as well as to see how often they discussed local issues.<\/p>\n\n\n\n

In the following discussion, we\u2019ll describe how we used human coders to capture particular kinds of rhetoric in a small sample of congressional Facebook posts and then applied those decisions across the entire set of posts using machine learning. We are also providing a dataset<\/a> that summarizes how often each member of Congress takes sides and \u201cgoes local\u201d on Facebook. (More about the dataset later.)<\/p>\n\n\n\n

Putting together a supervised classification project starts with researchers deciding what they want to measure. In our case, we wanted to capture expressions of political support and opposition directed at key political figures and groups. These included Donald Trump, Barack Obama, Hillary Clinton and Democrats\/liberals or Republicans\/conservatives more generally. We also sought to measure how often politicians focus on local topics by capturing references to any place, group, individual(s) or events in the home state or district of the politician who created the Facebook post.<\/p>\n\n\n\n

Once we decided on these key measures, we put together a coding instrument: a computer interface that showed each Facebook post in our training dataset, some information about when it was posted and who posted it, and a set of questions for human coders to answer. The full interface looked like this:<\/p>\n\n\n\n

\"\"<\/a><\/figure>\n\n\n\n

We used Amazon\u2019s Mechanical Turk<\/a> online outsourcing service to hire coders to complete much of the classification. Overall, Mechanical Turk workers examined 11,000 Facebook posts by members of Congress, and we classified an additional 1,100 ourselves using the same system. (Every post classified via Mechanical Turk was evaluated by five separate Mechanical Turk coders, while every post classified at the Center was evaluated by two of the Center\u2019s researchers.) This human classification project helped us assemble the training dataset: a large sample of posts that had been transformed from words into quantifiable kinds<\/em> of rhetoric.<\/p>\n\n\n\n

When assembling training data, researchers sometimes take a random sample of their entire dataset. But in this case, we wanted to be sure that the posts we selected for human classification contained our key concepts of interest. To do this, we used keyword over-samples to increase the rate at which posts expressing political support or opposition showed up in the training data. For example, we used regular expressions<\/a> like \u201c[Ll]iberal*|[Dd]emocrat*|[Pp]rogressiv*\u201d to select posts that might contain support or opposition to Democrats and\/or liberals, and randomly selected those posts at a higher proportion than they would otherwise appear. Those words appeared in 6% of the Facebook posts overall, but were present in 32% of the posts in the training sample we asked Mechanical Turk workers to classify. Once the coders classified those posts, we used probability weights to adjust for the oversampling in subsequent machine learning and analysis.<\/p>\n\n\n\n

There\u2019s an important question to acknowledge at this point: Did the human coders agree with each other? To find out, we computed Cohen\u2019s Kappa<\/a> \u2014 a common measure of interrater reliability that ranges from 0.0 (no agreement other than by chance) to 1.0 (perfect agreement) \u2014 across all of the items we planned to use in our analysis. Between the two Pew Research Center coders, we calculated values of kappa that ranged between 0.64 and 0.93 across particular items. Next, we collapsed the five Mechanical Turk classification decisions<\/a> for each measure into a single value. We then resolved the differences between the researchers here at the Center and compared our final decisions with those made by the aggregated Mechanical Turk workers. The kappa scores for that comparison ranged from 0.65 to 1.0, showing that, when aggregated, Mechanical Turk workers made coding decisions that were comparable to those made by our in-house experts. The full set of kappa values are shown here<\/a>.<\/p>\n\n\n\n

Once we assembled the training data and checked to ensure that the Center\u2019s coders and the Mechanical Turk coders were on the same page, it was time to train the machine learning models. Our goal was to take information about which words go with which concepts (according to our training data) and apply those lessons across all of the posts that our coders didn\u2019t read and evaluate. When training and evaluating the models, we accounted for the keyword oversampling described above, downweighting cases that were disproportionately more likely to appear in the training data.<\/p>\n\n\n\n

When applying models based on training data, it may be tempting to use the raw text that you want to measure. But there are several pitfalls associated with doing so. Namely, those models can become overfit<\/a> to the training data, picking up on noise instead of signal. To minimize that risk, we cleaned the text using a variety of different strategies, taking particular care to remove a wide range of stopwords<\/a>, including the names of politicians, states, cities, counties, months and other terms that might bias our model predictions toward idiosyncratic features of the training data. This helped protect against the risk that our models might learn to classify posts based on features associated with who<\/em> made the post, instead of what<\/em> they said in the post.<\/p>\n\n\n\n

The machine learning model we used (called XGBoost<\/a>) used a specific set of features <\/em>\u2014 attributes of the text \u2014 to transform the known human decisions into predictions. The specific features we used are described in the methodology section<\/a> of our report. When predicting whether a post contained political support or opposition, we trained the models separately for posts created by Democrats and Republicans, but for local issues we used the entire training set.<\/p>\n\n\n\n

One especially useful aspect of supervised machine learning is that you can directly assess how well the model works and how sensitive it is to variations in your data through a process known as cross-validation. By training the model with a slightly reduced subset of the human coded data \u2014 and holding out a small portion for testing \u2014 we can see whether or not the model accurately characterizes the human coded data that it never saw when it was being trained.<\/p>\n\n\n\n

Using the 11,000 posts coded by Mechanical Turk workers, we split the data into five equally sized portions and trained the model five separate times, each time omitting a different 20% of the data so we could check how well the model did. Not only did this process help determine whether the models worked, but it also helped us determine prediction thresholds that provided us with the greatest balance between precision and recall<\/a>. The full results of the cross validation process are shown here<\/a>. Finally, in addition to cross-validation, we also compared our model\u2019s predictions for the 1,100 posts that Center researchers labeled themselves, to make sure that our models agreed not only with the Mechanical Turk workers, but our own judgments as well.<\/p>\n\n\n\n

\"\"<\/a><\/figure>\n\n\n\n

Once we had our predictions, we decided to step back from the individual posts and take a look at the overall rates at which congressional Facebook posts took sides or went local. To do so, we computed the overall weighted proportions of posts that contained each of our topics, based on the machine learning models as well as the Mechanical Turk coders and in-house researchers, and compared the three different overall estimates. Happily, they all seemed to match closely<\/a>.<\/p>\n\n\n\n

Finally, we used these data to look at patterns in which members of Congress took sides<\/a>, who went local<\/a> and how the Facebook audience reacted<\/a> to different kinds of posts. But our research doesn\u2019t have to be the last word on this. We think other researchers and scholars interested in Congress might have their own ideas about how to use these data, so we\u2019re publishing a dataset that contains our estimates of the rate at which individual members of Congress expressed opposition, expressed support or discussed local topics on Facebook. The dataset covers the full 114th Congress and the first year of the 115th Congress and is available here<\/a>. (You\u2019ll just need to create an account first.)<\/p>\n\n\n\n

Please cite the dataset<\/a> as: van Kessel, Patrick, Adam G. Hughes, and Solomon Messing. 2018. \u201cTaking Sides on Facebook: How Congressional Outreach Changed Under President Trump.\u201d Dataset: Pew Research Center.<\/p>\n","protected":false},"excerpt":{"rendered":"

Several posts on this blog have examined unsupervised methods of natural language processing. These algorithms and models can help\u2026<\/p>\n","protected":false},"author":655,"featured_media":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"sub_headline":"","sub_title":"","_prc_public_revisions":[],"_ppp_expiration_hours":0,"_ppp_enabled":false,"ai_generated_summary":"","relatedPosts":[],"datacite_doi":"","datacite_doi_citation":"","_prc_seo_qr_attachment_id":0,"spoken_article_player_enabled":true,"displayBylines":true,"footnotes":"","prc_watchers":[],"_prc_fork_parent":0,"_prc_fork_status":"","_prc_active_fork":0},"categories":[353],"bylines":[936,927],"collection":[],"_post_visibility":[],"decoded-category":[531,530],"formats":[],"_fund_pool":[],"languages":[],"regions-countries":[],"research-teams":[524],"workflow-status":[],"class_list":["post-111675","decoded","type-decoded","status-publish","hentry","category-data-science","bylines-adam-hughes","bylines-patrick-van-kessel","decoded-category-coding-how-to","decoded-category-data-science","research-teams-decoded"],"label":"Decoded","post_parent":0,"word_count":1505,"canonical_url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/2018\/08\/28\/using-supervised-machine-learning-to-quantify-political-rhetoric\/","art_direction":{"A1":{"id":126051,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.28.2018_featured.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.28.2018_featured.png?w=564&h=317&crop=1","width":564,"height":317,"caption":"","chartArt":false},"A2":{"id":126051,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.28.2018_featured.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.28.2018_featured.png?w=268&h=151&crop=1","width":268,"height":151,"caption":"","chartArt":false},"A3":{"id":126051,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.28.2018_featured.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.28.2018_featured.png?w=194&h=110&crop=1","width":194,"height":110,"caption":"","chartArt":false},"A4":{"id":126051,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.28.2018_featured.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.28.2018_featured.png?w=268&h=151&crop=1","width":268,"height":151,"caption":"","chartArt":false},"XL":{"id":126051,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.28.2018_featured.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.28.2018_featured.png?w=720&h=405&crop=1","width":720,"height":405,"caption":"","chartArt":false},"social":{"id":126051,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.28.2018_featured.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.28.2018_featured.png?w=1200&h=628&crop=1","width":1200,"height":628,"caption":"","chartArt":false}},"_embeds":[],"watchers":[],"table_of_contents":[],"datacite_doi":"","prc_seo_data":{"title":"Using supervised machine learning to quantify political rhetoric","description":"Several posts on this blog have examined unsupervised methods of natural language processing. These algorithms and models can help\u2026","og_title":"Using supervised machine learning to quantify political rhetoric","og_description":"Several posts on this blog have examined unsupervised methods of natural language processing. These algorithms and models can help\u2026","schema_type":"Article","noindex":false,"canonical_url":"","primary_terms":{"category":1},"custom_schema":[],"og_image":126051,"indexnow_submitted_at":null,"gsc_index_status":null},"prepublish_checks":{},"jetpack_sharing_enabled":true,"relatedPostsOrdered":[],"bylinesOrdered":[{"key":"_1m3vemyj8","termId":936},{"key":"_y2z9sz577","termId":927}],"acknowledgementsOrdered":[],"_links":{"self":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111675","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded"}],"about":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/types\/decoded"}],"author":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/users\/655"}],"replies":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/comments?post=111675"}],"version-history":[{"count":2,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111675\/revisions"}],"predecessor-version":[{"id":138553,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111675\/revisions\/138553"}],"wp:attachment":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/media?parent=111675"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/categories?post=111675"},{"taxonomy":"bylines","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/bylines?post=111675"},{"taxonomy":"collection","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/collection?post=111675"},{"taxonomy":"_post_visibility","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_post_visibility?post=111675"},{"taxonomy":"decoded-category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded-category?post=111675"},{"taxonomy":"formats","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/formats?post=111675"},{"taxonomy":"_fund_pool","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_fund_pool?post=111675"},{"taxonomy":"languages","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/languages?post=111675"},{"taxonomy":"regions-countries","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/regions-countries?post=111675"},{"taxonomy":"research-teams","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/research-teams?post=111675"},{"taxonomy":"workflow-status","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/workflow-status?post=111675"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}