{"id":111670,"date":"2018-09-05T16:00:00","date_gmt":"2018-09-05T21:00:00","guid":{"rendered":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/\/\/__trashed\/"},"modified":"2024-04-14T04:10:42","modified_gmt":"2024-04-14T09:10:42","slug":"__trashed","status":"publish","type":"decoded","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/2018\/09\/05\/__trashed\/","title":{"rendered":"Q&A: What works and what doesn\u2019t when studying social media data"},"content":{"rendered":"\n
\"\"<\/a>
(Pew Research Center photo illustration by Abby Budiman)<\/figcaption><\/figure>\n\n\n\n

Ever since Friendster and Myspace gained popularity in the early 2000s, social scientists have been interested in studying the impact of social media sites. More than a decade later, the use of social media platforms is widespread in U.S. society<\/a> \u2014 and so is research into the way Americans use these platforms.<\/p>\n\n\n\n

Pew Research Center<\/a> has enjoyed being part of this explosion of new research. We\u2019ve produced a number of reports using data science tools, including analyses about how members of Congress communicate with their constituents on Facebook<\/a>, how much of a role bots play on Twitter<\/a> and how Twitter users shared information about immigration<\/a> during the first month of the Trump presidency.<\/p>\n\n\n\n

These research projects have taught us a great deal about the benefits of studying data from social media, as well as the challenges. In the following Q&A, some of the Center\u2019s data science researchers share their lessons learned from our social media data research. The answers have been edited for space and clarity.<\/p>\n\n\n\n

What can the study of social media add to traditional surveys?<\/h4>\n\n\n\n

Dennis Quinn<\/strong>, data science analyst<\/em>: With polling, researchers can exercise precise control over the prompts their respondents are offered, and there is a vast body of research underlying survey methods. But the world is really big, and there\u2019s room for a lot of information. Social media can help fill in some gaps. Social media data are organic: People just made them up. This can make it hard to distinguish real meaning \u2014 signal \u2014 from noise. But if you do it right, it can tell you a lot about the way people express their views in the world, which helps color in what they would tell you.<\/p>\n\n\n\n

Kenneth Olmstead<\/strong>, research associate<\/em>: For an opinion to make it onto a social media platform, the person must first have that view, choose to put it out into the world and then shape how they want it to be expressed. In a survey, the respondent could be asked about something they have not put much thought into. Compared with surveys, social media data are an expression of attitudes in a way that\u2019s less controlled by the researcher.<\/p>\n\n\n\n

Adam Hughes<\/strong>, computational social scientist<\/em>: Social media data are far from perfect, as we know. That said, a biased and somewhat murky signal can still be informative. For example, when Twitter users post hashtags, they appear to be focused on the particular topic. This action tells us something about what Twitter users are paying attention to, even if we can\u2019t necessarily say who they are, why they started paying attention or what they make of it.<\/p>\n\n\n\n

What are some of the limitations of studying social media?<\/h4>\n\n\n\n

Onyi Lam<\/strong>, computational social scientist<\/em>: Since social media data are not usually representative of the public as a whole, it may not be worthwhile to collect a random sample in the hopes of saying something about the general public. A better approach may be to collect data from a specific, well-defined group, such as members of Congress or news media organizations.<\/p>\n\n\n\n

Dennis Quinn<\/strong>: Organic data can be very messy, and they often require a lot of planning and strategy to collect. As a result, you sometimes find yourself making advanced analytical decisions at the first stage of the process. It\u2019s just a fact of big data that research design, data collection and analysis all meld together. Sometimes you realize in the final stage of a product that you made a big mistake in the first. <\/strong>The way to deal with this is to adopt a more cyclical, more iterative approach to your research: Collect a little data and analyze it. This lets you make your mistakes early, in small doses, before you make them later, in big doses. Striking the right balance between linear planning and iterative development is one of the biggest challenges.<\/p>\n\n\n\n

Stefan Wojcik<\/strong>, computational social scientist<\/em>: You\u2019re also limited to how the platform designs its interface. It constrains the conversation in various ways that don\u2019t necessarily maximize what a researcher would want to maximize.<\/p>\n\n\n\n

Kenneth Olmstead<\/strong>: Access to data is a huge challenge. Many researchers turn to Twitter for data because it is largely available \u2014 with some limitations \u2014 and mostly public. But Twitter is used by a relatively small portion<\/a> of the population. Facebook, on the other hand, is used by a larger portion of the population, but most of what is posted to Facebook is unavailable to researchers. Understanding that available data is often not the entire picture of that platform is key to analyzing the data.<\/p>\n\n\n\n

Skye Toor<\/strong>, data science assistant<\/em>: One common misconception is that the size of data is a major obstacle. In reality, it\u2019s the analysis that is usually more challenging. In most cases, the marginal cost of adding more data is low. The marginal cost of doing more analysis is much higher. That\u2019s where more time is spent.<\/p>\n\n\n\n

Mike Barthel<\/strong>, senior researcher<\/em>: The biggest question \u2014 and one we can\u2019t always answer \u2014 is always, \u201cHow do these data translate to people?\u201d On social media, each account could be a person, but it could also be a duplicate account or could be an automated account, and it may have very sparse information about what that person is like, demographically or otherwise.<\/p>\n\n\n\n

The study of social media often involves new, evolving tools and computer learning techniques. How do you determine which tools and techniques meet your standards for publication?<\/h4>\n\n\n\n

Adam Hughes<\/strong>: The most important part of adopting a new method is validation. For example, we check to ensure that our machine learning models correspond with human judgments about what social media posts are trying to convey. If a predictive model performs much worse than human content coders do \u2014 or if it were no more informative than flipping a coin \u2014 we wouldn\u2019t use it.<\/p>\n\n\n\n

We also value transparency. The tools we use should be well-documented and, when possible, open source. If a model or technique does not have a detailed explanation, underlying code or academic research supporting it, we approach it with lots of skepticism. We also make every effort to validate the algorithms we use, no matter where they come from.<\/p>\n\n\n\n

Stefan Wojcik<\/strong>: I think it\u2019s not so much about the tools themselves being improper for publication, but the improper application<\/em> of them. People sometimes apply things like topic models<\/a> to random samples of Twitter data and then make claims about what the broader public is talking about. One needs to think carefully about the population, how much text they produce and how it can be reasonably compared.<\/p>\n\n\n\n

When approaching a project involving social media data, do you start with a research question or a dataset? Is there a general rule of thumb?<\/h4>\n\n\n\n

Stefan Wojcik<\/strong>: It can work both ways, but it\u2019s usually better to start with a research question. Otherwise, you end up rudderless. You need some frame of reference to begin a study, because it allows you to make tradeoffs in your research design that ultimately maximize your ability to understand the thing you want to understand. But if you just start with data, there\u2019s nothing to maximize. Data are just data.<\/p>\n\n\n\n

Onyi Lam<\/strong>: This is a contentious question! I think different researchers have different preferences, but ultimately, we all have to go back and forth between refining the question and understanding the data from a given data source.<\/p>\n\n\n\n

Adam Hughes<\/strong>: We usually start with research questions to avoid post-hoc reasoning. It\u2019s easy to get a big dataset, mine the data for patterns and then come up with stories about what the patterns mean. But that approach can result in stories that don\u2019t generalize beyond the particular dataset you are looking at. The better approach is to draw on your substantive knowledge to develop hypotheses before doing any data analysis.<\/p>\n\n\n\n

Skye Toor<\/strong>: Start collecting data with a plan of what you want to analyze both for research integrity reasons and in order to keep the scope of any project manageable.<\/p>\n\n\n\n

Mike Barthel<\/strong>: It depends how familiar the data source and analysis techniques are. For areas we\u2019ve delved into several times already, we can go in with a research question knowing the limitations we\u2019re likely to face. <\/strong>For new data sources, we\u2019re likely to spend extensive time understanding their characteristics before we narrow in on a research question we can actually answer. Of course, we\u2019d never explore a data source in the first place unless we think it\u2019d be of some use to our general research agenda.<\/p>\n\n\n\n

What are some of the biggest technical challenges you have faced when working with large amounts of social media data?<\/h4>\n\n\n\n

Onyi Lam<\/strong>: Cleaning the data is no doubt the biggest challenge!<\/p>\n\n\n\n

Skye Toor<\/strong>: The biggest challenge isn\u2019t the size of the data. It\u2019s the quality.<\/p>\n\n\n\n

Adam Hughes<\/strong>: When analyzing Facebook posts, we found a huge number of apparent duplicates. The Facebook API (Application Programming Interface \u2014 a common method for retrieving large amounts of data from a website<\/a>) didn\u2019t explain why the duplicates were there. We ended up building an entirely separate machine learning model to detect and remove them.<\/p>\n\n\n\n

Mike Barthel<\/strong>: Broadly speaking, the challenge is always that you can\u2019t simply dive into the data and start analyzing. You need to check to make sure every step executed the way you thought it would be, that nothing got lost or corrupted and that everything is working. In survey data, I can always go in and look at an individual row to find out why data is wonky. I can\u2019t do that with big data sets. So all the checks need to be automated, and they take time.<\/p>\n\n\n\n

Kenneth Olmstead<\/strong>: Compute time can be an issue. In the report writing process we often find we need the data sliced in some new way or rerun differently for some reason. With very large datasets, that can often mean having to wait days for the data to run before we can put it back in the report.<\/p>\n\n\n\n

What are some of the challenges of writing research reports that rely on data science tools but are intended for a general audience?<\/h4>\n\n\n\n

Mike Barthel<\/strong>: Survey work is standardized enough at this point that we can simply meet industry standards and then mention the details of how we did what we did when we explain our methodology. For instance, we rarely mention in the body of a Pew Research Center report that survey data are weighted (though you would always find that level of detail in the methodology section). For data science tools, everything is still new and there aren\u2019t yet universally accepted industry standards for how to use machine learning, so we want to be fully transparent about our decisions and their limitations at the outset. We try to give more detail about how we got to our conclusions. As a result, it can be hard to provide our major findings without those caveats \u2014 which can mean that the substance of the findings is a little harder to follow.<\/p>\n\n\n\n

Adam Hughes<\/strong>: One of the biggest challenges is that error in machine learning models is not as easy to describe as a survey\u2019s margin of error. We report statistics like precision and recall (which convey information about a predictive model\u2019s false positive and false negative rate), and we provide information about the agreement between a model\u2019s decisions and human decisions. But for many readers, these quantities may be harder to interpret than a statement about a particular point estimate\u2019s uncertainty and the margin of error applied to probability survey sampling. So it might be the case that readers become overconfident in a particular finding.<\/p>\n\n\n\n

Jargon is definitely an issue. The language of data science is often unfriendly to outsiders. To explain what we mean by \u201csupport vector machine,\u201d \u201crandom effects,\u201d \u201cword2vec,\u201d and related terms, we write terminology sections in our reports that explain what we are talking about.<\/p>\n\n\n\n

Dennis Quinn<\/strong>: Metaphors help a lot, as do short, declarative sentences. Sometimes you really have to break something down step-by-step in a way that might make you look silly to an expert \u2014 but you just have to do that. If you\u2019re too afraid to look silly by over-explaining something, then some of your readers aren\u2019t going to understand it.<\/p>\n","protected":false},"excerpt":{"rendered":"

Ever since Friendster and Myspace gained popularity in the early 2000s, social scientists have been interested in studying the impact of\u2026<\/p>\n","protected":false},"author":655,"featured_media":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"sub_headline":"","sub_title":"","_prc_public_revisions":[],"_ppp_expiration_hours":0,"_ppp_enabled":false,"ai_generated_summary":"","relatedPosts":[],"_prc_fork_parent":0,"_prc_fork_status":"","_prc_active_fork":0,"datacite_doi":"","datacite_doi_citation":"","_prc_seo_qr_attachment_id":0,"spoken_article_player_enabled":true,"displayBylines":true,"footnotes":"","prc_watchers":[]},"categories":[353],"bylines":[683],"collection":[],"_post_visibility":[],"decoded-category":[530],"formats":[],"_fund_pool":[],"languages":[],"regions-countries":[],"research-teams":[524],"workflow-status":[],"class_list":["post-111670","decoded","type-decoded","status-publish","hentry","category-data-science","bylines-paul-hitlin","decoded-category-data-science","research-teams-decoded"],"label":"Decoded","post_parent":0,"word_count":2099,"canonical_url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/2018\/09\/05\/__trashed\/","art_direction":{"A1":{"id":126032,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/09.05.2018_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/09.05.2018_feature.png?w=564&h=317&crop=1","width":564,"height":317,"caption":"","chartArt":false},"A2":{"id":126032,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/09.05.2018_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/09.05.2018_feature.png?w=268&h=151&crop=1","width":268,"height":151,"caption":"","chartArt":false},"A3":{"id":126032,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/09.05.2018_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/09.05.2018_feature.png?w=194&h=110&crop=1","width":194,"height":110,"caption":"","chartArt":false},"A4":{"id":126032,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/09.05.2018_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/09.05.2018_feature.png?w=268&h=151&crop=1","width":268,"height":151,"caption":"","chartArt":false},"XL":{"id":126032,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/09.05.2018_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/09.05.2018_feature.png?w=700&h=394&crop=1","width":700,"height":394,"caption":"","chartArt":false},"social":{"id":126032,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/09.05.2018_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/09.05.2018_feature.png?w=700&h=394&crop=1","width":700,"height":394,"caption":"","chartArt":false}},"_embeds":[],"watchers":[],"table_of_contents":[],"datacite_doi":"","prc_seo_data":{"title":"Q&A: What works and what doesn\u2019t when studying social media data","description":"Ever since Friendster and Myspace gained popularity in the early 2000s, social scientists have been interested in studying the impact of\u2026","og_title":"Q&A: What works and what doesn\u2019t when studying social media data","og_description":"Ever since Friendster and Myspace gained popularity in the early 2000s, social scientists have been interested in studying the impact of\u2026","schema_type":"Article","noindex":false,"canonical_url":"","primary_terms":{"category":1},"custom_schema":[],"og_image":126032,"indexnow_submitted_at":null,"gsc_index_status":null},"prepublish_checks":{},"jetpack_sharing_enabled":true,"relatedPostsOrdered":[],"bylinesOrdered":[{"key":"_tlvcehu72","termId":683}],"acknowledgementsOrdered":[],"_links":{"self":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111670","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded"}],"about":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/types\/decoded"}],"author":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/users\/655"}],"replies":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/comments?post=111670"}],"version-history":[{"count":2,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111670\/revisions"}],"predecessor-version":[{"id":138545,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111670\/revisions\/138545"}],"wp:attachment":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/media?parent=111670"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/categories?post=111670"},{"taxonomy":"bylines","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/bylines?post=111670"},{"taxonomy":"collection","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/collection?post=111670"},{"taxonomy":"_post_visibility","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_post_visibility?post=111670"},{"taxonomy":"decoded-category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded-category?post=111670"},{"taxonomy":"formats","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/formats?post=111670"},{"taxonomy":"_fund_pool","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_fund_pool?post=111670"},{"taxonomy":"languages","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/languages?post=111670"},{"taxonomy":"regions-countries","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/regions-countries?post=111670"},{"taxonomy":"research-teams","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/research-teams?post=111670"},{"taxonomy":"workflow-status","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/workflow-status?post=111670"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}