{"id":111691,"date":"2018-08-13T18:00:00","date_gmt":"2018-08-13T23:00:00","guid":{"rendered":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/\/\/making-sense-of-topic-models\/"},"modified":"2024-04-14T04:10:45","modified_gmt":"2024-04-14T09:10:45","slug":"making-sense-of-topic-models","status":"publish","type":"decoded","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/2018\/08\/13\/making-sense-of-topic-models\/","title":{"rendered":"Making sense of topic models"},"content":{"rendered":"\n
\"\"<\/a>
(Related posts: <\/em>An intro to topic models for text analysis<\/em><\/a>, <\/em>Overcoming the limitations of topic models with a semi-supervised approach<\/em><\/a>, Interpreting and validating topic models<\/em><\/a>, <\/em>How keyword oversampling can help with text analysis<\/em><\/a> and <\/em>Are topic models reliable or useful?<\/em><\/a>)<\/em><\/figcaption><\/figure>\n\n\n\n

In my first post about topic models<\/a>, I discussed what topic models are, how they work and what their output looks like. The example I used trained a topic model on open-ended responses to a survey question about what makes life feel fulfilling and examined three topics in particular:<\/p>\n\n\n\n

\"\"<\/a><\/figure>\n\n\n\n

By looking at the top words for each topic, we can clearly see that Topic 4 is related in some way to hobbies; Topic 5 seems to pertain to charity and giving back to society; and Topic 88 has something to do with spending time pursuing and contributing to more serious things, like work. However, we also notice some quirks, which raise an important question: How do we figure out what these topics are<\/em>, exactly? As it turns out, that can be a very difficult question to answer.<\/p>\n\n\n\n

Translating topics to concepts<\/strong><\/h4>\n\n\n\n

Let\u2019s say we find Topic 5 present in enough documents that it\u2019s clearly capturing something unique about a subset of our corpus \u2014 something we\u2019d like to analyze. Again, the most common words in Topic 5 are \u201cgiving,\u201d \u201cwoman,\u201d \u201cgrateful,\u201d \u201ccharitable,\u201d \u201cgiving community,\u201d \u201cnonprofit,\u201d and \u201cphilosophy.\u201d What exactly could such an analysis tell us? What is <\/em>Topic 5? If we\u2019re interpreting this topic as being about charity and giving, does the word \u201cwoman\u201d really belong in it? What about \u201cphilosophy\u201d? When working with topic models, it\u2019s important to remember that these algorithms cannot guarantee that the words in each topic will be related to one another conceptually \u2014 <\/em>only that they frequently occur together in your data for some reason.<\/em><\/p>\n\n\n\n

Text data naturally involve the variety, complexity, and context of language. So, if you think about it, it\u2019s completely reasonable that topic modeling algorithms may pick up on noise or quirky, idiosyncratic correlations of the words in our documents. However, if we wish to use our topic model to make classifications and measure different concepts in our documents, then it\u2019s also reasonable to be rather concerned about false positives arising from words in our topics that don\u2019t align with how we want to interpret them<\/em>. Since they don\u2019t align with the general concept that we think the model is picking up \u2014 the concept that we want to use the model to measure \u2014 we might consider words like these to be conceptually spurious<\/em>. For one reason or another, they are associated in our data with the other words in the topic, but these associations are driven by something other than semantics. That can be a result of where the data came from, perhaps, or how the data were collected.<\/p>\n\n\n\n

When analyzing survey responses, for example, conceptually spurious words can appear not because they\u2019re related to other words in a topic semantically, but because they\u2019re demographically correlated. That is, certain types of respondents may mention some themes more than others, and these themes can sometimes coincide together frequently in their responses even though they\u2019re not conceptually related.<\/p>\n\n\n\n

Another topic in our model highlights this problem particularly well, consisting of the following words: \u201cgrandchild\u201d, \u201cgrand\u201d, \u201cchild grandchild\u201d, \u201cgrand child\u201d, \u201cchild\u201d, \u201cflorida\u201d<\/strong>, \u201cchild grand\u201d, \u201cgrandchild great\u201d, and \u201cgreat grandchild\u201d. Clearly, one of these words is not like the others. Having trained our topic model on open-ended responses from a nationally representative survey, in this analysis we\u2019re interested in using our topics to measure what<\/em> respondents talk about so that we can then characterize the population in terms of who<\/em> talks about what. Of course, that means that we need to make sure that our measures don\u2019t conflate the who<\/em> and the what. <\/em>So while it\u2019s insightful to find that certain respondents commonly mention both grandchildren and Florida, we probably don\u2019t want both of these concepts present together in a single topic.<\/p>\n\n\n\n

Conceptually spurious words may seem like a minor problem, but they can profoundly impact the results of your analysis. Even just a few extra words can have surprisingly dramatic consequences. Assuming that we want to use Topic 5 to measure whether or not a respondent mentions charity or giving, we probably want to remove several out-of-place terms: \u201cwoman,\u201d \u201cgrateful,\u201d and \u201cphilosophy.\u201d Unfortunately, with most common topic modeling algorithms, this is not an easy task. The words and their distributions across each topic are difficult to modify without a fair amount of programming and statistical knowledge. For the sake of simplicity, though, let\u2019s assume that we can roughly approximate whether a document has a topic or not simply by checking to see if it contains any of the top words in the topic.<\/p>\n\n\n\n

If we do this for the top words in Topic 5 (giving, woman, grateful, charitable, giving community, nonprofit, philosophy), we find that there are 163 responses in our corpus that match to at least one of these keywords. But if we remove the three words that seem unrelated to the concept of giving or charity (woman, grateful, philosophy), our findings change dramatically. Just 60 documents now match \u2014 less than half<\/em> the number that we got from the words in the raw topic model output.<\/p>\n\n\n\n

This highlights a major limitation of topic modeling: the \u201ctopics\u201d they produce are often not necessarily topics in the traditional sense of the term, and even one out-of-place word can make the difference between a model that accurately measures whether documents mention a coherent concept and one that vastly overestimates how often that concept actually appears in the corpus.<\/p>\n\n\n\n

There is no magic number of topics<\/strong><\/h4>\n\n\n\n

Conceptually spurious words are not the only potential problem with topic models. Another issue arises from the simple fact that researchers must specify a predetermined number of topics without knowing what that number should be. Regardless of the number you pick, many of your topics may come out looking like this:<\/p>\n\n\n\n

\"\"<\/a><\/figure>\n\n\n\n

Topic 106 above is an example of what some of us at Pew Research Center have come to colloquially refer to as an \u201cundercooked\u201d topic. Rather than capturing one coherent theme, we can see a variety of themes embedded in this topic, all mixed together. How might we interpret this group of words as a single theme? Is this topic about security, finance, success, health, or relationships? Here, our model appears to be grouping multiple distinct topics into a single topic. Accordingly, we may look at this topic and decide to re-run the model with a greater number of topics so it has the space to break these topics apart.<\/p>\n\n\n\n

However, in the very same model<\/em>, we also have Topic 15, an example of an \u201covercooked\u201d topic. Rather than being too broad or confounded to be interpretable, this topic is too granular. We might be inclined to give it a label like \u201cmentioning your job or career,\u201d but in that case, we would probably want to include other words like \u201ccareer,\u201d \u201cwork,\u201d and \u201cprofession.\u201d In fact, the word \u201ccareer\u201d actually exists elsewhere in our model in Topic 80 \u2014 another potentially \u201cundercooked\u201d topic \u2014 and if we want to bring those words together into a single topic, we might actually need to reduce <\/em>the total number of topics to encourage the model to merge them.<\/p>\n\n\n\n

Even then, the topics may not condense the way we want them to. In fact, \u201cjob\u201d and \u201ccareer\u201d may simply not co-occur frequently enough in our documents for the very reason we want them to be in the same topic together \u2014 they\u2019re more or less synonyms, and likely to be substituted for one another. Depending on our data, the use of both words in a single document could be redundant and therefore uncommon, making it impossible for our model to connect the dots between the two words. Indeed, based on the output of this model, it seems that both words occur frequently alongside a wide variety of other terms (\u201cfamily\u201d and \u201cwork\u201d in particular), but not alongside each other enough for the model to pick up on it.<\/p>\n\n\n\n

No matter which particular algorithm and parameters we\u2019ve used, or how many topics we\u2019ve asked a model to produce, we\u2019ve found that undercooked and overcooked topics are largely unavoidable. Metrics like perplexity<\/a> and coherence<\/a>can help guide you towards a number of topics that minimizes this problem, but no \u201cperfect\u201d number of topics exists that can eliminate it entirely. This is because while topic modeling algorithms are great at identifying clusters of words that frequently co-occur, they do not actually understand the context in which those words occur.<\/p>\n\n\n\n

As researchers, we\u2019re interested in measuring the distribution of certain themes in our documents, but some of those themes may be general, while others might be nuanced, detailed and specific. Topic models can find useful exploratory patterns, but they\u2019re unable to reliably capture context or nuance. They cannot assess how topics conceptually relate to one another; there is no magic number of topics; and they can\u2019t say how the topics should be interpreted. Of course, supervised classification algorithms can make use of more sophisticated linguistic models to overcome some of these limitations, but this requires training data that can be time-consuming and costly to collect. And the very promise of unsupervised topic modeling is that it\u2019s fast, easy, and avoids the need for manual coding.<\/p>\n\n\n\n

In future posts, I\u2019ll explore potential ways to overcome the limitations of topic models and assess the extent to which we can actually use them to reliably measure concepts in a corpus.<\/p>\n","protected":false},"excerpt":{"rendered":"

Topic models can produce clusters of words that characterize written documents. But how do we figure out what those clusters mean, exactly?<\/p>\n","protected":false},"author":655,"featured_media":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"sub_headline":"","sub_title":"","_prc_public_revisions":[],"_ppp_expiration_hours":0,"_ppp_enabled":false,"ai_generated_summary":"","relatedPosts":[],"_prc_fork_parent":0,"_prc_fork_status":"","_prc_active_fork":0,"datacite_doi":"","datacite_doi_citation":"","_prc_seo_qr_attachment_id":0,"spoken_article_player_enabled":true,"displayBylines":true,"footnotes":"","prc_watchers":[]},"categories":[353],"bylines":[927],"collection":[],"_post_visibility":[],"decoded-category":[530],"formats":[],"_fund_pool":[],"languages":[],"regions-countries":[],"research-teams":[524],"workflow-status":[],"class_list":["post-111691","decoded","type-decoded","status-publish","hentry","category-data-science","bylines-patrick-van-kessel","decoded-category-data-science","research-teams-decoded"],"label":"Decoded","post_parent":0,"word_count":1630,"canonical_url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/2018\/08\/13\/making-sense-of-topic-models\/","art_direction":{"A1":{"id":126052,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/Aug.13.2018_featured.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/Aug.13.2018_featured.png?w=564&h=317&crop=1","width":564,"height":317,"caption":"","chartArt":false},"A2":{"id":126052,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/Aug.13.2018_featured.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/Aug.13.2018_featured.png?w=268&h=151&crop=1","width":268,"height":151,"caption":"","chartArt":false},"A3":{"id":126052,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/Aug.13.2018_featured.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/Aug.13.2018_featured.png?w=194&h=110&crop=1","width":194,"height":110,"caption":"","chartArt":false},"A4":{"id":126052,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/Aug.13.2018_featured.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/Aug.13.2018_featured.png?w=268&h=151&crop=1","width":268,"height":151,"caption":"","chartArt":false},"XL":{"id":126052,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/Aug.13.2018_featured.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/Aug.13.2018_featured.png?w=720&h=405&crop=1","width":720,"height":405,"caption":"","chartArt":false},"social":{"id":126052,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/Aug.13.2018_featured.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/Aug.13.2018_featured.png?w=1200&h=628&crop=1","width":1200,"height":628,"caption":"","chartArt":false}},"_embeds":[],"watchers":[],"table_of_contents":[],"datacite_doi":"","prc_seo_data":{"title":"Making sense of topic models","description":"Topic models can produce clusters of words that characterize written documents. But how do we figure out what those clusters mean, exactly?","og_title":"Making sense of topic models","og_description":"Topic models can produce clusters of words that characterize written documents. But how do we figure out what those clusters mean, exactly?","schema_type":"Article","noindex":false,"canonical_url":"","primary_terms":{"category":1},"custom_schema":[],"og_image":126052,"indexnow_submitted_at":null,"gsc_index_status":null},"prepublish_checks":{},"jetpack_sharing_enabled":true,"relatedPostsOrdered":[],"bylinesOrdered":[{"key":"_0xx95nhvc","termId":927}],"acknowledgementsOrdered":[],"_links":{"self":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111691","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded"}],"about":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/types\/decoded"}],"author":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/users\/655"}],"replies":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/comments?post=111691"}],"version-history":[{"count":2,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111691\/revisions"}],"predecessor-version":[{"id":138555,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111691\/revisions\/138555"}],"wp:attachment":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/media?parent=111691"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/categories?post=111691"},{"taxonomy":"bylines","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/bylines?post=111691"},{"taxonomy":"collection","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/collection?post=111691"},{"taxonomy":"_post_visibility","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_post_visibility?post=111691"},{"taxonomy":"decoded-category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded-category?post=111691"},{"taxonomy":"formats","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/formats?post=111691"},{"taxonomy":"_fund_pool","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_fund_pool?post=111691"},{"taxonomy":"languages","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/languages?post=111691"},{"taxonomy":"regions-countries","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/regions-countries?post=111691"},{"taxonomy":"research-teams","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/research-teams?post=111691"},{"taxonomy":"workflow-status","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/workflow-status?post=111691"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}