{"id":111547,"date":"2019-12-03T15:47:00","date_gmt":"2019-12-03T20:47:00","guid":{"rendered":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/\/\/using-fixed-and-random-effects-models-for-panel-data-in-python\/"},"modified":"2024-04-14T04:10:40","modified_gmt":"2024-04-14T09:10:40","slug":"using-fixed-and-random-effects-models-for-panel-data-in-python","status":"publish","type":"decoded","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/2019\/12\/03\/using-fixed-and-random-effects-models-for-panel-data-in-python\/","title":{"rendered":"Using fixed and random effects models for panel data in Python"},"content":{"rendered":"\n
\"\"<\/a><\/figure>\n\n\n\n

Identifying causal relationships from observational data is not easy. Still, researchers are often interested in examining the effects of policy changes or other decisions. In those analyses, researchers will face any number of analytical decisions, including whether to use fixed or random effects models to control for variables that don\u2019t change over time.<\/p>\n\n\n\n

Let\u2019s consider an example. Suppose we\u2019re interested in estimating the effect that a government grant might have had on firms\u2019 product quality (as examined in this previous study<\/a>). In addition to controlling for observed variables like the number of employees the firms had at different time points in the study period, we might also want to control for unobserved<\/em> variables, such as the management quality of the firms.<\/p>\n\n\n\n

Assuming that the firms\u2019 management quality is constant over time, we can use regression models <\/a>to try to account for those unobserved factors \u2014 but there isn\u2019t always consensus about the best way to do so. Specifically, researchers often must decide whether to use a fixed or random effects approach in an analysis like this.<\/p>\n\n\n\n

In this post, we\u2019ll discuss some of the differences between fixed and random effects models when applied to panel data \u2014 that is, data collected over time on the same unit of analysis \u2014 and how these models can be implemented in the programming language Python.<\/p>\n\n\n\n

Fixed vs. random effects in panel data<\/strong><\/h4>\n\n\n\n

Broadly speaking, the distinction between a fixed effects approach and a random effects approach concerns the correlation \u2014 or lack thereof \u2014 between unobserved variables and observed variables. To highlight this difference, let\u2019s go back to the example cited above.<\/p>\n\n\n\n

The key issue in deciding between the two approaches is whether or not the unobserved variables in our analysis \u2014 in this case, the firms\u2019 management quality \u2014 might be correlated with observed variables. We might use a fixed effects approach if we think that these variables are<\/em> correlated \u2014 for example, if we think firms\u2019 management quality has a role in determining whether the firms receive a grant. But we might use a random effects approach if we think the two variables are not correlated.<\/p>\n\n\n\n

Fixed effects help capture the effects of all variables<\/strong> that don\u2019t change over time. In other words, anything else that does not change over time at the firm level, such as its location, would be captured by these fixed effects terms in the model. That means we cannot separately estimate the effect of firms\u2019 location on their performance.<\/p>\n\n\n\n

This is quite restrictive for some applications, so researchers who might be interested in studying the effect of time-invariant variables may want to choose the random effects framework instead, even though these models impose stronger assumptions about the unobserved effects.<\/p>\n\n\n\n

Using Python to implement the models<\/strong><\/h4>\n\n\n\n

Next, we\u2019ll illustrate how to implement panel data analysis in Python, using a built-in dataset on firms\u2019 performance from the `linearmodels` library that follows from the example discussed above. Note that `linearmodels` is only supported in Python 3.<\/p>\n\n\n\n

import numpy as np\nimport pandas as pd\nfrom linearmodels import PanelOLS\nfrom linearmodels import RandomEffects<\/code><\/code><\/pre>\n\n\n\n
To implement a random effects model, we call the RandomEffects method and assign the firm code and year columns as the indexes in the dataframe.<\/p>\n\n\n\n
from linearmodels.datasets import jobtraining\ndata = jobtraining.load()\nyear = pd.Categorical(data.year)\ndata = data.set_index([\u2018fcode\u2019, \u2018year\u2019])\ndata[\u2018year\u2019] = year<\/code><\/pre>\n\n\n\n
For the dependent variable, we use the change in scrap rate between periods as a proxy of the product quality. For the independent variables, we include the grant status in period t (=1 if received grant) and the number of employees at the firm.<\/p>\n\n\n\n
exog_vars = [\u2018grant\u2019, \u2018employ\u2019]\nexog = sm.add_constant(data[exog_vars])\nmod = RandomEffects(data.clscrap, exog)\nre_res = mod.fit()\nprint(re_res)<\/code><\/pre>\n\n\n\n
\"\"<\/a><\/figure>\n\n\n\n
To implement the fixed effects model, we use the PanelOLS method, and set the parameter `entity_effects` to be True.<\/p>\n\n\n\n
mod = PanelOLS(data.clscrap, exog)<\/p>\n\n\n\n
re_res = mod.fit()<\/p>\n\n\n\n
print(re_res)<\/p>\n\n\n\n
\"\"<\/a><\/figure>\n\n\n\n
The results are quite different between the fixed and random effects models, but neither is statistically significant. However, to the extent that you think the unobserved effect of the firms is uncorrelated with whether the firms received the grant, the random effects model is more appropriate.<\/p>\n\n\n\n
Equivalence of fixed effects model and dummy variable regression<\/strong><\/h4>\n\n\n\n
Estimating a fixed effects model is equivalent to adding a dummy variable for each subject or unit of interest in the standard OLS model. To illustrate equivalence between the two approaches, we can use the OLS method in the statsmodels library, and regress the same dependent variable on the categorized variable of firm, and other independent variables:<\/p>\n\n\n\n
data = jobtraining.load()\ndata[\u2018year\u2019] = pd.Categorical(data.year)\nFE_ols = smf.ols(formula=\u2019clscrap ~ 1 + grant + employ + C(fcode)\u2019, data = data).fit()\nprint(FE_ols.summary())<\/code><\/code><\/pre>\n\n\n\n
\"\"<\/a><\/figure>\n\n\n\n
The results from the dummy regression show the separately estimated effect of each firm on change in scrap rate. This is sometimes useful when we want to focus on specific units. In addition, we can compute some sample averages of these estimates to get a sense of how much variation there is across firms.<\/p>\n","protected":false},"excerpt":{"rendered":"
Identifying causal relationships from observational data is not easy.  Still, researchers are often interested in examining the effects of policy changes.<\/p>\n","protected":false},"author":655,"featured_media":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_crdt_document":"","sub_headline":"","sub_title":"","_prc_public_revisions":[],"_ppp_expiration_hours":0,"_ppp_enabled":false,"ai_generated_summary":"","relatedPosts":[],"_prc_fork_parent":0,"_prc_fork_status":"","_prc_active_fork":0,"datacite_doi":"","datacite_doi_citation":"","_prc_seo_qr_attachment_id":0,"spoken_article_player_enabled":true,"footnotes":""},"categories":[357],"bylines":[819],"collection":[],"_post_visibility":[],"decoded-category":[531,532],"formats":[],"_fund_pool":[],"languages":[],"regions-countries":[],"research-teams":[524],"class_list":["post-111547","decoded","type-decoded","status-publish","hentry","category-survey-methods","bylines-onyi-lam","decoded-category-coding-how-to","decoded-category-survey-methods","research-teams-decoded"],"label":"Decoded","post_parent":0,"word_count":859,"canonical_url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/2019\/12\/03\/using-fixed-and-random-effects-models-for-panel-data-in-python\/","art_direction":{"A1":{"id":125926,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/12.3.2018_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/12.3.2018_feature.png?w=564&h=317&crop=1","width":564,"height":317,"caption":"","chartArt":false},"A2":{"id":125926,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/12.3.2018_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/12.3.2018_feature.png?w=268&h=151&crop=1","width":268,"height":151,"caption":"","chartArt":false},"A3":{"id":125926,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/12.3.2018_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/12.3.2018_feature.png?w=194&h=110&crop=1","width":194,"height":110,"caption":"","chartArt":false},"A4":{"id":125926,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/12.3.2018_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/12.3.2018_feature.png?w=268&h=151&crop=1","width":268,"height":151,"caption":"","chartArt":false},"XL":{"id":125926,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/12.3.2018_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/12.3.2018_feature.png?w=700&h=394&crop=1","width":700,"height":394,"caption":"","chartArt":false},"social":{"id":125926,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/12.3.2018_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/12.3.2018_feature.png?w=700&h=394&crop=1","width":700,"height":394,"caption":"","chartArt":false}},"_embeds":[],"table_of_contents":[],"datacite_doi":"","prc_seo_data":{"title":"Using fixed and random effects models for panel data in Python","description":"Identifying causal relationships from observational data is not easy.  Still, researchers are often interested in examining the effects of policy changes.","og_title":"Using fixed and random effects models for panel data in Python","og_description":"Identifying causal relationships from observational data is not easy.  Still, researchers are often interested in examining the effects of policy changes.","schema_type":"Article","noindex":false,"canonical_url":"","primary_terms":{"category":42},"custom_schema":[],"og_image":125926,"indexnow_submitted_at":null,"gsc_index_status":null},"prepublish_checks":{},"jetpack_sharing_enabled":true,"relatedPostsOrdered":[],"bylinesOrdered":[{"key":"_jer2gm47n","termId":819}],"acknowledgementsOrdered":[],"_links":{"self":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111547","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded"}],"about":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/types\/decoded"}],"author":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/users\/655"}],"replies":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/comments?post=111547"}],"version-history":[{"count":2,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111547\/revisions"}],"predecessor-version":[{"id":138536,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111547\/revisions\/138536"}],"wp:attachment":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/media?parent=111547"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/categories?post=111547"},{"taxonomy":"bylines","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/bylines?post=111547"},{"taxonomy":"collection","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/collection?post=111547"},{"taxonomy":"_post_visibility","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_post_visibility?post=111547"},{"taxonomy":"decoded-category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded-category?post=111547"},{"taxonomy":"formats","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/formats?post=111547"},{"taxonomy":"_fund_pool","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_fund_pool?post=111547"},{"taxonomy":"languages","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/languages?post=111547"},{"taxonomy":"regions-countries","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/regions-countries?post=111547"},{"taxonomy":"research-teams","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/research-teams?post=111547"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}