{"id":111603,"date":"2019-06-12T15:05:00","date_gmt":"2019-06-12T20:05:00","guid":{"rendered":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/\/\/using-tidy-verse-tools-with-pew-research-center-survey-data-in-r\/"},"modified":"2024-04-14T04:10:37","modified_gmt":"2024-04-14T09:10:37","slug":"using-tidy-verse-tools-with-pew-research-center-survey-data-in-r","status":"publish","type":"decoded","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/2019\/06\/12\/using-tidy-verse-tools-with-pew-research-center-survey-data-in-r\/","title":{"rendered":"Using tidyverse tools with Pew Research Center survey data in R"},"content":{"rendered":"\n
\"\"<\/a>
(Illustration by Selena Qian\/Pew Research Center)<\/figcaption><\/figure>\n\n\n\n

(Related post: <\/em>How to analyze Pew Research Center survey data in R<\/em><\/a>)<\/em>

Last year, I wrote an introductory blog post<\/a> about how to access and analyze Pew Research Center survey data with R, a free, open-source software for statistical analysis. The post showed how to perform tasks using the survey<\/a> package.<\/p>\n\n\n\n

The survey<\/code> package is ideal if you\u2019re analyzing survey data and need variance estimates that account for complex sample designs, such as cluster samples or stratification. However, the first step of any data analysis is often exploring the data with simple but powerful summaries, such as crosstabs and plots. This usually requires significant re-coding of variables as well as data cleaning and other manipulation tasks that can be difficult and counterintuitive. Fortunately, something called the \u201ctidyverse<\/a>\u201d is here to help.<\/p>\n\n\n\n

In this post, I\u2019ll show how to use tidyverse<\/code> tools to do exploratory analyses of Pew Research Center survey data. (These tools, however, can be used with data from any source.)<\/p>\n\n\n\n

What is the “tidyverse”?<\/h4>\n\n\n\n

R has a large community of users, many of whom make their own code freely available to other R users in the form of packages<\/a> that implement specialized kinds of analyses or enable other programming tasks.<\/p>\n\n\n\n

R packages are typically hosted on The Comprehensive R Archive Network<\/a> (CRAN), but are also available on other primarily open-source code repositories like github<\/a>, gitlab<\/a> or bioconductor<\/a>. One of the nice things about R is that anyone can build a package and contribute to the software\u2019s development. But with so many packages (there were 14,366 on CRAN as of June 12, 2019), there are often many different ways to tackle the same challenge. This can lead to inconsistencies that are often confusing, making it harder on new users or those looking to develop their R skills.<\/p>\n\n\n\n

Enter the tidyverse<\/code>, a collection of R packages designed for data science that share a consistent design philosophy and produce code that can be more easily understood and shared. The tidyverse<\/code> has a growing community<\/a> of users, contributors and developers that aim to help R users access and learn these free, open-source tools. The book R for Data Science by Hadley Wickham and Garrett Grolemund walks through many key components of how to use these tools to do data science in R and can be accessed online<\/a> for free.<\/p>\n\n\n\n

Packages<\/h4>\n\n\n\n

For this post, we\u2019ll need the tidyverse<\/code> and haven<\/a> packages. The haven<\/code> package is the tidyverse\u2019s solution for importing and exporting data from several different formats, including SPSS (the format in which most Center datasets are currently released), SAS and Stata.<\/p>\n\n\n\n

The only functions from the haven<\/code> package we’ll use here are read_sav()<\/code> and as_factor().<\/code> All other functions referenced in this post come from packages like dplyr<\/code>, forcats <\/code>or ggplot2<\/code>, which are loaded automatically with the tidyverse<\/a>package.<\/p>\n\n\n\n

If you don\u2019t already have these packages, you\u2019ll need to install them in order to run the code in this post. Even if you have<\/em> already installed them, you can update them by running the below code:<\/p>\n\n\n\n

install.packages(\"tidyverse\")\ninstall.packages(\"haven\")<\/code><\/code><\/pre>\n\n\n\n
Next, load the packages in with the library()<\/code> function:<\/p>\n\n\n\n
library(tidyverse)\n#loads all \"core\" tidyverse packages like\n#dplyr, tidyr, forcats, and ggplot2\nlibrary(haven)<\/code><\/code><\/pre>\n\n\n\n
Pipes<\/h4>\n\n\n\n
One of the most appealing features of the tidyverse is its emphasis on making code easier to read and understand. A key tool for improving the readability of tidyverse code is the pipe<\/em>, a set of three characters that looks like this: %>%<\/code>. Placing a pipe to the right of an object (such as data %>%<\/code>) tells R to take the object on its left and send it to its right. This allows you to chain multiple commands together without getting lost in a hailstorm of parentheses.<\/p>\n\n\n\n
Let\u2019s use a pop culture dataset to illustrate this point. The starwars<\/code> dataset that comes with the tidyverse package contains information on the characters from the movies, made available through the Star Wars API<\/a>. Suppose we want to filter the dataset to all characters for whom a height is known, arrange the dataset in order of decreasing height and then change the height variable from centimeters to inches.<\/p>\n\n\n\n
Here is the code to do this without using pipes:<\/p>\n\n\n\n
mutate(arrange(filter(starwars, !is.na(height)), desc(height)), height = 0.393701 * height)<\/code><\/code><\/pre>\n\n\n\n
And here\u2019s the code with the pipe:<\/p>\n\n\n\n
starwars %>%\nfilter(!is.na(height)) %>%\narrange(desc(height)) %>%\nmutate(height = 0.393701 * height)<\/code><\/code><\/pre>\n\n\n\n
As far as R is concerned, these two pieces of code do exactly the same thing. But using a series of pipes makes it much easier for humans to read and understand exactly what\u2019s going on, and in what order. The code above can be easily read as a series of steps: take the starwars dataset, filter it to cases where height is not missing, arrange the remaining cases by height in descending order, and, finally, change the height variable to inches.<\/p>\n\n\n\n
Readability makes it easier to share your code with other people. Even if you\u2019ll never share your code with anyone else, this approach will make it easier for you to understand your own code if you ever need to revisit it weeks, months or even years later when the details are no longer fresh in your memory. (This can also help you or your team avoid technical debt<\/a>.)<\/p>\n\n\n\n
Beyond readability, pipes make your code easier to maintain. It\u2019s easy to remove a step, insert a new step or change the order when all of the commands are in a sequential series of pipes, rather than nested.<\/p>\n\n\n\n
Loading the data into R<\/h4>\n\n\n\n
Let\u2019s shift gears and return our attention to using tidyverse<\/code> tools to analyze Pew Research Center survey data. You can download the Center\u2019s datasets via this link<\/a>, and you can learn more here<\/a> about the kind of data we release and how to access it.<\/p>\n\n\n\n
For the purposes of this backgrounder, we\u2019ll use data from our April 2017 Political survey<\/a>. We\u2019ll walk through how to use tidyverse<\/code> tools to calculate weighted survey estimates (specifically, approval of President Donald Trump) broken out by other variables in our dataset (education, race\/ethnicity and generation).<\/p>\n\n\n\n
The first step is to load the dataset into your R environment. Almost all of the datasets available to download from the Center are stored as .sav (SPSS) files. The haven package can import datasets created by SPSS and a number other programs such as Stata and SAS.<\/p>\n\n\n\n
There are a few features of SPSS datasets that require special handling when loading data into R. SPSS files often contain variables with both character labels and numeric codes that are not necessarily sequential (e.g. for party ID, 1 = Republican, 2 = Democrat, 9 = Refused). SPSS also allows variables to have multiple, user-defined missing values. This sort of variable metadata is helpful, but it doesn\u2019t quite align with any of R\u2019s standard data types.<\/p>\n\n\n\n
The haven<\/code> package handles these differences by importing variables in a custom format called \"labelled\"<\/code> and leaving the user to decide which of the numeric codes or value labels he or she wants to work with in R.<\/p>\n\n\n\n
We\u2019ll import the data using haven\u2019s read_sav()<\/code> function. We’ll set user_na = TRUE<\/code> to ensure that responses such as “Don’t know” or “Refused” aren\u2019t automatically converted to missing values. In this case, we also want to convert any labelled<\/code> variables into factors<\/a> so that we can work with value labels instead of the numeric codes (e.g. “Republican” instead of “1”). We do this by piping (%>%<\/code>) the entire dataset to haven’s as_factor()<\/code> function which converts any labelled variables in the dataset to factors.<\/p>\n\n\n\n
Apr17 <- read_sav(\"Apr17 public.sav\",\nuser_na = TRUE) %>%\nas_factor()<\/code><\/code><\/pre>\n\n\n\n
Adding variables with mutate<\/mark><\/code><\/h4>\n\n\n\n
The first question (q1<\/code>) on the April 2017 survey asked whether or not respondents approved of Trump\u2019s performance as president so far. The next question (q1a<\/code>) asked respondents how strongly they approved or disapproved (“Very strongly,” “Not so strongly,” “Don’t know\/Refused (VOL.)”). We want to use these two variables to create a new variable that combines q1<\/code> and q1a<\/code>. This sort of re-coding is extremely common when working with survey data, but it can be a surprisingly difficult task when using only base R functions. The dplyr<\/code>package, which is loaded automatically as part of the tidyverse<\/code> package, includes a number of tools that make these sorts of data processing tasks much easier.<\/p>\n\n\n\n
The most general of these is the mutate()<\/a> function, which either adds new variables to a data frame or modifies existing variables. We\u2019ll use mutate()<\/code> in the code below to create a variable called trump_approval. The actual re-coding is done with the case_when()<\/a> function, which evaluates a series of conditions and returns the value associated with the first one that applies.<\/p>\n\n\n\n
Apr17 <- Apr17 %>%\nmutate(trump_approval = case_when(\n\nq1 == \"Approve\" & q1a == \"Very strongly\" ~ \"Strongly approve\",\nq1 == \"Approve\" & q1a != \"Very strongly\" ~ \"Not strongly approve\",\nq1 == \"Disapprove\" & q1a == \"Very strongly\" ~ \"Strongly disapprove\",\nq1 == \"Disapprove\" & q1a != \"Very strongly\" ~ \"Not strongly disapprove\",\nq1 == \"Don't know\/Refused (VOL.)\" |\nq1a == \"Don't know\/Refused (VOL.)\" ~ \"Refused\"\n) #this parentheses closes call to\n#case_when and sends it to\n#fct_relevel with %>%\n%>%\nfct_relevel(\"Strongly approve\",\n\"Not strongly approve\",\n\"Not strongly disapprove\",\n\"Strongly disapprove\",\n\"Refused\"\n) #this parentheses closes our call to fct_relevel\n) #this parentheses closes our call to mutate<\/code><\/code><\/pre>\n\n\n\n
We supply a series of formulas to the case_when()<\/code> function. Each formula is an \u201cif-then\u201d statement, where the left side (everything to the left of ~<\/code>) describes a logical condition and the right side provides the value to return if that condition is true. As a reminder, you can type ?case_when() to access the help page (this works for any function in R, not just case_when()<\/code>).<\/p>\n\n\n\n
The first line of the call to case_when()<\/code> is: q1 == \"Approve\" & q1a == \"Very strongly\" ~ \"Strongly approve\".<\/code> This can be read to mean that respondent said “Approve” to q1<\/code> and answered “Very strongly” to q1a<\/code>, so we\u2019ll code them as “Strongly approve” in our new trump_approval<\/code> variable. We use five different clauses to make these new categories and add the trump_approval variable to our dataset.<\/p>\n\n\n\n
Because case_when()<\/code> returns character variables when used this way, the last step is to pipe the variable to fct_relevel()<\/a> from the forcats<\/code> package. This converts trump_approval<\/code> to a factor and orders the categories from “Strongly approve” to “Refused.”<\/p>\n\n\n\n
We can run the table<\/code> command on our new variable to verify that everything worked as intended.<\/p>\n\n\n\n
table(Apr17$trump_approval, Apr17$q1)\n##                          \n##                  Approve      Disapprove              Don't know\/          \n##                                                    Refused (VOL.)\n##Strongly approve          476          0                         0\n##Not strongly approve      130          0                         0\n##Not strongly disapprove     0        143                         0\n##Strongly disapprove         0        676                         0\n##Refused                     0          0                        76<\/code><\/pre>\n\n\n\n
Having created our new trump_approval<\/code> variable, we’d like to see how it breaks down according to a few different demographic characteristics: educational attainment (educ2<\/code> in the Apr17 dataset), race\/ethnicty (racethn<\/code>) and generation (gen5<\/code>).<\/p>\n\n\n\n
But educ2<\/code> has nine categories, some of which don\u2019t have very many respondents. We thus want to collapse them into fewer categories with larger numbers of respondents. We can do this with the forcats<\/code> function fct_collapse()<\/code>. We use mutate()<\/code> again to make new variables.<\/p>\n\n\n\n
Let\u2019s first check the answer options of educ2 to see the categories we need to collapse. Since we used as_factor()<\/code> when we read the dataset in, educ2<\/code> is a factor variable. So, we can see the answer options by using the levels()<\/code>function.<\/p>\n\n\n\n
levels(Apr17$educ2)\n\n## [1] \"Less than high school (Grades 1-8 or no formal schooling)\"                                \n## [2] \"High school incomplete (Grades 9-11 or Grade 12 with NO diploma)\"                         \n## [3] \"High school graduate (Grade 12 with diploma or GED certificate)\"                          \n## [4] \"Some college, no degree (includes some community college)\"                                \n## [5] \"Two year associate degree from a college or university\"                                   \n## [6] \"Four year college or university degree\/Bachelor's degree (e.g., BS, BA, AB)\"              \n## [7] \"Some postgraduate or professional schooling, no postgraduate degree\"                      \n## [8] \"Postgraduate or professional degree, including master's, doctorate, medical or law degree\"\n## [9] \"Don't know\/Refused (VOL.)\"<\/code><\/pre>\n\n\n\n
Now that we know the levels, we can collapse them into fewer categories. The first argument to fct_collapse()<\/code> is the variable whose categories you want to collapse, in our case educ2<\/code>. We assign new categories on the left, like “High school grad or less”, in the below example. We use the c()<\/code> function to tell fct_collapse()<\/code> these categories belong in our new “High school grad or less” category. Note that you need to use the exact names of these categories.<\/p>\n\n\n\n
Apr17 <- Apr17 %>%\nmutate(educ_cat = fct_collapse(educ2,\n\"High school grad or less\" = c(\n\"Less than high school (Grades 1-8 or no formal schooling)\",\n\"High school incomplete (Grades 9-11 or Grade 12 with NO diploma)\",\n\"High school graduate (Grade 12 with diploma or GED certificate)\"\n),\n\"Some college\" = c(\n\"Some college, no degree (includes some community college)\",\n\"Two year associate degree from a college or university\"\n),\n\"College grad+\" = c(\n\"Four year college or university degree\/Bachelor's degree (e.g., BS, BA, AB)\",\n\"Some postgraduate or professional schooling, no postgraduate degree\",\n\"Postgraduate or professional degree, including master's, doctorate, medical or law degree\"\n)\n) #this parentheses closes our call\n#to fct_collapse\n) #this parentheses closes our call to mutate<\/code><\/pre>\n\n\n\n
Getting weighted estimates with group_by and summarise<\/h4>\n\n\n\n
Now that we have created and re-coded the variables we want to estimate, we can use the dplyr<\/code> functions group_by()<\/a> and summarise()<\/a> to produce some weighted summaries of the data. We can use group_by()<\/code> to tell R to group the Apr17 dataset by these variables and then summarise()<\/code> to create summary statistics among those groups.<\/p>\n\n\n\n
To make sure our estimates are representative of the population, we need to use the survey weights (variable named weight<\/code>) included in our dataset, as is the case in almost all of the Center’s datasets. For the total sample, we can calculate weighted percentages by adding up the respondent weights for each category and dividing by the sum of the weights for the whole sample.<\/p>\n\n\n\n
The first step is to group_by()<\/code> trump_approval and then use summarise()<\/code> to get the sum of the weights for each group. Then we use mutate()<\/code> to convert these totals into percentages.<\/p>\n\n\n\n
So the first step in code is this:<\/p>\n\n\n\n
trump_approval <- Apr17 %>%\ngroup_by(trump_approval) %>%\nsummarise(weighted_n = sum(weight))<\/code><\/code><\/pre>\n\n\n\n
We can see that group_by()<\/code> and summarise()<\/code> gives us a table with one row for each group and one column for the summary variables that we created. If we look at the trump_approval<\/code> object that we just created, we can see we have a column for each of the categories in our trump_approval<\/code> variable (that we passed to group_by()<\/code>) and a column we created called weighted_n<\/code>. The weighted_n<\/code>column is the weighted sum of each category in trump_approval<\/code>.<\/p>\n\n\n\n
But since we want show proportions, we have another step to do. We\u2019ll use mutate()<\/code> to add a column called weighted_group_size<\/code> that is the sum of weighted_n<\/code>. Then we just divide weighted_n<\/code> by weighted_group_size<\/code> and store that in a column called weighted_estimate<\/code>, which gives us our weighted proportions.<\/p>\n\n\n\n
trump_approval <- Apr17 %>% \n##group by trump_approval to calculate weighted totals\n##by taking the sum of the weights\n  group_by(trump_approval) %>% \n  summarise(weighted_n = sum(weight)) %>% \n##add the weighted_group_size to get the total weighted n and \n##divide weighted_n by weighted_group_size to get the proportions \n  mutate(weighted_group_size = sum(weighted_n), \n         weighted_estimate = weighted_n \/ weighted_group_size\n         )\n\ntrump_approval## # A tibble: 5 x 4\n\n##trump_approval weighted_n weighted_group_size weighted_estimate\n##  <fct>          <dbl>               <dbl>             <dbl>\n## 1 Strongly\n## approve         1293.                4319.            0.299 \n## 2 Not strongly\n## approve          408.                4319.            0.0945\n## 3 Not strongly\n## disapprove       458.               4319.             0.106 \n## 4 Strongly\n## disapprove      1884.               4319.            0.436 \n## 5 Refused        275.               4319.            0.0636<\/code><\/pre>\n\n\n\n
To add a subgroup variable and get a weighted crosstab, we can use the same procedure with one slight addition. Instead of going straight from summarise()<\/code>to mutate()<\/code> and adding our group sizes and proportions, we have to tell mutate()<\/code> to calculate the weighted_group_size<\/code> of educ_cat<\/code>. This gives the weighted number of respondents that are in each category of educ_cat<\/code> (“High school grad or less,” “Some college,” “College grad+,” “Don’t know\/Refused (VOL.)”)<\/p>\n\n\n\n
trump_estimates_educ <- Apr17 %>% \n##group by educ and trump approval to get weighted n's per group\n  group_by(educ_cat, trump_approval) %>% \n##calculate the total number of people in each answer and education category using survey weights (weight)\n  summarise(weighted_n = sum(weight)) %>% \n##group by education to calculate education category size\n  group_by(educ_cat) %>%\n##add columns for total group size and the proportion\n  mutate(weighted_group_size = sum(weighted_n),\n         weighted_estimate = weighted_n\/weighted_group_size)\n\ntrump_estimates_educ## # A tibble: 17 x 5\n## # Groups:   educ_cat [4]\n## educ_cat  trump_approval  weighted_n weighted_group_\u2026 weighted_estima\u2026\n##    <fct>       <fct>                <dbl>            <dbl>            <dbl>\n##  1 High schoo\u2026 Strongly appro\u2026     550.             1710.            0.322 \n##  2 High schoo\u2026 Not strongly a\u2026     207.             1710.            0.121 \n##  3 High schoo\u2026 Not strongly d\u2026     221.             1710.            0.129 \n##  4 High schoo\u2026 Strongly disap\u2026     593.             1710.            0.347 \n##  5 High schoo\u2026 Refused             140.             1710.            0.0817\n##  6 Some colle\u2026 Strongly appro\u2026     404.             1337.            0.302 \n##  7 Some colle\u2026 Not strongly a\u2026     111.             1337.            0.0833\n##  8 Some colle\u2026 Not strongly d\u2026     128.             1337.            0.0959\n##  9 Some colle\u2026 Strongly disap\u2026     605.             1337.            0.453 \n## 10 Some colle\u2026 Refused              88.1            1337.            0.0659\n## 11 College gr\u2026 Strongly appro\u2026     336.             1258.            0.267 \n## 12 College gr\u2026 Not strongly a\u2026      89.9            1258.            0.0715\n## 13 College gr\u2026 Not strongly d\u2026     109.             1258.            0.0870\n## 14 College gr\u2026 Strongly disap\u2026     676.             1258.            0.537 \n## 15 College gr\u2026 Refused              47.0            1258.            0.0373\n## 16 Don't know\u2026 Strongly appro\u2026       2.97             13.3           0.224 \n## 17 Don't know\u2026 Strongly disap\u2026      10.3              13.3           0.776<\/code><\/pre>\n\n\n\n
This may seem like a lot of work to create one weighted crosstab, and there are definitely easier ways to do this if that\u2019s all you need. However, it gives us a great deal of flexibility to create any number of other summary statistics besides weighted percentages. Because the summaries are stored as a data.frame<\/code>, it\u2019s easy to convert them into graphs or use them in other analyses. (Technically the output is a tibble<\/a>, tidyverse’s approach to data frames.)<\/p>\n\n\n\n
Perhaps most usefully, we can use this approach to create summaries for a large number of demographics or outcome variables simultaneously. In this example, we\u2019re not only interested in breaking out trump_approval<\/code> by educ_cat<\/code>, but also the racethn<\/code> and gen5<\/code> variables. With some reshaping of the data using the gather()<\/code> function, we can use the procedure to get all of our subgroup summaries at once.<\/p>\n\n\n\n
Rearranging data with gather()<\/mark><\/strong><\/code><\/h4>\n\n\n\n
The first step to making the process a bit simpler is to take the Apr17 dataset and select down to only the columns we need for our analysis using the select()<\/code>function. The select()<\/code> function is like a Swiss army knife of keeping, rearranging or dropping (using -<\/code>) columns. There are also a number of helper functions like starts_with()<\/code>, ends_with()<\/code>and matches()<\/code> that make it easy to select a number of columns with a certain naming pattern instead of naming each column explicitly. You can also rename variables with select()<\/code> by supplying a new variable name before an =<\/code> and the old variable name.<\/p>\n\n\n\n
Apr17 <- Apr17 %>%\nselect(resp_id = psraid,\nweight,\ntrump_approval,\neduc_cat, racethn, gen5)<\/code><\/pre>\n\n\n\n
This line of code uses the select()<\/code> function to limit the Apr17 dataset to only the columns we need for our analysis. Just to illustrate how renaming works, we\u2019ll change the name of the respondent identifier from psraid<\/code> to resp_id<\/code>. We\u2019ll retain the survey weight (weight<\/code>), the trump_approval<\/code> variable and the subgroup variables we are interested in (educ_cat<\/code>, racethn<\/code> and gen5<\/code>).<\/p>\n\n\n\n
head(Apr17)\n\n## # A tibble: 6 x 6\n##   resp_id weight trump_approval     educ_cat         racethn     gen5      \n##     <dbl>  <dbl> <fct>              <fct>            <fct>       <fct>     \n## 1  100005  2.94 Strongly disappro\u2026 College grad+  Black, non\u2026 Silent (1\u2026\n## 2  100010  1.32 Not strongly appr\u2026 Some college   Hispanic    Boomer (1\u2026\n## 3  100021  1.24 Strongly disappro\u2026 College grad+  White, non\u2026 Silent (1\u2026\n## 4  100028  4.09 Strongly approve   Some college   White, non\u2026 Boomer (1\u2026\n## 5  100037  1.12 Refused            College grad+  White, non\u2026 Boomer (1\u2026\n## 6  100039  6.68 Strongly disappro\u2026 High school gra\u2026 Black, non\u2026 Boomer (1\u2026<\/code><\/pre>\n\n\n\n
The next step is to rearrange the data in a way that makes it easy to calculate the weighted summary statistics by each demographic group. We\u2019ll do this with the gather()<\/a> function, which reshapes data from \u201cwide\u201d format to \u201clong\u201d \u2014 an extremely useful function when working with data. (Recently, the team behind the tidyr<\/code> package introduced functions called  pivot_longer()<\/a><\/code> and <\/a>pivot_wider()<\/a><\/code>. These functions will not replace gather()<\/code> and spread()<\/code>, but are designed to be more useful and intuitive.)<\/p>\n\n\n\n
In wide format, there is one row per person and one column per variable. In long format, there are multiple rows per person: one for each of the demographic variables we want to analyze. The separate demographic columns are replaced by a pair of columns; a \u201ckey\u201d column and a \u201cvalue\u201d column. Here, the \u201ckey\u201d column is called subgroup_variable<\/code>, and it identifies which demographic variable is associated with that row (one of educ_cat<\/code>, racethn<\/code>, or gen5<\/code>). The “value” column is called subgroup<\/code> and identifies the specific demographic category to which the person belongs.<\/p>\n\n\n\n
Apr17_long <- Apr17 %>%\ngather(key = subgroup_variable, value = subgroup,\neduc_cat, racethn, gen5)<\/code><\/pre>\n\n\n\n
In this code, we\u2019re telling gather()<\/code> to name the key variable \"subgroup_variable\"<\/code> and to name the value column \"subgroup.\"<\/code> The remaining arguments specify the names of the columns that we want to gather into rows. After the gather()<\/code> step, we\u2019ve gone from a wide dataset with 1,501 rows to a long dataset 4,503 rows (1,501 respondents x 3 demographic variables).<\/p>\n\n\n\n
\"\"<\/a><\/figure>\n\n\n\n
If you want to read more about gather()<\/code>, I recommend starting with this tweet<\/a>by Alison Hill<\/a> from the WeAreRLadies<\/a> twitter account. You can also go to the tidy data<\/a> chapter in the previously mentioned R for Data Science<\/em> book.<\/p>\n\n\n\n
Now that we have our data arranged in this format, getting weighted summaries for all three subgroup variables is just a matter adding another grouping variable. Whereas before we grouped by educ_cat<\/code>, we can now use our newly created subgroup_variable<\/code> and subgroup<\/code> columns instead.<\/p>\n\n\n\n
trump_estimates <- Apr17_long %>%\n#group by subgroup_variable, subgroup, and trump approval\ngroup_by(subgroup_variable, subgroup, trump_approval) %>%\n#calculate the total number of people in each answer and education #category using survey weights (weight)\nsummarise(weighted_n = sum(weight)) %>%\n#group by subgroup only to calculate subgroup category size\ngroup_by(subgroup) %>%\n#add columns for total group size and the proportion\nmutate(weighted_group_size = sum(weighted_n),\nweighted_estimate = weighted_n\/weighted_group_size)<\/code><\/code><\/pre>\n\n\n\n
Since we are only really interested in the proportions, we\u2019ll remove the weighted_total and weighted_group_size columns using the select() function. Including the \u2014 before a column name tells select() to drop that column.<\/p>\n\n\n\n
trump_estimates <- trump_estimates %>%\nselect(-weighted_n, -weighted_group_size)<\/code><\/pre>\n\n\n\n
Here\u2019s what we end up with.<\/p>\n\n\n\n
trump_estimates\n## # A tibble: 70 x 4\n## # Groups: subgroup [14]\n## subgroup_variable subgroup trump_approval weighted_estima\u2026\n##     <chr>               <chr>                <fct>              <dbl>\n## 1 educ_cat           College grad+      Strongly approve        0.267\n## 2 educ_cat           College grad+      Not strongly appr\u2026     0.0715\n## 3 educ_cat           College grad+      Not strongly disa\u2026     0.0870\n## 4 educ_cat           College grad+      Strongly disappro\u2026      0.537\n## 5 educ_cat           College grad+          Refused            0.0373\n## 6 educ_cat         Don't know\/Refuse\u2026   Strongly approve       0.0203\n## 7 educ_cat         Don't know\/Refuse\u2026   Strongly disappro\u2026     0.0704\n## 8 educ_cat         High school grad \u2026   Strongly approve        0.322\n## 9 educ_cat         High school grad \u2026   Not strongly appr\u2026      0.121\n## 10 educ_cat        High school grad \u2026   Not strongly disa\u2026      0.129\n## # \u2026 with 60 more rows<\/code><\/pre>\n\n\n\n
Arranging the data this way makes it easy to create plots or tables of the variables of interest. Below is an example of a simple plot of the estimates we created in this post. We use the filter()<\/a> function to remove the \u201cRefused\u201d category in the trump_approval<\/code> variable and in any of our subgroup variables. Then, we pipe that data into the plotting code from ggplot2<\/a>.<\/p>\n\n\n\n
trump_estimates %>%\nfilter(trump_approval != \"Refused\") %>%\nfilter(!(subgroup %in%\nc(\"Don't know\/Refused (VOL.)\", \"DK\/Ref\"))) %>%\nggplot(\naes(\nx = weighted_estimate,\ny = subgroup\n)\n) +\ngeom_point() +\nscale_x_continuous(limits = c(0, .8),\nbreaks = seq(0, .6, by = .2),\nlabels = scales::percent(\nseq(0, .6, by = .2), accuracy = 1)\n) +\nfacet_grid(cols = vars(trump_approval),\nrows = vars(subgroup_variable),\nscales = \"free_y\",\nspace = \"free\"\n) +\ntheme_bw() +\ntheme(axis.title.y = element_blank())<\/code><\/pre>\n\n\n\n
\"\"<\/a><\/figure>\n\n\n\n
Here is all the necessary code used in the post:<\/p>\n\n\n\n
#code from start to finish \n\n##install or update packages if neccessary\n#install.packages(\"tidyverse\")\n#install.packages(\"haven\") \n\n##load packages in \nlibrary(tidyverse) #loads all \"core\" tidyverse packages like dplyr, tidyr, forcats, and ggplot2\nlibrary(haven) \n\n##read dataset in with value labels (as_factor)\nApr17 <- read_sav(\"Apr17 public.sav\", user_na = TRUE) %>% as_factor()\n\n##create trump_approval variable and relevel it \nApr17 <- Apr17 %>% \n  mutate(trump_approval = case_when(\n    q1 == \"Approve\" & q1a == \"Very strongly\" ~ \"Strongly approve\",\n    q1 == \"Approve\" & q1a != \"Very strongly\" ~ \"Not strongly approve\",\n    q1 == \"Disapprove\" & q1a == \"Very strongly\" ~ \"Strongly disapprove\",\n    q1 == \"Disapprove\" & q1a != \"Very strongly\" ~ \"Not strongly disapprove\",\n    q1 == \"Don't know\/Refused (VOL.)\" | q1a == \"Don't know\/Refused (VOL.)\" ~ \"Refused\"\n  ) #this parentheses closes our call to case_when\n  %>% #and then sends it to fct_relevel with %>% \n    fct_relevel(\"Strongly approve\",\n                \"Not strongly approve\",\n                \"Not strongly disapprove\",\n                \"Strongly disapprove\",                             \n                \"Refused\"                \n    ) #this parentheses closes our call to fct_relevel\n  ) #this parentheses closes our call to mutate\n\n## collapse education variable into 3 categories\nApr17 <- Apr17 %>% \n  mutate(educ_cat = fct_collapse(educ2,\n                                 \"High school grad or less\" = c(\n                                   \"Less than high school (Grades 1-8 or no formal schooling)\",\n                                   \"High school incomplete (Grades 9-11 or Grade 12 with NO diploma)\",                         \n                                   \"High school graduate (Grade 12 with diploma or GED certificate)\"\n                                 ),\n                                 \"Some college\" = c(\n                                   \"Some college, no degree (includes some community college)\",                                \n                                   \"Two year associate degree from a college or university\"\n                                 ),\n                                 \"College grad+\" = c(\n                                   \"Four year college or university degree\/Bachelor's degree (e.g., BS, BA, AB)\",              \n                                   \"Some postgraduate or professional schooling, no postgraduate degree\",                      \n                                   \"Postgraduate or professional degree, including master's, doctorate, medical or law degree\"\n                                 )\n  ) #this parentheses closes our call to fct_collapse\n  ) #this parentheses closes our call to mutate\n\n##get trump_approval weighted totals\ntrump_approval <- Apr17 %>% \n  group_by(trump_approval) %>% \n  summarise(weighted_n = sum(weight))\n\n##get trump_approval weighted proportions\ntrump_approval <- Apr17 %>% \n  ##group by trump_approval to calculated weighted totals by taking the sum of the weights\n  group_by(trump_approval) %>% \n  summarise(weighted_n = sum(weight)) %>% \n  ##add the weighted_group_size to get the total weighted n and \n  ##divide weighted_n by weighted_group_size to get the proportions \n  mutate(weighted_group_size = sum(weighted_n), \n         weighted_estimate = weighted_n \/ weighted_group_size\n  )\n\n\n##get trump_approval by education \ntrump_estimates_educ <- Apr17 %>% \n  #group by educ and trump approval to get weighted n's per group\n  group_by(educ_cat, trump_approval) %>% \n  #calculate the total number of people in each answer and education category using survey weights (weight)\n  summarise(weighted_n = sum(weight)) %>% \n  #group by education to calculate education category size\n  group_by(educ_cat) %>%\n  #add columns for total group size and the proportion\n  mutate(weighted_group_size = sum(weighted_n),\n         weighted_estimate = weighted_n\/weighted_group_size)\n\n##select only columns interested in for this analysis\n###rename psraid to resp_id\nApr17 <- Apr17 %>% \n  select(resp_id = psraid, weight, trump_approval, educ_cat, racethn, gen5)\n\n##create Apr_17 long with gather\nApr17_long <- Apr17 %>% \n  #gather educ_cat, racethn, gen5 into two columns: \n  ##a key  called \"subgroup variable\" (educ_cat, racethn, gen5)\n  ##and a value called \"subgroup\" \n  gather(key = subgroup_variable, value = subgroup, educ_cat, racethn, gen5) \n\n##get weighted estimates for every subgroup \ntrump_estimates <- Apr17_long %>% \n  #group by subgroup_variable, subgroup, and trump approval to get weighted n of approval\/disapproval for all subgroup cats\n  group_by(subgroup_variable, subgroup, trump_approval) %>% \n  #calculate the total number of people in each answer and education category using survey weights (weight)\n  summarise(weighted_n = sum(weight)) %>% \n  #group by subgroup only to calculate subgroup category size\n  group_by(subgroup) %>%\n  #add columns for total group size and the proportion\n  mutate(weighted_group_size = sum(weighted_n),\n         weighted_estimate = weighted_n\/weighted_group_size)\n\n#only want proportions so select out total categories\ntrump_estimates <- trump_estimates %>% \n  select(-weighted_n, -weighted_group_size) \n\n##create plot\ntrump_estimates %>% \n  ##remove \"Refused\" category for Trump Approval\n  filter(trump_approval != \"Refused\") %>% \n  ##remove Refused categories in our subgroup values\n  filter(!(subgroup %in% c(\"Don't know\/Refused (VOL.)\", \"DK\/Ref\"))) %>% \n  ggplot(\n    aes(\n      x = weighted_estimate,\n      y = subgroup\n    )\n  ) +\n  geom_point() + \n  scale_x_continuous(limits = c(0, .8),\n                     breaks = seq(0, .6, by = .2),\n                     labels = scales::percent(seq(0, .6, by = .2), accuracy = 1)\n  ) +\n  facet_grid(cols = vars(trump_approval),\n             rows = vars(subgroup_variable), \n             scales = \"free_y\",\n             space = \"free\"\n  ) +\n  theme_bw() +\n  theme(axis.title.y = element_blank())<\/code><\/pre>\n\n\n\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
I wrote an introductory blog post about how to access and analyze Pew Research Center survey data with R, a free, open-source software for statistical analysis. The post showed how to perform tasks using the survey package.<\/p>\n","protected":false},"author":655,"featured_media":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"sub_headline":"","sub_title":"","_prc_public_revisions":[],"_ppp_expiration_hours":0,"_ppp_enabled":false,"ai_generated_summary":"","relatedPosts":[],"_prc_fork_parent":0,"_prc_fork_status":"","_prc_active_fork":0,"datacite_doi":"","datacite_doi_citation":"","_prc_seo_qr_attachment_id":0,"spoken_article_player_enabled":true,"displayBylines":true,"footnotes":"","prc_watchers":[]},"categories":[357],"bylines":[779],"collection":[],"_post_visibility":[],"decoded-category":[531,532],"formats":[],"_fund_pool":[],"languages":[],"regions-countries":[],"research-teams":[524],"workflow-status":[],"class_list":["post-111603","decoded","type-decoded","status-publish","hentry","category-survey-methods","bylines-nick-hatley","decoded-category-coding-how-to","decoded-category-survey-methods","research-teams-decoded"],"label":"Decoded","post_parent":0,"word_count":5051,"canonical_url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/2019\/06\/12\/using-tidy-verse-tools-with-pew-research-center-survey-data-in-r\/","art_direction":{"A1":{"id":125958,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/06.12.2019_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/06.12.2019_feature.png?w=564&h=317&crop=1","width":564,"height":317,"caption":"","chartArt":false},"A2":{"id":125958,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/06.12.2019_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/06.12.2019_feature.png?w=268&h=151&crop=1","width":268,"height":151,"caption":"","chartArt":false},"A3":{"id":125958,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/06.12.2019_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/06.12.2019_feature.png?w=194&h=110&crop=1","width":194,"height":110,"caption":"","chartArt":false},"A4":{"id":125958,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/06.12.2019_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/06.12.2019_feature.png?w=268&h=151&crop=1","width":268,"height":151,"caption":"","chartArt":false},"XL":{"id":125958,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/06.12.2019_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/06.12.2019_feature.png?w=720&h=405&crop=1","width":720,"height":405,"caption":"","chartArt":false},"social":{"id":125958,"rawUrl":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/06.12.2019_feature.png","url":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/06.12.2019_feature.png?w=1200&h=628&crop=1","width":1200,"height":628,"caption":"","chartArt":false}},"_embeds":[],"watchers":[],"table_of_contents":[],"datacite_doi":"","prc_seo_data":{"title":"Using tidyverse tools with Pew Research Center survey data in R","description":"I wrote an introductory blog post about how to access and analyze Pew Research Center survey data with R, a free, open-source software for statistical analysis. The post showed how to perform tasks using the survey package.","og_title":"Using tidyverse tools with Pew Research Center survey data in R","og_description":"I wrote an introductory blog post about how to access and analyze Pew Research Center survey data with R, a free, open-source software for statistical analysis. The post showed how to perform tasks using the survey package.","schema_type":"Article","noindex":false,"canonical_url":"","primary_terms":{"category":43},"custom_schema":[],"og_image":125958,"indexnow_submitted_at":null,"gsc_index_status":null},"prepublish_checks":{},"jetpack_sharing_enabled":true,"relatedPostsOrdered":[],"bylinesOrdered":[{"key":"_r6x699i4u","termId":779}],"acknowledgementsOrdered":[],"_links":{"self":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111603","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded"}],"about":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/types\/decoded"}],"author":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/users\/655"}],"replies":[{"embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/comments?post=111603"}],"version-history":[{"count":2,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111603\/revisions"}],"predecessor-version":[{"id":138527,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111603\/revisions\/138527"}],"wp:attachment":[{"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/media?parent=111603"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/categories?post=111603"},{"taxonomy":"bylines","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/bylines?post=111603"},{"taxonomy":"collection","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/collection?post=111603"},{"taxonomy":"_post_visibility","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_post_visibility?post=111603"},{"taxonomy":"decoded-category","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded-category?post=111603"},{"taxonomy":"formats","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/formats?post=111603"},{"taxonomy":"_fund_pool","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_fund_pool?post=111603"},{"taxonomy":"languages","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/languages?post=111603"},{"taxonomy":"regions-countries","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/regions-countries?post=111603"},{"taxonomy":"research-teams","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/research-teams?post=111603"},{"taxonomy":"workflow-status","embeddable":true,"href":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/workflow-status?post=111603"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}