{"id":111702,"date":"2018-06-27T12:21:00","date_gmt":"2018-06-27T17:21:00","guid":{"rendered":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/\/\/using-apis-to-collect-website-data\/"},"modified":"2024-04-14T04:10:45","modified_gmt":"2024-04-14T09:10:45","slug":"using-apis-to-collect-website-data","status":"publish","type":"decoded","link":"https:\/\/alpha.pewresearch.org\/pewresearch-org\/decoded\/2018\/06\/27\/using-apis-to-collect-website-data\/","title":{"rendered":"Using APIs to collect website data"},"content":{"rendered":"\n
\"\"<\/a><\/figure>\n\n\n\n

Many researchers are interested in obtaining new kinds of data directly from websites. But collecting large amounts of data from a website can be impractical. When done manually, this approach is prone to human error. When automated, it may violate a website\u2019s terms of service.<\/p>\n\n\n\n

Instead, researchers typically rely on a tool called an API, which stands for Application Programming Interface. Web APIs provide a means of communication between websites and users, structured by rules. In particular, they allow users to obtain clearly defined kinds of data quickly, by requesting it directly from the database underlying a particular website. (This isn\u2019t the only kind of API. For example, smartphone apps frequently use APIs to send data back and forth between your device and the application\u2019s database. In this blog post, however, we\u2019re focusing on APIs that deliver data from websites to users.)<\/p>\n\n\n\n

Many major companies, as well as government agencies, have created public APIs. These organizations provide easy access to their data to encourage developers to use their platforms. At Pew Research Center, we regularly use APIs to collect information for the studies we produce. For example, we\u2019ve used APIs in reports about online congressional communication<\/a>, bots in social media<\/a>, Google searches about guns<\/a>, science pages on Facebook<\/a>, and public comments on government policy<\/a>.<\/p>\n\n\n\n

How APIs provide data<\/h3>\n\n\n\n

APIs provide data in various formats, but JSON is the most popular. JSON is a useful file format for structuring data because it preserves more hierarchical structure and meta information about a dataset than tab- or comma-separated files (such as .tsv and .csv files). Depending on the programming language you\u2019re using for your data analysis, there are libraries that make JSON data manipulation intuitive. For example, in the R statistical language, you can use the jsonlite<\/code> library to transform data from APIs into familiar R objects and classes.<\/p>\n\n\n\n

APIs can be used in different programming languages<\/h3>\n\n\n\n

In order to interact with an API efficiently across a variety of computer programs written in different languages (such as Java, Python, R, etc), researchers rely on API \u201cwrappers.\u201d These tools \u2014 which exist for most popular APIs \u2014 allow a researcher to request data directly from an API while using their own programming language of choice. Below, we use the World Bank\u2019s API as an example and access the data using R.<\/p>\n\n\n\n

Example: How to interact with the World Bank\u2019s API<\/h3>\n\n\n\n

The World Bank API<\/a> gives researchers access to hundreds of variables about the health, wealth and culture of countries around the world. To show how to use this API, we can write a script to get a couple of variables and plot them. In this case, we\u2019ll search for variables using the identifying codes provided by the World Bank, and then pull the variables into memory in R.<\/p>\n\n\n\n

The code below uses the WDI<\/code> package<\/a>, an R package that includes a wrapper for the World Bank API. (A wrapper, as noted above, is a function or set of functions to execute calls to the API and convert the results into something more usable than what the API may return on its own.) To execute this code, you first have to install the WDI<\/code> package, as well as devtools<\/a><\/code>,ggplot2<\/a><\/code>, gganimate<\/a><\/code>, data.table<\/a><\/code>, and dplyr<\/a><\/code>. You can install these functions for R with the following commands:<\/p>\n\n\n\n

install.packages(c(\u201cWDI\u201d, \u201cggplot2\u201d, \u201cdevtools\u201d, \u201cdata.table\u201d, \u201cdplyr\u201d))\n<\/code><\/pre>\n\n\n\n
We\u2019ll also install the gganimate add-on package for animation:<\/p>\n\n\n\n
devtools::install_github(\u201cdgrtwo\/gganimate\u201d)<\/code><\/pre>\n\n\n\n
Now that these packages are installed, let\u2019s load them into working memory:<\/p>\n\n\n\n
library(WDI)\nlibrary(gganimate)\nlibrary(ggplot2)\nlibrary(data.table)\nlibrary(dplyr)\nlibrary(devtools)<\/code><\/pre>\n\n\n\n
The WDI<\/code> package provides a convenient way to search the World Bank API for variables that might be interesting for research. Below, we specify the word we want to search for after the \u201cstring\u201d field, and where to look for it \u2014 the \u201cname\u201d field. Doing so, we can see all the variables related to road infrastructure, immunization rates, and any other variables measured on a per capita basis.<\/p>\n\n\n\n
#This code to search the API\nWDIsearch(string=\u201droads\u201d, field=\u201dname\u201d)\nWDIsearch(string=\u201dimmunization\u201d, field=\u201dname\u201d)\nWDIsearch(string=\u201dper capita\u201d, field=\u201dname\u201d)<\/code><\/pre>\n\n\n\n
The search returns lists of variables and their corresponding codes. Now that we know the codes for the variables we\u2019re interested in, we can call for the data from the API. We do this with the WDI<\/code> function. Within this function call, we supply as arguments the countries we want (which is all of them), the variable codes, the first year of data, the last year of data, and an optional \u201cextra\u201d set of fields. The \u201cextra\u201d fields are country metadata \u2014 try turning this option off and on to see what happens.<\/p>\n\n\n\n
# Interesting Variables to plot\n# mortality per 1K women: SP.DYN.AMRT.FE\n# immunization rates of DPT: SH.IMM.IDPT\n# income per capita (constant): NY.ADJ.NNTY.PC.KD\n# population: SP.POP.TOTL\n# Getting the preliminary variables, using the code above\ndat<-data.table(WDI(country=\u201dall\u201d, indicator=c(\u201cSP.DYN.AMRT.FE\u201d, \u201cSH.IMM.IDPT\u201d, \u201cNY.ADJ.NNTY.PC.KD\u201d, \u201cSP.POP.TOTL\u201d), start=1980, end=2017, extra=T))<\/code><\/pre>\n\n\n\n
The data are loaded into memory as a data.table<\/code> object, which is a type of rectangular data frame that\u2019s easy to interact with and analyze. One problem is that the variable names are not easy to interpret, so let\u2019s convert them to something more intuitive. We do this with thedplyr<\/code> package, which provides a set of tools for manipulating data. It has a convenient and readable syntax. We will use the \u201crename\u201d command to rename the variables.<\/p>\n\n\n\n
ndat = dat %>%\n dplyr::rename(mortality = SP.DYN.AMRT.FE, immunization = SH.IMM.IDPT, \n income_cap = NY.ADJ.NNTY.PC.KD, population = SP.POP.TOTL)<\/span>\n<\/code><\/pre>\n\n\n\n
Next, let\u2019s remove the missing observations so we can plot the data. Of course, you should think carefully about which countries are missing and why.<\/p>\n\n\n\n
ndat = na.omit(ndat)\nndat = ndat[ndat$region!=\u201dAggregates\u201d, ]<\/code><\/pre>\n\n\n\n
\"\"<\/a><\/figure>\n\n\n\n
We can create a visual representation of the relationship between income and immunization rates for a particular year. We\u2019ll plot one point for each country, sized by the log of the country\u2019s population. We\u2019ll plot a LOESS<\/a> line of the general relationship, too.<\/p>\n\n\n\n
Here is a static plot using ggplot2<\/code>:<\/p>\n\n\n\n
p <- ggplot(filter(ndat, year==1980), aes(income_cap, immunization, size = log(population))) +\ngeom_point(aes(col=country), show.legend = FALSE) +\ngeom_smooth(aes(group = year), method = \u201cloess\u201d, show.legend = FALSE) + scale_x_log10()<\/code><\/pre>\n\n\n\n
\"\"<\/a><\/figure>\n\n\n\n
Finally, we\u2019ll create an animation of the same relationship over time. To do so, first install ImageMagick<\/a><\/code>. Then create a plot that includes all the years in the data:<\/p>\n\n\n\n
p <- ggplot(ndat, aes(income_cap, immunization, size = log(population), frame = year)) +\n geom_point(aes(col=country), show.legend = FALSE) + \n geom_smooth(aes(group = year), method = \u201cloess\u201d, show.legend = FALSE) + scale_x_log10()\ngganimate(p)<\/code><\/pre>\n\n\n\n
\"\"<\/a><\/figure>\n\n\n\n
Some useful APIs<\/h3>\n\n\n\n
In addition to the World Bank\u2019s API, here are some others that we\u2019ve interacted with and found useful:<\/p>\n\n\n\n