The global part of the analysis uses a different list of job titles meant to capture more general descriptions of the same occupations. The steps to create that list include:<\/p>\n\n
- \n
- Start with the list of BLS job titles<\/a> that had at least 100,000 people working in the occupation in the U.S.<\/li>\n
- Remove all occupations with ambiguous job functions (\u201call other\u201d, \u201cMisc.\u201d).<\/li>\n
- Split all titles with composite job functions into individual job titles (For example, \u201cmodels and demonstrators\u201d to \u201cmodels,\u201d \u201cdemonstrators\u201d).<\/li>\n
- Change plural words to singular (\u201cmodels\u201d to \u201cmodel\u201d) to standardize across occupations.<\/li>\n
- Manually inspect the list to ensure that the occupations were comprehensible and likely to describe human workers. This involved removing terms that might not apply to humans (such as tester, sorter) based on manual review of Google results.<\/li>\n
- Replace technical job titles with more general ones when possible to simplify translations and better represent searches. For example, instead of searching for \u201cpostsecondary teacher,\u201d the team searched for \u201cprofessor,\u201d and instead of \u201cchief executive,\u201d the team used \u201cCEO.\u201d<\/li>\n
- Use Google Trends to filter unpopular job titles relative to a reference occupation (\u201cchildcare worker\u201d), following the same procedure described above. Any terms with search intensity below the search intensity of the reference occupation were removed.<\/li>\n
- Select the top 100 terms with the most popular search intensity in Google in the U.S. within the past year.<\/li>\n
- Translate each job title and determine which form to use when multiple translations were available.<\/li>\n<\/ol>\n\n
Translations<\/h4>\n\n
To conduct the international analysis, the research team chose to examine image results within a subset of G20 countries, which collectively account for 63% of the global economy. These countries include Argentina, Australia, Brazil, Canada, France, Germany, India, Indonesia, Italy, Japan, Mexico, Russia, Saudi Arabia, South Africa, South Korea, Turkey, the United Kingdom and the United States. The analysis excludes the European Union because some of its member states are included separately and China because Google is blocked<\/a> in the country. Researchers used the official language of each country for each search (or the most popular language if there were multiple official languages), and worked with a translation service, cApStAn, to develop the specific search queries.<\/p>\n\n
To approximate search results for each country, researchers adjusted Google\u2019s country and language settings. For example, to search jobs in India, job titles were queried in the Hindi language with the country set to India. Several countries in the study share the same official language; for example, Argentina and Mexico both have Spanish as their official language. In these cases, researchers executed separate queries for each language and country combination. The languages used in the searches were: Modern Standard Arabic, English, French, German, Hindi, Indonesian, Italian, Japanese, Korean, Portuguese, Russian, Spanish and Turkish.<\/p>\n\n
Many languages spoken in these countries have gender-specific words for each occupation term. For example, in German, adding \u201cin\u201d to the end of the word \u201cmusiker\u201d (musician) gives a female connotation to the word. However, the word \u201cmusiker\u201d may not exclusively imply \u201cmale musician,\u201d and it is not the case that only male musicians can be referred to as \u201cmusiker.\u201d In consultation with the translation team, researchers identified the gender form of each job that would be used when a person of unknown gender is referenced, and searched for those terms. The male version is the default choice for most languages and occupations, but the translation team recommended using the feminine form for some cases when it was more commonly used. For example, researchers searched for \u201cnurse\u201d in Italian using the feminine term \u201cinfermi\u00e8re\u201d rather than the masculine \u201cinfirmier\u201d on the advice of the translation team.<\/p>\n\n
In addition, some titles do not have a directly equivalent title in another language. For example, the job term \u201ccompliance officer\u201d does not have an Italian equivalent. Finally, the same translated term can refer to different occupations in some languages. As a result, not all languages have exactly 20 search terms. Jobs lacking an equivalent translation in a given language were excluded.<\/p>\n\n
Data collection<\/h4>\n\n
To create the master dataset used for both analyses, researchers built a data pipeline to streamline image collection, facial recognition and extraction, and facial classification tasks. To ensure that a large number of images could be processed in a timely manner, the team set up a database and analysis environment on the Amazon Web Service (AWS) cloud, which enabled the use of graphics processing units (GPUs) for faster image processing. Building this pipeline also allowed the researchers to collect additional labeled training images relatively quickly, which they leveraged to increase the diversity of the training set in advance of classifying the image search results.<\/p>\n\n
Search results can be affected by the timing of the queries: Some photos could be more relevant during the time the query is executed, and therefore have a higher rank in the search results compared with searches at other times.<\/p>\n\n
There are a number of filters users can apply to the images returned by Google. Under \u201cTools,\u201d for example, users can signal to Google Image Search that they would like to receive images of different types, including \u201cFace,\u201d \u201cPhoto\u201d and \u201cClip Art,\u201d among other options. Users can also filter images by size and usage rights. For this study, researchers collected images using both the \u201cphoto\u201d and \u201cface\u201d filter settings, but the results presented in this report use the \u201cphoto\u201d filter only. Researchers made this decision because the \u201cphoto\u201d filter appeared to provide more diverse kinds of images than the \u201cface\u201d filter, while also excluding clip art and animated representations of jobs.<\/p>\n\n
Removing irrelevant queries<\/h4>\n\n
For occupations included across both the U.S. and international analysis term lists, some queries returned images that did not depict individuals engaged in the occupation being examined. Instead, they often returned images that showed clients or customers, rather than practitioners of the occupation, or depicted non-human objects. For example, the majority of image results for the term \u201cphysical therapist\u201d showed individuals receiving care rather than individuals engaging in the duties associated with being a physical therapist.<\/p>\n\n
To ensure the relevance of detected faces, researchers reviewed all of the collected images for each language, country and occupation combination. For the U.S. analysis, there were a total of 239 sets of images to review. For the international analysis, there were 1,800 sets of images to review. Queries were categorized into one of four categories based on the contents of the collected images.<\/p>\n\n
- \n
- \u201cPass\u201d: More than half of collected images depict only individuals employed in the queried occupation. Overall, 44% of jobs in the U.S. analysis and 43% of jobs in the global analysis fell into this category.<\/li>\n
- \u201cFail\u201d: The majority of collected images do not depict any face or depict faces irrelevant to the desired occupation. In many languages, the majority of collected images for the occupation \u201cbarber\u201d depict only people who have been to a barber, rather than the actual barber. In the analysis of international search results, this includes queries that return images of an occupation different from that initially defined by the English translation. For example, the Arabic translation of \u201cjanitor\u201d returns images of soccer goalies when queried in Saudi Arabia. Because the faces depicted in these images are not representative of the desired occupation, we categorize these queries as \u201cfail.\u201d A total of 31% of jobs in the U.S. analysis and 37% of jobs in the global analysis fell into this category.<\/li>\n
- \u201cComplicated\u201d: The majority of collected images depict multiple people, some of whom are engaged in the queried occupation and some of whom are not. For example, the term \u201cpreschool teacher\u201d and its translations often return images that feature not only a teacher but also students. These queries are categorized as \u201ccomplicated\u201d because of the difficulty in isolating the relevant faces. A total of 23% of jobs in the U.S. analysis and 17% of jobs in the global analysis fell into this category.<\/li>\n
- \u201cAmbiguous\u201d: Some queries do not fall into the other categories, as there is no clear majority of image type or it is unclear whether the people depicted in the collected images are engaged in the occupation of interest. This may occur if the term has many definitions, such as \u201ctrainer,\u201d which can refer to a person who trains athletes or various training equipment, or if the term has other usage in popular culture, such as the surname of a public figure (\u201cbaker\u201d) or the name of a popular movie (\u201ctaxi driver\u201d). Just 2% of jobs in the U.S. analysis and 3% of jobs in the global analysis fell into this category.<\/li>\n<\/ul>\n\n
To minimize any error caused by irrelevance of detected faces in collected images, we remove all queries categorized as \u201cfail,\u201d \u201ccomplicated\u201d or \u201cambiguous\u201d and only retain those queries categorized as \u201cpass.\u201d\n<\/a><\/p>\n\n
Machine vision for gender classification<\/h4>\n\n
Researchers used a method called \u201ctransfer learning\u201d to train a gender classifier, rather than using machine vision methods developed by an outside vendor. In some commercial and noncommercial alternative<\/a> classifiers, \u201cmultitask\u201d learning methods are used to simultaneously perform face detection, landmark localization, pose estimation, gender recognition and other face analysis tasks. The research team\u2019s classifier achieved high accuracy for the gender classification task, while allowing the research team to monitor a variety of important performance metrics.<\/p>\n\n
Face detection<\/h4>\n\n
Researchers used the face detector from the Python library dlib <\/em>to identify all faces in the image. The program identifies four coordinates of the face: top, right, bottom and left (in pixels). This system achieves<\/a> 99.4% accuracy on the popular Labeled Faces in the Wild<\/a> dataset. The research team cropped the faces from the images and stored them as separate files.<\/p>\n\n
Many images collected do not contain any individuals at all. For example, all images returned by Google for the German word \u201cBarmixer\u201d are images of a cocktail shaker product, even with the country search parameter set to Germany. To avoid drawing inference based on a small number of images, researchers included only queries that have at least 80 images downloaded and 50 images with at least one face detected in the analysis. Across different countries, the number of faces detected in the images varied. Hindi and Indonesian had the most detected faces. This means that their images tend to feature more people in them than other languages.<\/p>\n\n
The table below summarizes the number of queries, number of images and number of faces detected. Overall, researchers were able to collect over 95% of the top 100 images that we sought to download.<\/p>\n\n
Training the model<\/h4>\n\n
<\/a><\/p>\n\n
![\"Image](\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/social-trends\/wp-content\/uploads\/sites\/3\/2018\/12\/12.17.18_gender_jobs_methodology-00.png\")
<\/a><\/figure>\n\nRecently, research has provided evidence of algorithmic bias in image classification systems from a variety of high-profile vendors.[9. See Buolamwini, Joy and Timnit Gebru. 2018. \u201cGender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification<\/a>.\u201d Proceedings of Machine Learning Research.]\u00a0This problem is believed to stem from imbalanced training data that often overrepresents white men. For this analysis, researchers decided to train a new gender classification model using a more balanced image training set.<\/p>\n\n
However, training an image classifier is a daunting task because collecting a large labeled dataset for training is very time and labor intensive, and often is too computationally intensive to actually execute. To avoid these challenges, the research team relied on a technique called \u201ctransfer learning,\u201d which involves recycling large pretrained neural networks (a popular class of machine learning models) for more specific classification tasks. The key innovation of this technique is that lower layers of the pretrained neural networks often contain features that are useful across different image classification tasks. Researchers can reuse these pretrained lower layers and fine-tune the top layers for their specific application \u2013 in this case, the gender classification task.<\/p>\n\n
The specific pretrained network researchers used is VGG16<\/a>, implemented in the popular deep learning Python package Keras<\/em>. The VGG network architecture was introduced by Karen Simonyan and Andrew Zisserman in their 2014 paper \u201cVery Deep Convolutional Networks for Large Scale Image Recognition<\/a>.\u201d The model is trained using ImageNet, which has over 1.2 million images and 1,000 object categories. Other common pretrained models include ResNet and Inception. VGG16 contains 16 weight layers that include several convolution<\/a> and fully connected layers. The VGG16 network has achieved a 90% top-5 accuracy in ImageNet classification.[10. The top-5 accuracy is calculated by counting the times a predicted label matched the target label, divided by the number of data points evaluated for the five categories with the highest probabilities.]<\/p>\n\n
Researchers began with the classic architecture of the VGG16 neural network as a base, then added one fully connected layer, one dropout layer and one output layer. The team conducted two rounds of training \u2013 one for the layers added for the gender classification task (the custom model), and subsequently one for the upper layers of the VGG base model.<\/p>\n\n
Researchers froze the VGG base weights so that they could not be updated during the first round of training, and restricted training during this phase to the custom layers. This choice reflects the fact that weights for the new layers are randomly initialized, so if we allowed the VGG weights to be updated it would destroy the information contained within them. After 20 epochs of training on just the custom model, the team then unfroze four top layers of the VGG base and began a second round of training. For the second round of training, researchers implemented an early-stopping function. Early stopping checks the progress of the model loss (or error rate) during training, and halts training when validation loss value ceases to improve. This serves as both a timesaver and keeps the model from overfitting to the training data.<\/p>\n\n
In order to prevent the model from overfitting to the training images, researchers randomly augmented each image during the training process. These random augmentations included rotations, shifting of the center of the image, zooming in\/out, and shearing the image. As such, the model never saw the same image twice during training.<\/p>\n\n
Selecting training images<\/h4>\n\n
![\"Training](\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/social-trends\/wp-content\/uploads\/sites\/3\/2018\/12\/12.17.18_gender_jobs_methodology-01.png\")
<\/a><\/figure>\n\n![\"Validation](\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/social-trends\/wp-content\/uploads\/sites\/3\/2018\/12\/12.17.18_gender_jobs_methodology-02.png\")
<\/a><\/figure>\n\nImage classification systems, even those that draw on pretrained models, require a substantial amount of training and validation data. These systems also demand diverse training samples if they are to be accurate across demographic groups. To ensure that the model was accurate when it came to classifying the gender of people from diverse backgrounds, researchers took a variety of steps. First, the team located existing datasets used by researchers for image analysis. These include the \u201cLabeled Faces in the Wild\u201d (LFW) and \u201cBainbridge 10K U.S. Adult Faces\u201d datasets. Second, the team downloaded images of Brazilian politicians from a site that hosts municipal-level election results. Brazil is a racially diverse country, and that is reflected in the demographic diversity in its politicians. Third, researchers created original lists of celebrities who belong to different minority groups and collected 100 images for each individual. The list of minority celebrities focused on famous black and Asian individuals. The list of famous blacks includes 22 individuals: 11 men and 11 women. The list of famous Asians includes 30 individuals: 15 men and 15 women. Researchers then compiled a list of the most-populous 100 countries and downloaded up to 100 images of men and women for each nation-gender combination, respectively (for example, \u201cFrench man\u201d). This choice helped ensure that the training data included images that feature people from a diverse set of countries, balancing out the over-representativeness of white people in the training dataset. Finally, researchers supplemented this list with a set of 21 celebrity seniors (11 men and 10 women) to help improve model accuracy on older individuals. This allowed researchers to easily build up a demographically diverse dataset of faces with known gender and racial profiles.<\/p>\n\n
Some images feature multiple people. To ensure that the images were directly relevant, a member of the research team reviewed each face manually and removed irrelevant or erroneous faces (e.g., men in images with women). Researchers also removed images that were too blurry, too small, and those where much of the face was obscured. In summary, the training data consist of 14,351 men and 12,630 women in images. The images belong to seven different datasets.<\/p>\n\n
Model performance<\/h4>\n\n
To evaluate whether the model was accurate, researchers applied it to a subset equivalent to 20% of the image sources: a \u201cheld out\u201d set which was not used for training purposes. The model achieved an overall accuracy of 95% on this set of validation data. The model was also accurate on particular subsets of the data, achieving 0.96 positive predictive value on the black celebrities subset, for example.<\/p>\n\n
As a final validation exercise, researched used an online labor market<\/a> to create a hand-coded random sample of 996 faces. This random subset of images overrepresented men \u2013 665 of the images were classified as male. Each face was coded by three online workers. For the 924 faces that had consensus across the three coders, the overall accuracy of this sample is 88%. Using the value 1 for \u201cmale\u201d and 0 for \u201cfemale,\u201d the precision and recall of the model were 0.93 and 0.90, respectively. This suggests that the model performs slightly worse for female faces, but that the rates of false positives and negatives was relatively low. Researchers found that many of the misclassified images were blurry, smaller in size, or obscured.<\/p>\n\n
![\"Model](\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/social-trends\/wp-content\/uploads\/sites\/3\/2018\/12\/12.17.18_gender_jobs_methodology-03.png\")
<\/a><\/figure>\n\n![\"Comparison](\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/social-trends\/wp-content\/uploads\/sites\/3\/2018\/12\/12.17.18_gender_jobs_methodology-04.png\")
<\/a><\/figure>\n\n![\"Comparison](\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/social-trends\/wp-content\/uploads\/sites\/3\/2018\/12\/12.17.18_gender_jobs_methodology-05.png\")
<\/a><\/figure>\n\n![\"Comparison](\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/social-trends\/wp-content\/uploads\/sites\/3\/2018\/12\/12.17.18_gender_jobs_methodology-06.png\")
<\/a><\/figure>\n\n![\"Comparison](\"https:\/\/alpha.pewresearch.org\/pewresearch-org\/social-trends\/wp-content\/uploads\/sites\/3\/2018\/12\/12.17.18_gender_jobs_methodology.png\")
<\/a><\/figure>\n\n