Using Data Analysis & Visualization to Understand the Gender Gap in Literacy¶

By Rachel Walter, Fall 2019

Photo retrieved from Variety

Introduction¶

Motivation¶

I am a computer science major with a minor in women's studies. I thought it would be interesting to explore how these two fields can interact by learning how to analyze and visualize data with a critical focus on gender. This project will specifically investigate the gender gap in education globally, as measured by literacy rates of each country. We will also look into how location and gender interact to yield gender disparities.

By the end of this tutorial you will be able to:

1. Collect data from an online source
2. Clean & reorganize data
3. Find summary statistics for a dataset
4. Use map visualizations to see geographic trends in data
5. Graph and model the relationships between two sets of data
6. Train and Choose the Hyperparameters of a Classification Model
7. So much more!

This process will be able to be applied to a variety of datasets that distinguish location (i.e. country, city, continent) and and are sex disaggregated.

Required Libraries/Tools¶

The following libraries/tools will be used throughout the project. They will allow us to do and visualize more! If you do not have any of these libraries installed, you can install them into your development environmemt with the help of $ pip3 install [package]. You can find more information on them through their documentation linked below:

• pandas, a data analysis toolkit which helps us work with data frames - docs
• numpy, a package for scientific computing with special n-d arrays - docs
• folium for creating map visualizations - docs
• sklearn for regressions and modelling - docs
• matplotlib for graphing - docs
• requests for HTTP requests - docs
• BeautifulSoup for web-scraping - docs

Background Readings: Gender Gap in Education¶

Globally, access to education to dependent on a variety of factors including infrastructure development, prescence of conflict, socioeconomic status, and gender. In this case, I am going to look only at how gender (and location) affects education atainment. This, of course, may not paint a full picture since multiple forces can compound to privelege or oppress educational opportunity.

Readings below offer a perspective on known trends in the gender gap in education and why it exists:

• Mind the Gap: 5 Facts About The Gender Gap in Education
• UNESCO on Education and Gender Equality

Background Readings: Classification¶

When I was learning about classification and refining hyperparameters, I used the following resources

• GeeksForGeeks Regression & Classification
• Optimizing Hyperparameters in Random Forest Classification

Part 1: Data Collection¶

The first step of this project is to collect data. In my case, this meant choosing a data source that provided the information I needed for analyzing gender and education. I found that UNICEF keeps publicly available data on various global development measures surrounding children, such as childhood survival rates. I was able to find data on youth and adult literacy rates (sex disaggregated).

I updated the Excel spreadsheets to only contain the data tables, making it much easier to access the data. These updated .xlsx files are available to download from my GitHub for this project. We are going to start collection with one pandas dataframe for the adult dataset and the youth data set.

Explanation of Data¶

• This data set is collected across multiple years, ranging from 2005-2013
• Data was collected by UNESCO Institute for Statistics and Prepared by the Data and Analytics Section; Division of Data, Research and Policy, UNICEF
• Total, Male, and Female all represent the percentage of literacy in that population

Part 2: Data Processing¶

Initial Data Cleaning¶

Next we need to process the data little bit more to make it clean. This means everything will be well labelled and we will handle missing data. Other times you might want to reorganize your data table to be tidy, but since our tables are fairly simple and clean I am not going to reorganize.

We can see just from head() that there are extra columns that are all NaN and that the Source column all says "UNESCO Institute for Statistics," which is not useful for our analysis. We also have to work on re-labelling the Male and Female columns, which are currently "Unnamed : 7" and "Sex." Some countries are included in the table but have no actual data, such as Andorra.

Let us repeat this process on the adult data set.

Adding Country-Continent Mapping¶

As stated in the introduction, I am interested in how location and gender may interact to yield differences in education and literacy. Since each country only has one data point, it would be useful to be able to group countries geographically to see if this relationship exists. The easiest way to do this is to match each country to its respective continent. Do not worry, this is much easier than you might expect thanks to tools like BeautfulSoup.

I found a Wikipedia page) with a table which stores the country name, 2-character code, 3-character code, and continent code. I used Beautiful Soup and Pandas to create a dataframe from that table. The hardest part was using print(soup.prettify()) to find my data in HTML, however most tables are of the class "wikitable sortable," making it much easier to find and seperate out that data!

Now, as you can see above, we have a dataset that can match our ISO country codes with the continent codes (EU = Europe, SA = South America, AS = Asia, OC = Oceania, AF = Africa, and NA = North America). We will now make a few changes to the countrydf so that it can be combined with our main data sets.

Creating Combined Table¶

Now that both tables are tidy, I want to create a third table that will help me explore if the youth and adult scores are related for each country. However, we are not guarenteed to have all that data for every country. I am going to do a merge of youth and adult dataframes to create a new table of only countries that are represented in both datasets.

The alternate option to merging the tables would be be making a data table schema like the following:

| ISO Code | Countries and areas | Reference year(s) | Adult/Youth | Total | Male | Female |

Where Youth/Adult would be some 0/1 or Y/A value indicating whether that data would be for the youth or adult data set. This would be more in-line with tidy data theory. However, for my sake the ease of coding up visualizations, I am keeping my data in its less tidy form.

We know have tables that, for each country represented, store the ISO code, name, reference years, and literacy rates for each population/sub-population.

Part 3: Exploratory Analysis & Data Visualization¶

Now that our data is all set, we can begin analyzing it! There are a few beginning exploratory visualizations and analyses I want to do with our data, all of which will be described and demonstrated below.

General Statistical Analysis¶

Find the Min, Max, Mean, Median, and Standard Deviation for Literacy of Each Population¶

This is the basic statistics for any data set which can help us see the central tendency and get a better feel for our data. I am looking at the statistics for each population category (i.e. Youth_Total, Total, Boy, Girl, Male, and Female.

From these summary statistics, a few features appear. First and foremost, all of the statistics for female/girl were lower than the corresponding male populations. Another intesting thing to notice is that the mean is always less than the median, indicating there may be some lower end outliers bringing down the mean. This knowledge might come in handy later when seeing how region relates to gender and literacy.

Visualize Distributions Among Populations with Boxplots¶

If we want a visual of the distrbition within the literacy rate data setsn instead of the statistics number, we can also graph the box and whisker plot, as shown below on the total, male, and female literacy rate (seperated into the youth and adult categories):

Investigate Geographic Region Independently¶

Graph Total Literacy Percent on a Map¶

This priliminary visualization will help me visualize how literacy varies across the globe. Some countries, regions, or continents might have higher literacy rates. Seeing this mapped out will help us know later on in the in-depth analysis if it seems like location and gender interact.

I am using the Folium library to map the percentage of literacy. Note that you have to retrieve country map data in GeoJSON format before being able to map it out. Then, we can match our data's ISO codes to the IDs of countries' geographical areas to map out literacy rates visually.

Glancing over the map, it looks like South Asia and the Saharan/Sub-Saharan region in Africa have the lowest literacy rates. Most of Europe, Asia, and South America appear to have higher literacy rates.

Relationship Between Continent and Literacy¶

While on the topic of geographic location, let's use the continent categorization of data to see if there really are differences in literacy by continent. Once again, we will make use of the box and whisker plots to see the distribution of literacy rates for each continent.

We can see that many of the trends we noticed before on the map visualization are true. South America and Europe have the highest literacy rates. Asia and Oceania are generally high with some countries with lower literacy rates. However, Africa has by far the largest range in literacy rates and it's upper and lower quartile ranges are between 50-80%. To me, this indicates that location may have a relationship with literacy.

How might geography and gender interact?¶

Graph Female & Male Literacy Percent on a Map¶

I want to quickly visualize if the same geographic divides for literacy apply to the genders. For example, will both men and women's literacy be equally low in Africa, or does it impact women more? Will other countries/regions have much worse Female vs Male literacy?

Focusing on gender, it appears though both genders have lower literacy in Saharan and Sub-Saharan Africa and South Asia. However, on a whole women have lower literacy and have a new low-literacy region: the Arabian peninsula. These visualizations suggest to me that both gender and region affect literacy and they likely interact together.

Map the Gap in Men's and Women's Literacy¶

To answer a question raised above about the gaps between men's and women's literacy, we can map the difference between the literacy rates of both populations. For example, it is not fair to fault a country that has 55% female literacy if the men's literacy rate is only 56%. Looking at the differences in literacy might be a more precise way to identify/visualize gender disparities.

From this map, we can see that most places have around equal literacy rates for men and women. However, in areas in sub-Saharan Africa and South-West Asia/the Middle East have men with as much as a 34% more literate than women.

Investigate Gender Independently¶

Relationship Between Boy & Girl Literacy¶

My next phase of visualizations/data analysis is understanding whether the literacies of different population groups are correlated. For example, in the visualization directly below we want to see if boy literacy is related to girl literacy and to what extent. We would expect as boy's literacy increases, so would girl's literacy at an equal rate (i.e. a slope of 1 and, in an ideal world, an intercept at the origin). If this is not true, it could reveal gender disparities and we could use the slope to find the ratio of boy's to girl's educational attainment.

We can see that although the relationship appears to be positively linear, there is an interesting relationship between mens and women's literacy. The slope is about 1.35. In countries with low boy literacy rates, girl literacy starts even lower. However, when boy literacy rates get higher, girls are able to catch up because of the rate of change.

Relationship Between Adult Male & Adult Female Literacy¶

In the visualization directly below we want to see if adult male literacy is related to adult female literacy and to what extent. We would expect as men's literacy increases, so would women's literacy at an equal rate (i.e. a slope of 1 and, in an ideal world, an intercept at the origin). If this is not true, it could reveal gender disparities and we could use the slope to find the ratio of men's to women's educational attainment.

Adult and Youth literacy rates have an almost identical relationship. This means gender disparities are worse in countries where the male literacy is lower. This may be indicitive of other factors which vary country-to-country described in background readings related to causes for the educational gap. For example, poverty can make the gender gap in education worse, and countries that are more impoverished are likely to have lower literacy in general.

Investigate Age Independently¶

Relationship Between Child & Adult Literacy¶

The next population difference I wanted to look into was age as a factor of analysis. Were children more or less likely to be literate? If there are major differences between the two groups, it might imply changing access to education, educational quality, or the addition/removal of barriers to education.

This once again has a similar trend as the women vs men and girl vs boy correlations. Youth are more likely to be literate than adults until the gap closes as adult literacy increases. Since in general youth literacy is higher, this means that children are recieving more access to education that past generations.

How might age and gender interact?¶

Relationship Between Child & Adult Literacy (Sex Disaggregated)¶

If men have high literacy, will boys do as well? I wanted to do a secondary analysis of boys vs men and girls vs women, just in case the literacy trends are different for the different genders.

This once again has a similar trend as the adult vs youth correlations. Boys are more likely to be literate than men until the gap closes as men's literacy increases.

This once again has a similar trend as the women vs men and girl vs boy correlations. Girls are more likely to be literate than women until the gap closes as women's literacy increases.

Since both boys and girls have higher literacy rates than men and women, this may indicate new generations are having better access to education, etc. than the older generations.

Part 4: Deeper Analysis: Classification¶

We understand from preliminary analysis and visualization that it appears that age, location, and gender may be related and interact together to affect literacy. To see if we are able to model this, I thought it might be fun to see if we can create a classifer that when given literacy data can accurately predict the continent of origin. This might help us understand how closely literacy data and geographic location are connected.

Modify the Continent Encoding¶

In order to train our dataset with the randomforest classifier, we need to convert our continent code from strings to numbers. Panda's factorize does this very quickly for us!

Choosing Hyperparameters¶

Hyperparameters are basically variables that control how classification models are trained/created. You can optimize the hyperparameters to increase the accuracy of your classification model.

For random forest hyperparameters, I did research through this tutorial. The author said that fully optimizing the hyperparameters could take over 25 minutes to calculate. The author also showed that many of the default settings led to high level of accuracy. I decided to follow along with how the author used a validation curve to optimize one hyperparameter at a time. It trained with different parameters and plotted the accuracy of each input for the hyperparameters. I chose the optimal values for n_estimators, max_depth, min_samples_split, and min_samples_leaf. Find more information about what these hyperparameters do in the sklearn documentation. This process took a minute for each hyperparameter and was much faster than the full optimization.

Note: There is a variation in the optimization each time! I ended up choosing values that seemed to do best when run multiple times.

The gist of what you do is test how accurately a classification model performs when generated with different settings for an isolated hyperparameter. By always choosing the hyperparameter value which yields the most accurate classifications, you are optimizing your classification model so it can be its best! For ease of identifying the best hyperparameter settings, I graph the results of the training and testing accuracies.

Train Classification Model¶

We can see from our optimizations that the best values for hyperparameters appear to be the following:

• n-estimators = 300
• max_depth = 8
• min_samples_split = 15
• min_samples_leaf = 10

See How Well Our Model Performs: 5-fold cross-validation¶

In simple terms, a 5-fold cross-validation breaks up our dataset into 5 random groups. One group will be used to test the model, the nine others will be used to train the model. You will then get 5 scores of how well the classification performed for each data set. This can help us see how accurate our classification model is! For further reading on k-fold cross-validation, check out this resource.

We had to use 5 instead of any other k because "the least populated class in y (the continent data) has only 5 members." In order for Oceania (the continent with the fewest countries represented in the data) to be in each training/testing set, you can only split the data into 5 groups.

The highest accuracy it achieves is around 70%, while guessing at random would yield roughly 20% accuracy. While our model is not the best at classifying the continent based on the given data, it does show that there is a at least some relationship between literacy rates within continent groupings.

This imperfect classification perfomance is likely because some continents have similar data sets. An interesting follow up would be if only including certain literacy data, i.e. only using the adult literacy rates, yielded more or less accurate result.

Part 5: Insight & Policy Decision¶

Insight¶

From my analysis, a few trends become apparent:

• women in general have lower literacy than men
• adults may end up having lower literacy than children
• certain countries and regions have worse literacy rates and gaps in literacy than others
• in countries where literacy rates are already lower have greater gaps between youth and adult and between genders

Policy Recommendations¶

There is an obvious gap in literacy for women and girls. However, having a background of our dataset and a feminist analysis of the use of the girl in philanphropic work, I know that adult women are not always given the same aid as girls and may even be seen as a lost cause. There can be interventions that target women and girls, even strategic partnerships that build financial indpendence, job skills, educational attainment, and leadership experience to women and girls. A prime example of this is The Pad Project which helps provide menstrual products to women in rural areas in India, enabling girls to continue education and giving women and girls the important skills described previously for social capital development.

This data analysis reveals groups and countries/geographic areas where literacy is the lowest or there are large gaps between men and women. However, my analysis cannot begin to understand why the rate is low or why there is such a difference between men and women. It might be belief systems, lack of resources, or a state of crisis. These type of issues would have to be addressed before the countries could improve literacy rates.

How Can You Use What You Learned?¶

There are a lot of ways privilege and oppression can manifest themselves. It can be hard to fully see the impact of systems of inequality. By knowing how to collect, visualize, and analyze data, you are now able to see and communicate more about inequality & injustice. Another great example of this data-for-good approach I have seen recently is a collaboration between NPR and the Howard Center for Investigative Journalism on Heat & Health in American Cities.

By completing this tutorial, you have seen and been able to implement multiple ways of exploring how gender, age, and location are linked to educational inequality. Now it is up to you to choose data sources and problems important to you!