Exploratory Data Analysis( EDA ) is the process of analyzing and visualizing the data to get a better understanding of the data and glean insight from it. There are various steps involved when doing EDA but the following are the common steps that a data analyst can take when performing EDA:
This article focuses on EDA of a dataset, which means that it would involve all the steps mentioned above. Therefore, this article will walk you through all the steps required and the tools used in each step. Therefore, you would expect to find the followings in this article:
For beginners to EDA, if you do not have a lot of time and do not know where to start , I would recommend you to start with Tidyverse and ggplot2 . You can do almost everything about EDA with these 2 packages. There are various resources online like DataCamp , Setscholars , and books like Introduction to Data Science and so on.
In this article, I would walk you through the process of EDA through the analysis of the PISA score dataset which is available here .
Let’s get started, folks!!
Before importing the data into R for analysis, let’s look at how the data looks like:
When importing this data into R, we want the last column to be ‘numeric’ and the rest to be ‘factor’. With this in mind, let’s look at the following 3 scenarios:
df.raw <- read.csv(file ='Pisa scores 2013 - 2015 Data.csv', fileEncoding="UTF-8-BOM", na.strings = '..')df.raw1 <- read.csv(file ='Pisa scores 2013 - 2015 Data.csv')df.raw2 <- read.csv(file ='Pisa scores 2013 - 2015 Data.csv',na.strings = '..')
These are 3 ways of importing the data into R. Usually, one with go for the df.raw1 because it seems to be the most convenient way of importing the data. Let’s see the structure of the imported data:
df.raw1 <- read.csv(file ='Pisa scores 2013 - 2015 Data.csv')str(df.raw1)
There are 2 problems that we can spot immediately. The last column is ‘factor’ and not ‘numeric’ like what we desire. Secondly, the first column ‘Country name’ is encoded differently from the raw dataset.
df.raw2 <- read.csv(file ='Pisa scores 2013 - 2015 Data.csv',na.strings = '..')str(df.raw2)
The last column is now ‘numeric’. However, the name of the first column is not imported correctly.
df.raw <- read.csv(file ='Pisa scores 2013 - 2015 Data.csv', fileEncoding="UTF-8-BOM", na.strings = '..') str(df.raw)
As you can see, the first column is now named properly and the last column is ‘numeric’.
na.strings = '..'
This allows R to replace those blanks in the dataset with NA. This will be useful and convenient later when we want to remove all the ‘NA’s.
fileEncoding="UTF-8-BOM"
This allows R, in the laymen term, to read the characters as correctly as they would appear on the raw dataset.
install.packages("tidyverse") library(tidyverse)
We want to do a few things to clean the dataset:
df <- df.raw[1:1161, c(1, 4, 7)] %>% # Keep useful columns spread(key=Series.Code, value=X2015..YR2015.) %>% rename(Maths = LO.PISA.MAT, Maths.F = LO.PISA.MAT.FE, Maths.M = LO.PISA.MAT.MA, Reading = LO.PISA.REA, Reading.F = LO.PISA.REA.FE, Reading.M = LO.PISA.REA.MA, Science = LO.PISA.SCI, Science.F = LO.PISA.SCI.FE, Science.M = LO.PISA.SCI.MA ) %>% drop_na()
Now let’s see how the clean data looks like:
view(df)
install.packages("ggplot2") library(ggplot2)#Ranking of Maths Score by Countriesggplot(data=df,aes(x=reorder(Country.Name,Maths),y=Maths)) + geom_bar(stat ='identity',aes(fill=Maths))+ coord_flip() + theme_grey() + scale_fill_gradient(name="Maths Score Level")+ labs(title = 'Ranking of Countries by Maths Score', y='Score',x='Countries')+ geom_hline(yintercept = mean(df$Maths),size = 1, color = 'blue')
Similarly, we can rank by science score and reading score too, just change the name accordingly.
If we use the dataset above, we will not be able to draw a boxplot. This is because boxplot needs only 2 variables x and y but in the cleaned data that we have, there are so many variables. So we need to combine those into 2 variables. We name this as df2
df2 = df[,c(1,3,4,6,7,9,10)] %>% # select relevant columns pivot_longer(c(2,3,4,5,6,7),names_to = 'Score')view(df2)
Great! Now we can make boxplots
ggplot(data = df2, aes(x=Score,y=value, color=Score)) +
geom_boxplot()+
scale_color_brewer(palette="Dark2") +
geom_jitter(shape=16, position=position_jitter(0.2))+
labs(title = 'Did males perform better than females?',
y='Scores',x='Test Type')
geom_jitter() allows you to plot the data points on the plot.
You can play around with the code above to get various plots. For example, I can change from ‘color = Score’ to ‘fill=Score’:
ggplot(data = df2, aes(x=Score,y=value, fill=Score)) +
geom_boxplot()+
scale_fill_brewer(palette="Green") +
geom_jitter(shape=16, position=position_jitter(0.2))+
labs(title = 'Did males perform better than females?',
y='Scores',x='Test Type')
The plot looks a bit messy. A better visualisation would be to separate Subjects and Genders and plot them side by side.
Since we want to separate Subjects and Genders from a column containing ‘Subject.Gender’ (e.g. Maths.F), we need to use strsplit () to do this job for us
S = numeric(408) # create an empty vector for (i in 1:length(df2$Score)) { S[i] = strsplit(df2$Score[i],".",fixed = TRUE) }
Now S is a list of 408 components, each of which has 2 sub-components ‘Subject’ and ‘Gender’. We need to transform S into a data frame with 1 column of Subject and 1 column of Gender . We will name this data frame as df3
df3 = S%>%unlist() %>% matrix(ncol = 2, byrow = TRUE)%>% as.data.frame()view(df3)
We now need to combine this df3 with df2 we created earlier, and name the result as df4
df4 = cbind(df2,df3) colnames(df4) = c('Country','Score','Value','Test','Gender') df4$Score = NULL # since the 'Score' column is redundant view(df4)
Awesome! Now the data looks clean and neat. Let’s create multiple plots with the use of facet_wrap() function in ggplot2
ggplot(data = df4, aes(x=Test,y=Value, fill=Test)) +
geom_boxplot()+
scale_fill_brewer(palette="Green") +
geom_jitter(shape=16, position=position_jitter(0.2))+
labs(title = 'Did males perform better than females?',
y='Scores',x='Test')+
facet_wrap(~Gender,nrow = 1)
Here, we categorized the plot by Gender, hence facet_wrap(~Gender,nrow = 1)
We can also categorize the plot by Test by changing facet_wrap(~Test,nrow = 1)
ggplot(data = df4, aes(x=Gender,y=Value, fill=Gender)) +
geom_boxplot()+
scale_fill_brewer(palette="Green") +
geom_jitter(shape=16, position=position_jitter(0.2))+
labs(title = 'Did males perform better than females?',
y='Scores',x='')+
facet_wrap(~Test,nrow = 1)
By looking at these plots, we can make some insights about the performance of males and females. Generally, males performed better in Science and Maths, but females performed better in Reading. However, it would be naive for us to make a conclusion only after looking at the boxplot. Let’s dive deeper into the data and see any other insights we can get after manipulating the dataset.
Since I want to compare the performance of Males and Females in each subject across all participating countries, I would need to calculate the % difference in terms of the score for each subject between males and females and then plot it out to visualize.
How would I do this in R?Use mutate() function
Let’s look at the original clean data set that we have, df
We will now use the mutate() function to calculate, for each country, the % difference between Males and Females for each subject.
df = df %>% mutate(Maths.Dif = ((Maths.M - Maths.F)/Maths.F)*100, Reading.Dif = ((Reading.M - Reading.F)/Reading.F)*100, Science.Dif = ((Science.M - Science.F)/Science.F)*100, Total.Score = Maths + Reading + Science, Avg.Dif = (Maths.Dif+Reading.Dif+Science.Dif)/3 )view(df)
Now let’s plot this out to visualize the data better
##### MATHS SCORE ##### ggplot(data=df, aes(x=reorder(Country.Name, Maths.Dif), y=Maths.Dif)) + geom_bar(stat = "identity", aes(fill=Maths.Dif)) + coord_flip() + theme_light() + geom_hline(yintercept = mean(df$Maths.Dif), size=1, color="black") + scale_fill_gradient(name="% Difference Level") + labs(title="Are Males better at math?", x="", y="% difference from female")
This plot represents the % difference in scores using Females as reference . A positive difference means males scored higher, while a negative difference means males scored lower . The black line represents the mean difference across all countries.
Some interesting insights one can draw from here:
We can do the same thing for Reading and Science score.
df = df[,c(1,3,4,6,7,9,10)] #select relevant columns
To create correlation plot, simply use cor():
res = cor(df[,-1]) # -1 here means we look at all columns except the first columnres
We can calculate p-value to see whether the correlation is significant
install.packages("Hmisc") library("Hmisc")res2 <- rcorr(as.matrix(df[,-1))
The smaller the p-value, the more significant the correlation.
The purpose of this section about correlation plot is to introduce to you how to calculate correlation between variables in R. For this dataset, it is obvious that all the variables are correlated
To visualize
install.packages("corrplot") library(corrplot) corrplot(res, type = "upper", order = "hclust", tl.col = "black", tl.srt = 45)
The stronger the color and the bigger the size, the higher the correlation. The result is similar to the one we got earlier: All the variables are intercorrelated.
That is it! Hope you guys enjoyed and picked up something from this article. While this guide is not exhaustive, it will more or less give you some ideas of how to do some basic EDA with R.
If you have any questions, feel free to put them down in the comment section below. Thank you for your read. Have a great day and happy programming!!!
我来评几句
登录后评论已发表评论数()