# Exploratory Data Analysis (EDA)

EDA is a critical part of every data science project. It is the process of understanding the data, and its limitations, before you start to model it. We'll focus on simple counts and summaries, and then start "grokking" relationships in the data through cross-tabulation which we'll structure for visualisation.

You should have already run the introductory [01.3-WSL-1-EDA.Rmd](https://dsbristol.github.io/dst/assets/code/01.3-WSL-1-EDA.Rmd) worksheet and become familiar with working with an RStudio RMarkdown document.

# 0.0 Requirements

Its good practice to get the requirements right at the top. The following solution checks for the requirements, and then installs them if they are not present.

```{r}
if(!require("gplots")) install.packages("gplots")
if(!require("readr")) install.packages("readr")
library("gplots")
library("readr")
```

# 1.0 DATA
Here we're going to go a little further and explore a new dataset, the [KDD99](http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html) dataset. Read about the competition [task specification](http://kdd.ics.uci.edu/databases/kddcup99/task.html).

We use the [10%](http://kdd.org/cupfiles/KDDCupData/1999/kddcup.data_10_percent.zip) and [column names](http://kdd.org/cupfiles/KDDCupData/1999/kddcup.names) files, which you can download directly. We use the opportunity of exploring **external data** to make some suggestions for your group assessment project structure.

Q: What information would be useful here? What would you like to see in an exemplar web resource?

### Notes on the data

Cyber security data is often very weird. These data were generated in a competition setting where teams were hacking one another. The connection activity was then recorded on the internet connection, classified by what generated that traffic, and turned into "features", i.e. a data frame. Because the computers are "doing something" all the time, there is "normal" traffic in here, but there is a very large number of "cyber attack" related traffic, which is unrealistic of real data. Also the ability to classify everything your computer is doing to obtain "true labels" is unusual.

The details are very involved but all we really care about is that there are some labels, and our task as data scientists is to see whether the classification task of identifying attacks is feasible.

## 1.1 Data Import

**DATA ORGANISATION**: It is very helpful to keep the project structure that I use, so your code "just runs". Therefore work in a directory ("workshops") and keep your data in a directory called "data" in the PARENT DIRECTORY, e.g. both inside "dst", so that your file structure looks like this:

- dst
  - dst/data
  - dst/workshops

. This maps better to how we will structure the Assessments, which are fussier again because of their group project nature.

**OBTAINING DATA**: It is essential that it is clear how to obtain the data used in an analysis. It is OK to have manual steps if they are clearly described, but automation is best.

To automate this, DIFFERENTLY AND BETTER THAN THE RECORDING, we will use internal R functions which work regardless of file system. We would like a cross-platform (Windows/Linux/Mac) solution, and this is provided by the `file.path` function in R, rather than specifying folder locations completely. For example, Windows would use `..\\data\\file`, whereas Mac/Linux would be `../data/file`. Run `?file.path` to learn more.

This could look like this:

```{r}
if(!dir.exists(file.path("..","data"))) dir.create(file.path("..","data"))
if(!file.exists(file.path("..","data","kddcup.data_10_percent.zip"))) download.file("http://kdd.org/cupfiles/KDDCupData/1999/kddcup.data_10_percent.zip", destfile=file.path("..","data","kddcup.data_10_percent.zip"))
if(!file.exists(file.path("..","data","kddcup.names"))) download.file("http://kdd.org/cupfiles/KDDCupData/1999/kddcup.names",destfile=file.path("..","data","kddcup.names"))
```

Aside: An even better implementation would define, or find, a function like `safedircreate`, along the lines of:

```{r}
safedircreate<-function(...)
  if(!dir.exists(file.path(...))) dir.create(file.path(...))
```

This could be used as part of a `safedownloadfile` function, etc so that we only ever specify the location once. The above function is used as `safedircreate("..","data")` in this context. 

Q: When should you implement this? Can you do it so that it is reusable for your assessments?

Code checking question: is this guaranteed to always work? What would make this completely robust? Does R provide that tool already?

#### See Question Q1 in Block 1 Portfolio

about how to generate data.

## 1.2 Exploratory Data Analysis (1D)

The data are read in as follows:
```{r}
kddata<-as.data.frame(read_csv(file.path("..","data","kddcup.data_10_percent.zip"),col_names=FALSE)) ## Ignore the warnings - there is a bug with the header
kddnames=read.table(file.path("..","data","kddcup.names"),sep=":",skip=1,as.is=T)
colnames(kddata)=c(kddnames[,1],"normal") # here we fix the bug with the header
goodcat=names(which(table(kddata[,"normal"])>1))
kddata=kddata[kddata[,"normal"]%in%goodcat,]
```

Q: How important are these problems? What should we do about them? If we wanted to stop this warning, or all warnings, how would we do it? And when should we?

Lets take a look:
```{r}
head(kddata)
```
And get a summary of the data:
```{r}
summary(kddata)
```

Q: What can you "understand" about the data from this? What is worrying about it?

We need to pay attention to variables that:

* are not numeric
* Have surprising or large ranges, where the choice of linear scale may be inappropriate
* Don't vary much or at all

## Activity 1: EDA for the "normal" class label

The $Y$ value is a multiple-category label called "normal" in the data. This is the attack type, with value "normal" being "not an attack" and everything else being an attack.

Q: How did I know this? How would you find it out from the data source?

* We will make a barplot of the "normal" column, using an appropriate scaling. 
* Use the inline help (?par). 
* Consider the scaling of the "y" axis.

```{r}
par(las=2) # Ask R to plot perpendicular to the axes 
barplot((sort(table(kddata[,"normal"]))),log="y") # Log axis
```

Now we'll examine the labels separately. This is a way to make a list for each class:
```{r}
labs=unique(as.character(kddata[,"normal"]))
names(labs)=labs
kddlist=lapply(labs,function(x){
  kddata[kddata[,"normal"]==x,1:41]
})
```

The Tidyverse data objects have nice ways to examine these sorts of conditional statements.

Q: What does `lapply` do here? What sort of object is kddlist? Is this what we want?

## 1.3 Exploratory Data Analysis (1D)

## Activity 2: Heatmap of the mean for each variable within each category

```{r}
kddmean=t(sapply(kddlist,function(x)colMeans(x[,c(1,5:41)])))
library("gplots")
heatmap.2(log(kddmean+1),margins =c(9,15),trace="none",cexCol = 0.5,main="Heatmap 1")
mycols=c("dst_bytes","src_bytes","duration","dst_host_svd_count","dst_host_count","srv_count","count")
```

Q: What sort of objects are `kddmean` and `kddlist`? Why?
Q: Why has `trace="none"` been passed into heatmap.2? How do we find out what it does?
Q: What is the "+1" doing in the log?

We can obtain the top 7 features by their variance across the classes in this plot using:
```{r}
head(sort(apply(log(kddmean+1),2,var),decreasing=TRUE),7)
```

Q: What does high variance for a feature mean? How might it be important for downstream work?

Now we want to standardize the features and repeat the *heatmap*.
```{r}
kddfreq=apply(kddmean,2,function(x)x/(sum(x)+1)) ## Divide each column ()
kddfreq[!is.finite(kddfreq)]=0
kddfreq[is.nan(kddfreq)]=0
heatmap.2(kddfreq,margins =c(9,15),trace="none",cexCol = 0.5,main="Heatmap 2")
```

Q: What is this standardization? What is the "+1" doing in the frequency calculation? (We cover this later in the course)

The corresponding variance in the features for this heatmap is:
```{r}
head(sort(apply(kddfreq,2,var),decreasing=TRUE),7)
```

#### See Question Q2 in Block 1 Portfolio

about what these results mean.

## Activity 3: Comparison for the categorical variables

We will now make a *table* of the interaction between the class label and the *categorical* variables. 

```{r}
mycategorical=colnames(kddata)[2:4]
classlist=lapply(mycategorical,function(mycat){
  table(kddata[,c(mycat,"normal")])
})
for(i in 1:3) heatmap.2(log(classlist[[i]]+1),margins =c(9,15),trace="none",main=mycategorical[i])
```

Q: Some things to reflect on:

* What about scale here? Explore different choices of scaling function.
* Describe some key features that you see in the data.
* Are any of the classes *informative*? 
* In what sense are they informative?

## Activity 4: Variability

* We want to learn about the *variability* in the labels with the function "sd", given the mean. 
* For that we must *represent* variability in an appropriate format.
* It is again convenient to construct the matrices that we want to plot.

```{r}
kddsd=t(sapply(kddlist,function(x){
  apply(x[,c(1,5:41)],2,sd)
}))
heatmap.2(log(kddsd/(kddmean+0.01)+1),margins =c(9,15),trace="none",main="Heatmap 3")
```

Q: Some reflection:

* Again, what interpretations can you make about the data?
* What would happen if we do not scale the s.d. to the individual mean entry? 
* What if we use a linear instead of a log scale?
* Is high variability good, or bad, for inference?

#### See Question Q3 in Block 1 Portfolio

about what these results mean.

## Activity 5: Final thoughts and reflection

Q: Some final thoughts:

* The dataset contains attacks that are not listed here. 
* Consider what the above Exploratory Data analysis might mean for the hopes of detecting different properties of attack.
* How you might go about making a model that will perform well out-of-sample, *comparing* "normal" to other classes of "attack"?
* You might choose to explore this further in Assessment 0.