session2.Rmd

---
title: "Session 2 -- Introduction to other data structures"
---

> #### Aims
>
> * To learn about the categorical data class - *factors* 
> * To understand more complex R data structures like data frames, lists etc
> * To demonstrate the R markdown


## Factors

R has a special vector class for dealing with catergorical data - the *factor*.
Categorical data might be something such as sex - "Male", "Female" - or tumour
stage - "Stage1", "Stage2", "Stage3", etc. Critically, the possible entries, or
levels, of a factor are limited. 

Factors are particularly useful when generating plots or running statistical
analyses.

While factors look and behave like character vectors, they are actually stored
by R as an integer vector, so it is important to be careful with them when
treating them as strings or transforming them to other data types.

### Creating a factor

To create a factor we simply use the function `factor` to turn a standard
character vector into a factor.

```{r}
x <- c("orange", "apple", "apple", "orange")
x
fruit <- factor(x) # convert vector x into a factor
fruit
class(x)
class(fruit)
```

### Factors store the categories as "levels" 

Once created, factors can only contain a pre-defined set values, known as
levels. To examine what levels a particular factor has we can use the 
command `levels`:

```{r}
levels(fruit)
```

Here we can see that `fruit` has two levels: "apple" and "orange". These are
now the only valid possible values. If we try to change one of the values to
something else it will result in a missing value.

```{r}
fruit[1] <- "apple"
fruit[2] <- "banana"
fruit
```

### The values of the factors are actually stored as integers

R is actually storing these factors as integers (1, 2, 3, 4 ...). When the factor
is printed to the console for our benefit, the integers are replaced with the
corresponding level label. In this way factors also represent a data storage
solution, they can store large amounts of complex categorical data but only
use a small amount of memory. We can see this if we convert the factor to a
numeric vector using `as.numeric`.

```{r}
x <- c("low", "high", "medium", "high", "low", "medium", "high")
response <- factor(x)
as.numeric(x)
response
as.numeric(response)
```

### We can change the order of the levels

For plotting or statistical analysis the order of the levels matters. In 
plotting the data pertaining to the different levels will be plotted in
the order of the levels. In statistical analyses, such as linear modelling, 
the first level will often be automatically chosen as the reference.

Therefore, we might what to specify the order of the levels. In our current
`response` vector the order of the levels is "high", "low", "medium". We 
would probably want to switch this to "low", "medium", "high". We can do
this in the `factor` function.

```{r}
response <- factor(x, levels = c("low", "medium", "high"))
response
```

### Converting factors to plain vector types

We can convert factors back to plain vector types - character, integer etc - 
using the "as" family of functions. `as.character` will return the original
category values. `as.numeric` will return a vector of the level index for each
value.

```{r}
response
as.character(response)
as.numeric(response)
```

It is important to be careful with this when your categories look like numbers.
For example if we have a categorical vector of years and we want to convert this
to a numeric vector of years (so that perhaps we can do some arithmetic with it).

```{r}
x <- c(1987, 2002, 2019, 2004, 1983, 1982, 1999, 2004)
dateOfBirth <- factor(x)
dateOfBirth
as.character(dateOfBirth)
as.numeric(dateOfBirth)
```

To get the result we want we need to do it in two stages:

```{r}
dob <- as.character(dateOfBirth)
dob <- as.numeric(dob)
dob
age <- 2023 - dob
age
```

## Beyond the atomic vector - matrices, data.frames and lists

![Image source:http://venus.ifca.unican.es/Rintro/dataStruct.html](./images/dataStructures.png)


R has many data structures. These include:

- [x] atomic vector
- [ ] matrix
- [ ] data frame
- [ ] list

### Matrices 

In R matrices are tables of values with the same data type. They are an
extension of the atomic vector.

The matrix can be viewed as a collection of vectors:

* Matrix rows and columns are vectors  
* All the vectors hold identical data type  
* The vectors must have the same length
  
Matrices are widely using in statistical analyses and modelling.

```{r}
y <- matrix(data = 1:12, nrow = 4, ncol = 3)
y
class(y)
typeof(y)
```

In contrast to vectors, matrices and data frames are two dimensional data
structures.

* First dimension: rows  
* Second dimension: columns

We can find out about the size of the matrix using the following functions:

* dim(): Provides the dimensions of an object  
* nrow(): gives number of rows  
* ncol(): gives number of columns

```{r}
dim(y)
nrow(y)
ncol(y)
```

As with atomic vectors we can select a subset of the values of a matrix using
the `[]` operator.  However, as the matrix is a two dimensional data structure
you must provide both the rows and the columns.

The general syntax is `matrix[ rows , columns ]`:

To extract the value in the first row and the second column:

```{r}
y[1, 2]
```

If we want an entire row or an entire column, we can omit the column or row
index respectively.

```{r}
y[1, ]
y[, 3]
```

As with atomic vectors, we can use a vector within the `[]` operator to select
multiple values:

```{r}
y[c(1, 3), 2:3]
```

Like the atomic vector, matrix can hold only one type of data, if we try to mix
two or more different data type, R implicitly convert into one type:

```{r}
typeof(y)
y
y[2, 3]
y[2, 3] <- "A"
y
typeof(y)
```

### Lists

R’s simplest structure that combines data of different types is a list. Lists
are very flexible data structures that can hold anything.  A list is simply a
collection of multiple objects. The objects in the list can be of different
types and different lengths.  A list can even be a collection of lists.

```{r}
w <- c(TRUE, FALSE)
x <- 1:10
y <- c("a", "b", "c")
z <- matrix(1:12, nrow = 3, ncol = 4)

myList <- list(w, x, y, z)
myList
```

`myList` has four elements and when printed out like this looks quite
strange at first sight. Note how each of the elements of a list is referred to
by an index within 2 sets of square brackets. This gives a clue to how you can
access individual elements in the list:

1. `[]`: Standard subscript operator, works like in a vector. The output is
   still a list. 
2. `[[]]`: This operator can only take one index number at a time and the
   output will be the data structure for that element.

```{r}
length(myList)
myList[1]
myList[4]
myList[1:3]
```


```{r error = TRUE}
length(myList)
myList[[1]]
myList[[4]]
myList[[1:3]]
```

We can also name the elements in the list. This provides another way to access them: 

3. "$": to extract an element from a list by name.

```{r}
namedList <- list(
    city = c("Kampala", "London", "Paris"),
    population = c(1.51, 8.8, 2.1)
)

namedList

namedList$city
namedList$population
names(namedList)
```

You can modify lists either by adding additional elements or modifying existing
ones.

```{r}
namedList$city[2] <- "New York"
namedList$country <- c("Uganda", "USA", "France")
namedList
```

Lists can be thought of as a ragbag collection of things without a very clear
structure. You probably won’t find yourself creating list objects of the kind
we’ve seen above when analyzing your own data. However, the list provides the
basic underlying structure to the data frame that we’ll be using throughout the
rest of this course.

The other area where you’ll come across lists is as the return value for many
of the statistical tests and procedures such as linear regression that you can
carry out in R.

To demonstrate, we'll run a t-test comparing two sets of samples drawn from
subtly different normal distributions. We'll use the `rnorm()` function for
creating random numbers based on a normal distribution.

```{r}
sample_1 <- rnorm(n = 100, mean = 5, sd = 1)
sample_2 <- rnorm(n = 100, mean = 7, sd = 1)

result <- t.test(x = sample_1, y = sample_2)

is.list(result)
result
names(result)
result$statistic
result$p.value
```

### Data frames

The most commmonly used data structure is the `data.frame`. This can be though
of as a table of values. Unlike a matrix each column can have a different data
class, but each value in that column must be of the same data class.  Although
it is easiest to think of the data.frame as a table, and R will present and
treat it as a 2 dimensional table, it is actually a special type of list in
which all the elements are vectors of the same length. 

```{r} 
dat <- data.frame(
    city = c("Kampala", "London", "Paris"),
    population = c(1.51, 8.8, 2.1)
)
dat
class(dat)
dim(dat)
```

For the purposes of demonstration and testing, R provides many built-in data
sets, most of which are represented as data frames.  You can list all the
avilable built-in data sets with the `data()` command.

We will look at the `iris` data set:

* data frame  
* 150 observations (rows)    
* 5 variables (columns)  
    * Sepal.Length  
    * Sepal.Width  
    * Petal.Length  
    * Petal.Width  
    * Species  

To bring one of these internal data sets to the fore, you can just start using
it by name.

```{r}
head(iris)
```


You can also get help for a data set such as iris in the usual way.

```{r eval = FALSE}
?iris # get help for iris data
```
This reveals that iris is a rather famous old data set of measurements taken by
the esteemed British statistician and geneticist, Ronald Fisher (he of Fisher’s
exact test fame).

Viewing data frames is made easier with these functions:

* head(): Shows first 6 lines of a data frame  
* tail(): Shows first 6 lines of a data frame  
* View(): Shows data frame in excel like format  

```{r}
head(iris)
```

```{r}
tail(iris)
```

```{r eval = FALSE}
View(iris)
```

A data frame is a special type of list so you can access its columns in the
same way as we saw previously for lists.

```{r}
names(iris)
iris$Petal.Width
```
We can further subset the values in that column to, say, return the
first 10 values only.

```{r}
iris$Petal.Length[1:10]
```

As with the matrix, the `[, ]` syntax can be used to access both rows and
columns.

```{r}
iris[1:4, 1:3]
iris[c(1, 4, 7),  c(2, 4)]
```

To extract values from a data frame, row/column names may also be used

```{r}
iris[c(1, 4, 7),  c("Sepal.Width", "Species" )]
```

We can also use conditional sub-setting to extract the rows that meet certain
conditions, e.g. all the rows with Sepal.Length of 5.

```{r}
iris[iris$Sepal.Length == 5, ]
```
One can use & (and), | (or) and ! (not) logical operators for complex
sub-setting.

```{r}
iris[iris$Sepal.Length == 5 & iris$Species == "setosa", ]
```

A data frame is the most common data structure you will encounter as a
beginner. As you can see from the examples above, the syntax is tricky and not
intuitive at all. In order to overcome this problem, R has a package called
[tidyverse ](https://www.tidyverse.org/). The tidyverse package combines eight
other packages into one. Two important packages in tidyverse are `dplyr` and
`ggplot2`

* dplyr: Makes working with data frames fun and easy  
* ggplot2: Creates beautiful plots with ease.

We will start working with tidyverse in the next session.

* **`summary()`**  is a very useful function that summarises each column in a data frame

```{r}
summary(iris)
```

## Exercises

:::exercise

1. Using `mtcars` dataset answer the following.

    a. How many rows and columns in the `mtcars` data set?
    b. How many cars in the `mtcars` data set have 8 cylinders?
    c. How many cars in the `mtcars` data set have 6 cylinders and more than 3
    gears?

<details><summary>Answer</summary>

1a. How many rows and columns in the `mtcars` data set?

```{r ex_1, purl = FALSE}
nrow(mtcars)
ncol(mtcars)
```

1b. How many cars in the `mtcars` data set have 8 cylinders?

```{r}
sum(mtcars$cyl == 8)
```

1c. How many cars in the `mtcars` data set have 6 cylinders and more than 3
gears?

```{r}
sum(mtcars$cyl == 6 & mtcars$gear > 3)
```
</details>
:::


:::exercise

2. Extract 3rd and 5th rows with 1st and 3rd columns from `mtcars` data set.

<details><summary>Answer</summary>

```{r ex_2, purl = FALSE}
mtcars[c(3, 5), c(1, 3)]
```
</details>
:::


:::exercise
3. From `airquality` data set
    a. Identify the data structure of the first row by extracting it?
    b. Identify the data structure of the first column by extracting it?

<details><summary>Answer</summary>

3a. Identify the data structure of the first row by extracting it?

```{r ex_3, purl = FALSE}
class(airquality[1, ])
```

3b. Identify the data structure of the first column by extracting it?

```{r}
class(airquality[, 1])
is.vector(airquality[, 1])
is.data.frame(airquality[, 1])
```
</details>
:::


:::exercise
4. Using `airquality` data set
    a. Show first 10 rows 
    b. Show last 2 rows
    c. list all the column names available

<details><summary>Hint</summary>
Used `help` on `head` and `tail` to find out about their arguments.
</details>

<details><summary>Answer</summary>

4a. Show first 10 rows 

```{r ex_4, purl = FALSE}
head(airquality, n = 10)
```

4b. Show last 2 rows

```{r}
tail(airquality, n = 2)
```

4c. list all the column names available

```{r}
names(airquality)
colnames(airquality)
```
</details>
:::


:::exercise
5. Using `airquality` data set, can you test Ozone production and solar
   radiation are correlated and answer the following. Hint: Use`cor.test`
   function.  
    a. Get correlation coefficient confidence interval
    b. what is the p-value?
    c. What is the estimated correlation coefficient?


<details><summary>Answer</summary>

5a. Get correlation coefficient confidence interval

```{r ex_5, purl = FALSE}
results <- cor.test(airquality$Ozone, airquality$Solar.R)
results$conf.int
```

5b. what is the p-value?

```{r}
result$p.value
```

5c. What is the estimated correlation coefficient?

```{r}
results$estimate
```
</details>
:::