A2-data.qmd

# Data {#data}

This chapter describes the datasets used throughout the book as listed in `r ifelse(knitr::is_html_output(), '@tbl-datalist-html', '@tbl-datalist-pdf')`. 

```{r}
#| message: false
#| echo: false
library(readr)
library(dplyr)

datasets <- read_csv("misc/datasets.csv") |> 
  dplyr::select(-`Other Notes`)
```

::: {.content-visible when-format="html"}

```{r}
#| message: false
#| echo: false
#| label: tbl-datalist-html
#| tbl-cap: "List of data sets and their sources used in the book examples."
library(gt)
library(gtExtras)
library(gtsummary)
datasets |> gt() |> 
  cols_width(Name~px(150), 
             Description~px(250),
             Source~px(100), 
             Analysis~px(100))
```
:::

::: {.content-visible when-format="pdf"}

```{r}
#| message: false
#| echo: false
#| label: tbl-datalist-pdf
#| tbl-cap: "List of data sets and their sources used in the book examples."
library(kableExtra)
datasets |>
  dplyr::select(Name, Description, Analysis) |>
  knitr::kable(format="latex", booktabs = T) |>
  kable_paper(full_width = F) |>
  kable_styling(font_size = 8) |>
  column_spec(1, width = "2cm") |>
  column_spec(2, width = "5cm") |>
  column_spec(3, width = "4cm") 
```
:::

## Australian Football League Women

### Description {-}

The `aflw` data is from the 2021 Women's Australian Football League. These are average player statistics across the season, with game statistics provided by the [fitzRoy](https://jimmyday12.github.io/fitzRoy/) package. If you are new to the game of AFL, there is a nice explanation on [Wikipedia](https://en.wikipedia.org/wiki/Women%27s_Australian_rules_football). 

### Variables {-}

```{r}
#| label: load-data-libraries
#| echo: FALSE
#| message: FALSE
#| warning: FALSE
library(mulgar)
library(dplyr)
```

```{r}
#| label: data-aflw
#| echo: FALSE
#| message: FALSE
#| warning: FALSE
data(aflw)
glimpse(aflw)
```

### Purpose {-}

The primary analysis is to summarise the variation using principal component analysis, which gives information about relationships between the statistics or skills sets common in players. One also might be tempted to cluster the players, but there are no obvious clusters so it could be frustrating. At best one could partition the players into groups, while recognising there are no absolutely distinct and separated groups.

### Source {-}

See the information provided with the [fitzRoy](https://jimmyday12.github.io/fitzRoy/) package.

### Pre-processing {-}

The code for downloading and pre-processing the data is available at the [mulgar](https://dicook.github.io/mulgar/) website in the `data-raw` folder. The data provided by the fitzRoy package was pre-processed to reduce the variables to only those that relate to player skills and performance. It is possible that using some transformations on the variables would be useful to make them less skewed. 

## Bushfires

### Description {-}

This data was collated by Weihao (Patrick) Li as part of his Honours research at Monash University. It contains fire ignitions as detected from satellite hotspots, and processed using the [spotoroo](https://tengmcing.github.io/spotoroo/) package, augmented with measurements on weather, vegetation, proximity to human activity. The cause variable is predicted based on historical fire ignition data collected by County Fire Authority personnel.

### Variables {-}

```{r}
#| echo: false
library(mulgar)
data(bushfires)
glimpse(bushfires)
```

### Purpose {-}

The primary goal is to predict the cause of the bushfire using the weather and distance from human activity variables provided. 

### Source {-}

Collated data was part of Weihao Li's Honours thesis, which is not publicly available. The hotspots data was collected from @jaxa, climate data was taken from the Australian Bureau of Meteorology using the `bomrang` package [@R-bomrang], wind data from @wind and @windasos, vegetation data from @forest, distance from roads calculated using @OpenStreetMap, CFA stations from @cfa, and campsites from @recreation. The cause was predicted from training data provided by @fireorigin.

### Pre-processing {-}

The 60 variables are too many to view with a tour, so it should be pre-processed using principal component analysis. The categorical variables of FOR_TYPE and FOR_CAT are removed. It would be possible to keep these if they are converted to dummy (binary variables).


## Australian election data

### Description {-}

This is data from a study on the relationship between voting patterns and socio-demographic characteristics of Australian electorates reported in @eechidna. These are the predictor variables upon which voting percentages are modelled. There are two years of data in `oz_election_2001` and `oz_election_2016`.

### Variables {-}

```{r}
#| code-fold: false
#| eval: false
load("data/oz_election_2001.rda")
load("data/oz_election_2016.rda")
glimpse(oz_election_2001)
```

### Purpose {-}

The tour is used to check for multicollinearity between predictors, that might adversely affect the linear model fit. 

### Source {-}

The data was compiled from  Australian Electoral Commission (AEC) and the Australian
38 Bureau of Statistics (ABS). Code to construct the data, and the original data are available at https://github.com/jforbes14/eechidna-paper. 

### Pre-processing {-}

Considerable pre-processing was done to produce these data sets. The original data was wrangled into tidy form, some variables were log transformed to reduce skewness, and a subset of variables was chosen. 

## Palmer penguins

### Description {-}

This data measure four physical characteristics of three species of penguins.  

### Variables {-}

|Name      | Description     |
|:---------|:----------------|
|`bl`      | a number denoting bill length (millimeters) |
`bd`      | a number denoting bill depth (millimeters) |
|`fl`      | an integer denoting flipper length (millimeters) |
|`bm`      | an integer denoting body mass (grams) |
|`species` | a factor denoting penguin species (Adélie, Chinstrap and Gentoo) |

### Purpose {-}

The primary goal is to find a combination of the four variables where the three species are distinct. This is also a useful data set to illustrate cluster analysis.

### Source {-}

Details of the penguins data can be found at [https://allisonhorst.github.io/palmerpenguins/](https://allisonhorst.github.io/palmerpenguins/), and @R-palmerpenguins is the package source.

### Pre-processing {-}

The data is loaded from the `palmerpenguins` package. The four physical measurement variables and the species are selected, and the penguins with missing values are removed. Variables are standardised, and their names are shortened.


```{r}
#| label: penguins_echo
#| message: FALSE
#| eval: FALSE
#| echo: TRUE
#| code-fold: FALSE
library(palmerpenguins)
penguins <- penguins %>%
  na.omit() # 11 observations out of 344 removed
# use only vars of interest, and standardise
# them for easier interpretation
penguins_sub <- penguins[,c(3:6, 1)] %>% 
  mutate(across(where(is.numeric),  ~ scale(.)[,1])) %>%
  rename(bl = bill_length_mm,
         bd = bill_depth_mm,
         fl = flipper_length_mm,
         bm = body_mass_g) %>%
  as.data.frame()
save(penguins_sub, file="data/penguins_sub.rda")
```


## Program for International Student Assessment 

### Description {-}

The `pisa` data contains plausible scores for math, reading and science of Australian and Indonesian students from the 2018 testing cycle.  The plausible scores are simulated from a model fitted to the original data, to preserve privacy of the students.

### Variables {-}

|Name      | Description     |
|:---------|:----------------|
|`CNT`      | country, either AUS for Australia or IDN for Indonesia |
|`PV1MATH`-`PV10MATH` | plausible scores for math |
|`PV1READ`-`PV10READ` | plausible scores for reading |
|`PV1SCIE`-`PV10SCIE` | plausible scores for science |

### Purpose {-}

Primarily this data is useful as an example for dimension reduction.

### Source {-}

The full data is available from https://www.oecd.org/pisa/. There are records of the student test scores, along with survey data from the students, their households and their schools.

### Pre-processing {-}

The data was reduced to country and the plausible scores, and filtered to the two countries. It may be helpful to know that the SPSS format data was used, and was read into R using the `read_sav()` function in the `haven` package. 

## Sketches 

### Description {-}

This data is a subset of images from https://quickdraw.withgoogle.com. The subset was created using the quickdraw R package at https://huizezhang-sherry.github.io/quickdraw/. It has 6 different groups: banana, boomerang, cactus, crab, flip flops, kangaroo. Each image is 28x28 pixels. The `sketches_train` data would be used to train a classification model, and the unlabelled `sketches_test` can be used for prediction. 

### Variables {-}

|Name      | Description     |
|:---------|:----------------|
|`V1-V784`      | grey scale 0-255 |
|`word` | what the person was asked to draw, NA in the test data |
|`id` | unique id for each sketch |

### Purpose {-}

Primarily this data is useful as an example for supervised classification, and also dimension reduction.

### Source {-}

The full data is available from https://quickdraw.withgoogle.com. 

### Pre-processing {-}

It is typically useful to pre-process this data into principal components. This code can also be useful for plotting one of the sketches in a recognisable form:

```{r}
#| code-fold: false
#| label: fig-sketches
#| fig-cap: One of the sketches in the subset of training data.
#| fig-width: 1
#| fig-height: 1
#| out-width: 20%
#| fig-align: center
library(mulgar)
library(ggplot2)
data("sketches_train")
set.seed(77)
x <- sketches_train[sample(1:nrow(sketches_train), 1), ]
xm <- data.frame(gry=t(as.matrix(x[,1:784])),
        x=rep(1:28, 28),
        y=rep(28:1, rep(28, 28)))
ggplot(xm, aes(x=x, y=y, fill=gry)) +
  geom_tile() +
  scale_fill_gradientn(colors = gray.colors(256, 
                                     start = 0, 
                                     end = 1, 
                                     rev = TRUE )) +
  ggtitle(x$word) +
  theme_void() + 
    theme(legend.position="none")
```

## `multicluster`

```{r}
library(mulgar)
data("multicluster")
```

### Description {-}

This data has 10 numeric variables, and a class variable labelling groups.

### Variables {-}

|Name      | Description     |
|:---------|:----------------|
|`group`   | cluster label |
|`x1-x10`  | numeric variables |

### Purpose {-}

The primary goal is to find the different clusters. 

### Source {-}

This data is originally from http://ifs.tuwien.ac.at/dm/download/multiChallenge-matrix.txt, and provided as a challenge for non-linear dimension reduction. It was used as an example in Lee, Laa, Cook (2023) https://doi.org/10.52933/jdssv.v2i3.

## `clusters`, `clusters_nonlin`, `simple_clusters`

```{r}
library(mulgar)
data("clusters")
data("clusters_nonlin")
data("simple_clusters")
```

### Description {-}

This data has a various number of numeric variables, and a class variable labelling the clusters.

### Variables {-}

|Name      | Description     |
|:---------|:----------------|
|`x1-x5`   | numeric variables |
|`cl`      | cluster label |

### Purpose {-}

The primary goal is to find the different clusters. 

### Source {-}

Simulated using the code in the `simulate.R` file of the `data-raw` directory of the `mulgar` package.

## `plane`, `plane_nonlin`, `box`

```{r}
library(mulgar)
data("plane")
data("plane_nonlin")
data("box")
```

### Description {-}

This data has a varying number of numeric variables. 

### Variables {-}

|Name      | Description     |
|:---------|:----------------|
|`x1-x5`   | numeric variables |

### Purpose {-}

The primary goal is to understand how many dimensions the data spreads out. 

### Source {-}

Simulated using the code in the `simulate.R` file of the `data-raw` directory of the `mulgar` package.

## `c1` - `c7`

### Description {-}

This data has a varying number of numeric variables, and a variety of cluster shapes. 

### Variables {-}

|Name      | Description     |
|:---------|:----------------|
|`x1`, `x2`, ...   | numeric variables |

### Purpose {-}

The primary goal is to detect the different clusters. 

### Source {-}

These are challenge data sets, so the code to simulate them is not made available.

## Fashion MNIST

### Description {-}

This data is a subset of images from https://github.com/zalandoresearch/fashion-mnist. Each image is 28x28 grayscale image. The training set has 60,000 images, and the test set has 10,000 images.

### Variables {-}

|Name      | Description     |
|:---------|:----------------|
|`x`       | arrays of grey scale values 0-255 |
|`y`       | label 0-9, corresponding to T-shirt/top, Trouser, Pullover, Dress, Coat, Sandal, Shirt, Sneaker, Bag, Ankle boot |

### Purpose {-}

Primarily this data is useful as an example for neural network modeling, following the tutorial at https://tensorflow.rstudio.com/tutorials/keras/classification.

### Source {-}

The data is available from https://github.com/zalandoresearch/fashion-mnist. 

## `risk_MSA`

### Description {-}

The data was collected in Australia in 2015 [@risk-survey] and includes six types of risks (recreational, health, career, financial, safety and social) with responses on a scale from 1 (never) to 5 (very often).

### Variables {-}

```{r}
#| label: data-risk
#| echo: FALSE
#| message: FALSE
#| warning: FALSE
library(MSA)
data("risk", package = "MSA")
glimpse(as_tibble(risk))
```

### Purpose {-}

This data is useful for the demonstration of clustering methods, it was also used in @msabook.

### Source {-}

The data is available from https://homepage.boku.ac.at/leisch/MSA/. 

## Peripheral Blood Mononuclear Cells 

### Description {-}

The data was described in @chen2023, which is available through the R package `Seurat` (@seurat1, @seurat2, @seurat3, @seurat4). Here the data has been pre-processed following the tutorial at https://satijalab.org/seurat/articles/pbmc3k_tutorial.html, and the first 50 PCs are made available in the data file `pbmc_pca_50.rds` which is read into R using the `readRDS()` function.

### Variables {-}

|Name      | Description     |
|:---------|:----------------|
|`PC_1`-`PC_50`       |  Principal component scores |


### Purpose {-}

The purpose is to understand the clustering of cell types, relative to clustering in the gene expression. Here, our purpose is to determine if the low-dimensional representation provided by NLDR is an accurate representation of the clustering, as understood from using the tour on the PCs. We ignore the cell type labels, and focus on the geometric shapes of clumps and clusters in the high dimensions.

### Source {-}

The data can be downloaded and pre-processed following https://satijalab.org/seurat/articles/pbmc3k_tutorial.html. 


<!--

## Notes to self

There were fifteen datasets listed in chapter 7 of the first edition. Several
of these were related to networks which we are not including this time. I have
tried to give a mix of things on a variety of topics. It would be nice to
see if there's updated versions of the "tips" 
(there seems to be a lot of noise in the literature here and no open datasets) and 
Di's music data (perhaps we could scrape our own spotify accounts to get an equivalent), there are also a few audio challenge datasets like [FSD50K](https://zenodo.org/record/4060432#.YCHksxMzbWY) . I think
it would be useful to have more unstructured data sets like natural text that we have used for 1010.


### Other possible sources for data {-}

There are now many search engines available for datasets that originate
from research contexts that list licensing information and DOIs:

* https://zenodo.org (mostly ecology/biology)
* https://datadryad.org/stash (mostly biology)
* https://dataverse.harvard.edu/dataverse/harvard/ (mostly social sciences, but has a mixture of things)
* There's also a big list of datasets here: https://docs.google.com/spreadsheets/d/1ejOJTNTL5ApCuGTUciV0REEEAqvhI2Rd2FCoj7afops/edit#gid=0 (all psychology related)
* TidyTuesday
    * https://github.com/rfordatascience/tidytuesday/blob/master/data/2022/2022-03-29/readme.md Would need to rearrange data to look at count, participation, revenue, expenditure
* gapminder
* accounting records
* Lyn's ecology data
* learningtower, yowie

-->