-
Notifications
You must be signed in to change notification settings - Fork 2
/
001-introduction.Rmd
72 lines (33 loc) · 6.81 KB
/
001-introduction.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
\mainmatter
# Introduction {#intro}
## The origin of data
Data is proliferating, collected as part of scientific research (whether measuring sub-atomic particles or galaxies in the early universe, plant growth or the migration of animals, people's socio-economic characteristics or where they live and work in cities), and by the tools that are used to conduct the business of the world.
Today, data about _you_ might be collected when you:
* Buy a cup of coffee at a café. This generates at least three types of data: about the type of drink you ordered, about your coffee habits through the rewards program, and about the financial transaction—which are used by the store to measure sales and by you to manage your personal budget.
* Use a streaming service to watch a movie or listen to some music. The service provider builds a profile about your preferences, which it can then use to make recommendations as to what you might also enjoy. The service can also use the aggregated data from all subscribers to evaluate the success of the content on their platform, and pay the creators of the content.
* Renew your driver's license. Information about you such as date of birth, height, weight, and current address is recorded, which you can then use as a piece of identification to authenticate who you are. The licensing agency can also use the data in aggregate to estimate how many people will be renewing their licenses each month, and often share the data (securely, of course!) with other agencies. In British Columbia, the agency that licenses drivers shares elements of the database, such as your current address, with the provincial electoral agency who uses that information to maintain an up-to-date list of eligible voters in the province. [@EBC_Annual_201718, page 70]
* Travel to your place of work or study, captured through the movement of your cell phone. Aggregated location and movement can be used for a variety of purposes, including use by GPS systems to indicate areas of road congestion, or to let you know when the next bus will arrive at your stop.
* Respond to a survey, perhaps asking for your opinions about the platforms of various political parties and the people who represent those parties.
## Analyzing data: the data science process
A lot has been written about the benefits of organizations being "data-driven", when good data analysis leads to better decisions. There is no limit to the contexts where data analysis is being applied. In the private sector, businesses can compete on analytics [@Davenport_Harris_2007], including retailers [@stitchfix_algo] and supply chain management [@Ashton_2018]. It can also be used in the public sector: data analysis can inform
public health strategies during a pandemic [@Polonsky_etal_2019].
The process of going from raw data, collected during processes like those described above, to support decision making is often called data science. Data science combines the academic disciplines of statistics, computer programming, and the subject matter (be it astronomy, psychology, economics, or business). [@Conway_Venn_2010]
The typical data science process has been described in this model [@Wickham_Cetinkaya-Rundel_Grolemund2023]:
The generalist data scientist has a hand in all the steps through this workflow [@Monkman_bird_2019]; in this book we will focus on the Import to Tidy to Transform steps. We will also see how "Visualise" is an important part of Exploratory Data Analysis\index{exploratory data analysis}, a strategy for identifying potential problems with the data prior to cleaning.
## Data in the wild
Most data science and statistics textbooks use example data sets that are nicely formatted and easy to import. Anscombe's Quartet [@Anscombe_1973] is a famous example of a small data set that was designed to produce specific statistical results; it is included in the base version of R [@R-base]. Other data sets might be excerpted to use as teaching examples [@R-gapminder], chosen for their statistical properties [@Horst_Presmanes_Hill_Gorman_2022], compiled as examples to demonstrate a methodology or technique [@R-modeldata], or used to clean, validate, and aggregate still-larger data collections [@R-Lahman].
But in real life the data we find ourselves working with is, more often than not, entirely different from those text book examples.
> _Classroom data are like teddy bears; real data are like a grizzly with salmon blood dripping out of its mouth._ ^[quote attributed to Jenny Bryan, "Teach Data Science and They Will Come," Joint Statistical Meetings, 2015]
* Data can be stored in different formats–spreadsheets, databases, software-specific file formats, websites, and PDF files\index{Portable Document Format (PDF)}. We need to have tools to access the data, no matter the file format.
* The data might be stored in an untidy layout [@tidydata], or inconsistently structured.
* We might receive data from multiple sources, so we end up with two files that have the same value but are coded in different ways. For example, my home province in Canada can be represented as "British Columbia", "B.C.", or "BC". All three are accurate, but these variations can cause problems if they are inconsistent across the data sources (or even more problematically, in the same variable). And don't get me started on date formats—we'll save that for later.
* Data might have as-of-yet undetected errors. It might be incomplete, inconsistent, or it might have data entry mistakes and typos.
* It might have bias...and there are many different types of bias.
* The data may come from an administrative source, and collected for a purpose other than research. In these cases, you may need to transform that data to fit the requirements of the research question.
This book will introduce steps necessary to ensure that you can assemble your data in a way that allows you to undertake the rest of the data science process, including sharing the data with other data analysts who might seek to use the data in further analysis.
## Data collection
Before we can start our data science workflow—before we get to the "Import" step—someone has to collect the data. This book _won't_ cover the process of collecting data, a topic both broad and deep. Included in that would be determining the research question (or hypothesis) or business need that the data will support, designing the experiment or the business process to be captured (both could include a survey questionnaire), and undertaking the experiment (including fieldwork) or implementing the business data collection. The details of the data collection process are often domain-specific, so I encourage you to seek out reference materials that are tailored for your chosen subject area.
<!--
This file by Martin Monkman is licensed under a Creative Commons Attribution 4.0 International License.
-->