-
Notifications
You must be signed in to change notification settings - Fork 5
/
index.Rmd
2658 lines (1959 loc) · 131 KB
/
index.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
---
title: "Open Case Studies: Exploring global patterns of obesity across rural and urban regions"
css: style.css
output:
html_document:
includes:
in_header: GA_Script.Rhtml
self_contained: yes
code_download: yes
highlight: tango
number_sections: no
theme: cosmo
toc: yes
toc_float: yes
pdf_document:
toc: yes
word_document:
toc: yes
---
<style>
#TOC {
background: url("https://opencasestudies.github.io/img/icon-bahi.png");
background-size: contain;
padding-top: 240px !important;
background-repeat: no-repeat;
}
</style>
<!-- Open all links in new tab-->
<base target="_blank"/>
<div id="google_translate_element"></div>
<script type="text/javascript" src='//translate.google.com/translate_a/element.js?cb=googleTranslateElementInit'></script>
<script type="text/javascript">
function googleTranslateElementInit() {
new google.translate.TranslateElement({pageLanguage: 'en'}, 'google_translate_element');
}
</script>
```{r setup, include=FALSE}
knitr::opts_chunk$set(include = TRUE, comment = NA, echo = TRUE,
message = FALSE, warning = FALSE, cache = FALSE,
fig.align = "center", out.width = '90%')
library(here)
library(knitr)
library(magrittr)
remotes::install_github("benmarwick/wordcountaddin", type = "source", dependencies = TRUE)
remotes::install_github("alistaire47/read.so")
library(wordcountaddin)
library(read.so)
rmarkdown:::perf_timer_reset_all()
rmarkdown:::perf_timer_start("render")
```
#### {.outline }
```{r, echo = FALSE}
knitr::include_graphics(here::here("img", "main_plot.png"))
```
####
#### {.disclaimer_block}
**Disclaimer**: The purpose of the [Open Case Studies](https://opencasestudies.github.io){target="_blank"} project is **to demonstrate the use of various data science methods, tools, and software in the context of messy, real-world data**. A given case study does not cover all aspects of the research process, is not claiming to be the most appropriate way to analyze a given dataset, and should not be used in the context of making policy decisions without external consultation from scientific experts.
####
#### {.license_block}
This work is licensed under the Creative Commons Attribution-NonCommercial 3.0 [(CC BY-NC 3.0)](https://creativecommons.org/licenses/by-nc/3.0/us/){target="_blank"} United States License.
####
#### {.reference_block}
To cite this case study please use:
Wright, Carrie and Meng, Qier and Jager, Leah and Taub, Margaret and Hicks, Stephanie. (2020). https://github.com/opencasestudies/ocs-bp-rural-and-urban-obesity. Exploring global patterns of obesity across rural and urban regions (Version v1.0.0).
####
To access the GitHub Repository for this case study see here: https://github.com/opencasestudies/ocs-bp-rural-and-urban-obesity.
You may also access and download the data using our `OCSdata` package. To learn more about this package including examples, see this [link](https://github.com/opencasestudies/OCSdata). Here is how you would install this package:
```{r, eval=FALSE}
install.packages("OCSdata")
```
This case study is part of a series of public health case studies for the [Bloomberg American Health Initiative](https://americanhealth.jhu.edu/open-case-studies).
***
The total reading time for this case study is calculated via [koRpus](https://github.com/unDocUMeantIt/koRpus) and shown below:
```{r, echo=FALSE}
readtable = text_stats("index.Rmd") # producing reading time markdown table
readtime = read.so::read.md(readtable) %>% dplyr::select(Method, koRpus) %>% # reading table into dataframe, selecting relevant factors
dplyr::filter(Method == "Reading time") %>% # dropping unnecessary rows
dplyr::mutate(koRpus = paste(round(as.numeric(stringr::str_split(koRpus, " ")[[1]][1])), "minutes")) %>% # rounding reading time estimate
dplyr::mutate(Method = "koRpus") %>% dplyr::relocate(koRpus, .before = Method) %>% dplyr::rename(`Reading Time` = koRpus) # reorganizing table
knitr::kable(readtime, format="markdown")
```
***
**Readability Score: **
A readability index estimates the reading difficulty level of a particular text. Flesch-Kincaid, FORCAST, and SMOG are three common readability indices that were calculated for this case study via [koRpus](https://github.com/unDocUMeantIt/koRpus). These indices provide an estimation of the minimum reading level required to comprehend this case study by grade and age.
```{r, echo=FALSE}
rt = wordcountaddin::readability("index.Rmd", quiet=TRUE) # producing readability markdown table
df = read.so::read.md(rt) %>% dplyr::select(index, grade, age) %>% # reading table into dataframe, selecting relevant factors
tidyr::drop_na() %>% dplyr::mutate(grade = round(as.numeric(grade)), # dropping rows with missing values, rounding age and grade columns
age = round(as.numeric(age))
)
knitr::kable(df, format="markdown")
```
***
Please help us by filling out our survey.
<div style="display: flex; justify-content: center;"><iframe src="https://docs.google.com/forms/d/e/1FAIpQLSfpN4FN3KELqBNEgf2Atpi7Wy7Nqy2beSkFQINL7Y5sAMV5_w/viewform?embedded=true" width="1200" height="700" frameborder="0" marginheight="0" marginwidth="0">Loading…</iframe></div>
## **Motivation**
Body mass index ([BMI](https://www.cdc.gov/healthyweight/assessing/bmi/adult_bmi/index.html){target="_blank"}) is often used as a proxy for **adiposity** (the condition of having excess body fat) and is measured as an individual's weight in kilograms (kg) or pounds (lbs) divided by the individual's height in meters ($m^2$) squared.
There are different categories of weight defined using different BMI thresholds as defined by the [World Health Organization](http://www.euro.who.int/en/health-topics/disease-prevention/nutrition/a-healthy-lifestyle/body-mass-index-bmi){target="_blank"} (WHO), the [Centers for Disease Control and Prevention](https://www.cdc.gov/obesity/adult/defining.html){target="_blank"}, and the [National Institutes of Health](https://www.nhlbi.nih.gov/sites/default/files/media/docs/obesity-evidence-review.pdf){target="_blank"}.
For example, these are the categories defined by the WHO:
```{r, echo=FALSE, out.width='100%'}
knitr::include_graphics(here::here("img","obesity_classes.jpeg"))
```
###### [[source](https://www.ncbi.nlm.nih.gov/books/NBK535456/bin/bmi_WHO.jpg){target="_blank"}]
The following chart can help you identify your estimate of BMI:
```{r, echo=FALSE, out.width='100%'}
knitr::include_graphics(here::here("img","BMIchart.jpeg"))
```
###### [[source](http://diet.mayoclinic.org/diet/eat/what-is-your-bmi?xid=nl_MayoClinicDiet_20160426){target="_blank"}]
Recently, an [article](https://www.nature.com/articles/s41586-019-1171-x.pdf){target="_blank"} published in [*Nature*](https://www.nature.com){target="_blank"} evaluated and compared the BMI of populations in rural and urban communities around the world:
```{r, echo=FALSE, out.width='100%'}
knitr::include_graphics(here::here("img", "paper.png"))
```
#### {.reference_block}
NCD Risk Factor Collaboration (NCD-RisC). Rising rural body-mass index is the main driver of the global obesity epidemic in adults. *Nature* **569**, 260–264 (2019).
####
The article challenged the widely-held view that increased urbanization was one of the major reasons for increased global obesity rates.
This view came about because many countries around the world have shown increased urbanization levels in parallel with increased obesity rates.
In this article, however, the NCD-RisC argued that this might not be the case and that in fact for most regions around the world, BMI measurements are increasing in rural populations just as much, if not more so, than urban populations.
Furthermore, this study suggested that obesity has particularly increased in female populations in rural regions:
> "We noted a persistently higher rural BMI, especially for women."
In this case study, we will evaluate the data reported in this article to explore regional and gender specific differences in the obesity rates around the world in 1985 and 2017.
Most importantly we will test if there is a difference in obesity rates between rural and urban communities and if there has been a change in obesity over time for these regions, particularly for women.
To do this we will test if there is a difference between the average BMI for each group.
## **Main Questions**
***
#### {.main_question_block}
<b><u> Our main questions are: </u></b>
1) Is there a difference between rural and urban BMI estimates around the world? In particular, what does this difference look like for women?
2) How have BMI estimates changed from 1985 to 2017? In particular, what does this change over time look like for women?
3) How do different countries vary in terms of estimates of BMI? In particular, how does the United States compare to the rest of the world?
####
## **Learning Objectives**
***
The skills, methods, and concepts that students will be familiar with by the end of this case study are:
<u>**Data science Learning Objectives:**</u>
1. Importing data from a PDF (`pdftools`)
2. Subsetting and filtering data (`dplyr`)
3. Working with character strings (`stringr`)
4. Reshaping data into different formats (`tidyr`)
5. Applying functions to all columns of a tibble (`purrr`)
6. Creating data visualizations (`ggplot2`) with labels (`ggrepel`)
7. Combining multiple plots (`cowplot` and `patchwork`)
<u>**Statistical Learning Objectives:**</u>
1. Familiarity with the use of Quantile-Quantile plots to assess normality
2. Define and understand the utility of alpha and the p value
3. Describe the difference between nonparametric and parametric tests
4. Be able to identify paired data
5. Implementation of a [paired $t$-test](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5579465/){target="_blank"}
6. Interpretation of a [paired $t$-test](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5579465/){target="_blank"}
7. Implementation of a [Wilcoxon signed-rank test](http://www.biostathandbook.com/wilcoxonsignedrank.html){target="_blank"}
8.Interpretation of a [Wilcoxon signed-rank test](http://www.biostathandbook.com/wilcoxonsignedrank.html){target="_blank"}
9. Understanding of the need for multiple testing correction
In this case study, we will walk you through importing data from a pdf, cleaning, wrangling and visualizing the data, and <b> comparing two groups </b>.
We will use well-established and commonly used packages, including `stringr`, `tidyr`, `dplyr`, `purrr`, and `ggplot2` from the [`tidyverse`](https://www.tidyverse.org/){target="_blank"}.
Specifically, the tidyverse is
> "an opinionated collection of R packages designed for data science. All packages share an underlying philosophy and common APIs".
Another way of putting it is that it's a set of packages that are useful specifically for data manipulation, exploration, and visualization with a common philosophy.
While some students may be familiar with previous packages from the R programming language, these packages make data science in R especially efficient.
```{r, echo = FALSE, out.width = "20%"}
knitr::include_graphics("https://tidyverse.tidyverse.org/logo.png")
```
We will begin by loading the packages that we will need with a brief explanation:
```{r}
library(here)
library(pdftools)
library(stringr)
library(readr)
library(dplyr)
library(tibble)
library(magrittr)
library(glue)
library(purrr)
library(tidyr)
library(ggplot2)
library(ggrepel)
library(cowplot)
library(patchwork)
library(OCSdata)
```
<u>**Packages used in this case study:** </u>
Package | Use in this case study
---------- |-------------
[here](https://github.com/jennybc/here_here){target="_blank"} | to easily load and save data with relative paths
[pdftools](https://cran.r-project.org/web/packages/pdftools/pdftools.pdf){target="_blank"} | to read a text from pdf into R
[stringr](https://stringr.tidyverse.org/articles/stringr.html){target="_blank"} | to manipulate the text data
[readr](https://readr.tidyverse.org/){target="_blank"} | to manipulate the text data within the pdf into individual lines
[dplyr](https://dplyr.tidyverse.org/){target="_blank"} | to arrange/filter/select subsets of the data
[tibble](https://tibble.tidyverse.org/){target="_blank"} | to create data objects that we can manipulate with `dplyr`/`stringr`/`tidyr`/`purrr`
[magrittr](https://magrittr.tidyverse.org/articles/magrittr.html){target="_blank"} | to use the `%<>%` piping operator
[glue](https://www.tidyverse.org/blog/2017/10/glue-1.2.0/){target="_blank"} | to paste or combine character strings and data together
[purrr](https://purrr.tidyverse.org/){target="_blank"} | to perform functions on all columns of a tibble
[tidyr](https://tidyr.tidyverse.org/){target="_blank"} | to convert data from 'wide' to 'long' format
[ggplot2](https://ggplot2.tidyverse.org/) | to make visualizations with multiple layers
[ggrepel](https://cran.r-project.org/web/packages/ggrepel/vignettes/ggrepel.html){target="_blank"} | to allow labels in figures not to overlap
[cowplot](https://cran.r-project.org/web/packages/cowplot/vignettes/introduction.html){target="_blank"} and [patchwork](https://github.com/thomasp85/patchwork){target="_blank"} | to allow plots to be combined
[OCSdata](https://github.com/opencasestudies/OCSdata){target="_blank"} | to access and download OCS data files
___
The first time we use a function, we will use the `::` to indicate which package we are using (e.g. `stringr::str_detect()`).
Unless we have overlapping function names, this is not necessary, but we will include it here to be informative about where the functions we will use come from.
## **Context**
***
Previous work has shown higher BMI (>30) is associated with [higher rates of all-causes of mortality](https://www.nhlbi.nih.gov/sites/default/files/media/docs/obesity-evidence-review.pdf){target="_blank"}, as well as increased rates of type 2 diabetes, cancer, heart disease, and stroke.
However, there appears to be caveats associated with the use of a threshold like this to evaluate public health risk in various populations.
According to this [report](https://www.who.int/nutrition/publications/bmi_asia_strategies.pdf){target="_blank"}:
>...the associations between BMI, percentage of body fat, and body fat distribution differ across populations.
> In particular, in some Asian populations a specific BMI reflects a higher percentage of body fat.
> Some Pacific populations also have a lower percentage of body fat at a given BMI.
> The associations of BMI and comorbidities are probably not stable within populations over time.
##### [[source]](https://www.who.int/nutrition/publications/bmi_asia_strategies.pdf)
## **Limitations**
***
The measurement of BMI has some [limitations](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4890841/pdf/nt-50-117.pdf){target="_blank"} that are well recognized, as it does not account for the composition of body mass, the location of body fat, or the contribution of body frame size.
However, [BMI has been a useful health indicator](https://journals.lww.com/acsm-healthfitness/Fulltext/2016/07000/THE_BENEFITS_OF_BODY_MASS_INDEX_AND_WAIST.8.aspx#pdf-link){target="_blank"} for risk for many diseases and conditions particularly when combined with other risk factor information.
While [gender](https://www.genderspectrum.org/quick-links/understanding-gender/){target="_blank"} and [sex](https://www.who.int/genomics/gender/en/index1.html){target="_blank"} are not actually binary, the data presented that is used in this analysis only contain data for groups of individuals described as men or women.
## **What are the data?**
***
We will be using data located within a table of the [supplementary material](https://static-content.springer.com/esm/art%3A10.1038%2Fs41586-019-1171-x/MediaObjects/41586_2019_1171_MOESM1_ESM.pdf){target="_blank"} for the NCD-RisC paper referenced above.
This is a pdf that can be found freely available online. Here is a screenshot of the first few rows of this table:
```{r, echo = FALSE}
knitr::include_graphics(here::here("img","first_page.png"))
```
You can see that the data contain average BMI estimates for both men and women in countries at the national level, as well as the average BMI estimates for the rural and urban areas of these countries in both 1985 and 2017.
The data within the parentheses are the 95% [credible intervals](https://en.wikipedia.org/wiki/Credible_interval){target="_blank"} (CIs) for the average BMI estimates. The authors provide these CIs as a guide to understand how likely the estimate is for the true population mean BMI. A wider range suggests that the estimate is less accurate, as there are more possible values for the true mean with credible evidence. Notice that the mean BMI values are "age-standardised," which means that the mean values were adjusted for the different age distributions of the various countries so that the countries can be more fairly compared. For example, if one country has a population that is considerably younger, the mean BMI might be quite low (as younger individuals tend to have lower BMI values due to faster metabolic rates). One might falsely conclude that people in that country generally have lower BMI values than people in most other countries, however the lower overall BMI mean might simply be due to the fact that the country has a younger population than those other countries.
## **Data Import**
***
First let's download the data from the PDF file.
We use `file.exists()` from base R to check if the file we want to download already exists.
If not, then we download it using the function `utils::download.file()`. We also use the `here()` function of the `here` package to look specifically in the `raw` subdirectory of the `data` directory of the directory where our open RStudio project file is located. This helps us, as we do not need to write the full path to the document. This makes our code more reproducible for someone else, who may have a different overall file structure on their computer but downloads our case study files from GitHub and therefore has the same file structure specifically for the files included in our GitHub repository. Thus our commands about file locations still work on their local machine.
```{r}
if(!file.exists(here("data", "raw", "paper_supplement.pdf"))){
url <- "https://static-content.springer.com/esm/art%3A10.1038%2Fs41586-019-1171-x/MediaObjects/41586_2019_1171_MOESM1_ESM.pdf"
utils::download.file(url, here("data", "raw", "paper_supplement.pdf"))
}
```
We will save our data to a directory within our working directory called data. We will create subdirectories within this directory to organize our data. We can use the `here` function from the `here` package to make this process easier. The `here()` function allows us to specify the path or location of the document that we want to import, starting from the directory where an `.Rproj` file is located after creating a project in RStudio. In this case, we will import our files within subdirectories of a directory called `raw` of the `data` directory. (*Note the next chunk of code will only work for you if you pull the repository from GitHub and set up your file structure in the same way.*) If you had trouble downloading the data from the orginal sources you can download them from our [GitHub repository for this case study](https://github.com/opencasestudies/ocs-bp-rural-and-urban-obesity/tree/master/data) or use the `OCSdata` package:
```{r, eval=FALSE}
# install.packages("OCSdata")
library(OCSdata)
raw_data("ocs-bp-rural-and-urban-obesity", outpath = getwd())
# This will save the pdf file in a "OCSdata/data/raw/" subfolder
# in your current working directory
```
***
<details> <summary> Click here to see more about creating new projects in RStudio. </summary>
You can create a project by going to the File menu of RStudio like so:
```{r, echo = FALSE, out.width="60%"}
knitr::include_graphics(here::here("img", "New_project.png"))
```
You can also do so by clicking the project button:
```{r, echo = FALSE, out.width="60%"}
knitr::include_graphics(here::here("img", "project_button.png"))
```
See [here](https://support.rstudio.com/hc/en-us/articles/200526207-Using-Projects) to learn more about using RStudio projects and [here](https://github.com/jennybc/here_here) to learn more about the `here` package.
</details>
***
</details>
***
Now that we have downloaded the PDF file, we will read it in to R using the `pdftools` package:
```{r}
pdf_obesity <- pdftools::pdf_text(here("data", "raw", "paper_supplement.pdf"))
```
Let's take a look at the data -- the `summary()` function from base R helps us to look at the structure of R objects.
```{r}
summary(pdf_obesity)
```
We can see that we have 63 elements that are character strings.
You may also notice that the original PDF has 63 pages. Let's take a look at some of these elements and compare them to the pages in the original document.
```{r}
pdf_obesity[1] # this looks like the first page
pdf_obesity[2] # this looks like the second page
```
```{r, eval = TRUE}
str(pdf_obesity[52], nchar.max = 1600)
```
Yes, this looks like data in the table we want.
We are using the `str()` function to print just a portion of the data. The `str()` function compactly displays the structure of arbitrary R objects.
The `nchar.max` argument lets us define the maximal number of characters to show.
Note `print(pdf_obesity[52])` would show the entire contents of this page of the table.
Here is the PDF version again:
```{r, echo = FALSE}
knitr::include_graphics(here::here("img", "first_page.png"))
```
We can see that the output looks pretty similar to the pages of the pdf, but the spacing is a bit awkward.
Note that the way the data is displayed is partially influenced by the width setting of the RStudio window.
Now, we are interested in a supplementary Table 3, which has multiple pages and includes the same header on each page.
We can use that to determine what elements of our `pdf_obesity` character strings include our table.
We will use the `str_detect()` function from the `stringr` package to search for the elements that contain text that is consistently in the header.
The output of of this function will show which elements of the object (in this case pages of the pdf) include this pattern indicated as a `TRUE` or `FALSE`.
```{r}
stringr::str_detect(pattern = "Age-standardised mean BMI",
string = pdf_obesity)
```
We note, the "Age-standardised mean BMI" is part of the header in the table on every page.
The code above shows the pages that contain our table of interest (as `TRUE` values).
Now we will extract just the data for the table now and call it `rural_urban`.
To do this we will use the `str_subset()` function of the `stringr` package.
```{r}
rural_urban <- stringr::str_subset(pattern = "Age-standardised mean BMI",
string = pdf_obesity)
summary(rural_urban)
```
Here, we see there are 10 pages worth of elements in our `rural_urban` object.
Let's check the first and last page:
```{r, eval=TRUE}
str(rural_urban[1], nchar.max = 1600)
```
Using `[1]` shows just the first page of the 10 pages that contain the table.
Also, notice the new line `"\n"` values that indicate new lines.
```{r, echo = FALSE, eval = FALSE}
knitr::include_graphics(here::here("img","first_page_in_R.png"))
```
```{r, echo = FALSE}
knitr::include_graphics(here::here("img","first_page.png"))
```
This looks the same as the beginning... how about the end?
```{r}
str(rural_urban[10], nchar.max = 1800)
```
```{r, echo = FALSE }
knitr::include_graphics(here::here("img","last_page.png"))
```
Great! Our `rural_urban` object looks like it contains the entire Supplementary 3 table, as both the beginning and the end include the data we expected.
Now we will save this imported data for later use like so in a subdirectory of the `data` directory called `imported`:
```{r}
save(rural_urban, file = here::here("data", "imported", "imported_data.rda"))
```
## **Data Wrangling**
***
If you have been following along but stopped, we could load our imported data from the "data" directory like so:
```{r}
load(file = here::here("data", "imported", "imported_data.rda"))
```
***
<details> <summary> If you skipped the data import section click here. </summary>
First you need to install and load the `OCSdata` package:
```{r, eval=FALSE}
install.packages("OCSdata")
library(OCSdata)
```
Then, you may load the imported data using the following code:
```{r, eval=FALSE}
imported_data("ocs-bp-rural-and-urban-obesity", outpath = getwd())
load(here::here("OCSdata", "data", "imported", "imported_data.rda"))
```
If the package does not work for you, alternatively, an RDA file (stands for R data) of the data can be found [here](https://github.com//opencasestudies/ocs-bp-rural-and-urban-obesity/master/data/imported) or slightly more directly [here](https://raw.githubusercontent.com/opencasestudies/ocs-bp-rural-and-urban-obesity/master/data/imported/impoted_data.rda). Download this file and then place it in your current working directory within a subdirectory called `imported` within a subdirectory called `data` to copy and paste our code. We used an RStudio project and the [`here` package](https://github.com/jennybc/here_here) to navigate to the file more easily.
```{r}
load(here::here("data", "imported", "imported_data.rda"))
```
***
<details> <summary> Click here to see more about creating new projects in RStudio. </summary>
You can create a project by going to the File menu of RStudio like so:
```{r, echo = FALSE, out.width="60%"}
knitr::include_graphics(here::here("img", "New_project.png"))
```
You can also do so by clicking the project button:
```{r, echo = FALSE, out.width="60%"}
knitr::include_graphics(here::here("img", "project_button.png"))
```
See [here](https://support.rstudio.com/hc/en-us/articles/200526207-Using-Projects) to learn more about using RStudio projects and [here](https://github.com/jennybc/here_here) to learn more about the `here` package.
</details>
***
</details>
***
At this point we have large strings now for each page of the table, but this is not very convenient to work with.
Now, we will wrangle the data into a more usable form.
Ideally, we would like our data to be in some sort of tabular form.
### **Separate the data into lines**
***
First, it would be useful to separate the data that is currently in 10 giant string chunks into individual lines or rows of the table. To do this we can use the `read_lines()` function of the `readr` package which will separate the lines based on the `"\n"` new line characters.
```{r}
summary(rural_urban)
rural_urban <- readr::read_lines(rural_urban)
summary(rural_urban)
# now we have 461 lines
```
We can see that each string in the `rural_urban` object is now a single line from the table.
For example, line 6 shows the data for women in Afghanistan.
```{r}
rural_urban[6]
```
### **Removing excess white-space**
***
We also have a lot of white-space. Let's get rid of the excess white spaces using `str_squish()` also from the `stringr` package.
```{r}
rural_urban <- stringr::str_squish(rural_urban)
head(rural_urban)
```
Now it is much easier to see the data.
If we look at the part of our data that contains the end of the first page of the table and the start of the second we can see that the header information is repeated, as well as a line with the page number and an empty line, and a line that says "2 2". The "2 2" comes from importing the PDF file, these are the superscript "2"s in the $m^2$ strings in the table header - no need to worry about this. You may also notice some values that are `na*` and we will deal with this later.
#### {.scrollable }
```{r}
rural_urban[44:56] # scroll through!
```
####
Although the header was necessary on all of the pages of the pdf version of the table, we only need that information once in our data.
### **Removing unnecessary repeated header information**
***
So, let's remove all the header information and the page number lines from the `rural_urban` object, then we will make a single line header for the beginning.
One way to do this is to find all lines that include either "Women" or "Men" and only keep this data.
First, let's see how many lines include "Women" or "Men".
We can do this by first identifying the lines that contain these patterns and then check the length of the resulting vector for the line numbers that contain these patterns.
Note, the `"|"` indicates that we are looking for either "Women" or "Men".
#### {.scrollable }
```{r}
#scroll through the output!
stringr::str_which(string = rural_urban,
pattern = "Women|Men")
```
####
```{r}
length(stringr::str_which(string = rural_urban,
pattern = "Women|Men"))
```
Note that `str_which()` is case sensitive, so it would not work to use "women" as the pattern, and using "men" would return the lines that contain "Wo**men**" or "Ye**men**" etc. Try this out with `pattern = "women"` and `pattern = "men"` to see the result!
OK, so this looks correct.
This includes most lines but there are gaps where the header is located. It looks like out of the original 461 lines, there are 400 lines in our table that aren't headers.
```{r}
rural_urban <- str_subset(string = rural_urban,
pattern = "Women|Men")
```
We can check our data now using either `head()` from base R or `glimpse()` from the `tibble` and `dplyr` package. (Yup, the same function is available from both packages!) The `head()` function shows us the first rows or lines of the data, while the `glimpse()` function provides us information about the total size of the object and shows us the first line or row.
```{r}
dplyr::glimpse(rural_urban)
```
Now our `rural_urban` object 400 lines instead of 461, like it did before when it contained the redundant header information.
```{r}
head(rural_urban)
```
Great! So now our data looks much better but we need to add back our header and we would like this to only be a single line to make it easy to transform our data into a table or table-like object.
As a reminder, here is what our header used to look like:
```{r, echo = FALSE, out.width = '100%'}
knitr::include_graphics(here::here("img","last_page.png"))
```
We will add a new header after first dealing with some additional spacing and missing data issues.
### **Dealing with spacing**
***
First let's try splitting our header-less data into columns based on spaces using the `str_split()` function, where we specify that we are splitting the data based on the pattern of a space (the space is included in quotes). Referring to [str_split() documentation](https://www.rdocumentation.org/packages/stringr/versions/1.4.0/topics/str_split), the `simplify` argument is set to `FALSE` by default, where a list of character vectors is returned. If it is set to `TRUE` as we will be doing here, a character matrix is returned.
Here, we will take a look at just the first 10 lines using the `[1:10,]` syntax:
#### {.scrollable }
```{r}
#scroll through the output!
str_split(string = rural_urban,
pattern = " ",
simplify = TRUE)[1:10,]
```
####
This almost worked, but unfortunately country names that have spaces will be a problem.
For example, we can see that "American Samoa" has been divided into two columns and all subsequent columns are shifted.
Let's try this another way.
As we can see by looking at our `rural_urban` data, the country information is only present when the sex is Women. Let's try to extract the country information from the rows that contain data about Women.
```{r}
head(rural_urban)
```
To do this, we start by noticing that the sex always starts with either a capital "W" if the sex is female. Note, we need to use a space before the "W" otherwise we will split some of the country names if the names starts with "W".
Here, we will also introduce the concept of piping, which uses the `%>%` from the `magrittr` package, which allows us to pipe the output from one function to the input for another function.
This is really useful when we have multiple steps, which we will show soon.
First, we will select just the data for Women and Men separately:
```{r}
Women <- str_subset(string = rural_urban,
pattern = "Women")
Men <- str_subset(string = rural_urban,
pattern = "Men")
head(Women)
head(Men)
```
And then split the `Women` data based on the pattern `pattern = " Women"` (Note that the space before the word "Women" is included - so that it is not within the country data).
Here we take our `Women` dataset and pipe it into the first argument of the `str_split()` function, where then we can split on the pattern `" Women"`. We then pipe through the `unlist()` function so that the results are a vector instead of a list. We assign this new result to the `country_split` object.
```{r}
country_split <-
Women %>%
stringr::str_split(pattern = " Women") %>%
unlist()
head(country_split)
```
However, we can also do the same thing in a single step, where we replace the `Women` dataset with the code we used to create that dataset. So first, we select the "Women" and then we split the data by " Women" to create a new dataset all within the same chunk of code.
```{r}
country_split <-
str_subset(string = rural_urban,
pattern = "Women") %>%
stringr::str_split(pattern= " Women") %>%
unlist()
head(country_split)
```
Now, we can see that `Country` is always the odd rows and the BMI data for Women is the even rows.
We can select the odd rows using the <b> Modulus (Remainder from division) operator</b>.
This operator looks like this `%%`.
All odd values such as 3, 7, or 15 when divided by 2 would leave a remainder of 1.
We will select just these rows using the `filter()` function of `dplyr`.
```{r}
country <-
tibble(country_split) %>%
dplyr::filter(row_number() %% 2 == 1)
```
We can take a look to make sure that all the country names look as expected. You can scroll through the country names if you hover over the data.
```{r, eval= FALSE}
country
```
#### {.scrollable }
```{r, echo = FALSE}
as.data.frame(country)
```
####
Looks good! We have 200 countries represented in our dataset.
Great! Now we have a list of the countries that can be used for both the male and female data.
Remember the even rows are the rows with the BMI data for women.
These row values have a remainder of 0 when divided by 2.
Let's select that data too.
```{r}
Women_BMI <-
tibble(country_split) %>%
dplyr::filter(row_number() %% 2 == 0)
head(Women_BMI)
```
This looks similar to our `Men` data (although slightly different):
```{r}
head(Men)
```
Notice how the data in each row is in quotes. This is because these are character strings. We can check this by using the base `class()` function.
```{r}
class(Men)
```
Whereas the `Women_BMI` data is a tibble.
```{r}
class(Women_BMI)
```
Let's make the `Men` data into a tibble as well:
```{r}
Men <- tibble(Men)
```
It's always a good idea to check that your data objects are the size you expect when wrangling.
We can do so with the `dim()` function, which shows us the dimensions of data objects.
```{r}
dim(Women_BMI)
dim(Men)
head(Women_BMI)
head(Men)
```
Great! There are 200 rows of character strings with the BMI data like we expected to match the 200 countries in our data.
We do however need to add the word `" Women"` back to the `Women_BMI` data.
To do this we will use the `glue()` function of the `glue` package.
Here is a simple example of how this function glues two things together.
```{r}
where <-"the moon"
glue::glue("the cow jumped over {where}")
```
Now let's add the word "Women" to each row of the `Women_BMI` data using `glue`.
First let's pull the `country_split` data out as a vector to make this simpler when we `glue`. To do this we will use the `pull()` function of the `dplyr` package. This function will allow us to extract or pull out the single variable called `country_split` which contains the BMI values for women from within the tibble called `Women_BMI` which we just created by grabbing only the even rows of our original `country_split` tibble.
Then, let's use the `first()` function of the `dplyr` package to get a reminder of what just the first row of the `country_split` column of the `Women_BMI` data looks like.
```{r}
women_data <- Women_BMI %>%
pull(country_split)
dplyr::first(women_data)
```
Now we will paste the word `"Women"` in the beginning of each value in the vector using the `glue()` function.
Note that there will be a space in between.
This is because the space is already in front of the first BMI value.
```{r}
Women <- glue("Women{women_data}")
head(Women)
```
You could also do all of this in one step but its less obvious what is happening.
Here we can pull the `country_split` data out of the `Women_BMI` object and then paste `"Women"` in front of each string in one command.
```{r, eval = FALSE}
Women <- glue("Women{pull(Women_BMI,country_split)}")
```
If we try splitting our data by space again, will it have the expected number of columns? What about the rows that contain `na*` values?
Let's just take the `Men` data that contain `na*` values.
This column is called `rural_urban`.
```{r}
unknown <-
Men%>%
filter(str_detect(pattern ="na\\*",
string = Men))
head(unknown)
```
Now we can try splitting by a space
```{r}
tibble::as_tibble(str_split(pull(unknown, Men), " ",
simplify = TRUE))
```
So close! Notice that the `"na*"` values have shifted the subsequent values within the columns because typically there is a space between the BMI and the credible intervals.
Here we can see this data in our original pdf:
```{r, echo = FALSE, out.width="100%"}
knitr::include_graphics(here::here("img","missing_pdf.png"))
```
### **Dealing with NA values**
***
We need to replace our `na*` values with something that includes a space so that when we separate our data by space, we will have two values instead of one when we have an `na*`.
Therefore, `na* na*` should work.
So, in order to use the functions within the `stringr` package, our `Men` data needs to be of `character` class, so let's check that now.
```{r}
head(Men)
```
The `<chr>` indicates that our `Men` data is already `character`. But if this is not the case, we could use the ` as.character()` function that allows us to change factor data within to the character class.
We are also changing the structure so that the data is not within a column called `Men`.
```{r}
Men <- pull(Men, Men)
head(Men)
```
```{r}
Men <-
stringr::str_replace_all(string = Men,
pattern = "na\\*",
replacement = "na\\* na\\*")
# The \\ are necessary because the * is a special character
# The * would typically indicate any possible value,
# but here we actually want a "*" instead
# Thus the double backslash does that for us
# Here we are replacing all occurrences of the na* values
#(thus str_replace_all instead of str_replace) with na* na*.
```
Let's check that it worked...
```{r}
Men[20:30]
```
Great!
Now for the Women data object
```{r}
class(Women)
# the female data is already character type
Women <-
stringr::str_replace_all(string = Women,
pattern = "na\\*",
replacement = "na\\* na\\*")
Women[20:30]
```
Great, now we can split our data by spaces.
### **Splitting the data**
***
We will split our data finally by spaces using the `str_split()` function of the `stringr` package. We can check the rows with NA values by looking at rows 20-30.
```{r}
Men <- tibble::as_tibble(str_split(Men, " ",
simplify = TRUE))
Men[20:30,]
```
Great! It looks good!
Now for the `Women` data.
```{r}
Women <- as_tibble(str_split(Women, " ",
simplify = TRUE))
head(Women)
Women[20:30,]
```
We can see that our `na` values look correct.
Looks good!
### **Adding new header**
***
We can see from our pdf and our object called `header` what the header was like in the original pdf document. We need to add column names to our data based on the original data.
We'll wait to add `Country` to the header until we add the country information to our data.
```{r}
new_header <- c("Sex","National_BMI_1985",
"National_BMI_1985_CI", "Rural_BMI_1985",
"Rural_BMI_1985_CI", "Urban_BMI_1985",
"Urban_BMI_1985_CI", "National_BMI_2017",
"National_BMI_2017_CI","Rural_BMI_2017",
"Rural_BMI_2017_CI", "Urban_BMI_2017",
"Urban_BMI_2017_CI")
```
Let's change the names of our columns of our tibbles to this new header for our `Men` and `Women` data
```{r}
names(Women) <- new_header
names(Men) <- new_header
head(Women)
head(Men)
```
Now we will add our country data to both our `Men` and `Women` tibbles using the `bind_cols()` function. This will add the `country` as a new column to the `Women` data object on the left.
```{r}
Women <- dplyr::bind_cols(country, Women)
head(Women)
```
Similarly, for the data about Men, this will add the `country` as a new column to the `Men` data object on the left.
```{r}
Men <- dplyr::bind_cols(country, Men)
head(Men)
```
### **Changing variable names**
***
Here, we will change the variable name in the `country` dataset to `country` (currently it is called `country_split`).
We will also introduce the concept of the assignment pipe.
In this case our pipe operator looks like this `%<>%`.
Using this additional pipe requires another package, called `magrittr`.
The other simpler pipe options from this package are loaded with `tidyverse` (if you used `library(tidyverse)` which loads most tidyverse packages), but not this version.
The `>` portion of the pipe still behaves like a normal pipe, while the `<` portion of the pipe makes an assignment to whatever the `<`is pointing to, just like when we use the typical assignment operator `<-`.
```{r}
# library(magrittr)
# We can't use the `%<>%` unless we load the magrittr package
# We have already done this but we include this for illustrative purposes.
# Here we will just use the traditional assignment strategy
Women <- dplyr::rename(Women, Country = country_split)
```
Here, we have renamed the `country_split` variable to be called `Country`.
Here, we reassign `Men` using the pipe strategy.
```{r}
Men %<>% rename(Country = country_split)
```
This is equivalent to
```{r, eval=FALSE}
Men <- rename(Men, Country = country_split)
```
We have renamed the `country_split` variable to to be called `Country`.
We have also reassigned `Men` to the same data object. which has the `Country` variable renamed.
Let's take a look at our data for `Men` and `Women`:
```{r}
head(Women)
head(Men)
```
### **Joining the data**
***
Now that both of these tibbles have the same structure, we can combine our `Men` and `Women` data using the `bind_rows()` function of `dplyr`. This will combine all of the rows based on their similar column structure.
All of the male data will be first and all the female data will be second.