Scrape and Analize Olympics Data from Olympedia.org
Data was scraped from Olympedia.org in R (v4.3.2)1 using packages RCurl2, XML3, and tidyverse4, with an emphasis on team USA. Geographic data was geolocated using packages httr5 and tidyverse4, with geolocation services provided by API-Ninjas. Data analysis and visualization was performed with tidyverse4, and other supporting R code. Final figure generation was completed in ArcGIS Pro (v3.2.1)6.
This collection of original R scripts was developed with the intention of generating a geospatial visualization (i.e. a map) of USA Olympic gold medalist for submission to the University of Arizona, Data Visualization Challenge, 2024. All works herein are authored by me (Travis Zalesky) and are being provided to the GitHub community in the interest of transparency and repeatability.
A directory of links related to team USA was downloaded from olympedia.org/countries/USA. A subset of this directory was identified containing internal page links to detailed results for every event in which an American athlete competed, for every Olympics. Xml text was downloaded for each relevant page and the xml was subset to isolate the table of results. Results data was extracted from the xml table and saved as a data frame (df). Additionally, links to each American athlete's bio page were extracted and data was again downloaded as, and extracted from, xml text. A unique athlete ID number was generated for each. Athlete data was saved as a separate, large, df.
Dates were parsed for both athlete and event data from an ambiguous format using a custom function. Birthplace information for each athlete was extracted and separated from a text string and saved as an unambiguous City, State, Country format, being careful to remove extraneous punctuation and whitespaces. Birth country names were inferred from three letter International Olympic Committee (IOC) country codes (see Supporting Code). The events data was searched for bad Unicode characters and were updated as necessary with simplified text. The events data frame was then lengthened by identifying and extracting individual athletes from team events. The resulting df was then filtered to only include gold medalists and was summarized for each athlete by unique athlete ID. The gold medalists event data was joined to their biographical data by athlete ID. Next, the gold medalists were queried to identify athletes with missing birthplace data. Forty athletes with incomplete records were identified and additional biographical data was manually searched on the web and obtained from a variety of sources (see file "missing.csv"). Updated biographical data was then appended to the df as needed. Next the unique birthplaces of each athlete were extracted and summarized in a separate df which was then geolocated (see "Geolocate-API.R"). Finally, data was summarized using a variety of statistics and visualizations were generated using ggplot (a part of the tidyverse).
Each unique birthplace of all American gold medalists was queried through API-Ninjas free geolocation service (subscription required, terms apply). Geolocation results were returned as a string, from which latitudes and longitudes were extracted. Additional notes were generated, and full, detailed results were retained. Locations returning errors (primarily the result of bad Unicode characters) were manually queried and appended as necessary. Geolocation results were then saved and joined to the gold medalist df by City, State, and Country.
Additional supporting R code is given in "country codes wiki scrape.R" and "ggplot theme.R". These scripts (1) scrape IOC country codes and their associated countries names from Wikipedia, List of IOC country codes, and (2) set my preferred ggplot2 theme for general use data visualization.
Finishing and cartography was completed in ArcGIS Pro (v3.2.1)6.
Copyright 2024 Travis Zalesky
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Footnotes
-
R Core Team. (2021). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing. https://www.R-project.org/ ↩
-
Lang, D. T. (2024). RCurl: General Network (HTTP/FTP/...) Client Interface for R. https://CRAN.R-project.org/package=RCurl ↩
-
Lang, D. T. (2024). XML: Tools for Parsing and Generating XML Within R and S-Plus. https://CRAN.R-project.org/package=XML ↩
-
Wickham, H., Averick, M., Bryan, J., Chang, W., McGowan, L. D., François, R., Grolemund, G., Hayes, A., Henry, L., Hester, J., Kuhn, M., Pedersen, T. L., Miller, E., Bache, S. M., Müller, K., Ooms, J., Robinson, D., Seidel, D. P., Spinu, V., … Yutani, H. (2019). Welcome to the tidyverse. Journal of Open Source Software, 4(43), 1686. https://doi.org/10.21105/joss.01686 ↩ ↩2 ↩3
-
Wickham, H. (2023). httr: Tools for Working with URLs and HTTP. https://CRAN.R-project.org/package=httr ↩
-
ESRI 2023. ArcGIS Pro: Release 3. Redlands, CA: Environmental Systems Research Institute. ↩ ↩2