data_scraping_workflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data scraping workflow\n",
    "The workflow described here is divided into sections for each data provider, detailing the specific functions and processes used to collect data such as teams, players, standings, stats, shot maps, and positions. Each section includes information on how to generate and store the resulting data, ensuring a consistent and organized approach."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import libraries for data scraping\n",
    "import pvd_Sofascore as sofascore\n",
    "import pvd_Fotmob as fotmob\n",
    "import pvd_Fbref as fbref"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sofascore\n",
    "The URL of the league to be used is specified. The link includes the ID of the league and the current season. Several functions are executed to obtain specific data. Some of them take the output from a previous function as input. In each case, a .csv file is generated and will be found in the *data* folder, starting with *sofascore_*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# League URL to be used\n",
    "sofascore_league_url = 'https://www.sofascore.com/es/torneo/futbol/argentina/liga-profesional-de-futbol/155#id:57478'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get teams\n",
    "The teams from the league are obtained, a .csv file is exported, and the resulting dictionary is assigned to the variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "sofascore_teams = sofascore.get_teams_from_league(sofascore_league_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get events\n",
    "Just like with the teams, the events from the league are obtained."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "sofascore_events = sofascore.get_events_from_league(sofascore_league_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get players\n",
    "Based on the teams, the players that make up the roster of each are obtained. Similarly, a .csv file is exported, and the results are stored in a dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "sofascore_players = sofascore.get_players_from_teams(sofascore_teams)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get heatmaps\n",
    "For each player, the heatmap for the latest seasons (the current season and those from the past two years) is obtained."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "sofascore_heatmaps = sofascore.get_heatmap_from_players(sofascore_players)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get lineups\n",
    "For each event, the formation of each team is obtained, including the starting lineup and substitutions. Each position is assigned a code based on the line of play and lateral position. This code will then be used to assign an ordered pair for graphical representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "sofascore_lineups = sofascore.get_lineups_from_events(sofascore_events)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get results\n",
    "For each event, the result, goals, and the winning team (Home or Away) are obtained. Each event will have two records in the results, one for each involved team."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "sofascore_results = sofascore.get_results_from_events(sofascore_events)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fotmob\n",
    "Data retrieval from Fotmob starts with the provided league URL. Several functions are run to gather various data points, such as teams, players, shot maps, and positions. Some functions depend on data generated by others, creating a comprehensive dataset. The resulting data is exported as .csv files saved in the data folder, all prefixed with *fotmob_.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "fotmob_league_url = 'https://www.fotmob.com/es/leagues/112/overview/liga-profesional'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get teams\n",
    "This function retrieves the teams from the specified league using the provided URL. A file named *fotmob_teams.csv* is generated and saved in the *data* folder. The resulting output is stored in a dictionary for further use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "fotmob_teams = fotmob.get_teams_from_league(fotmob_league_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get players\n",
    "Based on the teams retrieved, this function gathers the players that make up each team’s roster. A .csv file is exported to the *data* folder with the filename *fotmob_players.csv*, and the results are stored in a dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "fotmob_players = fotmob.get_players_from_teams(fotmob_teams)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get shotmaps\n",
    "Using player data, this function retrieves the shot maps for each player. A .csv file is generated and saved in the *data* folder. The results are stored in a DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "fotmob_shotmaps = fotmob.get_shotmap_from_players(fotmob_players)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get positions\n",
    "This function captures each player's on-field positions throughout various matches. The data is exported to a .csv file, and the resulting DataFrame is ready for further analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fotmob_positions = fotmob.get_positions_from_players(fotmob_players)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fbref\n",
    "Fbref data extraction also begins with a specific league URL. Multiple functions are executed to collect different types of data, including teams, standings, stats, players, and squads. These functions may use the output from others to enhance the dataset. The extracted data is saved in .csv files located in the data folder, with filenames starting with *fbref_*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "fbref_league_url = 'https://fbref.com/es/comps/21/Estadisticas-de-Liga-Profesional-Argentina'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get teams\n",
    "This function extracts team information from the provided league URL. The data is saved in a .csv file named *fbref_teams.csv* within the *data* folder. The output is a dictionary for subsequent processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "fbref_teams = fbref.get_teams_from_league(fbref_league_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get standings\n",
    "This function retrieves the standings of the league, detailing the current positions of the teams. The information is exported to a .csv file. The standings data is organized for easy reference and further analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "fbref_standings = fbref.get_standings_from_league(fbref_league_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get stats\n",
    "This function gathers comprehensive statistical *data* for the league, including metrics for each team. A .csv file is created and stored in the data folder. The resulting DataFrame is ready for use in data analysis or visualization tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "fbref_stats = fbref.get_stats_from_league(fbref_league_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get players\n",
    "Using the team information, this function compiles a list of players from each team. The *data* is saved as a .csv file and structured in a dictionary format for additional use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "fbref_players = fbref.get_players_from_teams(fbref_teams)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get squads\n",
    "This function provides detailed squad information for each team, including player demographics and attributes. It outputs a .csv file for easy access, and the data is available in a DataFrame format for deeper analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "fbref_squads = fbref.get_squads_from_teams(fbref_teams)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get percentiles\n",
    "This function retrieves detailed percentile rankings for each player based on their performance metrics. It categorizes players into groups such as Offensive, Passing, and Defensive, depending on their roles, and outputs the data in a .csv file for easy access. The function also provides the data in a DataFrame format, enabling deeper analysis and comparisons among players based on their performance relative to others in similar positions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Estadísticas</th>\n",
       "      <th>Por 90</th>\n",
       "      <th>Percentil</th>\n",
       "      <th>player</th>\n",
       "      <th>player_id</th>\n",
       "      <th>clase</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Goles sin penalización</td>\n",
       "      <td>0.00</td>\n",
       "      <td>20.0</td>\n",
       "      <td>Elias Gomez</td>\n",
       "      <td>9f110faf</td>\n",
       "      <td>Ofensiva</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>npxG: Goles esperados (xG) sin contar penaltis</td>\n",
       "      <td>0.02</td>\n",
       "      <td>22.0</td>\n",
       "      <td>Elias Gomez</td>\n",
       "      <td>9f110faf</td>\n",
       "      <td>Ofensiva</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Total de disparos</td>\n",
       "      <td>0.52</td>\n",
       "      <td>36.0</td>\n",
       "      <td>Elias Gomez</td>\n",
       "      <td>9f110faf</td>\n",
       "      <td>Ofensiva</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Asistencias</td>\n",
       "      <td>0.09</td>\n",
       "      <td>51.0</td>\n",
       "      <td>Elias Gomez</td>\n",
       "      <td>9f110faf</td>\n",
       "      <td>Ofensiva</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>xAG: Exp. Objetivos asistidos</td>\n",
       "      <td>0.11</td>\n",
       "      <td>66.0</td>\n",
       "      <td>Elias Gomez</td>\n",
       "      <td>9f110faf</td>\n",
       "      <td>Ofensiva</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6995</th>\n",
       "      <td>Derribos</td>\n",
       "      <td>2.00</td>\n",
       "      <td>51.0</td>\n",
       "      <td>Pedro Velurtas</td>\n",
       "      <td>8585684e</td>\n",
       "      <td>Defensiva</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6996</th>\n",
       "      <td>Intercepciones</td>\n",
       "      <td>2.00</td>\n",
       "      <td>97.0</td>\n",
       "      <td>Pedro Velurtas</td>\n",
       "      <td>8585684e</td>\n",
       "      <td>Defensiva</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6997</th>\n",
       "      <td>Bloqueos</td>\n",
       "      <td>1.40</td>\n",
       "      <td>78.0</td>\n",
       "      <td>Pedro Velurtas</td>\n",
       "      <td>8585684e</td>\n",
       "      <td>Defensiva</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6998</th>\n",
       "      <td>Despeje</td>\n",
       "      <td>2.80</td>\n",
       "      <td>75.0</td>\n",
       "      <td>Pedro Velurtas</td>\n",
       "      <td>8585684e</td>\n",
       "      <td>Defensiva</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6999</th>\n",
       "      <td>Aéreos Ganados</td>\n",
       "      <td>3.20</td>\n",
       "      <td>99.0</td>\n",
       "      <td>Pedro Velurtas</td>\n",
       "      <td>8585684e</td>\n",
       "      <td>Defensiva</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>7000 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        Estadísticas Por 90  Percentil  \\\n",
       "0                             Goles sin penalización   0.00       20.0   \n",
       "1     npxG: Goles esperados (xG) sin contar penaltis   0.02       22.0   \n",
       "2                                  Total de disparos   0.52       36.0   \n",
       "3                                        Asistencias   0.09       51.0   \n",
       "4                      xAG: Exp. Objetivos asistidos   0.11       66.0   \n",
       "...                                              ...    ...        ...   \n",
       "6995                                        Derribos   2.00       51.0   \n",
       "6996                                  Intercepciones   2.00       97.0   \n",
       "6997                                        Bloqueos   1.40       78.0   \n",
       "6998                                         Despeje   2.80       75.0   \n",
       "6999                                  Aéreos Ganados   3.20       99.0   \n",
       "\n",
       "              player player_id      clase  \n",
       "0        Elias Gomez  9f110faf   Ofensiva  \n",
       "1        Elias Gomez  9f110faf   Ofensiva  \n",
       "2        Elias Gomez  9f110faf   Ofensiva  \n",
       "3        Elias Gomez  9f110faf   Ofensiva  \n",
       "4        Elias Gomez  9f110faf   Ofensiva  \n",
       "...              ...       ...        ...  \n",
       "6995  Pedro Velurtas  8585684e  Defensiva  \n",
       "6996  Pedro Velurtas  8585684e  Defensiva  \n",
       "6997  Pedro Velurtas  8585684e  Defensiva  \n",
       "6998  Pedro Velurtas  8585684e  Defensiva  \n",
       "6999  Pedro Velurtas  8585684e  Defensiva  \n",
       "\n",
       "[7000 rows x 6 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fbref.get_percentile_from_players(fbref_players)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}