notebooks/data-sources/gcsweb-ci/prow_archive_discovery.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "84450e02",
   "metadata": {},
   "source": [
    "# Prow Logs and GCS Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4f2b17c",
   "metadata": {},
   "source": [
    "### What do we have access to as data scientists when digging into the build artifacts?\n",
    "\n",
    "In this notebook we will demonstrate how to discover and interact with the data (logs) made availble on [GCS/origin-ci-test](https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/) as well as provide some simple EDA to help folks get started analyzing this data.\n",
    "\n",
    "This notebook is divided into 2 sections:\n",
    "\n",
    "1. Compare the different log files present throughout the archives and quantify how complete and comparable our log dataset is from build to build.\n",
    "1. Download a sample dataset of the events and build logs to perform some lite EDA.\n",
    "\n",
    "_Note: We will be collecting data from the \"origin-ci-test\" Bucket on Google Cloud Storage. But, after some out-of-notebook exploration it has become aparent that this is a massive amount of data that contains more than just the OpenShift CI logs we are intrested in here and programatically investigating that Bucket is not advised. Therefore, we recommend using the [web ui](https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/) to inspect what jobs are exposed and identify what is of interest to your analysis before collecting data via the google cloud stporage api. Here we will rely on web-scraping the UI to explore what's available to us based on what jobs are displayed on [TestGrid](https://testgrid.k8s.io/redhat-assisted-installer)._     "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d43e7580",
   "metadata": {},
   "source": [
    "## Compare availability of log files across a build"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f3eb5e1e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import requests\n",
    "import pandas as pd\n",
    "from bs4 import BeautifulSoup\n",
    "from google.cloud import storage\n",
    "from pathlib import Path\n",
    "from IPython.display import Markdown"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "280ae7eb",
   "metadata": {},
   "source": [
    "### Example to access a single set of Prow artifacts\n",
    "\n",
    "Let's make sure we understand how this works, and focus on a single job first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1df335d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "tab = '\"redhat-openshift-ocp-release-4.6-informing\"'\n",
    "job = \"periodic-ci-openshift-release-master-ci-4.6-upgrade-from-stable-4.5-e2e-gcp-upgrade\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2796a5be",
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.get(\n",
    "    f\"https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/{job}\"\n",
    ")\n",
    "soup = BeautifulSoup(response.text, \"html.parser\")\n",
    "list_of_builds = [x.get_text()[1:-1] for x in soup.find_all(\"a\")][1:-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f5781bfd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.6-upgrade-from-stable-4.5-e2e-gcp-upgrade/1383601003814719488'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response = requests.get(\n",
    "    f\"https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/{job}/{list_of_builds[1]}\"\n",
    ")\n",
    "response.url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "475853f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = BeautifulSoup(response.text, \"html.parser\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6fa2c7b1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[' ..',\n",
       " ' artifacts/',\n",
       " ' build-log.txt',\n",
       " ' finished.json',\n",
       " ' podinfo.json',\n",
       " ' prowjob.json',\n",
       " ' started.json']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[x.get_text() for x in soup.find_all(\"a\")]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "923a42e2",
   "metadata": {},
   "source": [
    "Great, we can now programmatically access the archives. Now, lets walk through all of the build archives for a single job and create a list of what they have on the first level of their directories.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "389751c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "build_data = {}\n",
    "\n",
    "for build in list_of_builds:\n",
    "    response = requests.get(\n",
    "        f\"https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/{job}/{build}\"\n",
    "    )\n",
    "    soup = BeautifulSoup(response.text, \"html.parser\")\n",
    "    artifacts = [x.get_text() for x in soup.find_all(\"a\")]\n",
    "    build_data[build] = artifacts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "20799616",
   "metadata": {},
   "outputs": [],
   "source": [
    "builds_info = pd.Series({k: len(v) for (k, v) in build_data.items()})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "bd8d5eb6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7    536\n",
       "dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "builds_info.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d634d02a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       " ..  artifacts/  build-log.txt  finished.json  podinfo.json  prowjob.json  started.json    536\n",
       "dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(build_data).apply(\" \".join).value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "ef560849",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   index    0\n",
       "0      7  1.0"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cent1 = builds_info.value_counts() / len(builds_info)\n",
    "cent1 = cent1.to_frame().reset_index()\n",
    "cent1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "277fe0b7-b660-4d57-b40c-b4989dfc265f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "100.0% of our records for this job appear to be complete and include the 'artifacts/' subdirectory, lets dig in and see what they contain."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Markdown(\n",
    "    \"{}% of our records for this job appear to be complete and include the 'artifacts/' subdirectory,\\\n",
    " lets dig in and see what they contain.\".format(\n",
    "        round(cent1.iloc[0, 1] * 100, 2)\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "1a0f0c2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "build_data = {}\n",
    "\n",
    "for build in list_of_builds:\n",
    "    response = requests.get(\n",
    "        f\"https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/{job}/{build}/artifacts\"\n",
    "    )\n",
    "    soup = BeautifulSoup(response.text, \"html.parser\")\n",
    "    artifacts = [x.get_text() for x in soup.find_all(\"a\")]\n",
    "    build_data[build] = artifacts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "3ab234df",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8    528\n",
       "7      4\n",
       "3      2\n",
       "6      1\n",
       "5      1\n",
       "dtype: int64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "artifact_info = pd.Series({k: len(v) for (k, v) in build_data.items()})\n",
    "artifact_info.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "0ae6a9be",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8</td>\n",
       "      <td>0.985075</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7</td>\n",
       "      <td>0.007463</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0.003731</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6</td>\n",
       "      <td>0.001866</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0.001866</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   index         0\n",
       "0      8  0.985075\n",
       "1      7  0.007463\n",
       "2      3  0.003731\n",
       "3      6  0.001866\n",
       "4      5  0.001866"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cent2 = artifact_info.value_counts() / len(artifact_info)\n",
    "cent2 = cent2.to_frame().reset_index()\n",
    "cent2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "aebc974c-a982-4644-b9f5-15bd991314de",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "The above shows us that there are about 98.51% of the artifacts dirs that have 8 items and 0.75% that have 7 items and so on (but it does not account for different combinations)."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Markdown(\n",
    "    \"The above shows us that there are about {}% of the artifacts dirs \"\n",
    "    \"that have {} items and {}% that have {} items and so on \"\n",
    "    \"(but it does not account for different combinations).\".format(\n",
    "        round(cent2.iloc[0, 1] * 100, 2),\n",
    "        cent2.iloc[0, 0],\n",
    "        round(cent2.iloc[1, 1] * 100, 2),\n",
    "        cent2.iloc[1, 0],\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "67daa19c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       " ..  build-resources/  e2e-gcp-upgrade/  release/  ci-operator-step-graph.json  ci-operator.log  junit_operator.xml  metadata.json    0.985075\n",
       " ..  build-resources/  release/  ci-operator-step-graph.json  ci-operator.log  junit_operator.xml  metadata.json                      0.007463\n",
       " ..  ci-operator.log  junit_job.xml                                                                                                   0.003731\n",
       " ..  build-resources/  ci-operator-step-graph.json  ci-operator.log  junit_operator.xml  metadata.json                                0.001866\n",
       " ..  e2e-gcp-upgrade/  release/  ci-operator.log  metadata.json                                                                       0.001866\n",
       "dtype: float64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(build_data).apply(\" \".join).value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb59947e",
   "metadata": {},
   "source": [
    "We can see from the results above that once we get down into the artifacts there is a far less uniformity to the data available to us for analysis. And this is all within a single job! Moving forward we will assume that this issue gets worse when comparing available artifacts across jobs and can dedicate a later notebook to proving out that assumption.  \n",
    "\n",
    "This heterogeneity of objects available for each build will make it somewhat difficult to use these sets of documents as a whole to compare different CI behaviour. At this point, it makes sense to consider looking only at the same document (log) across job where available. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f71ea321",
   "metadata": {},
   "source": [
    "## Collect Data\n",
    "\n",
    "### Build logs\n",
    "\n",
    "In the next section we are going to walkthrough accessing the `build-logs.txt` and the `events.json` as they appear to be nearly universally available. We will both download a small testing dataset as well show how to work directly with the data in memory.\n",
    "\n",
    "Now that we know what logs we want to collect its simpler to use the google cloud storage api to access or data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "b22122ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "def connect_storage(bucket_name):\n",
    "    storage_client = storage.Client.create_anonymous_client()\n",
    "    bucket = storage_client.bucket(bucket_name)\n",
    "    return {\"bucket\": bucket, \"storage_client\": storage_client}\n",
    "\n",
    "\n",
    "def download_public_file(client, source_blob_name):\n",
    "    \"\"\"Downloads a public blob from the bucket.\"\"\"\n",
    "    blob = client[\"bucket\"].blob(source_blob_name)\n",
    "    if blob.exists(client[\"storage_client\"]):\n",
    "        text = blob.download_as_text()\n",
    "    else:\n",
    "        text = \"\"\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "33215d04",
   "metadata": {},
   "outputs": [],
   "source": [
    "bucket_connection = connect_storage(\"origin-ci-test\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "f19b7f2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read data into memory\n",
    "build_log_data = {}\n",
    "for build in list_of_builds:\n",
    "    file = download_public_file(bucket_connection, f\"logs/{job}/{build}/build-log.txt\")\n",
    "    build_log_data[build] = file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "80a31a9b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\x1b[36mINFO\\x1b[0m[2021-04-17T19:58:56Z] ci-operator version v20210416-d902916        \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T19:58:56Z] Loading configuration from https://config.ci.openshift.org for openshift/release@master [ci-4.6-upgrade-from-stable-4.5] \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T19:58:57Z] Resolved SHA missing for master in https://github.com/openshift/release (will prevent caching) \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T19:58:57Z] Using explicitly provided pull-spec for release initial (registry.ci.openshift.org/ocp/release:4.5.37) \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T19:58:57Z] Using explicitly provided pull-spec for release latest (registry.ci.openshift.org/ocp/release:4.6.0-0.ci-2021-04-17-163254) \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T19:58:57Z] Using namespace https://console.build02.ci.openshift.org/k8s/cluster/projects/ci-op-y3cf82zn \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T19:58:57Z] Running [input:origin-centos-8], [input:ocp-4.5-upi-installer], [release:initial], [release:latest], [images], e2e-gcp-upgrade \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T19:58:58Z] Tagging origin/centos:8 into pipeline:origin-centos-8. \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T19:58:58Z] Tagging ocp/4.5:upi-installer into pipeline:ocp-4.5-upi-installer. \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T19:58:59Z] Importing release image initial.             \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T19:58:59Z] Importing release image latest.              \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:01:08Z] Imported release 4.5.37 created at 2021-03-31 06:05:56 +0000 UTC with 108 images to tag release:initial \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:01:16Z] Imported release 4.6.0-0.ci-2021-04-17-163254 created at 2021-04-17 16:33:19 +0000 UTC with 121 images to tag release:latest \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:01:16Z] Acquiring leases for test e2e-gcp-upgrade    \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:01:16Z] Running multi-stage test e2e-gcp-upgrade     \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:01:17Z] Running step e2e-gcp-upgrade-ipi-install-hosted-loki. \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:01:27Z] Step e2e-gcp-upgrade-ipi-install-hosted-loki succeeded after 10.064135461s. \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:01:27Z] Running step e2e-gcp-upgrade-ipi-conf.       \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:01:47Z] Step e2e-gcp-upgrade-ipi-conf succeeded after 20.05838057s. \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:01:47Z] Running step e2e-gcp-upgrade-ipi-conf-gcp.   \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:01:57Z] Step e2e-gcp-upgrade-ipi-conf-gcp succeeded after 10.065333292s. \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:01:57Z] Running step e2e-gcp-upgrade-ipi-install-monitoringpvc. \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:02:07Z] Step e2e-gcp-upgrade-ipi-install-monitoringpvc succeeded after 10.071902479s. \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:02:07Z] Running step e2e-gcp-upgrade-ipi-install-rbac. \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:02:17Z] Step e2e-gcp-upgrade-ipi-install-rbac succeeded after 10.051469409s. \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:02:17Z] Running step e2e-gcp-upgrade-ipi-install-install-stableinitial. \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] Logs for container test in pod e2e-gcp-upgrade-ipi-install-install-stableinitial: \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] Installing from release registry.build02.ci.openshift.org/ci-op-y3cf82zn/release@sha256:037ed15c79bbd6bee6f0d524d91790d2b368cba945e7ae6d64c984dc0f162b82 \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] level=info msg=\"Credentials loaded from environment variable \\\\\"GOOGLE_CLOUD_KEYFILE_JSON\\\\\", file \\\\\"/var/run/secrets/ci.openshift.io/cluster-profile/gce.json\\\\\"\" \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] level=info msg=\"Consuming Install Config from target directory\" \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] Will include manifests:                      \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] /tmp/secret/manifest_metrics_rb.yml          \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] /tmp/secret/manifest_creds.yml               \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] /tmp/secret/manifest_ds.yml                  \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] /tmp/secret/manifest_cm.yml                  \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] /tmp/secret/manifest_clusterrole.yml         \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] /tmp/secret/manifest_oauth_role.yml          \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] /tmp/secret/manifest_metrics_role.yml        \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] /tmp/secret/manifest_promtail_service.yml    \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] /tmp/secret/manifest_grafanacom_creds.yml    \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] /tmp/secret/manifest_sa.yml                  \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] /tmp/secret/manifest_promtail_cookie_secret.yml \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] /tmp/secret/manifest_role.yml                \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] /tmp/secret/manifest_cluster-monitoring-pvc.yml \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] /tmp/secret/manifest_psp.yml                 \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] /tmp/secret/manifest_01_ns.yml               \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] /tmp/secret/manifest_oauth_clusterrolebinding.yml \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] /tmp/secret/manifest_metrics.yml             \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] /tmp/secret/manifest_rolebinding.yml         \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] /tmp/secret/manifest_clusterrolebinding.yml  \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] level=warning msg=\"Found override for release image. Please be warned, this is not advised\" \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] level=info msg=\"Credentials loaded from environment variable \\\\\"GOOGLE_CLOUD_KEYFILE_JSON\\\\\", file \\\\\"/var/run/secrets/ci.openshift.io/cluster-profile/gce.json\\\\\"\" \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] level=info msg=\"Consuming Openshift Manifests from target directory\" \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] level=info msg=\"Consuming Master Machines from target directory\" \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] level=info msg=\"Consuming OpenShift Install (Manifests) from target directory\" \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] level=info msg=\"Consuming Worker Machines from target directory\" \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] level=info msg=\"Consuming Common Manifests from target directory\" \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] level=info msg=\"Creating infrastructure resources...\" \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] level=info msg=\"Waiting up to 20m0s for the Kubernetes API at https://api.ci-op-y3cf82zn-5a3d5.origin-ci-int-gce.dev.openshift.com:6443...\" \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] level=info msg=\"API v1.18.3+cdb0358 up\"      \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] level=info msg=\"Waiting up to 40m0s for bootstrapping to complete...\" \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] level=error msg=\"Cluster operator authentication Degraded is True with ConfigObservation_Error::IngressStateEndpoints_MissingEndpoints::RouteStatus_FailedHost::RouterCerts_NoRouterCertSecret: RouterCertsDegraded: secret/v4-0-config-system-router-certs -n openshift-authentication: could not be retrieved: secret \\\\\"v4-0-config-system-router-certs\\\\\" not found\\\\nConfigObservationDegraded: secret \\\\\"v4-0-config-system-router-certs\\\\\" not found\\\\nIngressStateEndpointsDegraded: No endpoints found for oauth-server\\\\nRouteStatusDegraded: route is not available at canonical host oauth-openshift.apps.ci-op-y3cf82zn-5a3d5.origin-ci-int-gce.dev.openshift.com: []\" \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] level=info msg=\"Cluster operator authentication Progressing is Unknown with NoData: \" \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] level=info msg=\"Cluster operator authentication Available is Unknown with NoData: \" \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] level=info msg=\"Cluster operator csi-snapshot-controller Progressing is True with _AsExpected: Progressing: Waiting for Deployment to deploy csi-snapshot-controller pods\" \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] level=info msg=\"Cluster operator csi-snapshot-controller Available is False with _AsExpected: Available: Waiting for Deployment to deploy csi-snapshot-controller pods\" \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] level=info msg=\"Cluster operator kube-storage-version-migrator Available is False with _NoMigratorPod: Available: deployment/migrator.openshift-kube-storage-version-migrator: no replicas are available\" \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] level=info msg=\"Cluster operator machine-config Progressing is True with : Working towards 4.5.37\" \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] level=error msg=\"Cluster operator machine-config Degraded is True with RenderConfigFailed: Unable to apply 4.5.37: openshift-config-managed/kube-cloud-config configmap is required on platform GCP but not found: configmap \\\\\"kube-cloud-config\\\\\" not found\" \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] level=info msg=\"Cluster operator machine-config Available is False with : Cluster not available for 4.5.37\" \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] level=info msg=\"Cluster operator openshift-apiserver Available is False with APIServices_Error: APIServicesAvailable: \\\\\"apps.openshift.io.v1\\\\\" is not ready: 503 (the server is currently unable to handle the request)\\\\nAPIServicesAvailable: \\\\\"authorization.openshift.io.v1\\\\\" is not ready: 503 (the server is currently unable to handle the request)\\\\nAPIServicesAvailable: \\\\\"build.openshift.io.v1\\\\\" is not ready: 503 (the server is currently unable to handle the request)\\\\nAPIServicesAvailable: \\\\\"image.openshift.io.v1\\\\\" is not ready: 503 (the server is currently unable to handle the request)\\\\nAPIServicesAvailable: \\\\\"oauth.openshift.io.v1\\\\\" is not ready: 503 (the server is currently unable to handle the request)\\\\nAPIServicesAvailable: \\\\\"project.openshift.io.v1\\\\\" is not ready: 503 (the server is currently unable to handle the request)\\\\nAPIServicesAvailable: \\\\\"quota.openshift.io.v1\\\\\" is not ready: 503 (the server is currently unable to handle the request)\\\\nAPIServicesAvailable: \\\\\"route.openshift.io.v1\\\\\" is not ready: 503 (the server is currently unable to handle the request)\\\\nAPIServicesAvailable: \\\\\"security.openshift.io.v1\\\\\" is not ready: 503 (the server is currently unable to handle the request)\\\\nAPIServicesAvailable: \\\\\"template.openshift.io.v1\\\\\" is not ready: 503 (the server is currently unable to handle the request)\\\\nAPIServicesAvailable: \\\\\"user.openshift.io.v1\\\\\" is not ready: 503 (the server is currently unable to handle the request)\" \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] level=info msg=\"Cluster operator openshift-controller-manager Progressing is True with _DesiredStateNotYetAchieved: Progressing: daemonset/controller-manager: observed generation is 0, desired generation is 8.\\\\nProgressing: daemonset/controller-manager: number available is 0, desired number available > 1\" \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] level=info msg=\"Cluster operator openshift-controller-manager Available is False with _NoPodsAvailable: Available: no daemon pods available on any node.\" \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] level=info msg=\"Cluster operator operator-lifecycle-manager-packageserver Available is False with : \" \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] level=info msg=\"Cluster operator operator-lifecycle-manager-packageserver Progressing is True with : Working toward 0.15.1\" \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] level=info msg=\"Pulling debug logs from the bootstrap machine\" \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] level=info msg=\"Bootstrap gather logs captured here \\\\\"/tmp/installer/log-bundle-20210417204954.tar.gz\\\\\"\" \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] level=fatal msg=\"Bootstrap failed to complete: failed to wait for bootstrapping to complete: timed out waiting for the condition\" \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] Setup phase finished, prepare env for next steps \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] Copying log bundle...                        \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] Removing REDACTED info from log...           \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] Copying required artifacts to shared dir     \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] {\"component\":\"entrypoint\",\"error\":\"wrapped process failed: exit status 1\",\"file\":\"prow/entrypoint/run.go:80\",\"func\":\"k8s.io/test-infra/prow/entrypoint.Options.Run\",\"level\":\"error\",\"msg\":\"Error executing test process\",\"severity\":\"error\",\"time\":\"2021-04-17T20:50:28Z\"} \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] error: failed to execute wrapped command: exit status 1 \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z]                                              \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] Step e2e-gcp-upgrade-ipi-install-install-stableinitial failed after 48m20.074374705s. \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:50:37Z] Running step e2e-gcp-upgrade-gather-gcp-console. \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:51:07Z] Step e2e-gcp-upgrade-gather-gcp-console succeeded after 30.058654549s. \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T20:51:07Z] Running step e2e-gcp-upgrade-gather-must-gather. \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T21:10:07Z] Step e2e-gcp-upgrade-gather-must-gather succeeded after 19m0.055587687s. \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T21:10:07Z] Running step e2e-gcp-upgrade-gather-extra.   \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T21:11:07Z] Step e2e-gcp-upgrade-gather-extra succeeded after 1m0.064604654s. \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T21:11:07Z] Running step e2e-gcp-upgrade-gather-audit-logs. \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T21:11:57Z] Step e2e-gcp-upgrade-gather-audit-logs succeeded after 50.057797698s. \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T21:11:57Z] Running step e2e-gcp-upgrade-ipi-deprovision-deprovision. \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T21:14:47Z] Step e2e-gcp-upgrade-ipi-deprovision-deprovision succeeded after 2m50.060881266s. \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T21:14:47Z] Releasing leases for test e2e-gcp-upgrade    \\n\\x1b[36mINFO\\x1b[0m[2021-04-17T21:14:48Z] Ran for 1h15m51s                             \\n\\x1b[31mERRO\\x1b[0m[2021-04-17T21:14:48Z] Some steps failed:                           \\n\\x1b[31mERRO\\x1b[0m[2021-04-17T21:14:48Z]   * could not run steps: step e2e-gcp-upgrade failed: \"e2e-gcp-upgrade\" pre steps failed: \"e2e-gcp-upgrade\" pod \"e2e-gcp-upgrade-ipi-install-install-stableinitial\" failed: the pod ci-op-y3cf82zn/e2e-gcp-upgrade-ipi-install-install-stableinitial failed after 48m15s (failed containers: test): ContainerFailed one or more containers exited \\n\\x1b[31mERRO\\x1b[0m[2021-04-17T21:14:48Z] Container test exited with code 1, reason Error \\n\\x1b[31mERRO\\x1b[0m[2021-04-17T21:14:48Z] ---                                          \\n\\x1b[31mERRO\\x1b[0m[2021-04-17T21:14:48Z] \\\\\"quota.openshift.io.v1\\\\\" is not ready: 503 (the server is currently unable to handle the request)\\\\nAPIServicesAvailable: \\\\\"route.openshift.io.v1\\\\\" is not ready: 503 (the server is currently unable to handle the request)\\\\nAPIServicesAvailable: \\\\\"security.openshift.io.v1\\\\\" is not ready: 503 (the server is currently unable to handle the request)\\\\nAPIServicesAvailable: \\\\\"template.openshift.io.v1\\\\\" is not ready: 503 (the server is currently unable to handle the request)\\\\nAPIServicesAvailable: \\\\\"user.openshift.io.v1\\\\\" is not ready: 503 (the server is currently unable to handle the request)\" \\n\\x1b[31mERRO\\x1b[0m[2021-04-17T21:14:48Z] level=info msg=\"Cluster operator openshift-controller-manager Progressing is True with _DesiredStateNotYetAchieved: Progressing: daemonset/controller-manager: observed generation is 0, desired generation is 8.\\\\nProgressing: daemonset/controller-manager: number available is 0, desired number available > 1\" \\n\\x1b[31mERRO\\x1b[0m[2021-04-17T21:14:48Z] level=info msg=\"Cluster operator openshift-controller-manager Available is False with _NoPodsAvailable: Available: no daemon pods available on any node.\" \\n\\x1b[31mERRO\\x1b[0m[2021-04-17T21:14:48Z] level=info msg=\"Cluster operator operator-lifecycle-manager-packageserver Available is False with : \" \\n\\x1b[31mERRO\\x1b[0m[2021-04-17T21:14:48Z] level=info msg=\"Cluster operator operator-lifecycle-manager-packageserver Progressing is True with : Working toward 0.15.1\" \\n\\x1b[31mERRO\\x1b[0m[2021-04-17T21:14:48Z] level=info msg=\"Pulling debug logs from the bootstrap machine\" \\n\\x1b[31mERRO\\x1b[0m[2021-04-17T21:14:48Z] level=info msg=\"Bootstrap gather logs captured here \\\\\"/tmp/installer/log-bundle-20210417204954.tar.gz\\\\\"\" \\n\\x1b[31mERRO\\x1b[0m[2021-04-17T21:14:48Z] level=fatal msg=\"Bootstrap failed to complete: failed to wait for bootstrapping to complete: timed out waiting for the condition\" \\n\\x1b[31mERRO\\x1b[0m[2021-04-17T21:14:48Z] Setup phase finished, prepare env for next steps \\n\\x1b[31mERRO\\x1b[0m[2021-04-17T21:14:48Z] Copying log bundle...                        \\n\\x1b[31mERRO\\x1b[0m[2021-04-17T21:14:48Z] Removing REDACTED info from log...           \\n\\x1b[31mERRO\\x1b[0m[2021-04-17T21:14:48Z] Copying required artifacts to shared dir     \\n\\x1b[31mERRO\\x1b[0m[2021-04-17T21:14:48Z] {\"component\":\"entrypoint\",\"error\":\"wrapped process failed: exit status 1\",\"file\":\"prow/entrypoint/run.go:80\",\"func\":\"k8s.io/test-infra/prow/entrypoint.Options.Run\",\"level\":\"error\",\"msg\":\"Error executing test process\",\"severity\":\"error\",\"time\":\"2021-04-17T20:50:28Z\"} \\n\\x1b[31mERRO\\x1b[0m[2021-04-17T21:14:48Z] error: failed to execute wrapped command: exit status 1 \\n\\x1b[31mERRO\\x1b[0m[2021-04-17T21:14:48Z] ---                                          \\n\\x1b[31mERRO\\x1b[0m[2021-04-17T21:14:48Z] Link to step on registry info site: https://steps.ci.openshift.org/reference/ipi-install-install-stableinitial \\n\\x1b[31mERRO\\x1b[0m[2021-04-17T21:14:48Z] Link to job on registry info site: https://steps.ci.openshift.org/job?org=openshift&repo=release&branch=master&test=e2e-gcp-upgrade&variant=ci-4.6-upgrade-from-stable-4.5 \\n'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "build_log_data[list(build_log_data.keys())[0]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "c8e401d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_counts(x):\n",
    "    \"\"\"\n",
    "    Gets counts for chars, words, lines for a log.\n",
    "    \"\"\"\n",
    "    if x:\n",
    "        chars = len(x)\n",
    "        words = len(x.split())\n",
    "        lines = x.count(\"\\n\") + 1\n",
    "        return chars, words, lines\n",
    "    else:\n",
    "        return 0, 0, 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "4ddd930b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>build_log_id</th>\n",
       "      <th>chars</th>\n",
       "      <th>words</th>\n",
       "      <th>lines</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1383510269677801472</td>\n",
       "      <td>17065</td>\n",
       "      <td>1188</td>\n",
       "      <td>120</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1383601003814719488</td>\n",
       "      <td>17063</td>\n",
       "      <td>1189</td>\n",
       "      <td>120</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1383691728313126912</td>\n",
       "      <td>17063</td>\n",
       "      <td>1189</td>\n",
       "      <td>120</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1383782446109036544</td>\n",
       "      <td>37601</td>\n",
       "      <td>2690</td>\n",
       "      <td>261</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1383873157873537024</td>\n",
       "      <td>16784</td>\n",
       "      <td>1167</td>\n",
       "      <td>120</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>531</th>\n",
       "      <td>1447844674805960704</td>\n",
       "      <td>20401</td>\n",
       "      <td>1485</td>\n",
       "      <td>223</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>532</th>\n",
       "      <td>1448025872249393152</td>\n",
       "      <td>20395</td>\n",
       "      <td>1480</td>\n",
       "      <td>222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>533</th>\n",
       "      <td>1448207267424899072</td>\n",
       "      <td>20353</td>\n",
       "      <td>1477</td>\n",
       "      <td>223</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>534</th>\n",
       "      <td>1448388543654137856</td>\n",
       "      <td>20443</td>\n",
       "      <td>1484</td>\n",
       "      <td>222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>535</th>\n",
       "      <td>1448569878565883904</td>\n",
       "      <td>20458</td>\n",
       "      <td>1486</td>\n",
       "      <td>222</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>536 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            build_log_id  chars  words  lines\n",
       "0    1383510269677801472  17065   1188    120\n",
       "1    1383601003814719488  17063   1189    120\n",
       "2    1383691728313126912  17063   1189    120\n",
       "3    1383782446109036544  37601   2690    261\n",
       "4    1383873157873537024  16784   1167    120\n",
       "..                   ...    ...    ...    ...\n",
       "531  1447844674805960704  20401   1485    223\n",
       "532  1448025872249393152  20395   1480    222\n",
       "533  1448207267424899072  20353   1477    223\n",
       "534  1448388543654137856  20443   1484    222\n",
       "535  1448569878565883904  20458   1486    222\n",
       "\n",
       "[536 rows x 4 columns]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Create a dataframe with char, words, and lines\n",
    "## count for the logs\n",
    "data = []\n",
    "for key, value in build_log_data.items():\n",
    "    chars, words, lines = get_counts(value)\n",
    "    data.append([key, chars, words, lines])\n",
    "\n",
    "df = pd.DataFrame(data=data, columns=[\"build_log_id\", \"chars\", \"words\", \"lines\"])\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e29e39fa",
   "metadata": {},
   "source": [
    "#### See the stats for chars, words, lines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "92e76008",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    5.360000e+02\n",
       "mean     3.516793e+04\n",
       "std      2.711030e+05\n",
       "min      2.120000e+03\n",
       "25%      2.041650e+04\n",
       "50%      2.052900e+04\n",
       "75%      2.809525e+04\n",
       "max      6.289343e+06\n",
       "Name: chars, dtype: float64"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[\"chars\"].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "64db4de5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count       536.000000\n",
       "mean       2382.583955\n",
       "std       19301.541247\n",
       "min         147.000000\n",
       "25%        1482.000000\n",
       "50%        1494.000000\n",
       "75%        1648.000000\n",
       "max      447981.000000\n",
       "Name: words, dtype: float64"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[\"words\"].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "c98b74f5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>lines</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>536.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>249.042910</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>815.830161</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>19.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>215.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>222.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>222.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>19034.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              lines\n",
       "count    536.000000\n",
       "mean     249.042910\n",
       "std      815.830161\n",
       "min       19.000000\n",
       "25%      215.000000\n",
       "50%      222.000000\n",
       "75%      222.000000\n",
       "max    19034.000000"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "describe = df[\"lines\"].describe()\n",
    "describe = describe.to_frame()\n",
    "describe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "055c9307-01b7-48a9-a531-42b043540b15",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "From the initial analysis above, we see that we have log files  with 19 lines to 19034 lines with a mean of ~249 lines. This suggest high variability. The next thing we could look  at would be the similarity between log files, performing analysis, templating, and clustering.  We will address those questions in an upcoming notebook."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Markdown(\n",
    "    \"From the initial analysis above, we see that we have log files \"\n",
    "    \" with {} lines to {} lines with a mean of ~{} lines. \"\n",
    "    \"This suggest high variability. The next thing we could look \"\n",
    "    \" at would be the similarity between log files, \"\n",
    "    \"performing analysis, templating, and clustering. \"\n",
    "    \" We will address those questions in an upcoming notebook.\".format(\n",
    "        round(describe.iloc[3, 0]),\n",
    "        round(describe.iloc[7, 0]),\n",
    "        round(describe.iloc[1, 0]),\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2c64849",
   "metadata": {},
   "source": [
    "### Events"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "eb64d84c",
   "metadata": {},
   "outputs": [],
   "source": [
    "build_events_data = {}\n",
    "for build in list_of_builds:\n",
    "    file = download_public_file(\n",
    "        bucket_connection, f\"logs/{job}/{build}/artifacts/build-resources/events.json\"\n",
    "    )\n",
    "    if file:\n",
    "        build_events_data[build] = json.loads(file)\n",
    "    else:\n",
    "        build_events_data[build] = \"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "fba810d0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "99.44029850746269\n"
     ]
    }
   ],
   "source": [
    "## Percentage of builds that have the events.json file\n",
    "count = 0\n",
    "for key, value in build_events_data.items():\n",
    "    if value:\n",
    "        count += 1\n",
    "percent = count * 100 / len(build_events_data)\n",
    "print(percent)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "3a9e67e4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UID</th>\n",
       "      <th>message</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>c694dae6-d31b-469c-b9ca-e139d20ef644</td>\n",
       "      <td>Running job periodic-ci-openshift-release-mast...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8174d743-feb7-4446-84f5-def97617fd3c</td>\n",
       "      <td>Running job periodic-ci-openshift-release-mast...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>e7545ab7-fa0a-473f-b719-4dbad5e01a65</td>\n",
       "      <td>created SCC ranges</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8a49821c-3ef6-4f71-ae64-a3faad6d8365</td>\n",
       "      <td>No matching pods found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>b97c3b80-4e2f-4e5e-8e4a-5e890c3cc4fd</td>\n",
       "      <td>found no controller ref for pod \"release-image...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>219</th>\n",
       "      <td>6e002c16-f8cc-4390-82cb-a6e7f521541e</td>\n",
       "      <td>Created container release</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>220</th>\n",
       "      <td>e0592c0a-de8b-4ab8-9aee-5fe0384dd51f</td>\n",
       "      <td>Started container release</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>221</th>\n",
       "      <td>946cf37e-9ce7-436a-8997-cd88a3ffeea8</td>\n",
       "      <td>Container image \"gcr.io/k8s-prow/sidecar:v2021...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>222</th>\n",
       "      <td>90fa202b-5ac5-4768-86ad-54767e6579d4</td>\n",
       "      <td>Created container sidecar</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>223</th>\n",
       "      <td>9d81659f-c5a9-4446-b86e-ddcc9e590dec</td>\n",
       "      <td>Started container sidecar</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>224 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      UID  \\\n",
       "0    c694dae6-d31b-469c-b9ca-e139d20ef644   \n",
       "1    8174d743-feb7-4446-84f5-def97617fd3c   \n",
       "2    e7545ab7-fa0a-473f-b719-4dbad5e01a65   \n",
       "3    8a49821c-3ef6-4f71-ae64-a3faad6d8365   \n",
       "4    b97c3b80-4e2f-4e5e-8e4a-5e890c3cc4fd   \n",
       "..                                    ...   \n",
       "219  6e002c16-f8cc-4390-82cb-a6e7f521541e   \n",
       "220  e0592c0a-de8b-4ab8-9aee-5fe0384dd51f   \n",
       "221  946cf37e-9ce7-436a-8997-cd88a3ffeea8   \n",
       "222  90fa202b-5ac5-4768-86ad-54767e6579d4   \n",
       "223  9d81659f-c5a9-4446-b86e-ddcc9e590dec   \n",
       "\n",
       "                                               message  \n",
       "0    Running job periodic-ci-openshift-release-mast...  \n",
       "1    Running job periodic-ci-openshift-release-mast...  \n",
       "2                                   created SCC ranges  \n",
       "3                               No matching pods found  \n",
       "4    found no controller ref for pod \"release-image...  \n",
       "..                                                 ...  \n",
       "219                          Created container release  \n",
       "220                          Started container release  \n",
       "221  Container image \"gcr.io/k8s-prow/sidecar:v2021...  \n",
       "222                          Created container sidecar  \n",
       "223                          Started container sidecar  \n",
       "\n",
       "[224 rows x 2 columns]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Analyzing the messages of a single build\n",
    "single_build = sorted(build_events_data.keys())[0]\n",
    "messages = [\n",
    "    (i[\"metadata\"][\"uid\"], i[\"message\"])\n",
    "    for i in build_events_data[single_build][\"items\"]\n",
    "]\n",
    "messages_df = pd.DataFrame(messages, columns=[\"UID\", \"message\"])\n",
    "messages_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "e7f4ca91",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count                           224\n",
       "unique                           89\n",
       "top       Created container sidecar\n",
       "freq                             13\n",
       "Name: message, dtype: object"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "messages_df[\"message\"].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "e438c850",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>message</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Created container sidecar</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Started container place-entrypoint</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Started container sidecar</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Container image \"gcr.io/k8s-prow/sidecar:v2021...</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Container image \"gcr.io/k8s-prow/entrypoint:v2...</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84</th>\n",
       "      <td>Add eth0 [10.130.47.13/23]</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>85</th>\n",
       "      <td>Successfully assigned ci-op-y3cf82zn/e2e-gcp-u...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>86</th>\n",
       "      <td>Successfully pulled image \"image-registry.open...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87</th>\n",
       "      <td>Pulling image \"image-registry.openshift-image-...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88</th>\n",
       "      <td>Successfully pulled image \"image-registry.open...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>89 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                index  message\n",
       "0                           Created container sidecar       13\n",
       "1                  Started container place-entrypoint       13\n",
       "2                           Started container sidecar       13\n",
       "3   Container image \"gcr.io/k8s-prow/sidecar:v2021...       13\n",
       "4   Container image \"gcr.io/k8s-prow/entrypoint:v2...       13\n",
       "..                                                ...      ...\n",
       "84                         Add eth0 [10.130.47.13/23]        1\n",
       "85  Successfully assigned ci-op-y3cf82zn/e2e-gcp-u...        1\n",
       "86  Successfully pulled image \"image-registry.open...        1\n",
       "87  Pulling image \"image-registry.openshift-image-...        1\n",
       "88  Successfully pulled image \"image-registry.open...        1\n",
       "\n",
       "[89 rows x 2 columns]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "messages_df[\"message\"].value_counts().reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "333c6ac1-e039-4511-bb21-58b281e5fd6c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "In the build data, we saw that about 99.44% builds have the events.json file. We further analyzed all the events that happened for a particular build and found the frequencies of the messages. We can repeat the process for all the other builds and find most common messages and perform further analysis."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Markdown(\n",
    "    \"In the build data, we saw that about {}% builds have the events.json file.\"\n",
    "    \" We further analyzed all the events that happened for a particular build\"\n",
    "    \" and found the frequencies of the messages. We can repeat the process for\"\n",
    "    \" all the other builds and find most common messages and perform further\"\n",
    "    \" analysis.\".format(round(percent, 2))\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4caa4f26",
   "metadata": {},
   "source": [
    "# Save sample data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "8176a8b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "path = \"../../../data/raw/gcs/build-logs/\"\n",
    "filename = \"sample-build-logs.parquet\"\n",
    "dataset_base_path = Path(path)\n",
    "dataset_base_path.mkdir(parents=True, exist_ok=True)\n",
    "build_logs = pd.DataFrame.from_dict(build_log_data, orient=\"index\", columns=[\"log\"])\n",
    "build_logs.to_parquet(f\"{path}/{filename}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "fc86d5ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "path = \"../../../data/raw/gcs/events/\"\n",
    "filename = \"sample-events.json\"\n",
    "dataset_base_path = Path(path)\n",
    "dataset_base_path.mkdir(parents=True, exist_ok=True)\n",
    "\n",
    "with open(f\"{path}/{filename}\", \"w\") as file:\n",
    "    json.dump(build_events_data, file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "790331a4",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "In this notebook, we demonstrated how to programmatically access the gcs openshift origins ci archives, pull specific logs types into our notebook for analysis and save them for later use. \n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}