LA 7 - Setting scripts on autopilot.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bonus round: Setting your script on autopilot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook helps you effectively transform your collection into a self-updating database. It describes the preparation for putting your script on autopilot. “Autopilot” here means that we will set a trigger to run your scripts periodically.\n",
    "\n",
    "Since we are not monitoring the execution of those scripts, we have to change the scripts so they record what's going on and alert us when something goes wrong. There are three elements we need to implement so we can set our scripts on autopilot. We have to create log files, add wrappers around our scripts that trigger the emails in case something goes wrong and finally configure an email account that sends the messages out. This note describes each of these three in turn. The transformation is then completed by writing a short execution script. Finally, we will see how to set a trigger, a so-called CRON job, that launches that execution script."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1: Creating log files\n",
    "### Preparatory thoughts\n",
    "Before we add the logging code to our scripts, let’s take a step back and think about the purpose of our logs in more detail. This will help us deduct how to set up the logging in a way that suits us best.\n",
    " \n",
    "The purpose of our logs is to provide an easily accessible record that allows us to locate bugs efficiently in the future. We may occasionally also just want to expect them to see that things are running as they should. Having them easily accessible suggests that we want to put our logs into a single, designated folder that we can find quickly (e.g. in the root directory of our project).\n",
    " \n",
    "To get to the source of a bug fast, we want to create separate log files for each script that we run. Having one log file per script ensures that errors will be printed at the bottom of the file. Our process has three steps: (i) collecting the text, (ii) parsing the text, and (iii) merging the updates to the existing database. Each of these steps should be its own script, complex steps probably more than one. Name each log file after the name of the script it is tracking.\n",
    " \n",
    "Finally, we may want to have a slim historical record to compare a failed script execution with the last successful one. One could do this two ways. First, by writing adding a piece to the script that copies the log file into a subfolder if and only if no error arose. The second, less elegant but probably easier solution is to overwrite log files only every few days, assuming we have time to check the bug before the log file for the last successful execution has been overwritten. For example, one could add the name of the weekday to the file name of the log file. That way, you will always have seven log files per executed script … and have six days to trouble shoot a bug before you lose your records of the last successful execution.\n",
    "\n",
    "\n",
    "### Adding logfiles in python\n",
    "Python comes with a <a href=\"https://docs.python.org/3/howto/logging.html\">module called “logging”</a> that allows us to write log files. Try this module to set up log files along the lines discussed above. Note that you can set what types of events are recorded in your log file.\n",
    " \n",
    "From the logging module’s manual:\n",
    "<table class=\"docutils align-center\">\n",
    "<colgroup>\n",
    "<col style=\"width: 24%\">\n",
    "<col style=\"width: 76%\">\n",
    "</colgroup>\n",
    "<thead>\n",
    "<tr class=\"row-odd\"><th class=\"head\"><p>Level</p></th>\n",
    "<th class=\"head\"><p>When it’s used</p></th>\n",
    "</tr>\n",
    "</thead>\n",
    "<tbody>\n",
    "<tr class=\"row-even\"><td><p><code class=\"docutils literal notranslate\"><span class=\"pre\">DEBUG</span></code></p></td>\n",
    "<td><p>Detailed information, typically of interest\n",
    "only when diagnosing problems.</p></td>\n",
    "</tr>\n",
    "<tr class=\"row-odd\"><td><p><code class=\"docutils literal notranslate\"><span class=\"pre\">INFO</span></code></p></td>\n",
    "<td><p>Confirmation that things are working as\n",
    "expected.</p></td>\n",
    "</tr>\n",
    "<tr class=\"row-even\"><td><p><code class=\"docutils literal notranslate\"><span class=\"pre\">WARNING</span></code></p></td>\n",
    "<td><p>An indication that something unexpected\n",
    "happened, or indicative of some problem in\n",
    "the near future (e.g. ‘disk space low’).\n",
    "The software is still working as expected.</p></td>\n",
    "</tr>\n",
    "<tr class=\"row-odd\"><td><p><code class=\"docutils literal notranslate\"><span class=\"pre\">ERROR</span></code></p></td>\n",
    "<td><p>Due to a more serious problem, the software\n",
    "has not been able to perform some function.</p></td>\n",
    "</tr>\n",
    "<tr class=\"row-even\"><td><p><code class=\"docutils literal notranslate\"><span class=\"pre\">CRITICAL</span></code></p></td>\n",
    "<td><p>A serious error, indicating that the program\n",
    "itself may be unable to continue running.</p></td>\n",
    "</tr>\n",
    "</tbody>\n",
    "</table>\n",
    "\n",
    "The default level is <code class=\"docutils literal notranslate\">WARNING</code>, which means that only events of this level and above will be tracked, unless the logging package is configured to do otherwise.\n",
    "\n",
    "Since we are only collecting a few dozen files even under the wasteful one-file-per-weekday solution described above, using the <code class=\"docutils literal notranslate\">DEBUG</code> is recommended. That mode will record anything that passes through your console. Reading it is as you would read your screen during the execution of your script. This level of detail helps us find the source of trouble faster than may be possible when, say, only the error message is recorded for us.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2: Creating alerts\n",
    "When we are watching our scripts execute, we notice when things go wrong, usually, once we spot some red ink at the bottom of the script’s computer window. When that script runs on a server, however, we just won’t see that it stopped, let alone why it stopped. In programming, defining specific actions subsequent to an event is called <a href=\"https://wiki.python.org/moin/HandlingExceptions\">“exception handling”</a>. To complete our scripts, we have to define what counts as an exception and how to handle it. In this tutorial, we will change your script’s setting to sending out the latest entries in our log file when exceptions occur.\n",
    "\n",
    "\n",
    "### Handling exceptions yourself\n",
    "Python does not take a guess when the commands you wrote down do not add up to a consistent order. Its default way of handling exceptions is simply to shut down there and then. It’s basically answering “... but I can’t” to whatever you asked your script to do. In exception handling, what we do is essentially ask the computer to try to execute something but to execute a different block of code in case there is trouble with this request.\n",
    "\n",
    "#### Try this\n",
    "The function used for the more nuanced approach to exception handling is literally called <a href=\"https://www.w3schools.com/python/python_try_except.asp\">“try”</a>. It comes with up to three additional blocks, namely “except”, “else” and “finally”.\n",
    "\n",
    "<div style=\"background:#EFF0F1; width:50%\">\n",
    "<code><font color=\"#0000cd\">try:\n",
    "  … this is the code you want to try … \n",
    "except:\n",
    "  … here is the exception handle you wrote … \n",
    "else:\n",
    "  … here is something that happens after smooth execution … \n",
    "finally:\n",
    "  … clean up, regardless of event history.\n",
    "\n",
    "print(<font color=\"#a52a2a\">\"This text is printed in all scenarios.\"</font>)</font></code>\n",
    "</div>\n",
    "\n",
    "Before we go into selected parts of this pseudo try function, note that the last print is always executed. This is thanks to Python’s code structure i.e. the indentation. Wrapping your main code into a try block moves it up one level in the indentation just like an if command, a while or a for loop. The program simply moves on upon completion. In the case above, it would just print the same text at the end of every execution, regardless of how successful or trouble the code we tried turned out to be.\n",
    "\n",
    "At this stage, nothing restricts you from simply wrapping an entire script as a single try block. Your scripts have been tested, there is no user interaction and thus they should run smoothly. Also, a breakdown at runtime is not the end of the world. If you wanted to isolate certain parts of a script from exceptions in other parts, you would try them in different try blocks. Splitting a script into several try blocks ensures that each one is attempted regardless of what happened to its preceding blocks.\n",
    "\n",
    "The interesting part is what to do once an exception occurs. This is done in the except block. It will be in this block where we will add the code that takes the last few lines from our log file and sends them to our email inbox. Note that you can specify different reactions to different exceptions by adding a second attribute to the command e.g.”except ErrorNrOne:”. See the below section on the different types of exceptions for more details and resources. \n",
    "\n",
    "When we run a script on autopilot, there may be some operations that we have to ensure that they always run no matter whether the script ran smoothly are was riddled with exceptions. Usefully, <a href=\"https://www.w3schools.com/python/python_try_except.asp\">the “finally” block</a> allows us to deposit code that is executed regardless of prior events. In our case, such code may involve removing temporary files, closing a selenium browser or other cleaning that insure that the next execution can start from a clean slate.\n",
    "\n",
    "#### Types of exceptions\n",
    "This is beyond our focus here, but Python is very flexible in exception handling. You can add more than one except clause to the same try block. This allows for different actions depending on the type of exceptions that have happened. Python comes with <a href=\"https://www.tutorialspoint.com/python/python_exceptions\">a sizeable list of exception types</a> already including the “NameError” (=the object you call does not exist in the working environment) or the “ImportError” (=an import statement fails). \n",
    "\n",
    "Beyond those standard exception types, the “raise” statement allows you to define your own exception types. You may choose to do that in functions that you wrote yourself. See the section “Raising an exception” in <a href=\"https://www.tutorialspoint.com/python/python_exceptions\">this tutorial</a> for a comprehensive introduction and example to the raise statement.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3: Allow python to email you\n",
    "\n",
    "list(host.name = \"smtp.gmail.com\",\n",
    "                            port=587,\n",
    "                            user.name=\"liberating.daemon@gmail.com\",\n",
    "                            passwd=\"thisisasillypassword\",\n",
    "                            ssl=T)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4: A single script to execute all scripts & send out errors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5: Set a periodic trigger\n",
    "For executing a script periodically, we have to set an alarm clock on our server. We will use something called a \"CRON job\" to do just that. A CRON job instructs the server to execute a file periodically. \n",
    "\n",
    "\n",
    "Use the <a href=\"https://crontab.guru/\">CRONtab Guru</a> to find the CRON schedule expression that suits you best."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}