Decorators, Closures and Wrappers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "****\n",
    "# Decorators, Closures and Wrappers in Python\n",
    "****"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### About this notebook\n",
    "Documention prepared by **Jesus Perez Colino**.\n",
    "Version 0.1, Released 25/11/2014, Alpha\n",
    "\n",
    "- This work is licensed under a [Creative Commons Attribution-ShareAlike 3.0 Unported License](http://creativecommons.org/licenses/by-sa/3.0/deed.en_US). This work is offered for free, with the hope that it will be useful.\n",
    "- **Summary**: This notebook is summary of **decorators** in Python. At the end of the notebook, two examples of memoization with decorators are showed.\n",
    "- **Reproducibility conditions**: the content in this technical note can be reproduce in your computer under the following versions of the Python's packages: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------------- Reproducibility conditions for this notebook -------------------\n",
      "Python version:     2.7.10 |Anaconda 2.3.0 (x86_64)| (default, Sep 15 2015, 14:29:08) \n",
      "[GCC 4.2.1 (Apple Inc. build 5577)]\n",
      "Numpy version:      1.9.2\n",
      "IPython version:    4.0.0\n",
      "-------------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "import IPython\n",
    "import numpy as np\n",
    "from sys import version \n",
    "print ' Reproducibility conditions for this notebook '.center(85,'-')\n",
    "print ('Python version:     ' + version).center(85)\n",
    "print ('Numpy version:      ' + np.__version__)\n",
    "print 'IPython version:    ' + IPython.__version__\n",
    "print '-'*85"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to Decorators\n",
    "\n",
    "When dealing with a large codebase, it’s very common to have a set of tasks that need to be performed by many different functions, usually before or after doing something more specific to the function at hand. The nature of these tasks is as varied as the projects that use them, but here are some of the more common examples of where decorators are used:\n",
    "\n",
    "- Access control \n",
    "- Cleanup of temporary objects\n",
    "- Error handling\n",
    "- Caching\n",
    "- Logging\n",
    "\n",
    "In all of these cases, there’s some boilerplate code that needs to be executed before or after what the function’s really trying to do. Rather than copying that code into each function, it’d be better if it could be written once and simply applied to each function that needs it. This is where decorators come in.\n",
    "\n",
    "Technically, decorators are just simple functions designed with one purpose: *accept a function and return a function*. The function returned can be the same as the one passed in or it could be completely replaced by something else along the way. \n",
    "\n",
    "The most common way to apply a decorator is using a special syntax designed just for this purpose. Here’s how you could apply a decorator designed to suppress any errors during the execution of a function:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import datetime\n",
    "from my_package import suppress_errors\n",
    "\n",
    "@suppress_errors\n",
    "def log_error(message, log_file='errors.log'):\n",
    "    \"\"\"Log an error message to a file.\"\"\"\n",
    "    log = open(log_file,'w')\n",
    "    log.write('%s\\t%s\\n' % (datetime.datetime.now(), message))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This syntax tells Python to pass the `log_error()` function as an argument to the `suppress_errors()` function, which then returns a replacement to use instead. It’s easier to understand what happens behind the scenes by examining the process used in older versions of Python, before the `@` syntax was introduced in Python 2.4:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import datetime\n",
    "from my_package import suppress_errors\n",
    "\n",
    "def log_error(message, log_file='errors.log'):\n",
    "    \"\"\"Log an error message to a file.\"\"\"\n",
    "    log = open(log_file, 'w')\n",
    "    log.write('%s\\t%s\\n' % (datetime.datetime.now(), message))\n",
    "    \n",
    "log_error = suppress_errors(log_error)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To understand what commonly goes on inside decorators like log_error(), it’s necessary to first examine one of the most misunderstood and underutilized features of Python, and many other languages as well: **closures**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Closures\n",
    "In a nutshell, *a closure is a function that’s defined inside another function but is then passed outside that function where it can be used by other code*. There are some other details to learn as well, but it’s still fairly abstract at this point, so here’s a simple example of a closure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4\n"
     ]
    }
   ],
   "source": [
    "def multiply_by(factor):\n",
    "    '''Return a function that multiplies values by the given factor'''\n",
    "    def multiply(value):\n",
    "        '''Multiply the given value by the factor already provided'''\n",
    "        return value * factor\n",
    "    return multiply\n",
    "\n",
    "times2 = multiply_by(2)\n",
    "print(times2(2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you call `multiply_by()` with a value to use as a multiplication factor, the inner `multiply()` gets returned to be used later on. \n",
    "\n",
    "Here’s how it would actually be used, which may help explain why this is useful. If you key in the previous code line by line from a Python prompt, the following would give you an idea as well about how this works:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10\n",
      "20\n",
      "200\n"
     ]
    }
   ],
   "source": [
    "times2 = multiply_by(2)\n",
    "print times2(5)\n",
    "print times2(10)\n",
    "print times2(100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50\n",
      "100\n",
      "1000\n"
     ]
    }
   ],
   "source": [
    "times10 = multiply_by(10)\n",
    "print times10(5)\n",
    "print times10(10)\n",
    "print times10(100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrappers\n",
    "Closures play an important role in the construction of wrappers, the most common use of decorators. **Wrappers** are *functions designed to contain another function, adding some extra behavior before or after the wrapped function executes*. \n",
    "\n",
    "In the context of the closure discussion, a **wrapper** *is the inner function,* while the wrapped function is passed in as an argument to the outer function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def suppress_errors(func):\n",
    "    \"\"\"Automatically silence any errors that occur within a function\"\"\"\n",
    "\n",
    "    def wrapper(*args, **kwargs):\n",
    "        try:\n",
    "            return func(*args, **kwargs)\n",
    "        except Exception:\n",
    "            pass\n",
    "        \n",
    "    return wrapper"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that here, the **decorator** `suppress_errors()` takes a function as its only argument, which isn’t executed until the inner wrapper function executes. By returning the wrapper instead of the original function, we form a **closure**, which allows the same function handle to be used even after `suppress_errors()` is done.\n",
    "\n",
    "Because the `wrapper` has to be called as if it were the original function, regardless of how that function was defined, *it must accept all possible argument combinations*. This is achieved by using *variable positional and keyword arguments* together and passing them straight into the original function internally. This is a very common practice with wrappers because it allows maximum flexibility, without caring what type of function it’s applied to.\n",
    "\n",
    "The actual work in the wrapper is quite simple: just execute the original function inside a `try/except block` to catch any errors that are raised. In the event of any errors, it just continues merrily along, implicitly returning None instead of doing anything interesting. It also makes sure to return any value returned by the original function, so that everything meaningful about the wrapped function is maintained.\n",
    "\n",
    "In this case, the wrapper function is fairly simple, but the basic idea works for many more complex situations as well. There could be several lines of code both before and after the original function is called, perhaps with some decisions about whether it is called at all. Authorization wrappers, for instance, will typically return or raise an exception without ever calling the wrapped function, if the authorization failed for any reason."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import functools\n",
    "def suppress_errors(func):\n",
    "    \"\"\"Automatically silence any errors that occur within a function\"\"\"\n",
    "    @functools.wrapper(func)\n",
    "    def wrapper(*args, **kwargs):\n",
    "        try:\n",
    "            return func(*args, **kwargs)\n",
    "        except Exception:\n",
    "            pass\n",
    "\n",
    "    return wrapper"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This layering allows `suppress_errors()` to accept arguments prior to being used as a decorator, but it removes the ability to call it without any arguments. Because that was the previous behavior, we’ve now introduced a backward incompatibility. The closest we can get to the original syntax is to actually call `suppress_errors()` first, but without any arguments.\n",
    "\n",
    "Here’s an example function that processes updates files in a given directory. This is a task that’s often performed on an automated schedule, so that if something goes wrong, it can just stop running and try again at the next appointed time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import datetime\n",
    "import os\n",
    "import time\n",
    "from my_package import suppress_errors\n",
    "\n",
    "@suppress_errors()\n",
    "def process_updated_files(directory, process, since=None):\n",
    "    \"\"\"\n",
    "    Processes any new files in a `directory` using the `process` function.\n",
    "    If provided, `since` is a date after which files are considered updated.\n",
    "\n",
    "    The process function passed in must accept a single argument: the absolute\n",
    "    path to the file that needs to be processed.\n",
    "    \"\"\"\n",
    "    \n",
    "    if since is not None:\n",
    "        # Get a threshold that we can compare to the modification time later\n",
    "        threshold = time.mktime(since.timetuple()) + since.microsecond / 1000000\n",
    "    else:\n",
    "        threshold = 0\n",
    "        \n",
    "    for filename in os.listdir(directory):\n",
    "        path = os.path.abspath(os.path.join(directory, filename))\n",
    "        if os.stat(path).st_mtime > threshold:\n",
    "            process(path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example of Decorator: Memoization\n",
    "\n",
    "To demonstrate how decorators can copy out common behavior into any function, let us consider what could be done to improve the efficiency of deterministic functions. Given such a function, it should be possible to cache the results of a given function call, so if it’s called with the same arguments again, the result can be looked up without having to call the function again.\n",
    "\n",
    "Using a cache, a decorator can store the result of a function using the argument list as its key. Dictionaries or lists can’t be used as keys in a dictionary, so only positional arguments can be taken into account when populating the cache. Thankfully, most functions that would take advantage of memoization are simple mathematical operations, which are typically called with positional arguments anyway:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def memoize(func):\n",
    "    \"\"\"\n",
    "    Cache the results of the function so it doesn't need to be called\n",
    "    again, if the same arguments are provided a second time.\n",
    "    \"\"\"\n",
    "    cache ={}\n",
    "    @functools.wraps(func)\n",
    "    def wrapper(*args):\n",
    "        if args in cache:\n",
    "            return cache[args]\n",
    "        \n",
    "        print('First time calling %s()' % func.__name__)\n",
    "        \n",
    "        result = func(*args)\n",
    "        cache[args] = result\n",
    "        return result\n",
    "    return wrapper"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, whenever you define a deterministic function, you can use the `memoize()` decorator to automatically cache its result for future use. Here’s how it would work for some simple mathematical operations. Again, given you keyed in the aforelisted stub, try the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "@memoize\n",
    "def multiply(x, y):\n",
    "    return x * y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First time calling multiply()\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "42"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "multiply(6, 7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "42"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "multiply(6, 7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "@memoize\n",
    "def factorial(x):\n",
    "    result = 1\n",
    "    for i in range(x):\n",
    "        result *= i+1\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First time calling factorial()\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "3628800"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "factorial(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3628800"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "factorial(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based in the same idea, but building the `@memoized` decorator using a class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import collections\n",
    "import functools\n",
    "\n",
    "class memoized(object):\n",
    "    def __init__(self, func):\n",
    "        self.func = func\n",
    "        self.cache = {}    \n",
    "    def __call__(self, *args):\n",
    "        if not isinstance(args, collections.Hashable):\n",
    "            # uncacheable: a dict, or a list, for instance.\n",
    "            return self.func(*args)\n",
    "        if args in self.cache:\n",
    "            return self.cache[args]\n",
    "        else:\n",
    "            value = self.func(*args)\n",
    "            print('First time calling %s()' % self.func.__name__)\n",
    "            self.cache[args] = value\n",
    "            return value\n",
    "    def __repr__(self):\n",
    "        '''Return the function's docstring.'''\n",
    "        return self.func.__doc__\n",
    "    def __get__(self, obj, objtype):\n",
    "        '''Support instance methods.'''\n",
    "        return functools.partial(self.__call__, obj)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "@memoized\n",
    "def factorial(x):\n",
    "    result = 1\n",
    "    for i in range(x):\n",
    "        result *= i+1\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First time calling factorial()\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "3628800"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "factorial(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3628800"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "factorial(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}