Intro to Tensorflow 1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What is TensorFlow?\n",
    "\n",
    "![](https://upload.wikimedia.org/wikipedia/commons/a/a4/TensorFlowLogo.png)\n",
    "\n",
    "\n",
    "TensorFlow is an open source software library released in 2015 by Google to make it easier for developers to design, build, and train deep learning models. TensorFlow originated as an internal library that Google developers used to build models in-house, and we expect additional functionality to be added to the open source version as they are tested and vetted in the internal flavor. Although TensorFlow is only one of several options available to developers, we choose to use it here because of its thoughtful design and ease of use. We’ll briefly compare TensorFlow to alternatives in the next section.\n",
    "\n",
    "At a high level, TensorFlow is a Python library that allows users to express arbitrary computation as a graph of data flows. Nodes in this graph represent mathematical operations, whereas edges represent data that is communicated from one node to another. Data in TensorFlow are represented as tensors, which are multidimensional arrays. Although this framework for thinking about computation is valuable in many different fields, TensorFlow is primarily used for deep learning in practice and research.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install\n",
    "As usual, we'll be using Conda to install TensorFlow. You might already have a TensorFlow environment, but check to make sure you have all the necessary packages.\n",
    "\n",
    "### OS X or Linux\n",
    "Run the following commands to setup your environment:\n",
    "\n",
    "    conda create -n tensorflow python=3.5\n",
    "    source activate tensorflow\n",
    "    conda install pandas matplotlib jupyter notebook scipy scikit-learn\n",
    "    pip install tensorflow\n",
    "\n",
    "### Windows\n",
    "And installing on Windows. In your console or Anaconda shell,\n",
    "\n",
    "    conda create -n tensorflow python=3.5\n",
    "    activate tensorflow\n",
    "    conda install pandas matplotlib jupyter notebook scipy scikit-learn\n",
    "    pip install tensorflow\n",
    "\n",
    "### Hello, world!\n",
    "Try running the following code in your Python console to make sure you have TensorFlow properly installed. The console will print \"Hello, world!\" if TensorFlow is installed. Don’t worry about understanding what it does. You’ll learn about it in the next section.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor 'Const_13:0' shape=() dtype=string>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hello_constant = tf.constant('Hello World!')\n",
    "hello_constant"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'Hello World!'\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "# Create TensorFlow object called tensor\n",
    "hello_constant = tf.constant('Hello World!')\n",
    "\n",
    "with tf.Session() as sess:\n",
    "        # Run the tf.constant operation in the session\n",
    "        output = sess.run(hello_constant)\n",
    "        print(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s analyze the Hello World script you ran. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tensor\n",
    "In TensorFlow, data isn’t stored as integers, floats, or strings. These values are encapsulated in an object called a tensor. In the case of hello_constant = tf.constant('Hello World!'), hello_constant is a 0-dimensional string tensor, but tensors come in a variety of sizes as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A is a 0-dimensional int32 tensor\n",
    "A = tf.constant(1234)\n",
    "\n",
    "# B is a 1-dimensional int32 tensor\n",
    "B = tf.constant([123,456,789])\n",
    "\n",
    "# C is a 2-dimensional int32 tensor\n",
    "C = tf.constant([ [123,456,789], [222,333,444] ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor 'Const_17:0' shape=(2, 3) dtype=int32>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "C "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "tf.constant() is one of many TensorFlow operations you will use in this lesson. The tensor returned by tf.constant() is called a constant tensor, because the value of the tensor never changes.\n",
    "\n",
    "### Session\n",
    "\n",
    "TensorFlow’s api is built around the idea of a computational graph, a way of visualizing a mathematical process.  Let’s take the TensorFlow code and turn that into a graph:\n",
    "\n",
    "![Session](https://d17h27t6h515a5.cloudfront.net/topher/2016/October/580feadb_session/session.png)\n",
    "\n",
    "A \"TensorFlow Session\", as shown above, is an environment for running a graph. The session is in charge of allocating the operations to GPU(s) and/or CPU(s), including remote machines. Let’s see how you use it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.Session() as sess:\n",
    "    output = sess.run(hello_constant)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code has already created the tensor, `hello_constant`, from the previous lines. The next step is to evaluate the tensor in a session.\n",
    "\n",
    "The code creates a session instance, `sess`, using `tf.Session`. The `sess.run()` function then evaluates the tensor and returns the results.\n",
    "\n",
    "## TensorFlow Input \n",
    "\n",
    "In the last section, a tensor was passed into a session and it returned the result. What if we want to use a non-constant? This is where `tf.placeholder()` and `feed_dict` come into place. In this section, we'll go over the basics of feeding data into TensorFlow.\n",
    "\n",
    "### tf.placeholder()\n",
    "\n",
    "Sadly you can’t just set `x` to your dataset and put it in TensorFlow, because over time you'll want your TensorFlow model to take in different datasets with different parameters. You need `tf.placeholder()`!\n",
    "\n",
    "`tf.placeholder()` returns a tensor that gets its value from data passed to the `tf.session.run()` function, allowing you to set the input right before the session runs.\n",
    "\n",
    "### Session's feed_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sudh\n"
     ]
    }
   ],
   "source": [
    "x = tf.placeholder(tf.string)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    output = sess.run(x, feed_dict={x: 'sudh'})\n",
    "    print(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the feed_dict parameter in `tf.session.run()` to set the placeholder tensor. The above example shows the tensor `x` being set to the string `\"Hello, world\"`. It's also possible to set more than one tensor using `feed_dict` as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test String\n",
      "123\n"
     ]
    }
   ],
   "source": [
    "x = tf.placeholder(tf.string)\n",
    "y = tf.placeholder(tf.int32)\n",
    "z = tf.placeholder(tf.float32)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    output_x = sess.run(x,feed_dict={x: 'Test String', y: 123, z: 45.67})\n",
    "    output_y = sess.run(y, feed_dict={x: 'Test String', y: 123, z:45.67})\n",
    "    print(output_x)\n",
    "    print(output_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: If the data passed to the `feed_dict` doesn’t match the tensor type and can’t be cast into the tensor type, you’ll get the error `“ValueError: invalid literal for...”`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## TensorFlow Math\n",
    "\n",
    "Getting the input is great, but now you need to use it. We're going to use basic math functions that everyone knows and loves - add, subtract, multiply, and divide - with tensors. (There's many more math functions you can check out in the [documentation](https://www.tensorflow.org/api_docs/python/math_ops/).)\n",
    "\n",
    "### Addition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7\n"
     ]
    }
   ],
   "source": [
    "x = tf.add(5, 2)  # 7\n",
    "with tf.Session() as sess:\n",
    "    output = sess.run(x)\n",
    "    print(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start with the add function. The `tf.add()` function does exactly what you expect it to do. It takes in two numbers, two tensors, or one of each, and returns their sum as a tensor.\n",
    "\n",
    "### Subraction and Multiplication"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = tf.subtract(10, 4) # 6\n",
    "y = tf.multiply(2, 5)  # 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `x` tensor will evaluate to `6`, because `10 - 4 = 6`. The `y` tensor will evaluate to `10`, because `2 * 5 = 10`. That was easy!\n",
    "\n",
    "### Converting Types\n",
    "\n",
    "It may be necessary to convert between types to make certain operators work together. For example, if you tried the following, it would fail with an exception:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "Input 'y' of 'Sub' Op has type int32 that does not match type float32 of argument 'x'.",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32mc:\\programdata\\anaconda3\\envs\\multi_class\\lib\\site-packages\\tensorflow\\python\\framework\\op_def_library.py\u001b[0m in \u001b[0;36m_apply_op_helper\u001b[1;34m(self, op_type_name, name, **keywords)\u001b[0m\n\u001b[0;32m    526\u001b[0m                 \u001b[0mas_ref\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0minput_arg\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mis_ref\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 527\u001b[1;33m                 preferred_dtype=default_dtype)\n\u001b[0m\u001b[0;32m    528\u001b[0m           \u001b[1;32mexcept\u001b[0m \u001b[0mTypeError\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0merr\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\programdata\\anaconda3\\envs\\multi_class\\lib\\site-packages\\tensorflow\\python\\framework\\ops.py\u001b[0m in \u001b[0;36minternal_convert_to_tensor\u001b[1;34m(value, dtype, name, as_ref, preferred_dtype, ctx, accept_symbolic_tensors, accept_composite_tensors)\u001b[0m\n\u001b[0;32m   1223\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[0mret\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1224\u001b[1;33m       \u001b[0mret\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mconversion_func\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mdtype\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mname\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mname\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mas_ref\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mas_ref\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1225\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\programdata\\anaconda3\\envs\\multi_class\\lib\\site-packages\\tensorflow\\python\\framework\\ops.py\u001b[0m in \u001b[0;36m_TensorTensorConversionFunction\u001b[1;34m(t, dtype, name, as_ref)\u001b[0m\n\u001b[0;32m   1017\u001b[0m         \u001b[1;34m\"Tensor conversion requested dtype %s for Tensor with dtype %s: %r\"\u001b[0m \u001b[1;33m%\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1018\u001b[1;33m         (dtype.name, t.dtype.name, str(t)))\n\u001b[0m\u001b[0;32m   1019\u001b[0m   \u001b[1;32mreturn\u001b[0m \u001b[0mt\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: Tensor conversion requested dtype float32 for Tensor with dtype int32: 'Tensor(\"Const_19:0\", shape=(), dtype=int32)'",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-35-22d5b2f298e0>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mtf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msubtract\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mconstant\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m2.0\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mtf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mconstant\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32mc:\\programdata\\anaconda3\\envs\\multi_class\\lib\\site-packages\\tensorflow\\python\\util\\dispatch.py\u001b[0m in \u001b[0;36mwrapper\u001b[1;34m(*args, **kwargs)\u001b[0m\n\u001b[0;32m    178\u001b[0m     \u001b[1;34m\"\"\"Call target, and fall back on dispatchers if there is a TypeError.\"\"\"\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    179\u001b[0m     \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 180\u001b[1;33m       \u001b[1;32mreturn\u001b[0m \u001b[0mtarget\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    181\u001b[0m     \u001b[1;32mexcept\u001b[0m \u001b[1;33m(\u001b[0m\u001b[0mTypeError\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    182\u001b[0m       \u001b[1;31m# Note: convert_to_eager_tensor currently raises a ValueError, not a\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\programdata\\anaconda3\\envs\\multi_class\\lib\\site-packages\\tensorflow\\python\\ops\\math_ops.py\u001b[0m in \u001b[0;36msubtract\u001b[1;34m(x, y, name)\u001b[0m\n\u001b[0;32m    341\u001b[0m \u001b[1;33m@\u001b[0m\u001b[0mdispatch\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0madd_dispatch_support\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    342\u001b[0m \u001b[1;32mdef\u001b[0m \u001b[0msubtract\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mx\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0my\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mname\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mNone\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 343\u001b[1;33m   \u001b[1;32mreturn\u001b[0m \u001b[0mgen_math_ops\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msub\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mx\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0my\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mname\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    344\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    345\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\programdata\\anaconda3\\envs\\multi_class\\lib\\site-packages\\tensorflow\\python\\ops\\gen_math_ops.py\u001b[0m in \u001b[0;36msub\u001b[1;34m(x, y, name)\u001b[0m\n\u001b[0;32m  11572\u001b[0m   \u001b[1;31m# Add nodes to the TensorFlow graph.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m  11573\u001b[0m   _, _, _op = _op_def_lib._apply_op_helper(\n\u001b[1;32m> 11574\u001b[1;33m         \"Sub\", x=x, y=y, name=name)\n\u001b[0m\u001b[0;32m  11575\u001b[0m   \u001b[0m_result\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0m_op\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0moutputs\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m  11576\u001b[0m   \u001b[0m_inputs_flat\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0m_op\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0minputs\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\programdata\\anaconda3\\envs\\multi_class\\lib\\site-packages\\tensorflow\\python\\framework\\op_def_library.py\u001b[0m in \u001b[0;36m_apply_op_helper\u001b[1;34m(self, op_type_name, name, **keywords)\u001b[0m\n\u001b[0;32m    561\u001b[0m                   \u001b[1;34m\"%s type %s of argument '%s'.\"\u001b[0m \u001b[1;33m%\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    562\u001b[0m                   (prefix, dtypes.as_dtype(attrs[input_arg.type_attr]).name,\n\u001b[1;32m--> 563\u001b[1;33m                    inferred_from[input_arg.type_attr]))\n\u001b[0m\u001b[0;32m    564\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    565\u001b[0m           \u001b[0mtypes\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0mvalues\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdtype\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mTypeError\u001b[0m: Input 'y' of 'Sub' Op has type int32 that does not match type float32 of argument 'x'."
     ]
    }
   ],
   "source": [
    "tf.subtract(tf.constant(2.0),tf.constant(1))  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's because the constant `1` is an integer but the constant `2.0` is a floating point value and subtract expects them to match.\n",
    "\n",
    "In cases like these, you can either make sure your data is all of the same type, or you can cast a value to another type. In this case, converting the `2.0` to an integer before subtracting, like so, will give the correct result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n"
     ]
    }
   ],
   "source": [
    "c = tf.subtract(tf.cast(tf.constant(2.0), tf.int32), tf.constant(1))   # 1\n",
    "with tf.Session() as sess:\n",
    "    output = sess.run(c)\n",
    "    print(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example\n",
    "\n",
    "Let's apply what you learned to convert an algorithm to TensorFlow. The code below is a simple algorithm using division and subtraction. Convert the following algorithm in regular Python to TensorFlow and print the results of the session. You can use `tf.constant()` for the values `10`, `2`, and `1`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4.0\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "\n",
    "x = tf.constant(10)\n",
    "y = tf.constant(2)\n",
    "z = tf.subtract(tf.divide(x, y), 1)\n",
    "\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    output = sess.run(z)\n",
    "    print(output)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TensorFlow Linear Functions\n",
    "\n",
    "The most common operation in neural networks is calculating the linear combination of inputs, weights, and biases. As a reminder, we can write the output of the linear operation as:\n",
    "\n",
    "![](https://d17h27t6h515a5.cloudfront.net/topher/2017/February/58a4d8b3_linear-equation/linear-equation.gif)\n",
    "\n",
    "Here, **W** is a matrix of the weights connecting two layers. The output **y**, the input **x**, and the biases **b** are all vectors.\n",
    "\n",
    "### Weights and Bias in TensorFlow\n",
    "\n",
    "The goal of training a neural network is to modify weights and biases to best predict the labels. In order to use weights and bias, you'll need a Tensor that can be modified. This leaves out `tf.placeholder()` and `tf.constant()`, since those Tensors can't be modified. This is where `tf.Variable` class comes in.\n",
    "\n",
    "### tf.Variable()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = tf.Variable(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `tf.Variable` class creates a tensor with an initial value that can be modified, much like a normal Python variable. This tensor stores its state in the session, so you must initialize the state of the tensor manually. You'll use the `tf.global_variables_initializer()` function to initialize the state of all the Variable tensors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "init = tf.global_variables_initializer()\n",
    "with tf.Session() as sess:\n",
    "    sess.run(init)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `tf.global_variables_initializer()` call returns an operation that will initialize all TensorFlow variables from the graph. You call the operation using a session to initialize all the variables as shown above. Using the `tf.Variable` class allows us to change the weights and bias, but an initial value needs to be chosen.\n",
    "\n",
    "Initializing the weights with random numbers from a normal distribution is good practice. Randomizing the weights helps the model from becoming stuck in the same place every time you train it. You'll learn more about this in the next lesson, gradient descent.\n",
    "\n",
    "Similarly, choosing weights from a normal distribution prevents any one weight from overwhelming other weights. We'll use the `tf.truncated_normal()` function to generate random numbers from a normal distribution.\n",
    "\n",
    "### tf.truncated_normal()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'Variable_7:0' shape=(120, 5) dtype=float32_ref>"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_features = 120\n",
    "n_labels = 5\n",
    "weights = tf.Variable(tf.truncated_normal((n_features, n_labels)))\n",
    "\n",
    "weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `tf.truncated_normal()` function returns a tensor with random values from a normal distribution whose magnitude is no more than 2 standard deviations from the mean.\n",
    "\n",
    "Since the weights are already helping prevent the model from getting stuck, you don't need to randomize the bias. Let's use the simplest solution, setting the bias to 0.\n",
    "\n",
    "### tf.zeros()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_labels = 5\n",
    "bias = tf.Variable(tf.zeros(n_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `tf.zeros()` function returns a tensor with all zeros.\n",
    "\n",
    "## TensorFlow Softmax\n",
    "\n",
    "The softmax function squashes it's inputs, typically called **logits** or **logit scores**, to be between 0 and 1 and also normalizes the outputs such that they all sum to 1. This means the output of the softmax function is equivalent to a categorical probability distribution. It's the perfect function to use as the output activation for a network predicting multiple classes.\n",
    "\n",
    "![](https://d17h27t6h515a5.cloudfront.net/topher/2017/February/58950908_softmax-input-output/softmax-input-output.png)\n",
    "\n",
    "We're using TensorFlow to build neural networks and, appropriately, there's a function for calculating softmax."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = tf.nn.softmax([2.0, 1.0, 0.2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Easy as that! `tf.nn.softmax()` implements the softmax function for you. It takes in logits and returns softmax activations.\n",
    "\n",
    "### Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.         0.33333334 0.33333334 0.         0.33333334 0.        ]\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "\n",
    "def run_2():\n",
    "    output = None\n",
    "    logit_data = [19,354,354,45,354,54]\n",
    "    logits = tf.placeholder(tf.float32)\n",
    "    \n",
    "    # Calculate the softmax of the logits\n",
    "    softmax = tf.nn.softmax(logits)   \n",
    "    \n",
    "    with tf.Session() as sess:\n",
    "        # Feed in the logit data\n",
    "        output = sess.run(softmax, feed_dict={logits: logit_data})\n",
    "\n",
    "    return output\n",
    "\n",
    "print(run_2())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## One-Hot Encoding\n",
    "\n",
    "Transforming your labels into one-hot encoded vectors is pretty simple with scikit-learn using `LabelBinarizer`. Check it out below!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TensorFlow Cross Entropy\n",
    "\n",
    "In the Intro to TFLearn lesson we discussed using cross entropy as the cost function for classification with one-hot encoded labels. Again, TensorFlow has a function to do the cross entropy calculations for us.\n",
    "\n",
    "![](https://d17h27t6h515a5.cloudfront.net/topher/2017/February/589b18f5_cross-entropy-diagram/cross-entropy-diagram.png)\n",
    "\n",
    "To create a cross entropy function in TensorFlow, you'll need to use two new functions:\n",
    "\n",
    "* `tf.reduce_sum()`\n",
    "* `tf.log()`\n",
    "\n",
    "### Reduce Sum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = tf.reduce_sum([1, 2, 3, 4, 5])  # 15"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `tf.reduce_sum()` function takes an array of numbers and sums them together.\n",
    "\n",
    "### Natural Log"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "l = tf.log(100.0)  # 4.60517"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function does exactly what you would expect it to do. `tf.log()` takes the natural log of a number.\n",
    "\n",
    "### Example\n",
    "\n",
    "Print the cross entropy using `softmax_data` and `one_hot_encod_label`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.35667497\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "softmax_data = [0.7, 0.2, 0.1]\n",
    "one_hot_data = [1.0, 0.0, 0.0]\n",
    "\n",
    "softmax = tf.placeholder(tf.float32)\n",
    "one_hot = tf.placeholder(tf.float32)\n",
    "\n",
    "# Print cross entropy from session\n",
    "cross_entropy = -tf.reduce_sum(tf.multiply(one_hot, tf.log(softmax)))\n",
    "\n",
    "with tf.Session() as session:\n",
    "    output = session.run(cross_entropy, feed_dict={one_hot: one_hot_data, softmax: softmax_data})\n",
    "    print(output)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mini-batching\n",
    "\n",
    "\n",
    "Mini-batching is a technique for training on subsets of the dataset instead of all the data at one time. This provides the ability to train a model, even if a computer lacks the memory to store the entire dataset.\n",
    "\n",
    "Mini-batching is computationally inefficient, since you can't calculate the loss simultaneously across all samples. However, this is a small price to pay in order to be able to run the model at all.\n",
    "\n",
    "It's also quite useful combined with SGD. The idea is to randomly shuffle the data at the start of each epoch, then create the mini-batches. For each mini-batch, you train the network weights with gradient descent. Since these batches are random, you're performing SGD with each batch.\n",
    "\n",
    "\n",
    "\n",
    "## Epochs\n",
    "\n",
    "An epoch is a single forward and backward pass of the whole dataset. This is used to increase the accuracy of the model without requiring more data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}