Single-Layer-Neural-Net-without-ML-Libraries.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural Network without ML Libraries ##\n",
    "Neural networks are a system which is used in deep learning. I wrote a single-layer neural network without using any Machine Learning Libraries. I only used numpy for math and matplotlib for data visualization. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import Libraries ###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define Data ###\n",
    "I will use fake data in this example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Data\n",
    "X = [[1,1,0,0,1,1,1,1],[1,1,0,0,0,1,1,1],[1,0,0,0,1,1,1,1],[0,1,1,1,0,0,0,1],[0,0,0,1,0,0,0,0],[0,0,0,1,1,0,0,0]]\n",
    "\n",
    "# Labels - where players choose to attack mostly\n",
    "Y = [1,1,1,0,0,0]\n",
    "\n",
    "# What are the different labels\n",
    "labels = [0, 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create A Neural Network ###\n",
    "A Neural Network can be simply created as three arrays: weights, biases, activations, because every layer is consist of matrix operations: Weight * a + bias.\n",
    "\n",
    "Note: NOTE: This function can create multilayer neural networks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def create_neural_net(layer_array, input_dims):\n",
    "    weights = []\n",
    "    biases = []\n",
    "    activations = []\n",
    "    \n",
    "    for i in range(len(layer_array)):\n",
    "        node_num = layer_array[i][0]\n",
    "        weights_of_layer = []\n",
    "        biases_of_layer = []\n",
    "        if i == 0:\n",
    "            last_layer_node_number = input_dims\n",
    "        else:\n",
    "            last_layer_node_number = layer_array[i-1][0]\n",
    "        \n",
    "        for n in range(0,node_num):\n",
    "            weights_of_node = []\n",
    "            for l in range(0, last_layer_node_number):\n",
    "                weights_of_node.append(1) \n",
    "            weights_of_layer.append(weights_of_node)\n",
    "            biases_of_layer.append(0)\n",
    "            \n",
    "        weights.append(weights_of_layer)\n",
    "        biases.append(biases_of_layer)\n",
    "        activations.append(layer_array[i][1])\n",
    "    return [weights, biases, activations]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test of create_neural_net() function ###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " weights: [[[1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1]]] \n",
      "\n",
      " biases: [[0, 0]] \n",
      "\n",
      " activations: ['sigmoid']\n"
     ]
    }
   ],
   "source": [
    "layer_array = [[len(labels), 'sigmoid']]\n",
    "input_dims = 8\n",
    "neural_net = create_neural_net(layer_array, input_dims)\n",
    "\n",
    "print(' weights:',neural_net[0],'\\n\\n biases:',neural_net[1],'\\n\\n activations:', neural_net[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Activation Functions ###\n",
    "Activation functions are functions that prevent values from getting too low or too high. There are many activation functions but I will wrote two of them: sigmoid and relu."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def sigmoid(x):\n",
    "    return 1 / (1 + np.exp(-x))\n",
    "    \n",
    "def sigmoid_deriv(x):\n",
    "    return x * (1 - x)\n",
    "\n",
    "def relu(x):\n",
    "    if x < 0:\n",
    "        return 0\n",
    "    else:\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predict Using The Neural Network ###\n",
    "NOTE: Predict and predict_ratio functions are designed to be used both for single-layer and multilayer neural nets. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def predict_ratio(data, neural_net):\n",
    "    weights = neural_net[0]\n",
    "    biases = neural_net[1]\n",
    "    activations = neural_net[2]\n",
    "    \n",
    "    layer_num = len(weights)\n",
    "    \n",
    "    for l in range(0, layer_num):\n",
    "        data = np.dot(weights[l], data)\n",
    "        for t in range(len(data)):\n",
    "            data[t] += biases[l][t]\n",
    "        if activations[l] == 'sigmoid':\n",
    "            data = sigmoid(data)\n",
    "        elif activations[l] == 'relu':\n",
    "            data = relu(data)\n",
    "        else:\n",
    "            # If not identified, do it with sigmoid\n",
    "            data = sigmoid(data)\n",
    "            print('activation function', activations[l], 'cannot be found. Sigmoid is used')   \n",
    "    return data\n",
    "\n",
    "def predict(data, neural_net):\n",
    "    data = predict_ratio(data, neural_net)\n",
    "    \n",
    "    class_num = len(data)\n",
    "    \n",
    "    highest_class = None\n",
    "    highest_class_probability = -1\n",
    "    \n",
    "    for i in range(0, class_num):\n",
    "        if highest_class == None:\n",
    "            highest_class = i\n",
    "            highest_class_probability = data[i]\n",
    "        elif data[i] > highest_class_probability:\n",
    "            highest_class = i\n",
    "            highest_class_probability = data[i]\n",
    "            \n",
    "    return highest_class, highest_class_probability"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0, 0.9933071490757153)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Of course, this prediction is absulutly wrong because we didn't trained the network yet\n",
    "predict(X[1], neural_net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train Network ###\n",
    "The most important feature of a neural net is learning.\n",
    "`train_network()` function is the function of learning, but how a neural net learns?\n",
    "\n",
    "#### Backpropagation ####\n",
    "Backpropagation is the most used technique for training. It changes weights by the weight's error. For example, if a weight changes the prediction too much in a wrong way, Backpropagation will decrease it. If a weight changes the prediction not enough in a right way, Backpropagation will increase it.\n",
    " \n",
    "For more info visit: https://en.wikipedia.org/wiki/Backpropagation\n",
    "\n",
    "#### Gradient Descent ####\n",
    "Gradient Descent is a optimization function that we use in deep learning.\n",
    "\n",
    "For more info watch: https://www.youtube.com/watch?v=IHZwWFHWa-w\n",
    "\n",
    "NOTE: `train_network()` function can only be used in single-layer networks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_network(X, Y, labels, neural_net, epochs=1000):\n",
    "    for epoch in range(0, epochs):\n",
    "        for d in range(0, len(X)):\n",
    "            prediction = predict_ratio(X[d], neural_net)\n",
    "            \n",
    "            # Calculate total error per label\n",
    "            true_prediction = []\n",
    "            for i in range(0, len(labels)):\n",
    "                true_prediction.append(0)\n",
    "            true_prediction[labels.index(Y[d])] = 1\n",
    "            \n",
    "            errors = []\n",
    "            for t in range(len(prediction)):\n",
    "                errors.append(true_prediction[t] - prediction[t]) \n",
    "            adjust_deriv = errors * sigmoid_deriv(prediction)\n",
    "            \n",
    "            for k in range(0, len(adjust_deriv)):\n",
    "                adjustment = np.dot(X[d], adjust_deriv[k])\n",
    "                neural_net[0][0][k] += adjustment\n",
    "    return neural_net"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "neural_net = train_network(X, Y, labels, neural_net, epochs=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results  ###\n",
    "You may see, neural network can classify the data correctly with a good accuracy rate, but there can be problem named 'Overfitting'. Overfitting means a neural network is fit on the data it trained and cannot classify other possible data points. For better results, train the net with more data and test with data not used in training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 0.9919398375371878)\n",
      "(1, 0.9936373425420446)\n",
      "(1, 0.9925307416847557)\n",
      "(0, 0.9903682943291514)\n",
      "(0, 0.9836167677309535)\n",
      "(0, 0.9876082070557368)\n"
     ]
    }
   ],
   "source": [
    "for i in range(len(X)):\n",
    "    print(predict(X[i], neural_net))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see the accuracy rate decreses when predicting a data point which is different than training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0, 0.589227580166354)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict([1,1,0,1,0,1,0,1], neural_net)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}