In this project I have built a neural network from scratch so that the people who are new to Neural Networks and Deep learning could understand how they work. Most of the times, no one will build Neural Networks from scratch. Often we use libraries like Pytorch and TensorFlow (ofcourse there are many others too) to build the neural networks. But building them from scratch will help in understanding the working behind these libraries.
simple neural network with one layer
We will build a simple neural network with just one layer. The dataset can be downloaded from here. It consists of three input features: GRE score, GPA, and the rank(numbered 1 through 4). The goal here is to predict if a student will be admitted to a graduate program based on his rank, gre and gpa scores. The first step of building is data cleaning.
Firstly, the categorical data must be removed. In this dataset rank is the categorical variable. Learn more about categorical variables here. Next, the data must be scaled such that they have mean as zero and standard deviation as one. Finally the data should be split into training and testing set so that we train on training on training data and test the model performance on testing data. The exact code with more decription can be found in the neural net from scratch.ipynb notebook
The next step after cleaning the data is model building. First we initialise the weights with size same as number of features. Next, we take each row from the training data and get the output after multiplying with weights and passing into activation function. The activation function we are using here is Sigmoid function. Next we calculate the error which is actual output minus obtained output error = actual_output - obtained_output. The formula for error term is error_term = error*sigmoid_prime(obtained_output). Next we add the error_term*input to a variable delta_of_w. The delta_of_w is summed for all the input values and finally we update the acutal weights. The formula to update the actual weights is original_weights += learn_rate*delta_of_w/n_records, where learn_rate is hyper-parameter that controls how much we are adjusting the weights of our network. We repeat this process for many iterations to reach to the optimal value of weights that predict the output accurately. See the notebook for more conceptual clarity and step by step descriptions.
After the model has been trained for 4000 iterations(also called epochs),Finally the model is tested is testing data and the accuracy obtained as 75%. That is really is great as we have used just one layer. The accuracy can be improved by adding more layers and tuning the hyperparameters.
Wondering how all the formulas came?? I have derivied them in the below image. In the below image in the final step while updating
the weights, I have written it as w_i = w_i + delta * x_i. I have not added learning rate and number of records here. It can be done by w_i = w_i + learn_rate * delta * x_i / n_records. n_records is the total number of records, we
divide by it to take the average of delta_w of all the inputs.
