Tensorflow Notes.txt

Tensor Flow Notes:-

Tensorflow provides multiple APIs. 
Lower level api -------------------> tensorflow core
Higher level api built on top of lower level api and are easier to learn and make repetitive task easier to implement.

TENSOR: central unit of data in tensorflow. Set of primitive values into an array of any number of dimensions.
RANK of TENSOR:-
3-------------------------------------------> #rank 0, scalar shape[]
[1,2,3]-------------------------------------> #rank 1, vector shape[3]
[[1,2,3],[4,5,6]]---------------------------> #rank 2, matrix shape[2,3]
[[[1,2,3]],[[7,8,9]]]-----------------------> #rank 3, shape[2,1,3]


Tensorflow core programs 2 sections:-
1) Building computational graph
2) Running computational graph

Computational graph: series of tensorflow operations arrranged into a graph of nodes.
Each node takes 0 or more tensors as input-------------> tensor as output
constant node: no input-------> output is a value that is stored internally

MNIST is like hello world for starting tensor flow it consists of number of labelled images and our goal is to build a tensorflow model to predict.
Data is split into 3 parts:-
1) 55,000 datapoints for training data
2) 10,000 datapoints for test data
3) 5,000 datapoints for validation data

MNIST dataset has 2 parts:-
1) Image of handwritten digit   --------- X
2) Corresponding label ------------------ Y

Each image is 28*28 pixels hence is big array of numbers.(28*28 == 784 numbers)

mnist.train.images is a tensor (an n-dimensional array) with a shape of [55000, 784]
The first dimension is an index into the list of images and the second dimension is the index for each pixel in each image. 
Each entry in the tensor is a pixel intensity between 0 and 1, for a particular pixel in a particular image.

Each image in MNIST has a corresponding label, a number between 0 and 9 representing the digit drawn in the image.
mnist.train.labels is a [55000, 10] array of floats.

Softmax Regression:-
If you want to assign probabilities to an object being one of several different things, softmax is the thing to do, 
because softmax gives us a list of values between 0 and 1 that add up to 1. 
Even later on, when we train more sophisticated models, the final step will be a layer of softmax.

2 steps of softmax regression
1) add up the evidence of our input being in certain classes(NOTE: weighted sum of pixel intensity. If the pixel matches then positive weight else negative weight)
2) convert that evidence into probabilities

Softmax: exponentiating its inputs and then normalizing them
exponentiation : that one more unit of evidence increases the weight given to any hypothesis multiplicatively.
Softmax then normalizes these weights, so that they add up to one, forming a valid probability distribution.

x = tf.placeholder(tf.float32, [None, 784])
W = tf.Variable(tf.zeros([784, 10]))
b = tf.Variable(tf.zeros([10]))
y = tf.nn.softmax(tf.matmul(x, W) + b)


In machine learning we typically define what it means for a model to be bad. We call this the cost, or the loss, 
and it represents how far off our model is from our desired outcome. We try to minimize that error, and the smaller the error margin, 
the better our model is.
CROSS-ENTROPY : is measuring how inefficient our predictions are for describing the truth

Using small batches of random data is called stochastic training -- in this case, stochastic gradient descent