An SVM practice on MNIST handwritten digits image dataset. Dataset includes 4,000 training examples and 1,000 testing examples.
Implementation of Gaussian kernel function including two voting scheme for multi-class classification:
- One versus one
For K classes(K = 10 in this handwritten digits classification task), K*(K-1)/2 classifiers are trained. Each one of them can classify between two classes, where positive class get+1vote and0vote for negative class. The prediction result is the class with max number of votes. - One versus the rest
For K classes, only K classifiers are trained. Each classifier can tell if a training example is in a class or not. Here I use a variant of the one-versus-the-rest scheme where the target value is modified so that the positive class has the value+1and the negative class has the value−1/(K − 1). This scheme is proposed by Lee et al.
Comparing the prediction result and true labels of testing examples, we can compute the accuracy of the multi-class SVM classifier. Using confusion matrix is a good way to show the result and also visualize it. An observation is that one-versus-one scheme gets a better accuracy.
- Accuracy of
One versus onescheme :0.943
- Accuracy of
One versus the restscheme :0.876
Besides the two voting schemes, another optional approach for multi-class classification can be DAG decision tree. DAG SVM is similar to one-versus-one scheme. Instead of voting to the positive class, DAG will always drop the negative class and follow a decision trace of length K (number of classes). By implementing DAG SVM, the accuracy rise about 0.1%.
- Moreover, when the sigma parameter in Gaussian kernel is tuned to 10 and the soft margin parameter "C" is tuned to 10, the multi-class classifier amazingly achieves a 100% accuracy over the 1,000 testing examples.
