Dr. Boxiang Wang, University of Iowa

Friday, October 16, 2020

 

Magic Cross-Validation for Support Vector Machines and Related Large Margin Classifiers

 

In this work we study the use of leave-one-out cross-validation (LOOCV) for the support vector machine (SVM) and related large margin classifiers. We argue that LOOCV does not have much higher variance than ten-fold or five-fold CV. A strong argument against the use of LOOCV is that its computation cost is seemingly prohibitively expensive, and hence people resort to  ten-fold CV or five-fold CV. We present a “magic” leave-one-out formula that is the foundation of a new and very efficient algorithm named “magicsvm” for fitting and tuning the kernel SVM. By magicsvm, the computational cost of LOOCV is of the same order of fitting a single SVM on the training data, and magicsvm is shown to be much faster than the state-of-the-art SVM solvers based on extensive simulations and benchmark examples. We show that the leave-one-out formula also provides a direct proof of the Bayes consistency of the large margin kernel classifiers. We also use the “magic” leave-one-out formula to construct an honest LOOCV estimator for estimating the post-selection generalization error of the kernel classifier. The honest LOOCV is nearly unbiased and has very competitive performance even when competing with the famous .632+ estimator.