Estimating the Expected Error of Empirical Minimizers for Model Selection

Download or read book Estimating the Expected Error of Empirical Minimizers for Model Selection written by Tobias Scheffer and published by . This book was released on 1998 with total page 26 pages. Available in PDF, EPUB and Kindle. Book excerpt: Abstract: "Model selection is considered the problem of choosing a 'good' hypothesis language from a given ensemble of models. Here, a 'good' model is one for which the true (or generalization) error of the hypothesis returned by a learner which takes the model as hypothesis language is low. The crucial part of model selection is to somehow assess the true error of the apparently best hypothesis (the empirical minimizer) of a model. In this paper, we discuss a new, very efficient approach to model selection. Our approach is inherently Bayesian, but instead of using priors on target functions or hypotheses, we talk about priors on error values -- which leads us to a new insightful characterization of the expected true error. Consequently, our solution is based on the prior of error values for the given problem which is, of course, unknown. But we show next that this prior can be estimated efficiently for a given learning problem by recording the empirical errors of a constant number of randomly drawn hypotheses. Using this estimated prior, our framework yields an estimate of the true error of the empirical minimizer of a model. We report on several controlled experiments (based on artificial problems and boolean concepts) which provide strong empirical evidence for the usefulness of the approach: In terms of accuracy, our algorithm becomes slightly superior to 10-fold cross-validation as the size of the models grows. In terms of time complexity and scalability, our algorithm is quite superior to cross-validation: Whie cross validation requires n invocations of the learner per model, a fast version of our algorithm is constant in the size of the models."