Journée Mathematical Foundations of Learning Theory
< précédent | suivant >
|Suboptimality of MDL and Bayes in Classification under Misspecification|
Peter Grünwald (Centrum voor Wiskunde en Informatica & Eurandom)
31 mai 2006
We show that forms of Bayesian and MDL learning that are often applied to classification problems can be “statistically inconsistent”. We present a classification model (a large family of classifiers) and a distribution such that the best classifier within the model has classification risk r, where r can be taken arbitrarily close to 0. Nevertheless, no matter how many data are observed, both the classifier inferred by MDL and the classifier based on the Bayesian posterior will make predictions with error much larger than r. If r is chosen not too small, predictions based on the Bayesian posterior can even perform substantially worse than random guessing, no matter how many data are observed. Our result can be re-interpreted as showing that, if a probabilistic model does not contain the data generating distribution, then Bayes and MDL do not always converge to the distribution in the model that is closest in KL divergence to the data generating distribution. We compare this result with earlier results on Bayesian inconsistency by Diaconis, Freedman and Barron.
This work is a follow-up on joint work with John Langford of the Toyota Technological Institute, Chicago, published at COLT 2004, available at www.grunwald.nl.