Reducing overfitting in wrapper-based search
Citation:
John Loughrey, 'Reducing overfitting in wrapper-based search', [thesis], Trinity College (Dublin, Ireland). School of Computer Science & Statistics, 2006, pp 136Download Item:
Loughrey TCD THESIS 7970 Reducing overfitting.pdf (PDF) 102.0Mb
Abstract:
The benefits of wrapper-based techniques for feature selection are well established. However, it is acknowledged that overfitting can occur in feature selection using the wrapper method when there is a limited amount of training data available. That is, feature subsets
that perform well on the training data may not perform as well on data not used in the training process. In this thesis we show that two stochastic search techniques (Simulated Annealing and Genetic Algorithms) that can be used for wrapper-based feature selection are susceptible to overfitting in this way. However, because of their stochastic nature, these algorithms can be stopped early to prevent overfitting. We present a framework called Stochastic Search with Early Stopping (SSES) that implements early-stopping for both of these stochastic search techniques. We believe that by controlling the intensity of search (using early stopping) we can reduce the effects of overfitting and increase the generalisation accuracy. We demonstrate that the problem of overfitting in feature weighting can be exacerbated if the feature weighting is fine grained. With greater representational power we risk learning not only the signal, but also the idiosyncrasies of the training data. Traditionally researchers have avoided overfitting by reducing the representational power of the Machine Learning algorithm. However, we believe that we can avoid overfitting while maintaining the representational power and show that the effects of both search intensity and increased representational power can be ameliorated by the early-stopping strategy we present. The SSES framework is evaluated over seven datasets and we find good results for both versions of the framework. We compare the framework to other feature selection algorithms used to deal with overfitting and find a favourable performance for our approach in each case. In the context of feature weighting, we find that while under normal search conditions feature weighting is outperformed by feature selection the opposite is true when using the SSES approach.
Author: Loughrey, John
Advisor:
Cunningham, PádraigQualification name:
Doctor of Philosophy (Ph.D.)Publisher:
Trinity College (Dublin, Ireland). School of Computer Science & StatisticsNote:
TARA (Trinity's Access to Research Archive) has a robust takedown policy. Please contact us if you have any concerns: rssadmin@tcd.ieType of material:
thesisAvailability:
Full text availableLicences: