Solutions to Instability Problems with Sequential Wrapper-based Approaches to Feature Selection

Abstract

It is generally accepted that Wrapper approaches will outperformFilter-based approaches to feature selection, particularly in situationswhere an adequate amount of data is available. What is often overlookedis that Wrapper approaches can be unstable. For instance, differentpartitionings of the training data can result in dierent routesthrough the search space and thus in dierent feature subsets beingselected. In this paper we illustrate examples of this problem and asolution based on the aggregation of several runs of a sequential searchis suggested. This is essentially an ensemble solution to instability infeature subset selection and it does seem to stabilise the process.