Calibrating Probability Density Forecasts with Multi-objective Search
Metadata:Show full item record
Citation:Carney, Michael; Cunningham, Padraig. 'Calibrating Probability Density Forecasts with Multi-objective Search'. - Dublin, Trinity College Dublin, Department of Computer Science, TCD-CS-2006-07, 2006, pp12
In this paper, we show that the optimization of density forecasting models for regression in machine learning can be formulated as a multi-objective problem.We describe the two objectives of sharpness and calibration and suggest suitable scoring metrics for both.We use the popular negative log-likelihood as a measure of sharpness and the probability integral transform as a measure of calibration. We show how optimization on negative log-likelihood alone often results in sub-optimal models. To solve this problem we introduce a multi-objective evolutionary optimization framework that can produce better density forecasts from a prediction users perspective. Our experiments show improvements over state-of-the-art approaches.
Publisher:Trinity College Dublin, Department of Computer Science
Series/Report no:Computer Science Technical Report