Hyperparameter Optimisation Methods For Transformer Neural Nets

Abstract

Previous studies comparing hyperparameter optimisation (HPO) algorithms are becoming stale, with the result that machine learning practitioners have little guidance as to which HPO methods are preferable for transformer models with text data. The present study is a step towards addressing this gap. It compares five popular HPO methods, under fair and reproducible conditions, evaluating their performance for hyperparameter tuning of a transformer model trained on text data. To the best of our knowledge, this is the first apples-to-apples comparison of these HPO methods, and also the first HPO comparison study using transformer neural nets with text data.