A quantile-based approach for hyperparameter transfer learning

Bayesian optimization (BO) is a popular methodology to tune the hyperparameters of expensive black-box functions. Despite its success, standard BO focuses on a single task at a time and is not designed to leverage information from related functions, such as the performance metric of the same algorithm tuned across multiple datasets. In this work, we introduce a novel approach to achieve transfer learning across different datasets as well as different metrics. The main idea is to reparametrize raw metrics as quantiles via the probability integral transform, and learn a mapping from hyperparameters to metric quantiles. We introduce two methods to leverage this estimation: a pure random search biased toward sampling lower quantiles, and a Gaussian process using such quantile estimate as a prior. We show that these strategies can combine the estimation of multiple metrics such as runtime and accuracy, steering the optimization toward cheaper hyperparameters for the same level of accuracy. Experiments on an extensive set of hyperparameter tuning tasks demonstrate significant improvements over several baselines.