Zero-shot Bayesian optimization with TabPFN: Competitive with state-of-the-art without per-task training

Bayesian hyperparameter optimization typically requires fitting a surrogate model to each new task, incurring per-task training cost that grows with the number of observations and limits deployment flexibility. We show that TabPFN v2 (Hollmann et al., 2025), a pretrained tabular foundation model never trained on Bayesian optimization data, can serve as a drop-in zero-shot BO surrogate, eliminating the per-task fitting step. On HPO-B v3-test (16 search spaces, 𝑛=430 matched cells, 50-trial budget), TabPFN-direct BO wins against 5 of 7 HPO-B baselines and ties the other 2 without any per-task training, matches DRE at half the evaluation budget (a 2× sample-efficiency advantage at budgets ≥25), and wins against the concurrent ZeroShotOpt (Meindl et al., 2025) on 13/16 HPO-B spaces at step 50 and 11/11 YAHPO-Gym scenarios. The enabling mechanism is calibration. TabPFN's predictive intervals achieve empirical coverage of 0.94–0.95 against the 95% target on both benchmarks, and calibrated uncertainty makes acquisition functions equivalent: EI, LogEI, and UCB variants produce optimization outcomes statistically equivalent to our UCB default under pre-specified paired equivalence tests, and a smaller cross-surrogate check on SMAC-GP is consistent with this pattern. The acquisition choice is therefore not a meaningful lever; practitioners can deploy any standard acquisition function without per-task selection, removing a tuning decision from the deployment pipeline. The full statistical protocol, effect sizes, and multiple-testing control are reported in §4.