Goldilocks: An active sampling bandit that's just right for multi-task forecasting

Neural networks have lead to improvements in demand forecast accuracy for supply chain and retailers. These neural networks have been designed and trained on data representing their particular use cases. We investigate the zero-shot performance of those deep learning models on retail dataset outside of their original use case. As such, we focus on the hypothesis that this zero-shot performance of deep learning models is linked to how we train a model and balance multiple tasks. To address this, we introduce a new active sampling bandit called Goldilocks that samples across multiple tasks, here corresponding to difference velocity groups, based on learnable samples that are not too hard, not too easy, but just right. For comparison, we also offer a novel Dynamic Importance Sampling (DIS), an extension of Static Importance Sampling (SIS) based on demand, used to train neural networks [7, 11]. A temperature hyperparameter in Goldilocks controls the algorithm's preference for harder problems, extending the idea behind DIS. We show out-of-sample convergence results on a public retail dataset called M5 to evaluate the zero-shot performance of the sampling strategies.