Regular balanced switchback designs for robust multi-unit online experimentation
2025
User-randomized A/B testing, while the gold standard for online experimentation, faces significant limitations when legal, ethical, or practical considerations prevent its use. Item-level randomization offers an alternative but typically suffers from high variance and low statistical power due to skewed distributions and limited sample sizes. We here introduce Regular Balanced Switchback Designs (RBSDs), a novel experimental framework that combines temporal and item-level randomization in a principled manner. RBSDs extend the switchback methodology of Bojinov et al. [2023] by incorporating multiple randomization designs [Masoero et al., 2024] to achieve double balance: a fixed fraction of items receive treatment at each time step, and all items receive treatment equally often across time periods. This balanced approach substantially reduces estimate variance compared to independent randomization schemes while maintaining unbiased estimation of average treatment effects under carryover effects. Using both realistic simulations based on e-commerce data patterns and theoretical analysis within a potential outcomes framework, we demonstrate that RBSDs achieve 40-60% reduction in standard errors compared to simple item randomization and 20-30% improvement over unbalanced switchback approaches. Our framework provides a principled solution for scenarios where user-randomization is infeasible, including pricing experiments and complex system testing where indirect effects through downstream systems must be captured.
Research areas