Stratification benefits in experimental design: Evidence from multiple contexts

This paper examines the effectiveness of stratification in experimental design using evidence from multiple large-scale experiments. We analyze data from experiments ranging from approximately 30,000 to 180,000 units across different business contexts. Our results show that pre-stratification and post-stratification achieve virtually identical precision improvements - largest in smaller samples (10% improvement in standard errors) and moderate but positive in larger samples (5% improvement). In more homogeneous populations, these benefits decrease substantially in small samples (24-63% reduction) but remain relatively stable in large samples (only 1-5% reduction). Post-stratification offers greater flexibility by allowing adjustments for any pre-treatment variable after randomization without risking accidental bias, while pre-stratification optimizes balance for chosen variables (reducing imbalances by 54-92%) but may create imbalance in variables negatively correlated with the stratification variables. The choice between methods ultimately depends on whether the primary goal is to improve precision, maintain implementation flexibility, or achieve visible balance in specific known covariates for the experimental context.