Incremental query optimizer statistics in Amazon Redshift

Accurate optimizer statistics are fundamental to query and ML-prediction performance in modern database systems, yet maintaining them poses a significant challenge for large-scale data warehouses. Traditional statistics collection relies on full table scans, which become prohibitively expensive as tables grow to billions of rows and beyond. This creates a critical tension: statistics must be kept current to ensure high-quality query plans and accurate ML predictions, but the cost of collecting them grows with data volume. Amazon Redshift, a fully managed, petabyte-scale cloud data warehouse, exemplifies this challenge as customers continuously ingest data from sources such as application logs, IoT telemetry, clickstream data, or use zero-ETL to ingest transactional and operational data. Incremental statistics collection addresses this challenge by updating statistics based solely on modified data, avoiding full table scans while maintaining accuracy. In this paper, we present the design and implementation of incremental statistics collection and maintenance based on data sketches in Amazon Redshift. This technique reduced the fleet-wide weekly compute time spent on collecting statistics for large tables by 40%, while producing statistics that are as accurate as, or even more accurate than before. Our experiences offer practical insights for database practitioners seeking to adopt incremental statistics in production systems at scale, particularly those employing PostgreSQL-style optimizer statistics.