The following are some of the major innovations contributing to these improvements in performance and total cost of ownership (TCO), including the built-in guardrails against regression.
BigQuery’s Self-Learning Engine: History-Based Optimizations (HBO)
One of the foundational capabilities of BigQuery’s autonomous query processor is history-based optimizations.
Traditional query optimizers rely on static statistics, metadata and cardinality estimates, which can be wildly inaccurate when faced with highly complex, multi-table joins and rapidly growing/changing data. Managing these is part of the critical path for administrators and automation has to be tailored to individual workloads to be effective and efficient. Even when everything is up-to-date and correct, queries can still have vastly different behavior due to natural data skew or changes in available compute resources. History-based optimizations change the paradigm: in addition to BigQuery’s already adaptive query execution that can change plans and resource allocations while a query is running, it learns from past executions and automatically applies additional optimizations for future executions.
It tracks runtime statistics of past queries to “remember” which optimizations were beneficial and continue to apply them, and learn from prior mistakes to ensure they are not repeated. When the same or similar query runs again, BigQuery automatically applies any additional optimization technique that is known to be beneficial and avoids those that can cause regressions.
No User Action Required & Built-in Safety Guardrails
History-based optimizations require no application rewrites, SQL modifications, or schema changes. Users literally do nothing, and their recurring dashboards, ELT pipelines, agentic workloads, or line of business modules run faster.
Crucially, this is a self-maintaining, self-improving closed-loop system with built-in safety guardrails. History-based optimizations only apply an optimization when there is high confidence it will improve performance. What if an optimization makes the wrong decision? This capability is inherently self-correcting. When an optimization is applied, the system measures the result. If the expected improvement is observed, the optimization is accepted. If it does not significantly improve performance—or worse, regresses or leads to failure—the optimization is immediately rejected, revoked, and never tried again for that query. This includes detecting data skew so queries that have parameter sensitive plans do not run into major performance issues when just a single parameter (aka WHERE clause) is changed.
The result? A decrease in execution times and a reduction of slots. One enterprise customer reported P90 execution times drop by up to 50%, with slot usage falling by up to 15% resulting in substantial price-performance improvements and no regressions.
