Dynamic scaling in action
In the previous elastic container service setup, the team allocated the maximum number of workers required for peak loads. This meant paying for peak capacity around the clock, regardless of actual usage.
Managed Airflow provides built-in dynamic scaling, eliminating the need for manual resource management. The environment automatically adjusts the number of workers based specifically on the workload demands. When tasks spike, the system scales up; when they drop, it scales down to save resources. Similarly, moving from fixed provisioning to dynamic scaling reduced monthly costs by an estimated 30%.
Reliability and DAG isolation
Achieving increased stability was a primary driver for Checkout.com’s migration since in the past, a single problematic DAG could affect its entire environment.
Managed Airflow introduced a number of critical architecture improvements:
-
DAG isolation: Each DAG runs in its own execution environment. If one DAG fails or consumes excessive resources, it doesn’t affect the entire environment.
-
Managed operations: Google Cloud handles patching and upgrades during scheduled windows, removing the need for manual upgrade management.
-
Improved visibility: Integration with Google Cloud Monitoring and Cloud Logging provides clear visibility into task execution. Engineers can now debug issues independently without escalating to the platform team.
Faster developer workflows
The migration also improved day-to-day workflows for Checkout.com’s data engineers.
- Faster deployments: Using Cloud Storage for DAGs enabled near-instant syncing.
- Simpler onboarding: Teams no longer needed platform support to create variables before onboarding.
- Modernizing dbt execution: One of the company’s most significant wins was changing how it runs dbt. Previously, its engineers had to manually install and manage complex virtual environments for every supported dbt version. By leveraging containerized dbt runs, Managed Airflow (Gen 3) eliminates dependency bottlenecks. This ensures complete dependency isolation, allowing teams to run any required dbt model with minimal setup and no manual infrastructure overhead.
- Environment updates: The company no longer needs to redeploy the entire Airflow environment to add new roles or update Python packages.
- AI-powered troubleshooting with Gemini Cloud Assist: In a self-managed environment, a failed task often triggered a frantic hunt through fragmented logs and metrics. With Managed Airflow, Checkout.com can initiate a Gemini investigation directly from its Airflow DAG UI in the Google Cloud console.
Gemini doesn’t just provide generic error messages; it generates a scorecard that evaluates different hypotheses with both supporting and contradictory evidence, which can drastically reduce mean time to recovery.
Conclusion
For Checkout.com, the move to Managed Airflow (Gen 3) marked a strategic shift, one that freed its engineers to focus on delivering value.
“With Managed Service for Apache Airflow, we’ve achieved significant improvements in efficiency, scalability, and reliability. Managed infrastructure, automated scaling, faster deployments, and isolated execution environments have transformed how we operate.” — Keisi Mancellari, Data Platform Engineer, Checkout.com
With a stable, scalable, and cost-efficient platform in place, Checkout.com is now able to focus on the future of its data pipelines, confident that its orchestration layer is ready for whatever comes next.
Learn more about Managed Service for Apache Airflow and how it can support your data platform.
Special thanks to the following contributors to this post: Serge Bouschet and Keisi Mancellari
