Over the past decade, Kubernetes established itself as the standard for orchestrating microservices and containerized applications. Today, we’re witnessing the next major architectural shift in distributed computing: AI workloads moving from isolated research prototypes into large-scale, multi-tenant production fleets. In this post, we’ll walk through the open-source stack that powers this shift and show you the sample manifests that you can try on your own Kubernetes clusters. You can also join us at KubeCon + CloudNativeCon North America 2026 to see these projects in action through live demos and technical sessions.
For modern engineering teams, production AI is rapidly moving beyond generic calls to closed, black-box APIs. Organizations are realizing that their long-term competitive advantage lies in domain specialization. Platform engineers, ML researchers, and AI builders want the autonomy to run high-performing open-weight models (such as the Gemma family). They want to adapt them to proprietary enterprise datasets by using reinforcement learning (RL) and serve them with predictable latency, high throughput, and maximum hardware efficiency.
Whether it’s scientific initiatives training domain models like DolphinGemma (an LLM helping scientists study how dolphins communicate) on acoustic data, legal platforms evaluating complex contracts, or engineering teams building autonomous coding agents, the objective is the same: achieving frontier-level task performance at a fraction of the serving cost and latency.
Yet, scaling RL post-training, stateful distributed inference, and autonomous agent execution on traditional container infrastructure introduces substantial hurdles:
-
Accelerator underutilization in RL: When you alternate between training and rollout generation, expensive GPUs and TPUs remain idle for most of the training run.
-
Cache-blind load balancing: When you use standard Kubernetes round-robin and least-connections routing, multi-turn agent calls are scattered across arbitrary pods and wipe out key-value (KV) cache locality.
-
Framework fragmentation: When you operate parallel, siloed compute clusters (such as bespoke Ray or Slurm setups) alongside Kubernetes, you fragment capacity and multiply maintenance toil.
-
Agent security in multi-tenant clusters: When you execute untrusted, model-generated shell commands and scripts in shared environments, you expose clusters to sandbox escapes and privilege escalation.
At KubeCon + CloudNativeCon North America 2026, Google Cloud is addressing these challenges by introducing and expanding a unified, open-source AI-native stack across Kubernetes and Google Kubernetes Engine (GKE) based on these components:
-
OpenRL (CNCF Sandbox): A Kubernetes-native post training service that simplifies RL post-training and maximizes accelerator utilization.
-
llm-d 1.0 (CNCF Sandbox): An open distributed inference control plane that provides prefix-cache-aware routing, disaggregated serving, and multi-tier KV cache management.
-
Ray on GKE with KubeRay (CNCF Sandbox): A framework that provides enterprise lifecycle management, elastic autoscaling, and Kueue-driven scheduling for distributed Ray compute and data processing.
-
Agent Substrate (CNCF Sandbox): An open orchestration runtime that provides lifecycle management and execution targets for multi-turn AI agent workflows.
-
gVisor (CNCF Sandbox): A kernel-level container sandbox that delivers strong system call isolation when you execute untrusted agent code in multi-tenant clusters.
-
TorchTPU: A backend that provides first-class, native PyTorch support across Cloud TPUs and GPUs inside upstream vLLM and SGLang.
Bridge reinforcement learning and Kubernetes with OpenRL
While supervised fine-tuning (SFT) has become standard practice, domain excellence especially in multi-step reasoning, mathematical proofing, and autonomous tool use requires RL.
Historically, it’s been difficult to bridge the world of ML experimentation with production Kubernetes fleets. ML researchers want to iterate rapidly in Python and PyTorch by using standard reward functions, while platform teams need multi-tenant governance, quota enforcement, and predictable scheduling.
To eliminate this friction, Google created OpenRL and contributed it to the CNCF Sandbox.
Architectural highlights of OpenRL
-
Clean 4-primitive Python abstraction: Researchers write standard Python training code that’s centered around four core primitives (forward_backward, optim_step, sample, and save_state). OpenRL translates these primitives into Kubernetes scheduling workflows without requiring developers to learn Kubernetes internals or manage infrastructure code.
-
Cooperative accelerator time-slicing: OpenRL interleaves training steps and rollout inferences on the same physical accelerators. This approach avoids the need to dedicate separate, static clusters to compute-bound training and latency-sensitive rollout generation.
-
Delta-weight synchronization: Shipping full multi-gigabyte model weights over cluster networking after every optimization step creates massive network saturation. OpenRL transmits only parameter deltas to rollout workers, eliminating rollout cold starts.
-
Zero-config dynamic resource allocation (DRA): OpenRL automatically profiles model context lengths, batch dimensions, and LoRA versus full fine-tuning configurations. It then dynamically bin-packs workloads across a heterogenous accelerator fleet by using native Kubernetes DRA claims.
Distributed inference at scale with llm-d 1.0
After you post-train a model, serving it serving it under interactive, bursty agent traffic presents a new set of distributed systems challenges. Standard ingress controllers treat LLM traffic as stateless HTTP requests, scattering sequential turns across different replicas and constantly rebuilding KV cache.
Google Cloud, Red Hat, IBM Research, CoreWeave, and NVIDIA co-founded the project, with contributions from over 500 engineers across 25+ organizations. Today, llm-d 1.0 is the CNCF Sandbox distributed inference control plane that transforms standalone inference engines (like vLLM and SGLang) into a coordinated, cluster-wide serving fabric.
