Running large-scale AI workloads on TPU and Google Kubernetes Engine (GKE) requires infrastructure that adapts as fast as your models change. Dynamic Slicing is generally available on GKE, which decouples logical TPU slice formation from physical nodepool provisioning.
Compared to creating a static nodepool for each TPU slice, dynamic slicing reduces AI workload startup times by up to 5x and workload recovery by 4.5x — maximizing TPU goodput, efficiency, and workload resilience at scale.
Beyond one workload, one nodepool
Previously, managing TPU capacity at GKE standard required separate, fixed TPU slice topology nodepools, increasing startup times and complexity while preventing multi-host nodepools from hosting multiple AI workloads. Changing workload needs meant manually deleting and recreating nodepools. Additionally, this tight coupling created large failure domains where a single hardware fault crashed an entire slice, slowing recovery while reprovisioning.
To overcome these challenges, we built GKE dynamic slicing, which decouples physical hardware provisioning from logical slice formation, dynamically stitching together TPU cubes using TPU optical circuit switching (OCS) technology to form various larger slice topologies (super-slicing), or partitioning TPU cubes into smaller slices for multiple workloads to run inside a nodepool (sub-slicing). By enabling workloads to assemble custom TPU slice topologies on demand without having to delete and recreate static nodepools, dynamic slicing reduces job startup latency, limits the blast radius of hardware failures, accelerates recovery of workloads, and improves fleet utilization and efficiency.
Dynamic slicing delivers several key improvements for platform teams:
-
Zero-downtime topology agility: Completely eliminates the rigid “delete-and-create” nodepool cycle, allowing to dynamically stitch together TPU cubes (16 VMs nodepools) to form large TPU slices, or carve out smaller-slices on the fly as an AI workload is submitted.
-
Up to 5x faster job startup: Slashes job startup times by up to five times by instantly assembling logical slice topologies from pre-provisioned TPU nodepools instead of waiting for a slow, new nodepool deployment.
-
Improved fleet utilization: Improves resource efficiency and eliminates idle capacity by enabling multi-tenant packing of smaller workloads within nodepools. Additionally, it lets you provision partially healthy nodepools. This way, you can utilize healthy hosts that would otherwise be unavailable.
-
Up to 4.5x faster recovery and fault tolerance: Accelerates recovery by isolating large slice failures to a TPU cube (a 16-VM 4x4x4 slice topology) and swapping unhealthy cubes in one to two minutes.
-
Enhanced visibility and control: Offers comprehensive insight into potential TPU partitions (slices) within each cube, covering real-time health and usage metrics to facilitate proactive, optimized workload scheduling.
Dynamic slicing architecture
GKE dynamic slicing is offered in two forms: super-slicing, to create larger scale TPU slices that exceed the size of a single TPU cube; and sub-slicing, which provisions smaller, independent slices from a single cube.
Dynamic super-slicing
GKE dynamic super-slicing utilizes TPU optical circuit switching (OCS) technology to dynamically connect TPU cubes (consisting of 16 VMs configured in a 4x4x4 slice topology) at the time of workload scheduling. By altering the OCS network configuration dynamically, larger slice topologies are created and tailored to the workload slice needs outlined in the Kubernetes workload manifest, significantly enhancing operational efficiency.
Super-slicing fundamentally relies on a baseline conversion of all available TPU capacity into uniform nodepools composed of 16-VM (4x4x4) TPU cubes. When a TPU slice topology is specified in Kubernetes workload YAML file, the GKE scheduler (or your custom scheduler) identifies the request and instantiates slice creation. This process uses the TPU optical circuit switch to stitch multiple 4x4x4 cubes together, dynamically instantiating the targeted slice topology and scheduling the workload directly onto the newly established super-slice.
