One endpoint, many accelerators
An agent fleet is not one workload. Interactive turns need low latency, background loops and reinforcement learning (RL) rollouts need high throughput, and demand swings by the minute. llm-d puts your accelerator pools behind a single endpoint, with one router, one KV cache index, and one flow-control policy in front of the whole fleet. Agents call a single API; the router picks the replica for each step and keeps a session’s context on the replicas that already hold it. Adding capacity, mixing accelerator generations, or moving a model to a different pool becomes a platform change, not an agent rewrite. TorchTPU brings Google Cloud TPUs into that fleet, running PyTorch models natively inside upstream vLLM and SGLang without model rewrites. For more, see the llm-d on Google Kubernetes Engine (GKE) TPU guide.
Routing that remembers context
The llm-d router builds on Gateway API Inference, the Kubernetes project for inference-optimized routing that we launched. On GKE, GKE Inference Gateway runs the llm-d router as a managed service, simplifying installation and management and extending routing across regions. The router tracks the replicas holding each cached prefix and weighs load and predicted latency when it picks one. When accelerator memory fills up, rather than dropping it, llm-d uses vLLM’s OffloadingConnector, Mooncake, or SGLang’s HiCache to move KV cache to host memory and then to SSD or Google Cloud Managed Lustre. This setup helps prevent prefix-eviction storms under heavy, multi-turn, and long-context conversational loads across mixed TPU and GPU fleets.
On an agentic coding workload on Ironwood TPU (TPU 7x), prefix-aware routing plus host memory offload cut median time-to-first-token (TTFT) by more than 85% and raised total throughput 2.4x versus a plain Kubernetes Service, as of October 2026 (source).
Disaggregation that upgrades safely
Prefill is compute-heavy and decode is memory-bound, so running both on the same replicas makes each wait on the other. llm-d runs prefill, decode, and multimodal encode as separate pools, with wide expert parallelism spreading large mixture-of-experts models across nodes. In 1.0, DisaggregatedSet — built on LeaderWorkerSet, the Kubernetes API for multi-host inference that we started — manages these pools as one unit. Rolling updates move prefill and decode together on matching versions while each phase scales independently. For more, see the disaggregated serving guide.
Async inference that runs alongside real-time inference
For latency-tolerant traffic, llm-d 1.0 introduces llm-d-async-processor, which pulls requests from queues like Google Cloud Pub/Sub or Redis and routes them alongside real-time traffic. Using queue and pool-dispatch gates, the processor checks available capacity and enforces strict priority lanes and quotas so background jobs run in slack capacity without starving interactive users.
Time-slicing to put idle RL capacity to work
During RL training, accelerators often sit idle 40% to 60% of the time during rollout generation or weight updates. llm-d now supports Google Cloud TPU and GPU time-slicing to interleave async inference into those quiet cycles, raising aggregate accelerator duty cycles from a ~40% baseline to >70% (source).
Benchmarks you can trust
Tuning a distributed inference stack across routers, KV cache tiers, and prefill/decode pools takes repeatable measurement under realistic multi-turn load. Prism is llm-d’s open-source analysis dashboard for exploring published benchmarks across llm-d’s well-lit paths, backed by standardized llm-d-benchmark recipes and inference-perf workloads. Inspect the exact configurations behind the numbers, or upload runs from your own cluster to compare throughput, latency, and cost curves side by side.
Running llm-d 1.0 on Google Cloud
llm-d builds on Kubernetes components you may already run: Gateway API, LeaderWorkerSet, Dynamic Resource Allocation, Prometheus, and OpenTelemetry. Running llm-d on Google Cloud brings these pieces together with managed infrastructure:
- GKE Inference Gateway is the llm-d router operating in Gateway Mode, a managed GKE Gateway that handles the L7 proxy and load balancing and calls the llm-d Endpoint Picker on every request, simplifying installation and management so you don’t run the proxy yourself. More importantly, GKE Inference Gateway is how you scale llm-d across clusters and regions with global failover and Model Armor safety checks.
- Autoscaling on inference signals. KEDA and Horizontal Pod Autoscaler (HPA) scale pools based on queue depth, saturation, and predicted latency, while Kueue enforces per-model quota on shared accelerators.
- Mixed accelerator node pools. Google Cloud TPU and GPU pools run behind the same llm-d endpoint, deployed with verified Helm charts.
- KV cache offloading across layers. Google Cloud Managed Lustre provides a shared remote tier beneath local TPU and GPU HBM and host CPU RAM so long-context workloads run without exhausting accelerator memory.
In production
This GKE llm-d stack carries production traffic, and it has been tested at fleet scale:
- Gemini platform (formerly Vertex AI) serves production traffic through GKE Inference Gateway, doubling its prefix cache hit rate from 35% to 70%, cutting TTFT by more than 35% for coding agents and cutting P95 TTFT by more than 50% under bursty chat traffic (source).
- In a published benchmark across a 17,000-node multi-region GKE deployment serving a mixture-of-experts model in the US and Europe, multi-cluster GKE Inference Gateway and the llm-d router scaled throughput 2.9x across three clusters with a 99.9% success rate and less than 1% routing overhead (source).
- In a published benchmark on A3 Mega cluster serving a 70B-parameter model, llm-d with multi-connector KV cache offloading to Google Cloud Managed Lustre in addition to Host CPU RAM, delivered an approximately 40% improvement in TTFT and a 30% reduction in end-to-end latency responding to 50k-token prompts compared to CPU RAM offload only, while reducing GPU-hour requirements by nearly 60% (source).
Organizations running llm-d on Google Cloud are putting these same capabilities to work in different ways, for instance prefix-aware routing to raise cache hit rates, open interfaces that fit an existing service mesh, and flow control to hold latency for priority tiers when traffic spikes.
