Search

cloud.google.com › blog › products › …

GKE Inference Gateway prefix caching accelerates AI inference | Google Cloud Blog

… Source: Principled Technologies GKE 3rd party Managed Kubernetes Service GKE Advantage Mean output token throughput 7,169.21 output tokens per second 6,042.05 output tokens per second 15.7% more output token throughput Mean time to first token TTFT 188.36 ms 2624.73 ms 92.8% less TTFT Mean inter-to… …

Jun 9, 2026 · Bob Tian

cloud.google.com › blog › products › …

Improving Ray Serve LLM on GKE throughput, latency | Google Cloud Blog

… For more details, check out the following resources: New from Anyscale: High Performance Distributed Inference with Ray Serve LLM Enable High Throughput on Ray Serve with KubeRay Serve an LLM with multi-cluster Ray Serve and GKE Inference Gateway Serve Gemma open models on GKE with Ray Posted in Co…

Jun 18, 2026 · Spencer Peterson