Distributed inference: 50 data centers, one service

Praful BhaidasnaVP, Product - F5 BIG-IP Next for Kubernetes | F5

Ask a telco architect to draw an AI cluster and you get racks, GPUs, the fabric, the storage, all of it. Ask them to draw how a request gets to that cluster and the pen stops. Traffic just arrives. It shows up at the doorstep, and the cluster takes it from there.

That blind spot is about to get expensive. For distributed inference, that gap can mean idle GPUs at one site and overloaded queues at another. A service provider with 50 facilities across a country is sitting on something a hyperscaler with two cannot buy: proximity to users, power diversity, a sovereign footprint, and fiber already in the ground.

That is the strongest argument for distributed inference, serving models from a network of sites instead of from one. But 50 sites only become one inference service if something decides, for every request, which of the 50 should answer it. Today, nothing does. That is the gap F5 is closing with NVIDIA, AMD, and other AI infrastructure providers, so the decision runs on telemetry from the accelerators and inference stacks service providers actually deploy.

Traffic doesn't arrive by magic

Here is what actually happens. A DNS lookup picks a site. A global server load balancer weighs proximity, health, and capacity and returns an address. Inside the site a proxy terminates the connection, checks who is asking, applies policy, and a load balancer picks a server. This chain is called application-aware routing. It has quietly carried every web and mobile service for two decades, and F5 built much of it.

The chain still works for AI. It just makes its decisions for the wrong reasons. The global server load balancer (GSLB) knows which site is closest and which is up. It does not know which site has the model loaded, whether that site's GPUs are saturated, or whether this request is a voice turn that cannot cross the country or a long agentic job that can go wherever capacity is cheap. So it picks the nearest healthy site, which may be the one running hot while an identical cluster 200 kilometers away sits half empty.

From application-aware to AI-aware routing

The jump is this: application-aware routing has to become AI-application-aware routing. Same chain, same DNS, same proxies. Two new inputs.

The first is telemetry from inside the AI infrastructure. Every cluster knows which models it is serving, how deep its queues are, how much KV cache is in use, how many requests are in flight, and how long a new one will wait for its first token. F5 BIG-IP Next for Kubernetes already sits at the front of those clusters and sees all of it, down to per-request token counts. Send that upward and F5 DNS solutions stop holding a map of where sites exist and starts holding a live picture of what each site can do right now.

F5 DNS solutions combine live inference telemetry from every site with the intent of the request to steer it to the site that can meet its SLA right now.

The second is the intent of the request. An LLM-aware router can classify a request before it is placed: which model it needs, how sensitive it is to latency, roughly how much work it represents, and which SLA tier the tenant is paying for. That intent becomes an input to the DNS answer instead of something discovered after the request has landed in the wrong place.

Combine the two and a service-level agreement (SLA) stops being a paragraph in a contract and becomes a routing rule. A latency SLA routes to the nearest site with the model loaded and headroom to spare. A cost SLA routes to the least expensive capacity that is free. A sovereignty requirement never leaves the country. The GSLB makes the decision it always could, now with the information it never had.

Three levels of decision, one answer

Placement is not one decision but three, nested inside each other.

• Globally, F5 DNS solutions choose the site or AI factory, now on live inference telemetry and request intent as well as proximity, health, and sovereignty.
• Regionally, the request is steered to a cluster with the right model and available capacity and crosses a tenant-specific security boundary that enforces isolation before any GPU is touched.
• Locally, inside the cluster, BIG-IP Next for Kubernetes applies identity, access, and traffic policy, and Endpoint Picker chooses the worker.

That last level is where routing stops being a path to the GPU and becomes part of inference performance. When an engine processes a prompt, the prefill stage fills the GPU's KV cache. If the next request shares that prefix and lands on a worker that still holds it, the work is reused. Land anywhere else and the GPU recomputes it. At light load nobody notices. Under production concurrency, the working set outgrows the cache, and placement starts deciding whether work is reused or rebuilt.

With BIG-IP Next for Kubernetes 2.4, Endpoint Picker weighs cache prefix, queue depth, running requests, KV-cache utilization, and expected time to first token before choosing a worker. It supports prefill/decode disaggregation and open inference technologies such as llm-d and the Kubernetes Gateway API Inference Extension. What matters is that the three levels share context. A site chosen for capacity does not hand off to a full cluster. A cluster chosen for a model does not send the request to a worker that has to rebuild its context from scratch.

The network that knows what the GPUs know

None of this requires a new platform beside the AI infrastructure. The global steering layer exists. The cluster-level intelligence exists and is in production with F5 customers today. The connecting piece, cluster telemetry, and request intent feeding a single DNS decision, is the work now underway.

So back to the architect with the pen. Fifty data centers do beat two for AI inference, but only if the network between them knows what the GPUs know. Then 50 sites stop being 50 islands of capacity and behave like one inference service that can honor an SLA, respect a border, and use every GPU already paid for.

Consider this the trailer. The full feature, how it is built, with which tools, and what it measured, is coming.

Learn more about how F5 is helping organizations improve distributed inference, AI traffic management, and GPU utilization across modern AI infrastructure.

About the Author

Praful BhaidasnaVP, Product - F5 BIG-IP Next for Kubernetes | F5

More blogs by Praful Bhaidasna