Let's play a quick game. You're running a global AI operation. A user in Germany asks your chatbot a simple question. Where should that request go?
- To the nearest available GPU, wherever it is?
- To the massively powerful, incredibly expensive new model you just deployed in a US data center?
- To a smaller, cheaper, CPU-based model hosted in Frankfurt that's perfectly capable of answering the question while respecting data residency laws?
If you don't have an immediate, confident answer, you're not alone. But you're also likely bleeding cash and exposing yourself to risk. The simple, often overlooked, mechanism of model routing is fast becoming the dividing line between a successful AI strategy and an expensive, insecure science project.
And our latest survey data shows that most organizations are on the wrong side of that line.
The underestimated hero of the AI stack
When we ask folks what their biggest challenges are in serving AI models, the usual suspects top the list: Security and guardrails (48%), GPU cost (43%), and Data residency (23%). But when we ask about the challenge of "routing traffic between multiple models or versions," it barely makes a blip, with only 13% citing it as a major concern.
This is a classic case of seeing the trees but missing the forest.
Organizations don't think they have a "routing" problem; they think they have a "cost" problem or a "security" problem. In reality, they have a routing problem that is creating their cost and security problems. They have the tools (API gateways, ingress controllers, service mesh) but they're using them as simple traffic cops, not as the strategic, context-aware brain of their AI operations.
How strategic routing solves your biggest AI headaches
Let's connect the dots. Intelligent, context-aware model routing isn't just about getting a request from point A to point B. It's about getting it to the right point B, based on cost, security, performance, and compliance.
Slaying the GPU cost dragon
GPU cost and utilization is the #2 challenge for a reason: those things are eye-wateringly expensive. Letting simple requests occupy a top-tier GPU is like using a sledgehammer to crack a nut. Strategic routing offers a direct solution:
- Cost-based routing: Steer requests to the most cost-effective compute resource. A simple query can go to a CPU-based model, reserving the expensive GPUs for complex, high-value tasks.
- Utilization-based routing: Balance the load across all available resources to maximize utilization and eliminate idle, cash-burning hardware.
Without this intelligence, you're just randomly throwing requests at your most expensive assets.
Building a real security perimeter
Security is the #1 challenge, and for good reason. How do you enforce guardrails? How do you inspect prompts for malicious input? You can't just wish a policy into existence. You have to physically (well, digitally) route the traffic through an enforcement point.
- Policy enforcement: Want to run a prompt through a guardrail? You need to route the request to that specific service first. This is fundamental to implementing the "prompt and response inspection" that 53% of organizations are trying to apply.
- Egress control: The only way to enforce "egress controls on outbound calls to external model providers" (a control used by 62% of respondents) is to have a routing layer that manages and inspects that traffic before it leaves your network.
Strategic routing is the difference between having security tools and actually using them effectively.
The data residency lifeline
For the 23% of organizations struggling with data residency and sovereignty, model routing is not just a nice-to-have; it's a compliance necessity. Geo-aware routing ensures that a request from a user in the EU is handled by a model running in an EU data center, keeping their data within the legal boundaries of GDPR.
The real culprit: Fragmented policies kill smart routing
So if routing is this magical solution, why isn't it working? Because you can't execute a global strategy with a local mindset.
Our data found an alarming (but unsurprising) correlation. The organizations struggling most with these top-tier AI challenges were the ones with fragmented policy management, where each team sets its own rules or "interprets" central guidelines in its own way.
[Side quest: read this treatise on Conway’s Law and its relationship to scaling AI then ask yourself why fragmented (per team) policies are part of the problem. Go ahead. I’ll wait.]
Yeah, you can't have a coherent, cost-saving, secure model routing strategy if your east coast team routes traffic differently from your west coast team. It breaks the whole model. To truly leverage the power of routing, you need a centralized control point that provides a consistent, global view of all traffic and policies.
The technical capabilities are already there in most organizations; over half are using API gateways and ingress controllers. The missing piece is the strategy.
It's (way past) time to stop thinking of routing as a mundane operational detail and start treating it as the strategic lever that will determine the success or failure of your entire AI investment.
About the Author
Lori MacVittie is a Distinguished Engineer and Chief Evangelist in F5’s Office of the CTO with deep expertise in application delivery, automation strategy, and infrastructure. She is known for turning complexity into clarity whether she’s defining guardrails for AI agents, dissecting brittle multicloud architectures, or probing the limits of scalable systems. She brings more than thirty years of industry experience across application development, IT architecture, and network and systems operations. Before joining F5, she served as an award-winning technology editor. MacVittie holds an M.S. in Computer Science and is a prolific author whose publications span security, cloud, and enterprise architecture. She is also an avid tabletop and video gamer with unapologetically strong opinions about cheese.
More blogs by Lori Mac VittieRelated Blog Posts
The secret to scaling AI is overcoming Conway’s Law
AI won’t scale by itself. Without a unified control plane that collapses tooling and ownership boundaries, inference becomes the most brittle and coordination-heavy tier.
Why model routing may become security infrastructure
Agentic AI exposes identity limits. Specialized models plus policy-based routing binds authority to model choice, reducing blast radius and governance complexity.
Your next SRE might be a machine
Control is concentrating at the point of intent: prompts, tokens, and routing. The architecture challenge is defining where AI can act, when it must defer, and how to reverse.
AI App Delivery Top 10: Insufficient traffic controls
Traditional rate limits and throttles miss the real blast radius. AI workloads fan out across models, vector stores, and services, driving latency spikes, cascading failures, and retries.
Inference is now the center of operational gravity
Multi-model operations and API-driven automation push control to inference-time layers—prompt, token/control, routing, and moderation—where cost, risk, and performance must be managed.
From packets to prompts: Inference adds a new layer to the stack
Inference is not training. It is not experimentation. It is not a data science exercise. Inference is production runtime behavior, and it behaves like an application tier.