AI App Delivery Top 10: Inability to handle latency

Lori Mac VittieDistinguished Engineer and Chief Evangelist | F5

Latency has always been the thing we all pretend is “just performance” right up until it starts breaking user trust, blowing up incident volume, and quietly turning your budget into a bonfire. The original ADC06 frames this correctly as an application delivery problem, because latency is rarely contained to one layer and it never stays polite. AI takes that dynamic and makes it louder, more expensive, and much harder to hide behind a spinner.

The reason is simple. AI experiences are interactive, and people experience delay differently when they are waiting on an answer that feels conversational. A slow page load is annoying. A slow “assistant” response feels like something is wrong, and the longer the pause gets, the more the user mentally reclassifies your system from “helpful” to “unreliable” without needing a dashboard to confirm it.

Impact on performance

In traditional application delivery, latency tends to show up as a slower response time that makes the application feel sluggish. Users complain, your NPS drops, your conversion rates take a hit, and the usual arguments start about whether it’s the network, the app, the database, or “the cloud.” That problem has always been real, but it has also been familiar, which means teams know how to reason about it.

With AI, latency stops feeling like slowness and starts feeling like incompetence. Users do not interpret a long pause as “a distributed system doing distributed system things.” They interpret it as hesitation, confusion, or failure, and once that perception lands, it tends to stick. The output might eventually arrive, but you are already working uphill because the system has signaled uncertainty instead of confidence.

Token-driven behavior makes this worse in a way that traditional apps never had to deal with. The prompt can grow over time because everyone keeps adding context “just in case,” and output size can balloon because models do what they do and keep talking unless you constrain them. Even when the infrastructure is fine, you can still end up with an experience that feels slow because the work expanded, not because the system regressed.

Impact on availability

Latency has always been a reliability problem hiding in a performance costume. When a dependency gets slow, timeouts trigger. When timeouts trigger, retries show up. When retries show up, load increases. When load increases, everything gets slower, and now you are watching a cascade that behaves like an outage even though nothing is technically dead.

AI systems are especially good at turning latency into availability incidents because they have more moving parts than the average application stack. You have the model endpoint, the gateway, the orchestration layer, retrieval services, tool execution systems, and often policy enforcement in-line with the request. When one element starts responding slowly, the experience degrades in messy, hard-to-diagnose ways.

Agentic workflows add another failure pattern that looks like unreliability even when it is simply delay. Agents tend to do multiple actions per user interaction, and they tend to do them sequentially because the next step depends on the result of the previous one. That means a slow tool call does not just slow down the system, it blocks progress. When enough sessions are blocked, queues build, concurrency collapses, and your system starts exhibiting symptoms that feel like downtime, including timeouts, abandoned sessions, and a rising tide of user complaints that sound irrational until you look at the traces.

Impact on scalability

Scalability in AI systems gets misunderstood because teams keep thinking in terms of request volume when the real constraint is often time per request. As latency rises, every session stays open longer, which ties up compute longer, which reduces effective throughput.

AI adds variability that makes this far more dramatic. Some requests are quick. Some requests include massive prompts, heavy retrieval, long outputs, and multiple tool calls. If you design for averages, the tail will punish you, and the tail in AI systems is not a rounding error. Tail latency becomes your user experience, because users do not remember the best response you ever produced.

Concurrency also behaves differently. AI workloads are often interactive and session-oriented, which means they do not behave like neat stateless API traffic. The system has to hold context, handle follow-up requests, and keep the experience coherent. If latency creeps up, you are not just serving fewer requests per second, you are serving fewer conversations per minute, and that distinction matters when your product value is defined by responsiveness.

Impact on operational efficiency

Latency taxes operations in a way that is both predictable and infuriating. The first response is usually to add capacity, because it is the fastest lever to pull and the easiest to justify. The second response is usually to add caching, because it feels clever and it can help. The third response is usually to add retries, because someone wants reliability, and now you have quietly built a system that turns slowness into amplified load.

Over time, latency creates an operating model where teams spend more time maintaining the delivery system than delivering features. AI compounds that because performance is end-to-end and cross-domain. You end up troubleshooting across teams and vendors and services that were never designed to be debugged as one coherent pipeline. Every investigation becomes a map of dependencies and timing spans, and the answer is rarely simple.

Best practices for mitigating latency challenges

The first and most important practice is treating AI delivery as delivery, not as a model problem. If you only optimize inference, you are improving the engine while ignoring the traffic, the roads, and the intersections. You need to reason about the entire path from user input to final answer, including the steps that happen before the model is called and the steps that happen after.

Routing becomes a meaningful lever again because AI workloads are sensitive to distance and to queue depth. Proximity between inference and retrieval matters, and so does choosing the right capacity pool for the right kind of request. A healthy endpoint is not always the best endpoint if it is overloaded or far away or paired with slow downstream dependencies. When the system gets busy, smart routing can be the difference between a usable experience and a slow-motion collapse.

Adaptive resource allocation is also mandatory, because AI demand is bursty and unpredictable, and you cannot capacity-plan your way out of that reality. The system needs mechanisms to control consumption under stress, including request shaping, graceful degradation, model selection strategies, and limits on how much orchestration a single interaction can trigger. Those are not just cost controls. They are stability controls.

Conclusion

Latency has always been a foundational risk in application delivery, and ADC06 called it out for good reason. AI simply makes it more visible and more damaging, because delay in an AI experience feels like unreliability, and unreliability kills adoption faster than almost anything else.

If you are building AI-enabled applications today, handling latency is part of making the system feel competent, part of keeping it available under real-world conditions, part of scaling without runaway costs, and part of keeping your ops teams from living in a constant state of forensic debugging. The old rules still apply, but the consequences are sharper, because users treat AI response time as a signal of whether the system can be trusted to work at all.

Read more about the Top 10 Application Delivery challenges faced by organizations across the globe.

About the Author

Lori Mac VittieDistinguished Engineer and Chief Evangelist | F5

Lori MacVittie is a Distinguished Engineer and Chief Evangelist in F5’s Office of the CTO with deep expertise in application delivery, automation strategy, and infrastructure. She is known for turning complexity into clarity whether she’s defining guardrails for AI agents, dissecting brittle multicloud architectures, or probing the limits of scalable systems. She brings more than thirty years of industry experience across application development, IT architecture, and network and systems operations. Before joining F5, she served as an award-winning technology editor. MacVittie holds an M.S. in Computer Science and is a prolific author whose publications span security, cloud, and enterprise architecture. She is also an avid tabletop and video gamer with unapologetically strong opinions about cheese.

More blogs by Lori Mac Vittie