When smart isn’t safe: What my reef tank taught me about AI, automation, and circuit breakers

F5 Ecosystem | August 05, 2025

Application delivery isn’t a nice-to-have in the era of AI and automation. It’s absolutely essential as load balancers, API gateways, and service meshes become the last line of defense when autonomous systems go off script.

Earlier this week, I lost a controller.

More specifically, I lost the Apex controller for my secondary system. It’s the automation brain responsible for handling part of my reef tank’s infrastructure, including two pumps that deliver fresh saltwater to the main system during water changes.

These pumps were configured with fallback values set to ON, a default safety measure in the event of system silence. So when the controller failed, they turned on and kept running. Unchecked. Uncoordinated. Unobserved.

The result? A near overflow of the main tank and a very panicked Lori trying to contain it. And a reminder that automation without boundaries isn’t smart. It’s risky.

And that reminder has everything to do with how we’re building modern IT architectures, especially as AI begins to take the wheel.

Automation is only as safe as its failure mode

What happened in the tank wasn’t due to a bug or hardware issue. The pumps did exactly what they were told. The problem was that when the controller failed, they kept going. There was no circuit breaker. No external watchdog. No smart fallback strategy.

This is exactly what can happen in an AI-driven IT system.

Imagine AI-generated policy updates being applied to live infrastructure. Or real-time traffic steering decisions based on a model that’s quietly failing. Or a deployment pipeline that keeps pushing changes because the controller never explicitly said “stop.”

In those moments, silence doesn’t mean approval. It means danger.

The role of load balancers as circuit breakers

In most enterprise architectures, we rely on load balancers, API gateways, and service meshes to act as our version of circuit breakers. They monitor for health, detect latency, apply retry logic, and cut off unhealthy systems before they bring down everything else.

That’s not just operational hygiene; it’s how we contain failure and preserve trust in autonomous systems.

When the control plane goes dark, the load balancer’s job is to stop traffic, not assume everything’s fine.

Sometimes “fail ON” is the right answer

Of course, not all systems should shut down in a failure. In my reef tank, the display tank circulation pumps are also automated but those are deliberately configured to fail ON.

Because without flow, oxygen levels drop and fish die.

This is the nuance that matters. Some systems must keep running no matter what, such as DNS, identity, and base telemetry. But process automation, like policy generation or staged deployments? Those need to fail OFF if the controller disappears.

And that’s where AI agents introduce real tension: they don’t always know when they’re wrong. That’s why the architecture around them must.

AI adoption is rising and with it the need for boundaries

Results from the 2025 F5 State of Application Strategy Report confirm what most of us are already seeing in the field: AI is moving from suggestion to action.

  • 77% of respondents said they want AI to generate suggestions for adjusting policies and configurations.
  • 45% want AI to automatically adjust those policies in real time to meet SLOs.
  • 55% want AI to execute scripts to deploy new/adjust existing configurations and policies.

That tells us something important: we’re entering the age of autonomous digital agents, and organizations are not backing away. They’re accelerating.

But AI that can write or modify configs without human review? That’s like leaving the pumps running with no oversight. It’s only a matter of time before something floods or breaks.

Conclusion: Build systems that know when to stop

AI is not optional. Neither are circuit breakers.

As we invite more intelligent agents into our systems—into traffic flows, configuration engines, and policy decisions—we need to equip the infrastructure beneath them with clear, automatic limits.

Load balancers. Gateways. Health checks. Retry logic. These aren’t just patterns; they’re guardrails for safe autonomy. That’s one of the driving reasons behind the development of the Application Delivery Top 10, to put a spotlight on the importance of application delivery to operating and securing a digital estate. Because it’s only going to become more important as automation overtakes manual operations in the role of controller of digital business.

Automation without discipline isn’t resilience.

It’s just chaos with a cron job.

Share
Tags: 2025

About the Author

Lori Mac Vittie
Lori Mac VittieDistinguished Engineer and Chief Evangelist

More blogs by Lori Mac Vittie

Related Blog Posts

The everywhere attack surface: EDR in the network is no longer optional
F5 Ecosystem | 11/12/2025

The everywhere attack surface: EDR in the network is no longer optional

All endpoints can become an attacker’s entry point. That’s why your network needs true endpoint detection and response (EDR), delivered by F5 and CrowdStrike.

F5 NGINX Gateway Fabric is a certified solution for Red Hat OpenShift
F5 Ecosystem | 11/11/2025

F5 NGINX Gateway Fabric is a certified solution for Red Hat OpenShift

F5 collaborates with Red Hat to deliver a solution that combines the high-performance app delivery of F5 NGINX with Red Hat OpenShift’s enterprise Kubernetes capabilities.

F5 accelerates and secures AI inference at scale with NVIDIA Cloud Partner reference architecture
F5 Ecosystem | 10/28/2025

F5 accelerates and secures AI inference at scale with NVIDIA Cloud Partner reference architecture

F5’s inclusion within the NVIDIA Cloud Partner (NCP) reference architecture enables secure, high-performance AI infrastructure that scales efficiently to support advanced AI workloads.

F5 Silverline Mitigates Record-Breaking DDoS Attacks
F5 Ecosystem | 08/26/2021

F5 Silverline Mitigates Record-Breaking DDoS Attacks

Malicious attacks are increasing in scale and complexity, threatening to overwhelm and breach the internal resources of businesses globally. Often, these attacks combine high-volume traffic with stealthy, low-and-slow, application-targeted attack techniques, powered by either automated botnets or human-driven tools.

Volterra and the Power of the Distributed Cloud (Video)
F5 Ecosystem | 04/15/2021

Volterra and the Power of the Distributed Cloud (Video)

How can organizations fully harness the power of multi-cloud and edge computing? VPs Mark Weiner and James Feger join the DevCentral team for a video discussion on how F5 and Volterra can help.

Phishing Attacks Soar 220% During COVID-19 Peak as Cybercriminal Opportunism Intensifies
F5 Ecosystem | 12/08/2020

Phishing Attacks Soar 220% During COVID-19 Peak as Cybercriminal Opportunism Intensifies

David Warburton, author of the F5 Labs 2020 Phishing and Fraud Report, describes how fraudsters are adapting to the pandemic and maps out the trends ahead in this video, with summary comments.

Deliver and Secure Every App
F5 application delivery and security solutions are built to ensure that every app and API deployed anywhere is fast, available, and secure. Learn how we can partner to deliver exceptional experiences every time.
Connect With Us