F5 AI Red Team: One-time gate or ongoing practice?

Keiko CheungProduct Marketing Intern | F5

What happens to your AI model's security score after you stop testing it?

Most teams assume the answer is "nothing" because they ran a red team assessment before launch, got a good score, and moved on. However, every organization is changing the same three things at once: speed, scale, and security. They want to ship AI features fast, keep them trustworthy, and do both across a growing footprint of models, agents, and integrations without any tradeoff.

But in practice, pushing hard on one almost always cannibalizes another. A team shipping a model that updates every few weeks doesn't have six weeks to spare for a manual review each time, so last month's results get treated as still valid or an unvalidated model stays exposed. Red teaming used to be a box you check before launch, but now static, point-in-time testing can no longer keep pace with how fast AI threats evolve.

Generic, one-size-fits-all testing can't keep up with risks specific to an organization's own workflows, industry, and regulatory footprint. That's exactly why practitioners need red teaming that's tuned to their bespoke environment, and why some customers are already running it integrated directly into their pipelines and infrastructure.

What CASI and ARS scores actually measure

Static testing can tell you a model passed once, but it can't tell you how that model holds up against sustained, sophisticated, or evolving attacks. That's why F5 Labs developed two objective measures, the Comprehensive AI Security Index (CASI) and the Agentic Resistance Score (ARS), to give security teams a clear way to evaluate models, track posture over time, and report risk to leadership. Here's what each one actually measures.

Comprehensive AI Security Index (CASI) – CASI is a composite metric designed to measure the overall security of a model (methodology here). It factors in things like how many attacks succeeded, how severe the consequences would have been, and how much effort it took an attacker to get there. The higher the score, the stronger the model's overall security posture.

Agentic Resistance Score (ARS) – ARS measures how the full AI system holds up when a team of autonomous attack agents actively tries to break it, extract sensitive information, map out the system's architecture, or social engineer the model. A high ARS means the system keeps its guardrails intact even under sustained, targeted pressure, not just a single attempt.

How we’re seeing CASI, ARS, and F5 AI Red Team used in practice

Most security teams already use traditional tools like Static Application Security Testing (SAST) to scan the source code, and Dynamic Application Security Testing (DAST) to test the running application. Neither tool was designed to test whether the AI model underneath the application can be tricked, manipulated, or gradually worn down.

A CASI score is accurate for the moment it's measured, but models and threats don't stand still. A model scoring around 93 in one testing cycle can score 73 to 77 a couple of months later. That’s a 15 –to 20-point swing with no code change on the customer's side at all, simply because the underlying model shifted. That's why we see some F5 AI Red Team customers build re-testing directly into their CI/CD release pipeline. Any time a new model version or fine-tuned variation ships, or enough time has passed, the CASI and ARS baseline gets re-checked instead of assuming last quarter's score still applies.

The CASI Leaderboard tracks how CASI scores for popular AI models have shifted month-over-month.

Custom intents regulation

A standard CASI run shows how a model holds up against a broad library of known attacks, but it won't automatically catch a risk tied to one specific new feature or legal requirement. For instance, a healthcare chatbot adding a symptom checker or a financial agent gaining the ability to initiate a wire transfer are both risks a generic test has no reason to look for. For scenarios like that, customers build custom scoped attack intents around a specific functionality or regulatory requirement.

Let’s say, a multinational contract tool might need to be tested against EU AI Act obligations in one region and a completely different PII standard elsewhere. Custom intents let teams test the exact rule or feature they need to prove compliance against instead of relying on general coverage.

Building an internal red-team center of excellence

Some larger organizations (including at least one major law firm) have gone a step further and built a small, dedicated internal team whose entire role focuses solely on red-teaming AI applications for the rest of the business. This team understands both the organization's threat landscape, its CASI and ARS results, and they work directly with product and vulnerability management teams as new AI features get built.

Security teams share CASI results with application developers to make a simple point that not every model carries the same security posture; assuming otherwise is how risk slips through. This closes a knowledge gap between InfoSec and DevOps teams who might not otherwise think about model level risk at all.

The takeaway here is that the threat landscape is shifting faster than most security teams can manually keep up with. AI red teaming is an ongoing practice that must move at the same pace as the systems it’s protecting. It needs to:

• Be continuous: Manual, weeks-long testing engagements can't match how fast models and agents change.
• Close the gap between finding a risk and fixing it: Red team results feed directly into remediation, turning testing from a static report into a driver of faster, prioritized action against real exposure.
• Give security teams a way to prove the work: Every assessment produces a CASI and ARS score, giving teams an objective, benchmarkable way to track posture over time and communicate risk clearly to leadership.

The security landscape right now is changing faster than anyone can predict, but with F5 AI Red Team, CASI, and ARS, security teams can have ongoing threat-informed guardrails.

Learn more by visiting the F5 AI Red Team and the F5 Labs CASI and ARS Leaderboards webpages.

About the Author

Keiko CheungProduct Marketing Intern | F5

More blogs by Keiko Cheung