F5 BIG-IP v21.0 brings enhanced AI data delivery and ingestion for S3 workflows

Hunter SmitSenior Product Marketing Manager

Mark MengerSolutions Architect

Fouad ChmainySolutions Architect

Amazon S3 changed the shape of modern infrastructure. By separating object storage from compute, S3 gave application teams durable, affordable capacity with a simple API, and just as importantly, a neutral point for data created in one place and consumed in another. That model has become a backbone for AI pipelines: training sets land in buckets, fine-tuning datasets roll through versioned prefixes, and knowledge corpora for retrieval-augmented generation (RAG) are pulled by indexers all day long.

As AI estates scale, pointing your app at S3-compliant object stores stops being enough; enterprises need a delivery layer that makes data movement predictable, fast, and secure. This is exactly the problem that AI data delivery addresses: ensuring the right data gets securely to object stores and is reliably delivered to AI model foundry workflows with the right underlying policies and resilience across sites. Doing so enables training, fine-tuning, and RAG workflows to stay on schedule. An application delivery controller (ADC) in front of storage is a business and technical necessity; it is a control point that protects object stores from surges, smooths throughput to ensure cluster reliability, and centralizes governance across hybrid multicloud S3 deployments.

Why should we place F5 BIG-IP in front of data stores?

Today’s AI storage looks like yesterday’s web traffic on steroids. Ingest is bursty (e.g., parallel writers, multi-part uploads), reads are random, and maintenance is constant (e.g., node rebuilds, compactions, tiering). These unpredictable patterns demand an architecture that decouples applications from the underlying storage, shielding workloads from disruptions and endpoint volatility. Without this dedicated delivery tier, enterprises face eight common issues:

1. Local resiliency challenges: A single slow or rebuilding node can disrupt jobs throughput. F5 BIG-IP mitigates this with targeted health monitors, such as S3 HEAD/GET checks on canary keys, policy-driven load balancing, and connection draining to ensure uninterrupted, long-running transfers during maintenance.
2. Manual global resiliency: F5 BIG-IP DNS automatically routes clients to the best site based on health, proximity, and business policies. It seamlessly handles cluster or regional failovers without requiring changes to application endpoints.
3. Object- and metadata-level quality of service and rate-limiting: BIG-IP inspects and discriminates traffic based on its bucket, object type, size, and tagging. Low-priority requests are rate-limited in favor of high-priority requests, avoiding timeouts and retries in congested or oversubscribed networks.
4. Risky upgrades: Storage software upgrades can disrupt workflows. BIG-IP enables safe blue/green or rolling upgrades by draining traffic from affected pools and temporarily pinning traffic to stable system versions until health checks pass.
5. Congested replication flows: Cross-region synchronization and cluster replication can face bottlenecks during high-demand periods. BIG-IP prioritizes and shapes traffic to ensure replication processes don’t hinder performance for AI model workloads.
6. Inconsistent TLS posture: BIG-IP centralizes TLS management by handling offload, re-encryption, or passthrough. It enforces consistent cipher policies, Perfect Forward Secrecy (PFS), and mTLS where needed, simplifying storage configurations and enabling transparent scaling with SSL acceleration on F5 rSeries and F5 VELOS.
7. Post quantum cryptography (PQC) defense: With BIG-IP, organizations cancomply with U.S. National Institute for Standards and Technology (NIST) and EU Digital Operational Resilience Act (DORA) recommendations by supporting PQC in transit and protecting data from attack by future quantum computers like “harvest now, decrypt later” exploits.
8. S3 protocol defense: BIG-IP detects and blocks S3 protocol attacks such as injections, privilege escalation, bucket takeover, credential or pre-signed URL reuse, and account takeover before they reach the backend.

S3 data delivery profiles come to BIG-IP v21.0

BIG-IP version 21.0 introduces S3 TCP/TLS profiles designed to easily turn a virtual server into a production-ready S3 front door. These profiles provide TCP settings for high-bandwidth paths, refine HTTP/S3 behavior to support multi-part uploads and retries, and implement security guardrails like strict method allowlists and optional mTLS. This streamlines the transition from cluster setup to fully operating AI pipelines, reducing setup time from days to minutes.

AI data delivery in action

The following demos illustrate how BIG-IP extends beyond faster setup to deliver lasting operational value. They highlight how S3 data delivery can be abstracted from infrastructure changes, dynamically shaped through programmable policies, and safeguarded against cluster stress. Together, these examples show how a dedicated delivery layer ensures AI pipelines remain predictable, scalable, and secure.

Demo 1: Lifecycle abstraction

This demo highlights how applications can stay connected and uninterrupted even as storage environments evolve. By abstracting endpoints, workloads remain resilient through cluster lifecycle changes, vendor changes, version upgrades, or cluster growth without requiring application reconfiguration, with observability that surfaces performance and compliance insights.

Demo 2: Data plane programmability

Here we show how policies and routing can be dynamically adjusted at the data plane level. This flexibility allows enterprises to enforce fine-grained controls, adapt traffic patterns to business needs, and support risk-managed transitions between environments.

Demo 3: Hot spot prevention

The final demo demonstrates how advanced monitoring and intelligent load balancing reduce hot spots and protect clusters under stress. By detecting failures and avoiding overloaded nodes before they disrupt workflows, enterprises can maintain consistent throughput and safeguard critical AI pipelines.

Design notes for practitioners

S3-compatible storage clusters power scalable AI, machine learning, and data-driven applications, but they come with inherent challenges like hot spot formation, node maintenance, shifting traffic patterns, and ensuring resiliency during high-demand workloads. BIG-IP offers flexible and programmable traffic management to address these challenges efficiently. Below are seven essential best practices to optimize BIG-IP for S3 environments and obtain scalability, performance, and maintenance isolation:

1. Leverage Intelligent load balancing algorithms: Avoid round-robin load balancing, which can create hot spots in S3 clusters due to uniform traffic distribution. Use adaptive algorithms such as dynamic ratio or least connections to intelligently balance traffic based on node health and workload capacity, ensuring predictable performance.
2. Deploy flexible HA architectures: This configuration isolates maintenance activities from client workloads, preventing downtime while achieving maximum fault tolerance. Active-active deployments allow multiple BIG-IP devices to handle traffic simultaneously, ensuring continuous availability and increasing throughput for heavy workloads.
3. Use configuration automation: Automation ensures faster deployment, simplifies scaling, and reduces the complexity of maintaining consistent configurations across environments. Use the BIG-IP Application Services 3 Extension (BIG-IP AS3) and BIG-IP Declarative Onboarding to automate BIG-IP provisioning and configuration.
4. Utilize advanced health monitors for S3 endpoints: Enable custom health monitors to check the availability and performance of individual S3 endpoints, such as response times, error rates, and node capacity utilization. BIG-IP can automatically reroute traffic away from degraded or offline nodes, ensuring resiliency and uninterrupted workloads even during maintenance.
5. Enhance data throughput with TCP/TLS optimizations: Optimize TCP configurations, such as window scaling and congestion control, to improve throughput for large AI/ML data transfers. BIG-IP’s S3-tuned TCP profile minimizes latency, ensuring encrypted data delivery is both secure and fast.
6. Leverage data plane programmability for traffic customization: Custom programmability enables granular traffic control, ensuring performance and flexibility for unique S3 workload requirements. Use BIG-IP’s event-driven iRules to dynamically route traffic based on headers, metadata, or maintenance status of individual nodes.
7. Enable locality-aware request routing: Use locality-aware routing policies to prioritize traffic to storage nodes based on geographic proximity or latency. This reduces network overhead and improves response times for globally distributed workloads reliant on S3 clusters.

A dedicated delivery tier for AI data pipelines

S3 brings universal, durable storage; F5 BIG-IP version 21.0 gives storage a first-class delivery tier tailored to AI. The result is high-performance data pipelines that keep models training, fine-tuning, and serving RAG workflows, without relying on fragile client logic or custom scripts. BIG-IP v21.0 is now available. For more information, read the BIG-IP version 21.0 announcement blog. Also, be sure to check out today’s press release.

F5’s focus on AI doesn’t stop here—explore how F5 secures and delivers AI apps everywhere.

About the Authors

Hunter SmitSenior Product Marketing Manager

More blogs by Hunter Smit

Mark MengerSolutions Architect

More blogs by Mark Menger

Fouad ChmainySolutions Architect

More blogs by Fouad Chmainy