The Importance of Network Segmentation for AI Factories

Within an AI cluster, we can logically segment resources with a tenant context, allowing for tenant quotas, resource consumption limits, host system security, and management role-based access control (RBAC). However, tenant contexts are not exposed with the basic network services providing AI cluster traffic ingress and egress with the rest of the AI factory network. Without this context, the bedrock of cybersecurity in the data center, network segmentation is blind. Typical methods to expose the necessary tenant contexts either significantly rob the AI factory of high-value compute or slow networking paths beneath required limits for service latency, bandwidth, or concurrency.  We face false choices between efficiently utilizing high-value AI factory resources and proper tenant integration with the network.

In public cloud infrastructures, orchestrated multi-tenant network designs are the foundation of all services in a cloud region and implemented with virtual private clouds (VPC). This virtualized network segmentation is key for security and resource metering. Public cloud providers maintain this function with teams of network software developers and specialized networking hardware, including SmartNICs and data processing units (DPUs). AI factory clusters in public clouds are designed to take advantage of the underlying infrastructure VPC network orchestration. The cost to maintain VPCs is quite substantial, but it is core to the public cloud business model.  

The question arises: How does an organization maximize its AI factory investment and dynamically schedule GPUs and compute without the same level of investment as a public cloud provider?

The industry’s first stop along this journey was to use server virtualization to create virtual machines (VMs). VMs utilize hardware pass-through to connect to segmented data center networks. Simply place all the virtual machines on the same VLANs, and we continue with operations as normal if we are only concerned about a single tenant in a single AI cluster.

VMs can also address GPU segmenting as GPU vendors support ways to subdivide GPU devices into sets of cores and memory, and then assign to specific virtual machines. However, segmenting GPU devices is not dynamic and requires a restart of the VM. Additionally, this design limits the ability to create a pool of GPU resources that can be metered across many tenants. These are significant drawbacks to this solution.

What happens to our AI factory clusters when they can no longer serve one tenant? The problem shifts to the data center. In AI factory clusters, by default, all network traffic egressing into the data center gets source network address translated (SNATed) to an individual cluster node IP address that the containerized workload that issues the network request happens to be running, effectively masking the true source. The traffic is then sourced from a network segment on which that node was deployed. For a multi-tenant cluster, this means we lose tenant context, and we get a mixed river of egress traffic from multiple tenants that’s impossible to sort out, secure, troubleshoot, or audit.

AI factories  must plan for a network feature rich, programable, secure, low latency solution that is scalable in both bandwidth and concurrency. Tenant contexts at Layer 2 (e.g., VLANs, VxLAN) and Layer 3 (e.g., Subnets, IPSEC interfaces) must be presented from within a cluster out to the AI factory network. Observability metrics, logs, and debugging tools must be available to NetOps.

Traditionally, many of these application tenancy and visibility solutions are provided by F5 BIG-IP. F5 BIG-IP Container Ingress Services (CIS) dynamically discovers Kubernetes services and exposes them to data centers as virtual servers, a configuration object BIG-IP administrators will be familiar with from configuring them to present physical servers and virtual machines. While BIG-IP does provide much of the requirements we are looking for in a solution, it does not manage egress traffic from the AI cluster into the AI factory network, which is needed for maintaining segmentation.

To address this issue, we designed F5 BIG-IP Next for Kubernetes, a solution for multi-tenant compute clusters built on top of our next-generation platform BIG-IP Next.

BIG-IP Next for Kubernetes is completely managed through the Kubernetes control plane and supports Kubernetes management authentication, RBAC for all declared resources, recognizes Kubernetes tenancy through namespaces to support the required network segmentation for both ingress and egress traffic. This is key for orchestration-first architectures like AI factories.

BIG-IP Next for Kubernetes provides a simplified way for NetOps to declare the mappings between Kubernetes namespaces and network segments. Dynamic route peering between the AI factory network and BIG-IP Next instances uses familiar route configuration syntax. NetOps teams have the unique ability to securely troubleshoot live network streams for cluster ingress and egress. SecOps teams gain per cluster tenant ingress and egress firewall access control lists (ACLs), distributed denial-of-service (DDoS), and IPS capabilities.

For the platform engineering teams, BIG-IP Next for Kubernetes unburdens compute resources by offloading network functions such as data ingress and egress traffic processing as well as Source NAT and firewalling. This helps with operational costs, while keeping services available and efficient.

BIG-IP Next for Kubernetes also supports Kubernetes Gateway API, the first community ingress API modeled for specific organizational roles in NetOps, platform engineering, DevOps, and MLOps. Through the Gateway API, BIG-IP Next for Kubernetes extends typical Layer 4 port based or Layer 7 HTTP(S) route ingress services to a suite for DevOps/MLOps, namely TCPRoute, UDPRoute, HTTPRoute, TLSRoute, and GRPCRoute—all of which are controlled through the same CI/CD automation in major AI frameworks.

From a management perspective, BIG-IP Next for Kubernetes keeps NetOps, SecOps, platform engineering, DevOps, and MLOps all working together effectively through Kubernetes API declarations. It is everything you expect from BIG-IP, but through the Kubernetes lens while supporting network segmentation.

Data processing units (DPU) have gained exponential traction within AI factories. Defined in an earlier blog post within our AI factory series, a DPU is a programmable processor designed to handle vast data movement and processing via hardware acceleration at network line rate. Through product innovation at F5 and collaboration with NVIDIA, we released BIG-IP Next for Kubernetes to offload data flows for ingress and egress traffic while supporting network segmentation, as well as security functions by deploying to the NVIDIA BlueField-3 DPUs using NVIDIA’s DOCA APIs. This maximizes the investment in an AI factory by ensuring AI clusters are “data fed.”