Testing Process and Environment
Each of the products went through the same multi-phase testing process that F5 has used in previous reports. This process consists of the following phases:
- Preliminary Testing: Create and validate the configuration for each Device Under Test (DUT) so that all DUTs manage the network traffic the same way.
- Exploratory Testing: This determines the best test settings for each device and reveals how well it performs in each type of test. The DUTs configuration is finalized during this phase.
- Final Testing: Each type of test is run multiple times. Testing is repeated until there are at least three good runs that consistently produced the best results. It can take many runs of a test to reach this standard of consistency.
- Determine Best Results: The three best test runs for each type of test are examined in detail to identify which one produced the best overall performance. The results of that best run for each type of test are what is used in this report.
In total, more than 50 test runs were conducted in order to produce these results.
The products we tested were in similar price bands, and consisted of:
- Citrix 14080 ($113,069)
- A10 4440S ($94,240)
- F5 BIG-IP i7800 ($85,000)
SSL Processing Tests
Secure Sockets Layer (SSL) encryption is used around the world to secure communications between users and applications. SSL is a standard encryption protocol available in every major operating system, web browser, smart phone, and so on. SSL technology helps make online shopping secure, enables secure remote access (SSL VPN) and much more—SSL is ubiquitous in commercial and consumer networking security solutions. SSL provides security using a combination of public key cryptography to share the cryptographic keys, and symmetric encryption (commonly RC4, 3DES, or AES) to actually encrypt the traffic. Both the key exchange and the various encryption algorithms are computationally-intensive, and require specialized hardware on the server side to achieve acceptable performance or large scale in nearly all commercial uses of SSL.
SSL Transactions per Second (TPS) performance is primarily a measure of the key exchange/handshake capacity of a device. Normally measured with small file sizes, this measures the handshake operations that occur at the start of every new SSL session. This operation is computationally-intensive and all major SSL offload vendors use specialized hardware to accelerate this task. For larger server responses and file sizes, the computational cost of the handshake operation is less relevant. Because the operation only occurs once at the beginning of a session the overhead is much less. A more balanced metric for comparison of performance is the throughput of encrypted traffic, also known as symmetric encryption or bulk crypto. Bulk crypto is a measure of the amount of data that can be encrypted and transferred in a given second.
There are different approaches to handling SSL traffic. Some devices will use specialized hardware only for the SSL handshake / key exchange, and then use the CPU for the ongoing ‘bulk’ encryption. Other devices have the advantage of using specialized hardware for both functions. The F5 iSeries is uniquely designed to optimally handle SSL connection setup and bulk throughput. By fully utilizing the advanced crypto hardware, F5 iSeries platforms have excellent transactional performance while simultaneously delivering large amounts of encrypted bulk throughput. This allows customers and system administrators to preserve CPU cycles for additional performance or functionality.
As usual, tests were conducted across a range of file sizes (128B, 5KB, 16KB, and 512KB) to demonstrate performance in a range of situations.
Tests were run using 384 bit key sizes, which is the size that is recommended by all reputable security agencies, using ECDH-ECDSA-AES128-SHA256 ciphers, which is one of the most common cypher algorithms available.
The iSeries platform continues F5’s leadership in delivering comprehensive SSL solutions for our customers—including being the first ADC to support dedicated hardware offload of ECDHE. As more businesses move to ECC cipher suites for perfect forward secrecy, the need for solutions that ensure app performance will continue to grow. Our performance testing shows that F5’s iSeries platforms maintain the highest levels of performance while supporting the broadest range of cipher suites going forward.