Simple answer -- No. Not in this world, and not in our lifetime. Throughout my day-to-day activities around high-speed clustering interconnects, I have seen many questions from different people and was entertained to see how marketing people try to convince us that the Earth is flat, the sun is circulating Earth and that a turtle is faster than a rabbit. In this article, I have tried to gather most of the typical interconnect-related questions and to provide the logical answers.
1. Are all InfiniBand alike?
No. InfiniBand includes five speeds - 10Gb/s (SDR), 20Gb/s (DDR), 40Gb/s (QDR), 56Gb/s (FDR) and 100Gb/s (EDR). The first three exist in the market today, with 40Gb/s (QDR) since early 2008. FDR and EDR were just announced this year by the IBTA and are on the vendors’ roadmaps. When one refers to InfiniBand, the appropriate speed should be mentioned, as well. Same as the way we refer to Ethernet – 100Mb, 1GigE, 10GigE, etc.
2. Can GigE be faster than InfiniBand?
No. GigE provides 1Gb/s, less than any InfiniBand speed. GigE node-to-node latency is in the range of 50us while InfiniBand is in the 1us range. Moreover GigE requires TCP as the transport which consumes lots of CPU cycles and therefore reduces the system/CPU efficiency to 50 percent or less. InfiniBand transport is done within the network hardware and therefore does not consume much from the CPU, and the system/CPU efficiencies are in the range of 85 percent or higher.
3. Can iWARP be faster than InfiniBand?
No. iWARP is a very complex solution for bringing RDMA capability to Ethernet. iWARP is really RDMAP over DDP over MPA over TCP, and this complexity is the reason for the higher CPU overhead and higher latency (6-10usec) when compared to InfiniBand. Moreover, InfiniBand provides 40Gb/s now, and iWARP solutions are 10Gb/s. We can create situations where iWARP will “show” better performance, and those situations require using InfiniBand 10Gb/s (that was introduced to the market in 2002) and not 40Gbs to make the bandwidth of iWARP and InfiniBand in par, and use different setups for the benchmarking, such as using more servers for iWARP and less servers for InfiniBand to compensate on the latency and CPU availability advantages of InfiniBand. This is what iWARP marketing folks typically do.
4. Can 10GigE be faster than InfiniBand?
No. Same reasons as discussed from the previous question.
5. Can 40GigE be faster than InfiniBand?
40GigE can deliver in par bandwidth as InfiniBand, but InfiniBand’s advantages in latency and CPU efficiency will continue to enable InfiniBand as the faster and more scalable and efficient solution. Current expectations are that InfiniBand will deliver 100Gb/s per port, before 40GigE will be in full production. InfiniBand has proven multiple times that solutions come to market shortly after the specification is completed. Just as a data point, the specification for 10GigE was done in 2002.
6. Can applications show better performance on Ethernet?
Not really. Applications that are latency/bandwidth/CPU efficiency dependent will run much faster on InfiniBand – you can see even 10x performance improvement using InfiniBand. Applications that don’t utilize the network can provide same level of performance on any interconnect.
7. Is 10GigE cheaper than InfiniBand?
No. At the moment, InfiniBand network is 2x or cheaper than 10GigE. InfiniBand has a lower cost than 10GigE and definitely better cost/performance.
I hope that the data here helps to dismiss some of the FUD created by marketing organizations. For more questions and answers, please e-mail HPC@mellanox.com.
In recent news, TMCnet reported that IBM (News - Alert) conducted a benchmark study that showed performance delivered by 10G Ethernet was better compared to InfiniBand Gb Ethernet for a range of high-performance computing applications.