We love InfiniBand. But it is no merely enough to simply install InfiniBand. We decided to test three different popular connection options with InfiniBand so we could better understand which method offers the best performance. We tested IPoIB with connected mode enabled (IPoIB-CM), IPoIB with connected mode disabled (IPoIB-UD), and SRP.
IPoIB stands for IP over InfiniBand. IPoIB is a TCP/IP networking stack that runs on top of InfiniBand fabric. Without IPoIB, you cannot do IP based network traffic over an InfiniBand network. Connected Mode means the connection is TCP (rather than UDP) style connection is used. Connected Mode allows for packets as large as 64KB.
When Connected Mode is disabled, the mode is called Unreliable Datagram. In Unreliable Datagram mode, UDP (rather than TCP) packets are used. Max packet size with IPoIB-UD is 2KB.
With both IPoIB-CM and IPoIB-UD, you need to use iSCSI to connect to the ZFSBuild2012 SAN.
SRP means SCSI over Remote Direct Memory Access Protocol. If you are really serious about performance and your environment supports SRP, you definitely want to use SRP. With SRP, you don’t use iSCSI or TCP/IP. SRP operates at fabric level. The ZFSBuild2012 server registers its available SRP targets with the InfiniBand subnet manager. Then your SRP enabled clients ask the subnet manager about the SRP target. From the client side, it looks like a local drive. Xen admins can create LVMs on the drive. Windows/Hyper-V admins can create a Clustered Shared Volume on the SRP drive.
SRP was the only way ZFSBuild2012 could deliver over 100,000 IOPS. Neither IPoIB option could deliver that level of performance. We expect this was due to the overhead of iSCSI and TCP/IP. If you cannot get SRP working, you will still see really good performance, but not the huge amount that SRP can deliver.
If you cannot use SRP, we recommend using IPoIB-UD (connected mode disabled). Connected mode seems to hurt performance.
All of the following benchmarks were run using the ZFSBuild2012 server with Nexenta 3.1.3. Write back caching was enabled for the shared ZVol in each test. Click here to read about benchmark methods. The 1Gbps performance of the ZFSBuild2012 is included as a point of reference within these graphs.
IOMeter 4k Benchmarks:
IOMeter 8k Benchmarks:
IOMeter 16k Benchmarks:
IOMeter 32k Benchmarks:
12 Comments to ZFSBuild2012 – InfiniBand Performance
Leave a Reply
You must be logged in to post a comment.