Mellanox SRP initiator in vSphere OFED 1 8.2.4 shows me a good performance.
But it has a issue with ZFS SRP target that cause problem to VM auto start function in ESXi host.
SRP is a native RDMA SCSI protocol that light-weight storage protocol and~ very old ancient one.
Mellanox iSER initiator in vSphere OFED 1.8.3 shows me a good performance.
IPoIB iSER initiator can support properly my ZFS iSER target that can support VM auto start function in ESXi host perfectly.
But above 2 of protocols in ESXi 6.x will be a journey to PSOD world.
IPoIB iSCSI was a bad performer in history.
They show me a only 450~650MB/s throughput with 2port ConnectX-2 HCA between ZFS IPoIB iSCSI target.
AND~ very high processor usage level will let you go to tour the Andromeda Galaxy above your roof!
I decide to move IPoIB iSCSI with vSphere OFED 2.4.0 and try SR-IOV function in ESXi 6.x environment.
Because vSphere OFED 2.4.0 can support SR-IOV and both IB and ETH mode or VPI.
vSphere OFED 2.4.0 show me a big improvement in IPoIB iSCSI performance with 2 of MHQH19B-XTR QDR HCAs.
Performance level was over 2GB/s in peak time...
But not like as RDMA protocol...
01.Mellanox vSphere OFED 2.4.0 iometer test in IPoIB iSCSI with 2 of MHQH19B-XTR and OmniOS ZFS iSER Target
Also SR-IOV support was impressive.
Absolutely, ConnectX-2 was EOL product and I build a custom configuration firmware with binary file version 2.9.1314 from IBM site to support SR-IOV.
Here is a result.
02. SR-IOV VF list in MHQH19B-XTR ConnectX-2 HCA
02. vSphere 6.0 Update 2 Build 4192238 PCI device lists
03. Windows Guest RDMA communication test
Mellanox show me a good product concept and also many functions in their brochure.
But it's almost impossible to get a good manuals and best practice guide like Dell or etc.
So many failures make a above test result.
Unstable driver!
Unstable firmware!
Undescripted driver options!
For what?
RDMA is a good concept and RDMA NIC show a excellent performance!
Mellanox said their product can support almost major OS environment.
But there ware always exist many bugs and limits in almost OS environments.
Latest product ConnectX-4,5 also useless in vSphere and Hyper-V environment, too.
It's almost EOL of vSphere 5.x and Mellanox show a beta level driver and laziest support for vSphere environment in historically.
vSphere OFED 1.8.3 beta IPoIB iSER initiator have a critical bug that was show in public some years ago.
I think vSphere OFED 1.8.3 beta IPoIB iSER initiator poor quality, tricky SRP initiator modification for experimental!
Everybody can find that how it was a critical that issue can find in resolved issue in vSphere Ethernet iSER dirver 1.9.10.5 release notes.
I'm waiting VMWorld in Aug. 2016, and expect vRDMA nic driver in new VMwareTools and stable IPoIB iSER initiator support.
I'll make a decision in near future to move or not...