HI,
I've been investigating SR-IOV support on ConnectX-3, specifically MT27500 with CentOS-7 as the host and CentOS-6.4 as the guest. I've create one VF on my card and expose it to the VM. I wasn't too clear on what to do within the guest, so I installed the entire Mellanox stack, version MLNX_OFED_LINUX-2.2-1.0.2 within the guest as well. I'm able to run ibv_devinfo inside the guest and it seems fine. I've also been able to run some benchmarks with Fluid Dynamics codes using verbs within the VMs. However, I am not sure if I've set up everything the correct way and if configuration is optimal. My queries:
1) I haven't set any specific GUIDS in the sysfs on the host as per section "4.15.6.2.1SR-IOV sysfs Administration Interfaces on the Hypervisor" of the Mellanox documentation.
2) I have had issues with RDMA connectivity and applications reporting no RDMA connection could be made. Restarting the vms and the hosts resolved the issue somehow,
3) I see around 20% drop in performance on a Fluid Dynamics code with the current setup. Is this expected ? Is there any fine tuning which can be done on the host and/or the guest ? My current mlx4_core.conf file looks like options
mlx4_core num_vfs=1 port_type_array=1,4 probe_vf=0
4) Should I really install the whole Mellanox OFED driver stack within the VM, or, is there some other driver I should be installing ?
Thanks!
/D