Are there any good resources that go more in depth into the analysis of tuning of the ConnectX3 FDR cards? Currently with ib_send_bw natively between the two adapters (connected directly / switchless) I'm getting around 1580 MB/s which seems to be far below what the VMware ESXi and other mellanox results claim for the card.
I'm wondering if there's any analysis on the effects of different BIOS settings: power management, hyperthreading, and etc. I've left the power management to be power friendly vs high performance mode, which I assume will simply let the CPUs run full throttle. Perhaps someone has some tuning observations they've made in the past with regards to the packaged performance benchmarks, ib_send_bw, ibv_rc_pingpong, etc.
Thanks