I am having issues with ConnectX-3 dual port 40GB adapters connected to a Brocade VDX6940 switch. It appears to be a problem with the physical layer negotiation. Randomly when I reboot the server, a port will not come up. I have two adapters in the server. I have seen this problem across 3 different physical servers thus far. I also have used two different VDX6940s, with no difference. I am hoping someone can point me to some debug tools to help track down the problem.
Server: Dell FC630
OS: Windows 2012 R2 with latest updates
NIC Firmware: 2.36.5080
NIC Driver: 5.10.11345.0
I am using Brocade 40G-QSFP-SR4 transceiver modules in both the switch and NIC, connected with 2 meter MTP/MTP 40GB fiber patch cables. I have swapped transceivers, cables, and there is no correlation to what port might not come up after a reboot.
Results from ibstat, on the ports that are working fine after a reboot, state = active, Physical State = LinkUp
On the ports that don't come up, state = down, Physical State = Disabled
When I do "show media" on the brocade switch, I see no RX power level on the port that is down. a "show interface" shows the interface up, line protocol down, link protocol down.
The only way to correct the problem is to pull the SFP out of the NIC port and reseat it. I have attempted to pull the SFP out of the switch port on the same link, but it does not bring the link up. Only reseating the SFP in the NIC will bring the link up.
Are there any other commands for windows that I can use to debug what is going on with the Mellanox adapter? Is there a way to reset the port via command line? These are purely Ethernet adapters, no IB.
Thanks.