Hello, I'm new to IB and have been working on it for two weeks now without success. Here are my hardware:
2x Dell R510 server
2x MT26428 Mellanox 40gig
1xcisco 3064x switch
I have installed Ubuntu server 14.04 on both servers.
Each server has one of the dual port IB cards installed.
I have installed the MLNX_OFED_LINUX-3.0-2.0.1-ubuntu14.04-x86_64.tgz drivers on both.
Once ofed installed I have plugged into the HCA and cisco switch a QSFP-H40G-CU3M-ESI.
There are no lights indicator on the switch nor on the HCA.
I reboot both server and run ibstat and get down and polling on both IB
I then run the command sudo opensm and get messages that port is down and still no indicator lights HCA or switch and Ibstat show down/polling.
Here are the trouble shooting steps I took.
-I grabbed one of the QSFP cables and plugged one end into port 50 of the cisco switch and I plugged the other end into port 51 of the cisco switch and the lights on the ports turn green which tells me the cable and the ports on the switch is good.
-I grabbed one of the QSFP cables and plug one into port 1 or serverA and the other end into port 1 of ServerB. I then reboot both machines and now I can see the green led lights solid on both HCA cards. I run ibstat and see that the physical state on port 1 is now linkup but the state is initializing. So now I run sudo opensm on one of the ServerA and it just sits there at Entering Master state but now the amber led lights on the HCA lit up solid. I go to serverB and run ibstat and I get the sate change to Active and Link is up. ServerA is stuck on Entering master state so I hit ctrl-C to close and I get the message exiting SM but my ibstat is still good for serverB and serverA. Both LED lights are on.
-Now I give ip address to both servers. I enter the command sudo ifconfig ib0 192.168.0.1 netmask 255.255.255.0 to serverA and 192.168.0.2 for serverB.
-At this point the two servers cannot ping each other using ip address nor using ibping -G guid
Firewall is OFF
Please see the following information attached.
In the end my main problem is that there is no link light on the HCA and switch when connected with the QSFP cable. I need the two machine to connect to the network and not just each other.
Please help!!!!