After messing with it again last night, I seem to have IPoIB configured as the MTU dropped to 2044 and the IPs that I assigned each node in their ifcfg-ib0 show up when i run "ifconfig ib0". Now I see the SUBNET UP message in my log but I still don't have connection. When I run ping 10.0.0.1 it times out, and when I try ibping -G <GUID> it times out as well.
I ran osmtest and got all of these errors,
and even a few more above that I couldn't fit into the screen shot.
My opensm.log on the head node running opensm shows that the subnet was up, but a number of errors showed up once I ran osmtest.
I can't really find anything in my setup that looks like an error, or anything that indicates where the error is. I am using older HCAs 26428s and when I ran mlnxofedinstall it said it couldn't update the firmware. Is this potentially a firmware issue?
I installed flexboth nodes HCA as well if it makes any difference.
Thanks,
Trevor