I am pretty new to Infiniband/VPI configurations and I am currently running 4 Nodes connected to an InfiniScale IS5022 unmanaged switch. One of the nodes is running Windows 10 with OpenSM and ConnectX-3 dual port HCA. The other 3 nodes are running VMware ESXi 6 with VMware Standard Switches using ConnectX-3 dual port HCAs for their uplinks. I have installed the appropriate OFED driver package, native driver package and MST package on all 3 VMware ESXi 6 hosts as well as the Windows OFED driver package on the Windows 10 host running the OpenSM instance. I know that everything is working correctly because I can create VLAN/PKey mappings using the partitions.conf file and appropriate configurations on the VMware Standard Switch port configurations and see that the hosts can communicate with each other with newly created VMware PortGroups with VLAN tagging defined in partitions.conf.
As a test, I created PortGroups with VLAN tags that have not been defined in the partitions.conf and validated that the hosts/guests configured for the VMware PortGroups were not able to communicate with each other. Then I created the VLAN/PKey mapping for that VMware PortGroup and the guests/hosts were able to communicate with each other without issue. The problem I am facing is that hosts outside of the VMware environment but connected to the same IB switch are NOT able to communicate with each other more specifically the host that is running the subnet manager. I validated that the Windows node is seeing the correct mapping leveraging "mlxtool dbg pkeys" command with the following output. I have configured the Windows 10 node with a Team and 5 VLAN tagged interfaces in VLAN 111, 112, 211, 212 and 213. The host was assigned and IP address in each one of the given VLANs/Partitions using IB miniport drivers.
C:\Program Files\Mellanox\MLNX_VPI\Tools>mlxtool dbg pkey
ConnectX IPoIB NIC: Lag1112__IBSwitch1__Port1
---------------- ----------------
| PKEY index | PKEY |
---------------- ----------------
| 0 | ffff |
| 1 | 806f |
| 2 | 8070 |
| 3 | 80d3 |
| 4 | 80d4 |
| 5 | 80d5 |
---------------- ----------------
ConnectX IPoIB NIC: Lag1112__IBSwitch1__Port2
---------------- ----------------
| PKEY index | PKEY |
---------------- ----------------
| 0 | ffff |
| 1 | 806f |
| 2 | 8070 |
| 3 | 80d3 |
| 4 | 80d4 |
| 5 | 80d5 |
---------------- ----------------
Below is the configuration of the partitions.conf configured for the given environment. I commented out the Multicast sections for each partition as I was seeing errors generated and complaining about invalid Multicast Group IDs in the osm.log. From the Windows 10 host, I ran the arp -a and I do not see any ARP entries for the 3 VMware ESXi hosts. What am I missing?
# Rate =
# 2 = 2.5 GBit/s
# 3 = 10 GBit/s
# 4 = 30 GBit/s
# 5 = 5 GBit/s
# 6 = 20 GBit/s
# 7 = 40 GBit/s
# 8 = 60 GBit/s
# 9 = 80 GBit/s
# 10 = 120 GBit/s
#
# MTU =
# 1 = 256
# 2 = 512
# 3 = 1024
# 4 = 2048
# 5 = 4096
#
#
#
Default=0x7fff, rate=7, mtu=5, scope=2, defmember=full:
ALL, ALL_SWITCHES=full;
Default=0x7fff, ipoib, rate=7, mtu=5, scope=2:
# mgid=ff12:401b::ffff:ffff # IPv4 Broadcast address
# mgid=ff12:401b::1 # IPv4 All Hosts group
# mgid=ff12:401b::2 # IPv4 All Routers group
# mgid=ff12:401b::16 # IPv4 IGMP group
# mgid=ff12:401b::fb # IPv4 mDNS group
# mgid=ff12:401b::fc # IPv4 Multicast Link Local Name Resolution group
# mgid=ff12:401b::101 # IPv4 NTP group
# mgid=ff12:401b::202 # IPv4 Sun RPC
# mgid=ff12:601b::1 # IPv6 All Hosts group
# mgid=ff12:601b::2 # IPv6 All Routers group
# mgid=ff12:601b::16 # IPv6 MLDv2-capable Routers group
# mgid=ff12:601b::fb # IPv6 mDNS group
# mgid=ff12:601b::101 # IPv6 NTP group
# mgid=ff12:601b::202 # IPv6 Sun RPC group
# mgid=ff12:601b::1:3 # IPv6 Multicast Link Local Name Resolution group
ALL=full, ALL_SWITCHES=full;
#
#
#
VLAN0111=0x006f, rate=7, mtu=5, scope=2, defmember=full:
ALL, ALL_SWITCHES=full;
VLAN0111=0x006f, ipoib, rate=7, mtu=5, scope=2:
# mgid=ff12:401b::ffff:ffff # IPv4 Broadcast address
# mgid=ff12:401b::1 # IPv4 All Hosts group
# mgid=ff12:401b::2 # IPv4 All Routers group
# mgid=ff12:401b::16 # IPv4 IGMP group
# mgid=ff12:401b::fb # IPv4 mDNS group
# mgid=ff12:401b::fc # IPv4 Multicast Link Local Name Resolution group
# mgid=ff12:401b::101 # IPv4 NTP group
# mgid=ff12:401b::202 # IPv4 Sun RPC
# mgid=ff12:601b::1 # IPv6 All Hosts group
# mgid=ff12:601b::2 # IPv6 All Routers group
# mgid=ff12:601b::16 # IPv6 MLDv2-capable Routers group
# mgid=ff12:601b::fb # IPv6 mDNS group
# mgid=ff12:601b::101 # IPv6 NTP group
# mgid=ff12:601b::202 # IPv6 Sun RPC group
# mgid=ff12:601b::1:3 # IPv6 Multicast Link Local Name Resolution group
ALL=full, ALL_SWITCHES=full;
#
#
#
VLAN0112=0x0070, rate=7, mtu=5, scope=2, defmember=full:
ALL, ALL_SWITCHES=full;
VLAN0112=0x0070, ipoib, rate=7, mtu=5, scope=2:
# mgid=ff12:401b::ffff:ffff # IPv4 Broadcast address
# mgid=ff12:401b::1 # IPv4 All Hosts group
# mgid=ff12:401b::2 # IPv4 All Routers group
# mgid=ff12:401b::16 # IPv4 IGMP group
# mgid=ff12:401b::fb # IPv4 mDNS group
# mgid=ff12:401b::fc # IPv4 Multicast Link Local Name Resolution group
# mgid=ff12:401b::101 # IPv4 NTP group
# mgid=ff12:401b::202 # IPv4 Sun RPC
# mgid=ff12:601b::1 # IPv6 All Hosts group
# mgid=ff12:601b::2 # IPv6 All Routers group
# mgid=ff12:601b::16 # IPv6 MLDv2-capable Routers group
# mgid=ff12:601b::fb # IPv6 mDNS group
# mgid=ff12:601b::101 # IPv6 NTP group
# mgid=ff12:601b::202 # IPv6 Sun RPC group
# mgid=ff12:601b::1:3 # IPv6 Multicast Link Local Name Resolution group
ALL=full, ALL_SWITCHES=full;
#
#
#
VLAN0211=0x00d3, rate=7, mtu=5, scope=2, defmember=full:
ALL, ALL_SWITCHES=full;
VLAN0211=0x00d3, ipoib, rate=7, mtu=5, scope=2:
# mgid=ff12:401b::ffff:ffff # IPv4 Broadcast address
# mgid=ff12:401b::1 # IPv4 All Hosts group
# mgid=ff12:401b::2 # IPv4 All Routers group
# mgid=ff12:401b::16 # IPv4 IGMP group
# mgid=ff12:401b::fb # IPv4 mDNS group
# mgid=ff12:401b::fc # IPv4 Multicast Link Local Name Resolution group
# mgid=ff12:401b::101 # IPv4 NTP group
# mgid=ff12:401b::202 # IPv4 Sun RPC
# mgid=ff12:601b::1 # IPv6 All Hosts group
# mgid=ff12:601b::2 # IPv6 All Routers group
# mgid=ff12:601b::16 # IPv6 MLDv2-capable Routers group
# mgid=ff12:601b::fb # IPv6 mDNS group
# mgid=ff12:601b::101 # IPv6 NTP group
# mgid=ff12:601b::202 # IPv6 Sun RPC group
# mgid=ff12:601b::1:3 # IPv6 Multicast Link Local Name Resolution group
ALL=full, ALL_SWITCHES=full;
#
#
#
VLAN0212=0x00d4, rate=7, mtu=5, scope=2, defmember=full:
ALL, ALL_SWITCHES=full;
VLAN0212=0x00d4, ipoib, rate=7, mtu=5, scope=2:
# mgid=ff12:401b::ffff:ffff # IPv4 Broadcast address
# mgid=ff12:401b::1 # IPv4 All Hosts group
# mgid=ff12:401b::2 # IPv4 All Routers group
# mgid=ff12:401b::16 # IPv4 IGMP group
# mgid=ff12:401b::fb # IPv4 mDNS group
# mgid=ff12:401b::fc # IPv4 Multicast Link Local Name Resolution group
# mgid=ff12:401b::101 # IPv4 NTP group
# mgid=ff12:401b::202 # IPv4 Sun RPC
# mgid=ff12:601b::1 # IPv6 All Hosts group
# mgid=ff12:601b::2 # IPv6 All Routers group
# mgid=ff12:601b::16 # IPv6 MLDv2-capable Routers group
# mgid=ff12:601b::fb # IPv6 mDNS group
# mgid=ff12:601b::101 # IPv6 NTP group
# mgid=ff12:601b::202 # IPv6 Sun RPC group
# mgid=ff12:601b::1:3 # IPv6 Multicast Link Local Name Resolution group
ALL=full, ALL_SWITCHES=full;
#
#
#
VLAN0213=0x00d5, rate=7, mtu=5, scope=2, defmember=full:
ALL, ALL_SWITCHES=full;
VLAN0213=0x00d5, ipoib, rate=7, mtu=5, scope=2:
# mgid=ff12:401b::ffff:ffff # IPv4 Broadcast address
# mgid=ff12:401b::1 # IPv4 All Hosts group
# mgid=ff12:401b::2 # IPv4 All Routers group
# mgid=ff12:401b::16 # IPv4 IGMP group
# mgid=ff12:401b::fb # IPv4 mDNS group
# mgid=ff12:401b::fc # IPv4 Multicast Link Local Name Resolution group
# mgid=ff12:401b::101 # IPv4 NTP group
# mgid=ff12:401b::202 # IPv4 Sun RPC
# mgid=ff12:601b::1 # IPv6 All Hosts group
# mgid=ff12:601b::2 # IPv6 All Routers group
# mgid=ff12:601b::16 # IPv6 MLDv2-capable Routers group
# mgid=ff12:601b::fb # IPv6 mDNS group
# mgid=ff12:601b::101 # IPv6 NTP group
# mgid=ff12:601b::202 # IPv6 Sun RPC group
# mgid=ff12:601b::1:3 # IPv6 Multicast Link Local Name Resolution group
ALL=full, ALL_SWITCHES=full;