activate in-band IP access to grid director 4036

Hi, my setup looks like

server2 ---seriallink--- 4036(192.168.20.254/24) -----QSFPcable----- server1(192.168.20.1/24)

Result: on server1 the switch is not reachable with ping/telnet/browser/arp. Communication between other IB devices via the switch works.

4036(config-if)# ip-address show

port selection     InfiniBand

ip                 192.168.20.254

mask               255.255.255.0

broadcast          192.168.20.255

Maybe a magic step like loading ipoib on the switch is needed? Or you have other ideas?