Hi Seol,
1) MLNX-OS Switch Configuration:
a. Have you created a VLAN and set the switchports to hybrid (or trunk) mode?
b. Have you Enable PFC on the switch and on the interfaces?
c. Does the "show dcb priority-flow-control" reports the accurate configuration?
2) Server configuration:
a. Do you have the /etc/modprobe.d/mlx4_en.conf file with the following options:
options mlx4_en pfctx=0x08 pfcrx=0x08
b. Did you restart openibd service ? (#/etc/init.d/openidb restart)
Please also verify under /sys/module/mlx4_en/parameters/pfcrx & pfctx files that the priority is set accordingly.
c. When you run the "ib_write_bw" from the client (Aprotos), does the IP 192.168.10.10 correspond to the VLAN: p1p1.100 on server Clotho?
3) Port priority counters from the Mellanox switch (via MLNX-OS CLI):
a. Does the interface(s) on the switch report any pause packets? (check for both interfaces from the servers).
ie: switch (config) # show interfaces ethernet 1/1 counters priority 3
Thank you,
Sophie.