Hi, we have a setup with some compute nodes:
Supermicro server with "Mellanox ConnectX-3 FDR Infiniband 56Gbps Controller" onboard (https://www.supermicro.com/products/system/2U/6028/SYS-6028TP-HTFR.cfm )
They are connected to a MSX6025F-1SFR.
Installed OS is Centos7.
If we try a ibping between two hosts, we recieve the following latency:
bping -G 0x002590ffff90XXXX
Pong from compute01(none) (Lid 19): time 0.174 ms
Pong from compute01(none) (Lid 19): time 0.201 ms
Pong from compute01(none) (Lid 19): time 0.242 ms
Pong from compute01(none) (Lid 19): time 0.206 ms
Pong from compute01(none) (Lid 19): time 0.200 ms
With
mpirun -iface ib0 -hosts compute02,compute03 -n 2 -ppn 1 IMB-MPI1 pingpong
we recieve the following results:
# PingPong
#---------------------------------------------------
# Benchmarking PingPong
# #processes = 2
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 1.54 0.00
1 1000 1.57 0.61
2 1000 1.53 1.25
4 1000 1.51 2.53
8 1000 1.50 5.07
16 1000 1.47 10.41
32 1000 1.57 19.49
64 1000 1.52 40.18
128 1000 1.58 77.33
256 1000 2.28 106.92
512 1000 2.40 203.25
1024 1000 2.63 370.75
2048 1000 3.14 622.11
4096 1000 3.66 1066.28
8192 1000 4.84 1613.00
16384 1000 6.36 2456.60
32768 1000 9.20 3396.35
65536 640 14.93 4186.30
131072 320 26.39 4737.12
262144 160 48.40 5165.32
524288 80 89.42 5591.29
1048576 40 172.22 5806.37
2097152 20 336.77 5938.73
4194304 10 671.45 5957.29
How do you think about results, shouldn't be the ibping latency lower and the t[usec] round about 0.7 with FDR?
Regards,
Volker