Thanks, Sophie Naudin
1. Swich configuration is Hybrid mode and enable pfc setting seems there is no problem.
switch-5e13ce [standalone: master] (config) # show dcb priority-flow-control
PFC enabled
Priority Enabled List :3
Priority Disabled List :0 1 2 4 5 6 7
TC Lossless
--- ----------
0 N
1 Y
2 Y
3 N
Interface PFC admin PFC oper
------------ -------------- -------------
Eth1/1 On Enabled
Eth1/2 Disabled Disabled
Eth1/3 Disabled Disabled
Eth1/4 Disabled Disabled
Eth1/5 On Enabled
Eth1/6 Disabled Disabled
Eth1/7 Disabled Disabled
Eth1/8 Disabled Disabled
Eth1/9 Disabled Disabled
Eth1/10 Disabled Disabled
Eth1/11 Disabled Disabled
Eth1/12 Disabled Disabled
Eth1/13 Disabled Disabled
Eth1/14 Disabled Disabled
Eth1/15 Disabled Disabled
Eth1/16 Disabled Disabled
Eth1/17 Disabled Disabled
Eth1/18 Disabled Disabled
Eth1/19 Disabled Disabled
Eth1/20 Disabled Disabled
Eth1/21 Disabled Disabled
Eth1/22 Disabled Disabled
Eth1/23 Disabled Disabled
Eth1/24 Disabled Disabled
Eth1/25 Disabled Disabled
Eth1/26 Disabled Disabled
Eth1/27 Disabled Disabled
Eth1/28 Disabled Disabled
Eth1/29 Disabled Disabled
Eth1/30 Disabled Disabled
Eth1/31 Disabled Disabled
Eth1/32 Disabled Disabled
Eth1/33 Disabled Disabled
Eth1/34 Disabled Disabled
Eth1/35 Disabled Disabled
Eth1/36 Disabled Disabled
2. According to your instruction Check it all.
mlx4_en.conf
openibd restart
/sys/module/mlx4_en/parameters/pfcrx & pfctx
pc1
root@Atropos:~# cat /proc/net/vlan/p1p1.100
p1p1.100 VID: 100 REORDER_HDR: 1 dev->priv_flags: 1
total frames received 0
total bytes received 0
Broadcast/Multicast Rcvd 0
total frames transmitted 8
total bytes transmitted 648
Device: p1p1
INGRESS priority mappings: 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0
EGRESS priority mappings: 0:3 1:3 2:3 3:3 4:3 5:3 6:3 7:3
root@Atropos:~# ib_write_bw -R --report_gbits --port=12500 -D 10
************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
RDMA_Write BW Test
Dual-port : OFF Device : mlx4_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
CQ Moderation : 100
Mtu : 2048[B]
Link type : Ethernet
Gid index : 0
Max inline data : 0[B]
rdma_cm QPs : ON
Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
Waiting for client rdma_cm QP to connect
Please run the same command with the IB/RoCE interface IP
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x0267 PSN 0xfa896c
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:10:20
remote address: LID 0000 QPN 0x026e PSN 0x5ce508
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:10:30
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
65536 447200 0.00 39.07 0.074512
---------------------------------------------------------------------------------------
pc2
root@ubuntu:/etc/modprobe.d# cat /proc/net/vlan/p3p1.100
p3p1.100 VID: 100 REORDER_HDR: 1 dev->priv_flags: 1
total frames received 0
total bytes received 0
Broadcast/Multicast Rcvd 0
total frames transmitted 8
total bytes transmitted 648
Device: p3p1
INGRESS priority mappings: 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0
EGRESS priority mappings: 0:3 1:3 2:3 3:3 4:3 5:3 6:3 7:3
root@ubuntu:/etc/modprobe.d# ib_write_bw -R --report_gbits 192.168.10.20 --port=12500 -D 10
---------------------------------------------------------------------------------------
RDMA_Write BW Test
Dual-port : OFF Device : mlx4_1
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
TX depth : 128
CQ Moderation : 100
Mtu : 2048[B]
Link type : Ethernet
Gid index : 0
Max inline data : 0[B]
rdma_cm QPs : ON
Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x026e PSN 0x5ce508
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:10:30
remote address: LID 0000 QPN 0x0267 PSN 0xfa896c
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:10:20
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
Conflicting CPU frequency values detected: 1199.980000 != 3799.960000. CPU Frequency is not max.
65536 447200 0.00 39.07 0.074512
---------------------------------------------------------------------------------------
3.
switch-5e13ce [standalone: master] (config) # show interfaces ethernet 1/1 counters priority 3
Rx
0 packets
0 unicast packets
0 multicast packets
0 broadcast packets
0 bytes
0 pause packets
0 pause duration milliseconds
Tx
0 packets
0 unicast packets
0 multicast packets
0 broadcast packets
0 bytes
0 pause packets
switch-5e13ce [standalone: master] (config) # show interfaces ethernet 1/1 counters priority 0
Rx
176634481 packets
176634354 unicast packets
34 multicast packets
93 broadcast packets
697235467340 bytes
0 pause packets
0 pause duration milliseconds
Tx
164914925 packets
164907341 unicast packets
7583 multicast packets
1 broadcast packets
648406932028 bytes
0 pause packets
switch-5e13ce [standalone: master] (config) # show interfaces ethernet 1/5 counters priority 3
Rx
0 packets
0 unicast packets
0 multicast packets
0 broadcast packets
0 bytes
0 pause packets
0 pause duration milliseconds
Tx
0 packets
0 unicast packets
0 multicast packets
0 broadcast packets
0 bytes
0 pause packets
switch-5e13ce [standalone: master] (config) # show interfaces ethernet 1/5 counters priority 0
Rx
164907378 packets
164907341 unicast packets
36 multicast packets
1 broadcast packets
645108150520 bytes
0 pause packets
0 pause duration milliseconds
Tx
176641795 packets
176634354 unicast packets
7416 multicast packets
25 broadcast packets
700768765837 bytes
0 pause packets
switch-5e13ce [standalone: master] (config) #
Still now it is not solve. Advise, Thanks!!