Hi,
I'm working on a testing environment where I want to test the QoS functionality in Mofed.
I do think that I have a good comprehension of Basic InfiniBand QoS (SL and VL), and I try to understand Advanced QoS by defining policies (/etc/opensm/qos-policy.conf), and especially the rate-limit functionality.
I basically have 2 groups (One EDR, one FDR) and I want to restrict the bandwidth when one node of one group is communicating with a node of the other group.
Here is my qos-policy.conf :
port-groups
port-group
name: nodeFDR
port-guid: 0x0002c9030032ac41, 0x0002c9030032ac70, 0x0002c9030032acb0, 0x0002c9030032acc0, 0x0002c9030032af00
end-port-group
port-group
name: nodeEDR
port-guid: 0xe41d2d0300f29d70, 0xe41d2d0300f29eac, 0xe41d2d0300f29fb4
end-port-group
end-port-groups
qos-levels
qos-level
name: DEFAULT
use: default QoS Level
sl: 5
rate-limit: 16
end-qos-level
qos-level
name: SDR1X
mtu-limit: 4
rate-limit: 2
sl: 2
end-qos-level
qos-level
name: QDR4X
mtu-limit: 4
rate-limit: 6
sl: 4
end-qos-level
end-qos-levels
qos-match-rules
qos-match-rule
source: nodeEDR
destination: nodeFDR
qos-level-name: SDR1X
end-qos-match-rule
qos-match-rule
source: nodeFDR
destination: nodeEDR
qos-level-name: QDR4X
end-qos-match-rule
end-qos-match-rules
So far so good, opensm seems to be okay with all that. Also When I do an saquery pathrecord, everything seems to be ok (I'm using ibtool from python-rdma) :
ibtool saquery PathRecord --sgid=fe80::e41d:2d03:f2:9fb4 --dgid=fe80::2:c903:32:ac41
PathRecord dump:
service_id...............0x0000000000000000
dgid.....................fe80::2:c903:32:ac41
sgid.....................fe80::e41d:2d03:f2:9fb4
dlid.....................6
slid.....................15
RawTraffic...............0
Reserved_353.............0
FlowLabel................0
HopLimit.................0
tclass...................0x00
Reversible...............1
NumbPath.................0
pkey.....................0xffff
qos_class................0x000
sl.......................2
MTUSelector..............2
mtu......................4
RateSelector.............2
rate.....................2
PacketLifeTimeSelector...2
pkt_life.................18
preference...............0
ReversePathPKeyMemberBit.0
Reserved_466.............0
Reserved_480.............0
As expected, I don't have the same result for the reverse path :
ibtool saquery PathRecord --dgid=fe80::e41d:2d03:f2:9fb4 --sgid=fe80::2:c903:32:ac41
PathRecord dump:
service_id...............0x0000000000000000
dgid.....................fe80::e41d:2d03:f2:9fb4
sgid.....................fe80::2:c903:32:ac41
dlid.....................15
slid.....................6
RawTraffic...............0
Reserved_353.............0
FlowLabel................0
HopLimit.................0
tclass...................0x00
Reversible...............1
NumbPath.................0
pkey.....................0xffff
qos_class................0x000
sl.......................4
MTUSelector..............2
mtu......................4
RateSelector.............2
rate.....................6
PacketLifeTimeSelector...2
pkt_life.................18
preference...............0
ReversePathPKeyMemberBit.0
Reserved_466.............0
Reserved_480.............0
Starting from here, I'm running benchmarks (both ib_send_bw and ib_write_bw) but I'm not speed limited. I'm running wirespeed.
Am I missing something here ?
Thank you,
Jérôme