Any experts about this?

Thank you for the answers to my questions!

One Correct:

Wrong command:

ip igmp snooping static-group 232.43.211.234 interface mlag-port-channel 1 source 192.168.1.1 192.168.1.2 192.168.1.3 192.168.1.4

Right command:

vlan 1 ip igmp snooping static-group 232.43.211.234 interface mlag-port-channel 1
Unfortunately, this does not work because the IPL interface Po11 can not be added to a static group.

mrouter problem:

When I set the mlag ports as mrouter, I do not get a multicast group, either dynamically or statically.

Here is the working configuration in a mlag:

#enable global

ip igmp snooping

#enable snooping via vlan
vlan 1 ip igmp snooping

#enable querier per vlan
vlan 1 ip igmp snooping querier

#set the igmp snooping querier ip address in Webui off to free ip 1.1.1.1
vlan 1 ip igmp snooping querier address 1.1.1.1

Hi ,

Could you please check if ib0 interface found under "ifconfig -a" ?

If not, i suggest the following:

1) invoke mst start -> mst status -> ifconfig and check again

2) Try to restart the interfaces:

- /etc/init.d/openibd restart

- opensm start or start the SM on the switch

3) If the above still not working , create interface manually :

vi /etc/sysconfig/network-scripts/ifcfg-ib0

NAME="ib0"

DEVICE="ib0"

ONBOOT=yes

BOOTPROTO=static

TYPE=Infiniband

IPADDR=<ip from the same subnet>

Thanks,

Samer

I have successfully enable SR-IOV on kvm for ConnectX-3 with KVM (InfiniBand).Speeds up to 28.6Gb/s between guest hosts by using the iperf tool,but speeds up to 14Gb/s between virtual machines.I found that although the virtual machine shows multiple queues in /proc/interrupts,only one queue is actually available.I have configured smp_affinity and disable the irqbalance service.How to enable VF multi-queue for SR-IOV on KVM?

Thanks !

vm host:

[root@host-09 ~]# cat /proc/interrupts | grep mlx4

45:        106         52         58         59         59         59         54         55   PCI-MSI-edge      mlx4-async@pci:0000:00:07.0

46:    2435659    2939965      41253      26523      49013      59796      56406      70341   PCI-MSI-edge      mlx4-1@0000:00:07.0

47:          0          0          0          0          0          0          0          0   PCI-MSI-edge      mlx4-2@0000:00:07.0

48:          0          0          0          0          0          0          0          0   PCI-MSI-edge      mlx4-3@0000:00:07.0

49:          0          0          0          0          0          0          0          0   PCI-MSI-edge      mlx4-4@0000:00:07.0

50:          0          0          0          0          0          0          0          0   PCI-MSI-edge      mlx4-5@0000:00:07.0

51:          0          0          0          0          0          0          0          0   PCI-MSI-edge      mlx4-6@0000:00:07.0

52:          0          0          0          0          0          0          0          0   PCI-MSI-edge      mlx4-7@0000:00:07.0

53:          0          0          0          0          0          0          0          0   PCI-MSI-edge      mlx4-8@0000:00:07.0

54:          0          0          0          0          0          0          0          0   PCI-MSI-edge      mlx4-9@0000:00:07.0

55:          0          0          0          0          0          0          0          0   PCI-MSI-edge      mlx4-10@0000:00:07.0

56:          0          0          0          0          0          0          0          0   PCI-MSI-edge      mlx4-11@0000:00:07.0

57:          0          0          0          0          0          0          0          0   PCI-MSI-edge      mlx4-12@0000:00:07.0

58:          0          0          0          0          0          0          0          0   PCI-MSI-edge      mlx4-13@0000:00:07.0

59:          0          0          0          0          0          0          0          0   PCI-MSI-edge      mlx4-14@0000:00:07.0

60:          0          0          0          0          0          0          0          0   PCI-MSI-edge      mlx4-15@0000:00:07.0

61:          0          0          0          0          0          0          0          0   PCI-MSI-edge      mlx4-16@0000:00:07.0

[root@host-09 ~]# cat /proc/irq/46/smp_affinity

02

[root@host-09 ~]# cat /proc/irq/47/smp_affinity

04

[root@host-09 ~]# cat /proc/irq/48/smp_affinity

08

[root@host-09 ~]# cat /proc/irq/49/smp_affinity

10

[root@host-09 ~]# cat /proc/irq/50/smp_affinity

20

[root@host-09 ~]# cat /proc/irq/51/smp_affinity

40

[root@host-09 ~]# cat /proc/irq/52/smp_affinity

80

[root@host-09 ~]# cat /proc/irq/53/smp_affinity

01

[root@host-09 ~]# cat /proc/irq/54/smp_affinity

02

[root@host-09 ~]# cat /proc/irq/55/smp_affinity

04

[root@host-09 ~]# cat /proc/irq/56/smp_affinity

08

[root@host-09 ~]# cat /proc/irq/57/smp_affinity

10

[root@host-09 ~]# cat /proc/irq/58/smp_affinity

20

[root@host-09 ~]# cat /proc/irq/59/smp_affinity

40

[root@host-09 ~]# cat /proc/irq/60/smp_affinity

80

[root@host-09 ~]# cat /proc/irq/61/smp_affinity

01

[root@host-09 ~]# ls -la /sys/class/net/ib0/queues/

total 0

drwxr-xr-x 4 root root 0 Jun 26 12:11 .

drwxr-xr-x 5 root root 0 Jun 26 12:11 ..

drwxr-xr-x 2 root root 0 Jun 26 12:11 rx-0

drwxr-xr-x 3 root root 0 Jun 26 12:11 tx-0

Hi Steve,

I'm using the latest FW and MOFED as well.

I've issued the command you advised, it returned the correct output, but priority levels are all stacked under TC0, which by default are spreaded over all the TCs.

The output:

mlnx_qos -i enp6s0f1 --pfc 0,0,0,1,0,0,0,0

DCBX mode: OS controlled

Priority trust state: pcp

Cable len: 7

PFC configuration:

priority    0   1   2   3   4   5   6   7

enabled     0   0   0   1   0   0   0   0  

tc: 0 ratelimit: unlimited, tsa: vendor

priority:  0

priority:  1

priority:  2

priority:  3

priority:  4

priority:  5

priority:  6

priority:  7

Thanks,

David

Stephen Yannalfo wrote:

> Could you get the board_id with "ibv_devinfo"

Sure; took a little time as I had to get a system running to put the HCA in ;-)

When I put some IB between systems at home years ago, the PCIe

would limit me to about DDR anyway (one 8x 1.0, and 4x 2.0 ...), so

i used DDR boards with available firmware. I have since upgraded

some bits of hardware, and wondering if I couldn't get QDR on 8x 2.0.

But it seems QDR was mostly deployed with QSFP connector, not

CX4, so the MHJH29 seems a bit of a black sheep from that era...

Thanks for your help !

Host is a Supermicro X9SRi, primary PCIe slot (16x 2.0). Running

CentOS 7.3

[root@localhost ~]# ibv_devinfo

hca_id: mlx4_0

        transport:                      InfiniBand (0)

        fw_ver:                         2.6.900

        node_guid:                      0002:c903:0002:173a

        sys_image_guid:                 0002:c903:0002:173d

        vendor_id:                      0x02c9

        vendor_part_id:                 26428

        hw_ver:                         0xA0

        board_id:                       MT_04E0120005

        phys_port_cnt:                  2

                port:   1

                        state:                  PORT_DOWN (1)

                        max_mtu:                4096 (5)

                        active_mtu:             4096 (5)

                        sm_lid:                 0

                        port_lid:               0

                        port_lmc:               0x00

                        link_layer:             InfiniBand

(second port same as first, nothing is plugged in).

2.6.9 feels old, the latest seems to be 2.9.1[000]

> And the part number with:

> > lspci | grep Mell       NOTE: the bus:dev.func of the device

> > lspci -s bus:dev.func -xxxvvv

That's one verbose lspci :-)

02:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev a0)

        Subsystem: Mellanox Technologies Device 0005

        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+

        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

        Latency: 0, Cache Line Size: 64 bytes

        Interrupt: pin A routed to IRQ 61

        NUMA node: 0

        Region 0: Memory at fba00000 (64-bit, non-prefetchable) [size=1M]

        Region 2: Memory at 38ffff000000 (64-bit, prefetchable) [size=8M]

        Capabilities: [40] Power Management version 3

                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)

                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-

        Capabilities: [48] Vital Product Data

                Product Name: Eagle QDR

                Read-only fields:

                        [PN] Part number: MHJH29-XTC          

                        [EC] Engineering changes: X5

                        [SN] Serial number: MT0821X00122           

                        [V0] Vendor specific: PCIe Gen2 x8   

                        [RV] Reserved: checksum good, 0 byte(s) reserved

                Read/write fields:

                        [V1] Vendor specific: N/A  

                        [YA] Asset tag: N/A                            

                        [RW] Read-write area: 111 byte(s) free

                End

        Capabilities: [9c] MSI-X: Enable+ Count=256 Masked-

                Vector table: BAR=0 offset=0007c000

                PBA: BAR=0 offset=0007d000

        Capabilities: [60] Express (v2) Endpoint, MSI 00

                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited

                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 75.000W

                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-

                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-

                        MaxPayload 256 bytes, MaxReadReq 512 bytes

                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

                LnkCap: Port #8, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited

                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-

                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-

                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

                LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-

                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported

                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-

                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-

                         Compliance De-emphasis: -6dB

                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-

                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-

        Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)

                ARICap: MFVC- ACS-, Next Function: 1

                ARICtl: MFVC- ACS-, Function Group: 0

        Kernel driver in use: mlx4_core

        Kernel modules: mlx4_core

00: b3 15 3c 67 06 04 10 00 a0 00 06 0c 10 00 00 00

10: 04 00 a0 fb 00 00 00 00 0c 00 00 ff ff 38 00 00

20: 00 00 00 00 00 00 00 00 00 00 00 00 b3 15 05 00

30: 00 00 00 00 40 00 00 00 00 00 00 00 0b 01 00 00

40: 01 48 03 00 00 00 00 00 03 9c ff 7f 00 00 00 78

50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

60: 10 00 02 00 01 8e 2c 01 20 20 00 00 82 f4 03 08

70: 00 00 82 00 00 00 00 00 00 00 00 00 00 00 00 00

80: 00 00 00 00 1f 00 00 00 00 00 00 00 00 00 00 00

90: 02 00 00 00 00 00 00 00 00 00 00 00 11 60 ff 80

a0: 00 c0 07 00 00 d0 07 00 05 00 8a 00 00 00 00 00

b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Hi Marc,

What is the best way to run Contrail vRouter on HCI environment where kernel will use same bonded ConnectX-5 EN ports with vRouter. I want to achieve maximum possible offload of ConnectX-5? SR-IOV, DPDK, SR-IOV + DPDK or else? If related to SR-IOV, should I use it with PF or VF?

Best regards,

Hi,

I have a problem with local invalid/send with invalid operation with ConnectX-4 NICs.

[863318.002031] mlx5_0:dump_cqe:275:(pid 31419): dump error cqe

[863318.002032] 00000000 00000000 00000000 00000000

[863318.002033] 00000000 00000000 00000000 00000000

[863318.002034] 00000000 00000000 00000000 00000000

[863318.002035] 00000000 09007806 25000178 000006d2

ofed version:

MLNX_OFED_LINUX-4.1-1.0.2.0 (OFED-4.1-1.0.2)

firmware version:

Querying Mellanox devices firmware ...

Device #1:

----------

  Device Type:      ConnectX4LX

  Part Number:      MCX4121A-ACA_Ax

  Description:      ConnectX-4 Lx EN network interface card; 25GbE dual-port SFP28; PCIe3.0 x8; ROHS R6

  PSID:             MT_2420110034

  PCI Device Name:  0000:81:00.0

  Base MAC:         0000248a07b37aa2

  Versions:         Current        Available    

     FW             14.20.1010     N/A          

     PXE            3.5.0210       N/A          

  Status:           No matching image found

I used the tools I developed, and it works in other device like cx3, qlogic. I'v checked the input parameter and make sure it's right. Please share me any suggestions about how to fix it.

Thanks

There is another script -

./mlnxofedinstall

that should be run in order to install drivers, not ./install.sh

Also there are some other issues (potentially removing mellanox-mlnxen.x86_64 and fixing pif-scan or pif-introduce not working) - you should probably check here for solution:

https://discussions.citrix.com/topic/383014-xenserver-70-mellanox-connectx-3-nic-infiniband/

Greetings,

Emil

Does your custom essay writing service really can help with Linux command or mellanox utility?

Hello David -

   I hope all is well...

Could you open a case with Mellanox Support so we can take a deeper look at this issue?

Thank you -

Steve

Hello David -

I hope all is well...

Please try the following :

# mlnx_qos -i eth2 -p 0,1,2,3,4,5,6,7

-p --prio_tc

This parameter is used to map priorities to Egress Traffic Class (ETC).

Note: By default, priority 0 is mapped to tc1, and priority 1 is mapped to tc0. All other priorities are mapped to the same TC.

Please let me know you results...

Thanks - steve

Hi Steve,

the default output is shown above, except the PFC setting from your previous post.

DCBX mode: OS controlled

Priority trust state: pcp

Cable len: 7

PFC configuration:

priority    0   1   2   3   4   5   6   7

enabled     0   0   0   1   0   0   0   0  

tc: 0 ratelimit: unlimited, tsa: vendor

priority:  0

priority:  1

priority:  2

priority:  3

priority:  4

priority:  5

priority:  6

priority:  7

And yes, when I try to set prio values to ETCs I get the errors.

The output with -p or with --prio_tc is listed above

sudo mlnx_qos -i enp6s0f1 -p 0,1,2,3,4,5,6,7

Netlink error: Bad value. see dmesg.

Thank you, David

Hi David,

Could you shed the light on what you are trying to achieve? Why you are configuring SR-IOV? Are you going to use VLAN? Are you planning to run VM on it? and then RDMA traffic over it? Understanding whole picture of the final result will be extremely helpful.

Hi,

yes I'm going to use VLAN, we have a smaller network of containers and we assign SR-IOV VFs to each of them. At the moment we are using 2 servers connected back-to-back. In the containers we are running bandwidth hungry and latency sensitive applications. By setting QoS rules we would like to reduce the message passing latencies of the latency sensitive traffic. Latency values are increasing due to the presence of the bandwidth hungry traffic. We need to use VLAN because of the value of the PCP header, which differentiates the types of traffic. However running 8 containers per server is not a real world scenario I think.

Note: There was a previous question which seemed to be solved. That was about how we cannot assign priority values to TCs of SR-IOV VFs -- furthermore VFs are completely not configurable by the mlnx_qos tool -- but according to our measurements the priority settings of the PF are inherited by the VFs, so it works out.

Thanks,

David

It seems that the RedHat has changed the naming conventions of some RPM packages. In the file "centos-release-7-5.*aarch64" rpm packages, the letter "a" is used instead of the full name "alternate".

So in the script, the checking will fail and refuse to install the package.

#    } elsif ($dist_rpm =~ /redhat-release-.*-7.5.*el7a|sl-release-7.5.*el7a|centos-release-7-5.*el7.centos.altarch/) {

    } elsif ($dist_rpm =~ /redhat-release-.*-7.5.*el7a|sl-release-7.5.*el7a|centos-release-7-5.*el7.centos.a*/) {

I tried this WA. Would you please give some official solution for this?

Thanks

Hello Romain -

I hope all is well...

    Do you have an email address so I can send you a link to the file?

thanks - steve

Thank you. I sent you a direct message with my e-mail.

BTW, naive question - any reason the link can't simply be made public?

(my guess would be, this is somehow a prerelease HCA and there was never any official release for it?)

Cordially & thanks for the great support,

Romain

Hello Community,

I have 2 Mellanox CX516A in same x86 host. I successfully use a Finisar QSFP28 100G LR  module plugged into each NIC and back-to-back cabled. Means NIC 1 port 0 goes to NIC 2 port 0, this is working fine.

I tried to place both QSFP28 modules into the same NIC and use back-to-back cabling. After I plugged in the second QSFP28 into port 1 I got the following error message:

[419335.854822] mlx5_core 0000:88:00.1: Port module event[error]: module 1, Cable error, Power budget exceeded

also the orange led is flashing.

My host running Ubuntu 18.04 with "MLNX_OFED_LINUX-4.3-3.0.2.1-ubuntu18.04-x86_64" driver version.

Do you have any hints how to bringup both QSFP28 in the same NIC ?

Hello Romain -

Service for this Hardware and FW has ended. So it will not be hosted on our site.

Many thanks for your support & patience...

Steve