Hi,

First of all, I'm glad to hear you are using mlnx_tune and I hope you find it at least a little bit helpful.

For the issue above, I agree the behavior isn't perfect but I would like to explain a few points:

1. For most (if not all) OS, it is next to impossible to query for the I/O non-posted prefetch value. This is why I choose to prompt the user of a possible issue (note the 'might' in the message).
2. The issue with I/O non-posted prefetch is only relevant when trying to run traffic at rates higher than 150Gb/s (bidirectional). This is only relevant for EDR, FDR (dualport over x16 PCI), 100GbE and 40/50/56GbE (dualport over x16 PCI). This is why the warning only shows for x16 devices. It is simply impossible to get high enough rate with shorter PCI width.

I hope this answers your concern. If not, I would be happy to hear your feedback and improve mlnx_tune.

Hi Tal,

Thanks for your response, from the author of mlnx_tune too!  Great to e-meet you!

Regarding your point 1: For most (if not all) OS, it is next to impossible to query for the I/O non-posted prefetch value. May I make a suggestion?  Given Mellanox's industry-wide weight, and the fact there are only two major COTS BIOS makers: AMI and Phoenix, both publish Windows/Linux based BIOS review tools, so perhaps calling such tools is a way to do more reliable queries for BIOS values, including the I/O no-posted prefetch value?

Regarding your point 2: The issue with I/O non-posted prefetch is only relevant when trying to run traffic at rates higher than 150Gb/s (bidirectional).  Indeed, we are using four EDR IB HCAs, with SB7700 IB switch.  We are pushing for more than 100Gbps traffic so we are in the territory that you mentioned

Finally, one suggestion: the -p comes with the following options, but AFAICT, the only Mellanox doc that explains somewhat is this HowTo Tune Your Linux Server for Best Performance using mlnx_tune Tool  published by the prolific Ophir in Oct 22, 2014.  IMHO, there is a lack of actual application examples for each profile.  An excellent one for illustration would be again Ophir's HowTo Configure and Test BeeGFS with RDMA.

                        ['HIGH_THROUGHPUT',

                        'IP_FORWARDING_MULTI_STREAM_THROUGHPUT',

                        'IP_FORWARDING_MULTI_STREAM_PACKET_RATE',

                        'IP_FORWARDING_SINGLE_STREAM',

                        'IP_FORWARDING_SINGLE_STREAM_0_LOSS',

                        'IP_FORWARDING_SINGLE_STREAM_SINGLE_PORT']

Could you please talk to him about adding this piece of info?  Since the naming of these profiles all start with IP, which means they apply to Ethernet mostly (IB traffic doesn't use kernel networking stack!).  So, perhaps somewhere there should be more explinations regarding

• Is this tuning tool really usable to IB fabric?  Ophir said in his 2014 post it is. My review of the Python script didn't convince me so.
• What if all my NIC interfaces use jumbo frames?  If so, what is the applicability of IP_FORWARDING_MULTI_STREAM_THROUGHPUT and IP_FORWARDING_MULTI_STREAM_PACKET_RATE?

I look forward to hearing your comments about the above.

Best,

Chin

#1 - constantly in progress. As with any other relevant parameter, I'll add it to mlnx_tune once there's a solution.

Regarding the documentation, next mlnx_tune version should be out by EOM September. I will update the mlnx_tune entry with recent changes and will expand upon the existing profiles.

Until than, mlnx_tune as a static system analyzer is good for both Ethernet and IB. The tuning part is currently only good for Ethernet, mostly because IB has excellent out of box performance.

Again, thanks for your feedback. It is highly appreciated.

Hi Tal,

Look forward to the next mlnx_tune.  Thanks for confirming my code-review conclusion.  Please talk to Ophir and encourage him to supplement his very useful HowTo Configure and Test BeeGFS with RDMA  the use of mlnx_tune. Given the popularity of Mellanox IB products in the HPC space, the addition will be valuable.

Best,

Chin

Hi Tal,

A quick supplement to #1, well-known server vendors may provide their own utilities for set/view BIOS values. Eg. Dell's OpenManage Deployment Kit.  Given many such large vendors work closely with Mellanox, that's another option for you and your users?

Best,

Chin

We are aware of such tools, but also would like mlnx_tune to be as independent as possible. I will consider checking if these exist on the system and use them if they are.

We are successfully using nvidia_peer_memory version 1.0 on a system running Scientific Linux 6.8 with the patch pasted below. The build problems you experienced originate from the fact that the compilation takes the printk header from the Mellanox OFED compat layer without having the compat/config.h macros defined.

Best regards,

Dorian

From f653387ae914271e9fd639a00af1c02daf8560e3 Mon Sep 17 00:00:00 2001

From: []

Date: Tue, 23 Aug 2016 13:58:56 +0200

Subject: [PATCH 1/2] Update Makefile

Take the defines from compat/config.h and define them on the

command line. This fixes a build problem with the latest RHEL6

kernels.

---

Makefile | 5 +++--

1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/Makefile b/Makefile

index c2d6a29..929a296 100644

--- a/Makefile

+++ b/Makefile

@@ -1,8 +1,9 @@

obj-m += nv_peer_mem.o

-OFA_KERNEL=$(shell (test -d /usr/src/ofa_kernel/default && echo /usr/src/ofa_kernel/default) || (test -d /var/lib/dkms/mlnx-ofed-kernel/ && ls -d /var/lib/dkms/mlnx-ofed-kernel/*/build))

+OFA_KERNEL = /usr/src/ofa_kernel/default

+DEFINES    = $(shell /bin/cat $(OFA_KERNEL)/compat/config.h | grep '\#define' | sed 's/\#define /-D/g' | sed 's/ /=/g' | tr '\n' ' ')

-EXTRA_CFLAGS +=-I$(OFA_KERNEL)/include/ -I$(OFA_KERNEL)/include/rdma

+EXTRA_CFLAGS +=-I$(OFA_KERNEL)/include/ -I$(OFA_KERNEL)/include/rdma $(DEFINES)

PWD  := $(shell pwd)

KVER := $(shell uname -r)

MODULES_DIR := /lib/modules/$(KVER)

--

2.7.4

Hi Tal,

My team and I were discussing which profile to use to tune for our usage scenarios; a team member asked me "why don't we just try different profiles and then test it out?"  I replied "well, based on my code review, I didn't see a way to restore the pre-tune system configuration. So, we may not wish to do that". 

Can an "restore" action be added to mlnx_tune?

Best,

Chin

I have an SR-IOV enabled installation of Openstack Kilo on RHEL 7.1 compute nodes.

I followed this document  (Mellanox-Neutron-Kilo-Redhat-InfiniBand - OpenStack) and it seemed to work partially.

I can see ib0 attached to the VM (I log in from console and run command "lspci" and "ip link")

But ib0 doesn't link up.

I  can make it work by using libvirt(not nova) directly,so I think this is a problem of Nova, eswitchd or neutron-mlnx-agent.

Error logs are here.

・On computer node

1. dmesg of VM

the message "ib0: multicast join failed for ff12:401b:8000:0000:0000:0000:ffff:ffff, status -22"  appears many times  after this.

I find it strange that  "Bringing up interface ib0:  [  OK  ]"  displays without " ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready"

mlx4_core: Mellanox ConnectX core driver v3.1-1.0.3 (29 Sep 2015)

mlx4_core: Initializing 0000:00:05.0

mlx4_core 0000:00:05.0: Detected virtual function - running in slave mode

mlx4_core 0000:00:05.0: Sending reset

mlx4_core 0000:00:05.0: Sending vhcr0

mlx4_core 0000:00:05.0: Requested number of MACs is too much for port 1, reducing to 64

mlx4_core 0000:00:05.0: HCA minimum page size:512

mlx4_core 0000:00:05.0: Timestamping is not supported in slave mode

mlx4_core: device is working in RoCE mode: Roce V1

mlx4_core: gid_type 1 for UD QPs is not supported by the devicegid_type 0 was chosen instead

mlx4_core: UD QP Gid type is: V1

NET: Registered protocol family 10

lo: Disabled Privacy Extensions

<mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v3.1-1.0.3 (29 Sep 2015)

<mlx4_ib> check_flow_steering_support: Device managed flow steering is unavailable for IB port in multifunction env.

mlx4_core 0000:00:05.0: mlx4_ib_add: allocated counter index 6 for port 1

mlx4_core 0000:00:05.0: mlx4_ib_add: allocated counter index 7 for port 2

microcode: CPU0 sig=0x206a1, pf=0x1, revision=0x1

platform microcode: firmware: requesting intel-ucode/06-2a-01

microcode: CPU1 sig=0x206a1, pf=0x1, revision=0x1

platform microcode: firmware: requesting intel-ucode/06-2a-01

Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba

[  OK  ]

mlx4_core 0000:00:05.0: mlx4_ib: multi-function enabled

mlx4_core 0000:00:05.0: mlx4_ib: operating in qp1 tunnel mode

knem 1.1.2.90mlnx: initialized

Setting hostname cbv-lsf4.novalocal:  [  OK  ]

Setting up Logical Volume Management:   7 logical volume(s) in volume group "rootvg" now active

[  OK  ]

Checking filesystems

Checking all file systems.

[/sbin/fsck.ext4 (1) -- /] fsck.ext4 -a /dev/mapper/rootvg-lv_root

/dev/mapper/rootvg-lv_root: clean, 134671/4915200 files, 2194674/19660800 blocks

Entering non-interactive startup

Calling the system activity data collector (sadc)...

Starting monitoring for VG rootvg:   7 logical volume(s) in volume group "rootvg" monitored

[  OK  ]

pps_core: LinuxPPS API ver. 1 registered

pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>

PTP clock support registered

mlx4_en: Mellanox ConnectX HCA Ethernet driver v3.1-1.0.3 (29 Sep 2015)

card: mlx4_0, QP: 0xa78, inline size: 120

Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16

card: mlx4_0, QP: 0xa80, inline size: 120

Default coalesing params for mtu:4092 - rx_frames:88 rx_usecs:16

Loading HCA driver and Access Layer:[  OK  ]

NOHZ: local_softirq_pending 08

ADDRCONF(NETDEV_UP): ib0: link is not ready

ib0: multicast join failed for ff12:401b:8000:0000:0000:0000:ffff:ffff, status -22

ip6tables: No config file.[WARNING]

Bringing up loopback interface:  [  OK  ]

Bringing up interface eth0:

Determining IP information for eth0...ib0: multicast join failed for ff12:401b:8000:0000:0000:0000:ffff:ffff, status -22

done.

[  OK  ]

Bringing up interface ib0:  [  OK  ]

2. /var/log/neutron/eswitchd

2016-08-25 13:21:35,989 DEBUG eswitchd [-] Handling message - {u'action': u'get_vnics', u'fabric': u'*'}

2016-08-25 13:21:35,989 DEBUG eswitchd [-] fabrics =['default']

2016-08-25 13:21:35,989 DEBUG eswitchd [-] vnics are {u'fa:16:3e:7d:d7:87': {'mac': u'fa:16:3e:7d:d7:87', 'device_id': u'afab526e-da36-44ee-8f5e-8743451bc8a4'}, 'fa:16:3e:d8:dd:a3': {'mac': 'fa:16:3e:d8:dd:a3', 'device_id': '7c5f4c1a-1492-4087-8eee-c54b91cc733b'}, '1a:5c:90:77:4f:88': {'mac': '1a:5c:90:77:4f:88', 'device_id': '0e3e7d62-b88f-4e9b-b685-116280c87f5a'}, u'fa:16:3e:4f:46:de': {'mac': u'fa:16:3e:4f:46:de', 'device_id': u'7b7e8f69-438c-4ec7-95fe-0d59f939fd19'}, 'fe:66:d7:3e:cb:ca': {'mac': 'fe:66:d7:3e:cb:ca', 'device_id': '7d4a002a-cfab-4189-9bff-b656c863592a'}}

2016-08-25 13:21:37,989 DEBUG eswitchd [-] Handling message - {u'action': u'get_vnics', u'fabric': u'*'}

2016-08-25 13:21:37,989 DEBUG eswitchd [-] fabrics =['default']

2016-08-25 13:21:37,990 DEBUG eswitchd [-] vnics are {u'fa:16:3e:7d:d7:87': {'mac': u'fa:16:3e:7d:d7:87', 'device_id': u'afab526e-da36-44ee-8f5e-8743451bc8a4'}, 'fa:16:3e:d8:dd:a3': {'mac': 'fa:16:3e:d8:dd:a3', 'device_id': '7c5f4c1a-1492-4087-8eee-c54b91cc733b'}, '1a:5c:90:77:4f:88': {'mac': '1a:5c:90:77:4f:88', 'device_id': '0e3e7d62-b88f-4e9b-b685-116280c87f5a'}, u'fa:16:3e:4f:46:de': {'mac': u'fa:16:3e:4f:46:de', 'device_id': u'7b7e8f69-438c-4ec7-95fe-0d59f939fd19'}, 'fe:66:d7:3e:cb:ca': {'mac': 'fe:66:d7:3e:cb:ca', 'device_id': '7d4a002a-cfab-4189-9bff-b656c863592a'}}

3. /var/log/neutron/mlnx-agent.log

2016-08-25 13:20:16.230 8881 DEBUG oslo_messaging._drivers.amqp [-] UNIQUE_ID is b3adf08a1ac24b8d83eee8f48f0e47aa. _add_unique_id /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py:264

2016-08-25 13:20:17.973 8881 DEBUG networking_mlnx.plugins.ml2.drivers.mlnx.agent.utils [req-e22856b5-392a-4794-ac1e-b34fdf0eb9e1 ] get_attached_vnics get_attached_vnics /usr/lib/python2.7/site-packages/networking_mlnx/plugins/ml2/drivers/mlnx/agent/utils.py:82

2016-08-25 13:20:19.974 8881 DEBUG networking_mlnx.plugins.ml2.drivers.mlnx.agent.utils [req-e22856b5-392a-4794-ac1e-b34fdf0eb9e1 ] get_attached_vnics get_attached_vnics /usr/lib/python2.7/site-packages/networking_mlnx/plugins/ml2/drivers/mlnx/agent/utils.py:82

2016-08-25 13:20:21.974 8881 DEBUG networking_mlnx.plugins.ml2.drivers.mlnx.agent.utils [req-e22856b5-392a-4794-ac1e-b34fdf0eb9e1 ] get_attached_vnics get_attached_vnics /usr/lib/python2.7/site-packages/networking_mlnx/plugins/ml2/drivers/mlnx/agent/utils.py:82

I think this is a similar issue to this post,but I can't find what to do the next.

Mellanox eSwitchd issue on Openstack Havana nova-compute

Could you tell me any ideas?

Thank you! This worked for me as well.

Additionally, I had to define the DEPMOD variable with 'DEPMOD=$(shell which depmod)' for the install target.

Hi David,

We dont manufacture 40GbE loopback cables and we don't have any recommendation for any of these cables.

Thnaks

Khwaja

Hi Muneyoshi,

I see that you are using an EOL version of Openstack. Is it possible for you to reproduce with the "Liberty" or "Mitaka" versions?

Thanks and regards,

~Martijn

Hi Martijn,

Unfortunately I can't upgrade a version of OpenStack because of  restrictions for middleware associated with OpenStack.

Is there any way to update eswitchd for Kilo version? I find that eswitchd is updated in Liberty(or later) release, but not for Kilo.

Hi Muneyoshi,

When you followed the installation Wiki document, did you also configured the SubnetManager through the "Manual OpenSM Configuration" section? Can you also share which HCA you are using?

Thanks and regards,

~Martijn

Correct. It is only MS-MPI and not Platform-MPI that that Mellanox WinOF supports since v2.1 and the reason is stemming from Microsoft compatibility requirements.

Hi Martijn,

>When you followed the installation Wiki document, did you also configured the SubnetManager through the "Manual OpenSM Configuration" section?

Yes, I copied the document setting like this. And I restarted the opensmd.

[root@xxxx neutron]# cat /etc/opensm/partitions.conf

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;

vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

vlan2=0x2, ipoib, sl=0, defmember=full : ALL;

vlan3=0x3, ipoib, sl=0, defmember=full : ALL;

vlan4=0x4, ipoib, sl=0, defmember=full : ALL;

vlan5=0x5, ipoib, sl=0, defmember=full : ALL;

vlan6=0x6, ipoib, sl=0, defmember=full : ALL;

[root@xxxx neutron]# cat /etc/opensm/opensm.conf

allow_both_pkeys TRUE

------------------------------------------------------------------

The number of vlan is the same as the number of sriov.

>Can you also share which HCA you are using?

Sure. Here are the results for ibstat and other hardware information commands.

----------------------------------------------

[root@xxxx neutron]# ibstat

CA 'mlx4_0'

        CA type: MT4103

        Number of ports: 2

        Firmware version: 2.34.5000

        Hardware version: 0

        Node GUID: 0x7cfe9003009b8ae0

        System image GUID: 0x7cfe9003009b8ae3

        Port 1:

                State: Active

                Physical state: LinkUp

                Rate: 56

                Base lid: 63

                LMC: 0

                SM lid: 24

                Capability mask: 0x02514868

                Port GUID: 0x7cfe9003009b8ae1

                Link layer: InfiniBand

        Port 2:

                State: Down

                Physical state: Polling

                Rate: 10

                Base lid: 0

                LMC: 0

                SM lid: 0

                Capability mask: 0x02514868

                Port GUID: 0x7cfe9003009b8ae2

                Link layer: InfiniBand

[root@xxxx neutron]# mst status

MST modules:

------------

    MST PCI module loaded

    MST PCI configuration module loaded

MST devices:

------------

/dev/mst/mt4103_pciconf0         - PCI configuration cycles access.

                                   domain:bus:dev.fn=0000:06:00.0 addr.reg=88 data.reg=92

                                   Chip revision is: 00

/dev/mst/mt4103_pci_cr0          - PCI direct access.

                                   domain:bus:dev.fn=0000:06:00.0 bar=0x90700000 size=0x100000

                                   Chip revision is: 00

[root@xxxx neutron]# flint -d /dev/mst/mt4103_pci_cr0 query

Image type:          FS2

FW Version:          2.34.5000

FW Release Date:     28.7.2015

Product Version:     02.34.50.00

Rom Info:            type=PXE version=3.4.521 devid=4103

                     type=UEFI version=14.7.24

Device ID:           4103

Description:         Node             Port1            Port2            Sys image

GUIDs:               7cfe9003009b8ae0 7cfe9003009b8ae1 7cfe9003009b8ae2 7cfe9003009b8ae3

MACs:                                     7cfe909b8ae1     7cfe909b8ae2

VSD:

PSID:                IBM2000110021

[root@xxxx neutron]# lspci |grep Mell

06:00.0 Network controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]

06:00.1 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]

06:00.2 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]

06:00.3 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]

06:00.4 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]

06:00.5 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]

06:00.6 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]

--------------------------------

It might be a firmware problem(not still up-to-date), but I need a conviction for updating a firmware because the firmware version is fixed by a rule of our project..Though I can make an exception if I have an understandable reason.

Thank you and best regards,

Muneyoshi

Check the MOFED release notes, "Table 11", item 3 may be related to your issue

http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_Release_Notes_3_3-1_0_4_0.pdf

I you have configured mlx4_en.conf file properly as suggested below then you should be able to use pfc properly

+++++++++++++++++++++++++++++++

/etc/modprobe.d/mlx4_en.conf:

options mlx4_en pfctx=0x08 pfcrx=0x08

++++++++++++++++++++++++++++++++++++

try also to restart the driver after configuration by using /etc/init.d/opeibd restart

Rather then this, if loging into the target lun still fails, you should be re-checking if iser parms on bith initiator & target are configured as stated in Mellanox community

 

Hi Muneyoshi,

Can you provide us the full eswitchd.log?

Thanks and regards,

~Martijn

This environment is VMware ESXi, not a Linux one.

mlx4_en parameter must need integer value, not a 0x08!