recommended tools to keep an HCA/NIC's firmware updated:
- Whilst installing the latest GA-released MLNX_OFED from our web site http://www.mellanox.com - you will have the option to upgrade the Mellanox adapter's to the latest suitable firmware as well
- Mellanox Firmware tools - MFT & mlxup are most common and recommanded to update and query adapter firmware
http://www.mellanox.com/page/mlxup_firmware_tool?ssn=lr6f81efuoijsfg39mko1hhi34
Hello,
I want to use SR-IOV function.
so I configure it as the "HowTo Configure SR-IOV for ConnectX-3 with KVM (InfiniBand) " instructed.
But when I run the command " mlxconfig -d /dev/mst/mt4099_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=4". There is some error happened and I can't find any information about it.
Here is the output:
Device #1:
----------
Device type: ConnectX3
PCI device: /dev/mst/mt4099_pciconf0
Configurations: Current New
SRIOV_EN False(0) True(1)
NUM_OF_VFS 8 4
Apply new Configuration? ? (y/n) [n] : y
Applying... Failed!
-E- Failed to set configuration: failed to set SRIOV params: Message receipt ack
I use 12 computers, at first there is two computers that could execute the command correctly. But there is ten computers output that the firmware is unsupported for the command
"mlxconfig -d /dev/mst/mt4099_pciconf0 q", so I update the firmware of all the computer as requested. After that, all the computer could execute the command "mlxconfig -d /dev/mst/mt4099_pciconf0 q". But even the first two computers that could execute the command " mlxconfig -d /dev/mst/mt4099_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=4" well could no longer running the command any more. It's very strange. I guess something was changed by the firmware update. Any help would be appreciate!!
Does the included ESXi 6.5 have RDMA support for the hypervisor back to storage? or is it just for a guest. I really am only using SRP at a HOST to Storage level, nothing inside of the vm is infiniband aware.
Mellanox, if you arent going to release SRP drivers again, would you consider releasing the source for the old ones and we will take it from here..
Probably I deleted tjis log but below is different log with for the same error:
Logs dir: /tmp/MLNX_OFED_LINUX-3.4-2.0.0.0-3.10.0-514.2.2.rt56.424.el7.x86_64/OFED.29661.logs
[32m
Below is the list of OFED packages that you have chosen
(some may have been added by the installer due to package dependencies):
[0m
ofed-scripts
mlnx-ofa_kernel
mlnx-ofa_kernel-modules
mlnx-ofa_kernel-devel
knem
kernel-mft
iser
srp
isert
mlnx-sdp
mlnx-rds
Build ofed-scripts 3.4 RPM
Running rpmbuild --rebuild --define '_topdir /tmp/MLNX_OFED_LINUX-3.4-2.0.0.0-3.10.0-514.2.2.rt56.424.el7.x86_64/mlnx_iso.29365/OFED_topdir' --define '_sourcedir %{_topdir}/SOURCES' --define '_specdir %{_topdir}/SPECS' --define '_srcrpmdir %{_topdir}/SRPMS' --define '_rpmdir %{_topdir}/RPMS' --define 'dist %{nil}' --target x86_64 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /tmp/MLNX_OFED_LINUX-3.4-2.0.0.0-3.10.0-514.2.2.rt56.424.el7.x86_64/mlnx_iso.29365/MLNX_OFED_SRC-3.4-2.0.0.0/SRPMS/ofed-scripts-3.4-OFED.3.4.2.0.0.src.rpm
Build mlnx-ofa_kernel 3.4 RPM
Running rpmbuild --rebuild --define '_topdir /tmp/MLNX_OFED_LINUX-3.4-2.0.0.0-3.10.0-514.2.2.rt56.424.el7.x86_64/mlnx_iso.29365/OFED_topdir' --define '_sourcedir %{_topdir}/SOURCES' --define '_specdir %{_topdir}/SPECS' --define '_srcrpmdir %{_topdir}/SRPMS' --define '_rpmdir %{_topdir}/RPMS' --nodeps --define '_dist .rhel7u3' --define 'configure_options --with-core-mod --with-user_mad-mod --with-user_access-mod --with-addr_trans-mod --with-mthca-mod --with-mlx4-mod --with-mlx4_en-mod --with-mlx4_vnic-mod --with-mlx5-mod --with-cxgb3-mod --with-cxgb4-mod --with-nes-mod --with-ehca-mod --with-qib-mod --with-ipoib-mod --with-ipath_inf-mod --with-amso1100-mod --with-ocrdma-mod --with-sdp-mod --with-srp-mod --with-rds-mod --with-iser-mod --with-e_ipoib-mod --with-9pnet_rdma-mod --with-9p-mod --with-cxgb3i-mod --with-cxgb4i-mod --with-isert-mod' --define 'KVERSION 3.10.0-514.2.2.rt56.424.el7.x86_64' --define 'K_SRC /lib/modules/3.10.0-514.2.2.rt56.424.el7.x86_64/build' --define '_prefix /usr' /tmp/MLNX_OFED_LINUX-3.4-2.0.0.0-3.10.0-514.2.2.rt56.424.el7.x86_64/mlnx_iso.29365/MLNX_OFED_SRC-3.4-2.0.0.0/SRPMS/mlnx-ofa_kernel-3.4-OFED.3.4.2.0.0.1.g30039f7.src.rpm
[31mFailed to build mlnx-ofa_kernel 3.4 RPM [0m
Collecting debug info...
[31mSee /tmp/MLNX_OFED_LINUX-3.4-2.0.0.0-3.10.0-514.2.2.rt56.424.el7.x86_64/OFED.29661.logs/mlnx-ofa_kernel-3.4.rpmbuild.log [0m
Hello,
I want to use SR-IOV function.
so I configure it as the "HowTo Configure SR-IOV for ConnectX-3 with KVM (InfiniBand) " instructed.
But when I run the command " mlxconfig -d /dev/mst/mt4099_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=4". There is some error happened and I can't find any information about it.
Here is the output:
Device #1:
----------
Device type: ConnectX3
PCI device: /dev/mst/mt4099_pciconf0
Configurations: Current New
SRIOV_EN False(0) True(1)
NUM_OF_VFS 8 4
Apply new Configuration? ? (y/n) [n] : y
Applying... Failed!
-E- Failed to set configuration: failed to set SRIOV params: Message receipt ack
I use 12 computers, at first there is two computers that could execute the command correctly. But there is ten computers output that the firmware is unsupported for the command
"mlxconfig -d /dev/mst/mt4099_pciconf0 q", so I update the firmware of all the computer as requested. After that, all the computer could execute the command "mlxconfig -d /dev/mst/mt4099_pciconf0 q". But even the first two computers that could execute the command " mlxconfig -d /dev/mst/mt4099_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=4" well could no longer running the command any more. It's very strange. I guess something was changed by the firmware update. Any help would be appreciate!!
Thanks for your reply. So, my own understanding is in-line with yours.
Regards,
Chin
Hi Oskar,
We need the contents of /tmp/MLNX_OFED_LINUX-3.4-2.0.0.0-3.10.0-514.2.2.rt56.424.el7.x86_64/OFED.29661.logs/mlnx-ofa_kernel-3.4.rpmbuild.log
Thanks and regards,
~Martijn
Hi Oskar,
We need the contents of /tmp/MLNX_OFED_LINUX-3.4-2.0.0.0-3.10.0-514.2.2.rt56.424.el7.x86_64/OFED.29661.logs/mlnx-ofa_kernel-3.4.rpmbuild.log
Can you please share that log file.
Thanks and regards,
~Martijn
can you give me your email. File is too big to paste here.
EAL: Detected 2 lcore(s)
EAL: Probing VFIO support...
EAL: PCI device 0000:06:00.0 on NUMA socket -1
EAL: probe driver: 15b3:1015 net_mlx5
PMD: mlx5.c:428: mlx5_pci_probe(): cannot access device, is mlx5_ib loaded? [ Yes it is loaded ]
EAL: PCI device 0000:06:00.1 on NUMA socket -1
EAL: probe driver: 15b3:1015 net_mlx5
PMD: mlx5.c:428: mlx5_pci_probe(): cannot access device, is mlx5_ib loaded?
EAL: No probed ethernet devices
PANIC in main():
Empty set of forwarding logical cores - check the core mask supplied in the command parameters
Section 4.3 of this article suggests that in-box ESXi drivers are RDMA-aware, which you can also conclude from the very fact one of the modules related to these drivers is called nlmx4(5)_rdma. That doesn't mean though that RDMA is actually going to be effectively used for host-to-storage access. For iSCSI, that would require implementing iSER layer, which the in-box driver clearly doesn't have (after all, it's explicitly called "Software iSCSI Adapter" and it's like 100 years old). And, if VMware NFS could utilise RDMA, I'm pretty sure they'd already told us about it.
So, someone has to do some work here, either Mellanox or VMware or both. But they're not eager to tell us anything, as you can clearly see from 1/2 year lack of response to this thread...
The patches provided by RH from link http://people.redhat.com/~alwillia/bz1299846/is solving the issue.
I have downloaded and installed (yum install *.rpm) the 3 user space packages (qemu-img, qemu-kvm and qemu-kvm-common) on the hypervisor.
The performance could be enhanced by as much as 90% and 65% in the case of 1KB and 4KB message size respectively.
qemu-img-1.5.3-105.el7_2.1.bz1299846.0.x86_64.rpm
qemu-kvm-1.5.3-105.el7_2.1.bz1299846.0.x86_64.rpm
qemu-kvm-common-1.5.3-105.el7_2.1.bz1299846.0.x86_64.rpm
Performance Known Issues#783496: When using a VF over RH7.X KVM, low throughput is expected.
http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_Release_Notes_3_3-1_0_0_0.pdf
I know the default native vlan on hybrid port is VLAN 1.
Do we have a command to configure/change this default native vlan on Mellanox switch?
No.
You need to create new VLAN, and assign it to the relevant ports.
Ophir.
Can you verify that the BIOS is enabled with virtualization?
See for example Dell HowTo Set Dell PowerEdge R730 BIOS parameters to support SR-IOV.
Refer to the BIOS manufacture manuals.
Ophir.
See some examples here for mlx5, I'm not sure if this is the same for mlx4.
Hi Kaijun,
Did you bind the devices to UIO/VFIO? If so, you should bind it back to kernel driver. Mellanox PMD doesn't use UIO/VFIO but its control path goes through regular kernel driver unlike Intel NICs.
Thanks,
Yongseok
hi all,
I've a a very basic setup, directly two boxes via two MHEH28-XTC and I cannot activate them.
One peculiar thing is I get (randomly & !often):
[85947.090496] AMD-Vi: Event logged [
[85947.090539] IO_PAGE_FAULT device=09:00.7 domain=0x0000 address=0x00000000f6ffb000 flags=0x0050]
[85947.298509] AMD-Vi: Event logged [
[85947.298550] IO_PAGE_FAULT device=09:00.7 domain=0x0000 address=0x00000000f6ffb000 flags=0x0050]
which is the card itself, judging by the device id
Would you have and share some thoughts please?
$ ./flint/mstflint -d 09:00.0 q # for both cards
-W- Running quick query - Skipping full image integrity checks.
Image type: FS2
FW Version: 2.9.1000
Device ID: 25408
Description: Node Port1 Port2 Sys image
GUIDs: 0008f104039a62a0 0008f104039a62a1 0008f104039a62a2 0008f104039a62a3
MACs: 000000000000 000000000001
VSD:
PSID: MT_04A0110001
$ ibstat
CA 'mlx4_0'
CA type: MT25408
Number of ports: 2
Firmware version: 2.9.1000
Hardware version: a0
Node GUID: 0x0008f104039a08dc
System image GUID: 0x0008f104039a08df
Port 1:
State: Initializing
Physical state: LinkUp
Rate: 10
Base lid: 1
LMC: 0
SM lid: 1
Capability mask: 0x0259086a
Port GUID: 0x0008f104039a08dd
Link layer: InfiniBand
Port 2:
State: Down
Physical state: Polling
Rate: 10
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x0259086a
Port GUID: 0x0008f104039a08de
Link layer: InfiniBand
in opensm log:
Jan 06 17:00:28 817185 [F6D5A700] 0x01 -> sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT): SubnGet(NodeInfo), attr_mod 0x0, TID 0x1cd1
Jan 06 17:00:28 817200 [F6D5A700] 0x01 -> sm_mad_ctrl_send_err_cb: ERR 3120 Timeout while getting attribute 0x11 (NodeInfo); Possible mis-set mkey?
many thanks
counters_ext is not upstream and is MOFED only.
See the below commands:
[standalone: master] > enable
[standalone: master] # configure terminal
[standalone: master] (config) # vlan 3
[standalone: master] (config vlan 3) exit
[standalone: master] (config) # interface ethernet 1/1
[standalone: master] (config interface ethernet 1/1) # switchport mode hybrid
[standalone: master] (config interface ethernet 1/1) # switchport access vlan 3
Verify with the following command:
config) # show interface switchport
Interface Mode Access vlan Allowed vlans
---------------------------------------------------------------------------------
Eth1/1 hybrid 3
Marlon