So you still only get 10G performance?
thanks
↧
Re: 40Gb/s IPoIB only gives 5Gb/s real throughput?!
↧
Re: Firmware for Voltaire 4036
Mellanox support does not give a login with my serial number. So for those who search http://h20564.www2.hpe.com/hpsc/swd/public/detail?swItemId=MTX_e36c6c6b4cd249c2b2edbf4aab
↧
↧
Does anyone know what the Max Junction temperature is for the MT27508 IC on a ConnectX-3
Does anyone know what the Max Junction temperature is for the MT27508 IC on a ConnectX-3, or where I can find it?
or failing that what is the max operating temperatures for ConnectX-3 cards, we are actutally using Dell Mezzanine card versions of the X-3
↧
Setting SR-IOV num_vfs for ConnectX-2 card
Hi,
I need to enable 4 VFs for a CX-2 card on a Power8 server
Following are the model details of my card (obtained via flint -d /dev/mst/mt26448_pci_cr0 q)
Device ID: 26448
PSID: IBM0F30001010
FW Version: 2.9.1326
lspci | grep Mellanox
0000:01:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
I could not find instructions to enable VFs for CX-2 cards (CX-3 and CX-4 have instructions).
I checked a few things to try and enable SR-IOV
- I believe the card I have supports SR-IOV as per the "features" section in this link: http://www.mellanox.com/page/products_dyn?product_family=20&mtag=connectx_en
- The next thing I checked was to see if SR-IOV is enabled for my firmware
flint -d /dev/mst/mt26448_pci_cr0 dc
>> [HCA] section does not have entries for SR-IOV or num_vfs
- Tried enabling it using MST mlxconfig
[root@pts00451 ~]# mlxconfig -d /dev/mst/mt26448_pci_cr0 q
Device #1:
----------
Device type: ConnectX2
PCI device: /dev/mst/mt26448_pci_cr0
-E- Failed to query device: /dev/mst/mt26448_pci_cr0. Device not supported.
It seems the MST version does not support CX-2 (I tried using older versions of MST with no luck)
- Looked to see if could install modified firmware (set num_vfs etc. in INI file)
Could not find a .tgz firmware image file for this IBM OEM card, firmware image here is a .bin file: http://www.mellanox.com/page/firmware_table_IBM_SystemP
Needed some help or instructions/pointers to enable VFs for my card
Thanks in advance!
Vaibhav
↧
40Gb/s IPoIB only gives 5Gb/s real throughput?!
I really need some expertise here:
For the current situation see update 2 below.
Update: I tested with Windows 2012 Clients to verify and I still get about 5.5 Gbit/s max.
Maybe someone has other 40Gbit adapters what are the speeds for you?
Update 2: The mainboard had 16x physical and only 2x electrical connection. (Special thx to Erez support admin for a quick and good answer)
After changing to a PCIe 3.0 8x lane I now get the following speed: (should still be 3x faster)
Update 3: One support admin suggested to not use passive copper, but to use optic fibre. After getting an 56Gbit Optical fibre IB cable I now get these results:
Which is still way below the advertised 40Gbit!
The story goes like this: Advertised 40Gbit , 32Gbit theoretical which is really only 25.6 Gbit according to Enez from Mellanox which turns out to be in Reality HALF-DUPLEX 16Gbit!
Do I make something wrong or is it just the way it works for customers of mellanox :/
If there is still something wrong how do i fix it?
OLD PART DO NOT READ: (READ UPDATE 3 instead)
I have two Windows 10 Machines with two MHQH19B-XTR 40 Gbit Adapters and a QSFP cable in between. The Vlan manager is opensm.
The connection should be about 32Gbits Lan. In reality i only get 5 Gbit performance. So clearly something is very wrong.
C:\Program Files\Mellanox\MLNX_VPI\IB\Tools>iblinkinfo
CA: E8400:
0x0002c903004cdfb1 2 1[ ] ==( 4X 10.0 Gbps Active/ LinkUp)==> 1 1[ ] "IP35" ( )
CA: IP35:
0x0002c903004ef325 1 1[ ] ==( 4X 10.0 Gbps Active/ LinkUp)==> 2 1[ ] "E8400" ( )
I tested my IPoIB with a program called lanbench and nd_read_bw:
nd_read_bw -a -n 100 -C 169.254.195.189
#qp #bytes #iterations MR [Mmps] Gb/s CPU Util.
0 512 100 0.843 3.45 0.00
0 1024 100 0.629 5.15 0.00
0 2048 100 0.313 5.13 0.00
0 4096 100 0.165 5.39 0.00
0 8192 100 0.083 5.44 0.00
0 16384 100 0.042 5.47 0.00
0 32768 100 0.021 5.47 100.00
..stays at 5.47 after that. with CPU util 100%
The processor is an intel core I7 4790k so it should not be at 100%. According to Taskmanager only 1 Core is actively used.
Firmware, Drivers, Windows 10 are up to date.
My goal is to get the fastest possible File sharing between two windows 10 machines.
What could be the problem here and how do I fix it?
After endless hours of searching I found out that vstat showed that I have a 10GBit connection.
C:\Users\Daniel>"C:\Program Files\Mellanox\MLNX_VPI\IB\Tools\vstat.exe"
hca_idx=0
uplink={BUS=PCI_E Gen2, SPEED=5.0 Gbps, WIDTH=x8, CAPS=5.0*x8} --> Looks good
MSI-X={ENABLED=1, SUPPORTED=128, GRANTED=10, ALL_MASKED=N}
vendor_id=0x02c9
vendor_part_id=26428
hw_ver=0xb0
fw_ver=2.09.1000
PSID=MT_0D90110009
node_guid=0002:c903:004e:f324
num_phys_ports=1
port=1
port_guid=0002:c903:004e:f325
port_state=PORT_ACTIVE (4)
link_speed=10.00 Gbps
link_width=4x (2)
rate=40.00 Gbps
real_rate=32.00 Gbps (QDR)
port_phys_state=LINK_UP (5)
active_speed=10.00 Gbps --> WHY?
sm_lid=0x0001
port_lid=0x0001
port_lmc=0x0
transport=IB
max_mtu=4096 (5)
active_mtu=4096 (5)
GID[0]=fe80:0000:0000:0000:0002:c903:004e:f325
What I should get is : (thx to erez)
PCI_LANES(8)*PCI_SPEED(5)*PCI_ENCODING(0.8)*PCI_HEADERS(128/152)*PCI_FLOW_CONT(0.95) = 25.6 Gbit
Can anyone help me with this problem?
↧
↧
Re: 40Gb/s IPoIB only gives 5Gb/s real throughput?!
Hello another support admin has suggested that my cable only supports 10Gbit.
That was a fair assesment. I have gotten a real 56 Gbit Infiniband cable yesterday.
Unfortunately still nowhere near 40 Gbit or 32Gbit IPOIB performance
Sending alone yields 16Gbit. It should perform better! Recieving works with 16 Gbit when used alone.
Mixed yields 8GBbit Send and 8Gbit Recieve.
Do you have any suggestion of what could be wrong?
↧
Re: 40Gb/s IPoIB only gives 5Gb/s real throughput?!
Yes 40Gb/s data rate, but sending 8 bits data in a 10 bit packet giving 32Gb/s max data thruput; however the PCIe bus will limit you to about 25Gb/s.
Keep in mind that the performance for hardware to hardware is better than software to software. I've only used Mellanox cards with Linux and the performance for hardware to hardware hits 25Gb/s with ConnectX-2 cards.
The IB equipment you are using has 4 pairs of wire running at 10Gb/s each - hence 40Gb/s total.
Real world file sharing, even with older 10Gb/s InfiniHost cards is better than 10Gb/s ethernet. My MAXIMUM performance tests (using the Linux fio program) are below. That being said we've avoided Windows file servers since at least Windows 2000 - the performance has been terrible compared to Linux; esp. when one factors in the cost of the hardware required.
I would suggest that you compare the exact servers using an ethernet link to see how it compares. In the end theoretical performance is nice - but what really matters is the actual software you are using. In my case going to 10Gb ethernet or QDR IB things like data replication (ZFS snapshots, rsync) went from 90 minutes to sub 3 minutes. It was often not the increased bandwidth but the lower latency (IOPs) that mattered. For user applications accessing the file server - compile times were only reduced by about 30% going to InfiniBand or 10Gb ethernet - but the ethernet is around 10x as expensive. I've not performance tested our Oracle database - but it went to 10Gb ethernet because my IB setup is for the students and I don't trust it yet on a "corporate" server.
In the case of file sharing you'll want to see if you're using the old ports 137 to 139 instead of 445 as that can impact performance.
Also - there is no way to exploit the exceptionally low latency of InfiniBand unless you've got SSDs or your data in RAM.
| Network | GB Data in 30 sec | Aggregate Bandwidth (MB/s, Gb/s) | Bandwidth (MB/s, Mb/s) | latency (ms) | iops |
|---|---|---|---|---|---|
| QDR IB 40Gb/s NFS over RDMA | 94 | 3,100, 25 | 802, 6.4 | 0.615 | 12,535 |
| DDR IB 20Gb/s NFS over RDMA | 24.4 | 834, 6.7 | 208, 1.7 | 2.4 | 3256 |
| SDR IB 10Gb/s NFS over RDMA | 22.3 | 762, 6.1 | 190, 1.5 | 2.57 | 2978 |
| QDR IB 40Gb/s | 16.7 | 568, 4.5 | 142, 1.1 | 3.4 | 2218 |
| DDR IB 20Gb/s | 13.9 | 473, 3.8 | 118, 0.94 | 4.1 | 1845 |
| SDR IB 10Gb/s | 13.8 | 470, 3.8 | 117, 0.94 | 4.2 | 1840 |
| 10Gb/s ethernet | 5.9 | 202, 1.6 | 51, 0.41 | 9.7 | 793 |
| 1Gb/s ethernet | 3.2 | 112, 0.90 | 28 | 17.8 | 438 |
| 100Mb/s ethernet | 346MB | 11.5 | 2.9 | 174 | 45 |
| 10Mb/s ethernet via switch | 36MB | 1.2 | 279kB/s | 1797 | 4 |
| 10Mb/s ethernet via hub | 33MB | 1.0 | 260kB/s | 1920 | 4 |
↧
Re: 40Gb/s IPoIB only gives 5Gb/s real throughput?!
Hi!
Don't use IPoIB mode.
Mellanox remove support RC mode (mtu=65535) on Windows.
RC mode use RDMA, but DM mode wasn't.
And ~
Microsoft don't want to use RC mode, but SMB Direct file share that use RDMA.
Jaehoon Choi
↧
Re: SN2100B and 56GbE
Hi Andrzej,
SN2700B does only supports 40g, not the 56GbE and below are the specification for the same.
MSN2700-BS2F Spectrum™ based 40GbE, 1U Open Ethernet Switch with MLNX-OS, 32 QSFP28
ports, 2 Power Supplies (AC), x86 CPU, Standard depth, P2C airflow, Rail Kit, RoHS6
Thanks
Khwaja
↧
↧
Re: Does anyone know what the Max Junction temperature is for the MT27508 IC on a ConnectX-3
Hello,
Mellanox HCAs’ temperature threshold is 120 [degrees centigrade]. Temperature measured on HCA lower or equal than this threshold is considered as proper temperature.
You can measure it also using mget_temp tool that comes with MFT (Mellanox firmware tool) that can be downloaded from Mellanox website:
http://www.mellanox.com/page/management_tools
Regards,
Viki
↧
Re: Setting SR-IOV num_vfs for ConnectX-2 card
Hi Vaibhav Sood,
As you know, ConnectX-2 is old device, so maybe it is a bit complicated to configure it with the newest firmware tools.
please try the following and update me if it is not working.
1. Enable SRIOV in BIOS.
2. Enable SRIOV in Linux Kernel:
Activate Intel VT-d in the kernel by appending the intel_iommu=on parameter to the kernel line of the kernel line in the /boot/grub/grub.conf file.
Example: #vim /boot/grub/grub.conf
default=0timeout=5splashimage=(hd0,0)/grub/splash.xpm.gzhiddenmenutitle Red Hat Enterprise Linux Server (2.6.32-36.x86-645) root (hd0,0) kernel /vmlinuz-2.6.32-36.x86-64 ro root=/dev/VolGroup00/LogVol00 rhgb quiet intel_iommu=on initrd /initrd-2.6.32-36.x86-64.img
3. Enable SRIOV in the firmware:
A. Copy the INI file locally or dump it from the device by running flint -d <mstdevice> dc >tmp/ini.ini.
B. Edit the INI file and add these params under HCA :
num_pfs = 1
total_vfs = 64
sriov_en = true
Example:
[root@]# vim /tmp/MCX354A-FCB_A1_SRIOV.ini --> Add the following lines under [HCA] section: num_pfs = 1 total_vfs = 64 sriov_en = truec. reburn the FW by running mlxburn -d mstdevice -fw <fw.mlx> -c /tmp/ini.ini
4. Enable SRIOV in the driver:
- A. Create the file /etc/modprobe.d/mlx4_core.conf and add the following content:
options mlx4_core num_vfs=64 port_type_array=2,2(which will open the driver with 64 VFs and Port1 = eth, port2=eth)
- If you need IB port_type shall be = 1
- B. Restart the driver.
5. Reboot the Machine.
6. How to check that SRIOV is working: Use LSPCI.
↧
Re: How to configure cards to run at 25Gbs
Hi,
your device is already with the requested speed 25000 after running "ethtool -s enp2s0f0 speed 25000 autoneg off",
does the other side (switch or Back to Back device ) configured to 25000 speed ?
I see that link is down, when the link raise up, the ethtool will show the configured speed.
Thanks,
Talat
↧
Re: Patch needed to activate ROCEV2 for Connect 3X 10G card
Hi Rama,
could you please tell me the device type, operation system and the kernel version that you are using.
for your information, RoCE v2 is supported ONLY in ConnectX®-3 Pro and ConnectX®-4 adapter cards.
please make sure that your device is ConnectX3-pro and not ConnectX3.
Thanks,
Talat
↧
↧
Re: questions regarding SRQ performance
Hi Sam,
For the above question please open a ticket with Mellanox support by sending the info to support@mellanox.com.
this need to be investigated in order to provide you with the most accurate reply.
Thank you,
Karen.
↧
Re: Patch needed to activate ROCEV2 for Connect 3X 10G card
we are using ConnectX3-pro with MLNX OFED stack, it is detecting ROCEV2 properly.
But we are not getting ROCEV2 parameter with 4.8 kernel without MLNX OFED stack.
I want to work with Connect X3 pro with ROCEV2 with out Mellanox OFED stack.
is it possible?
↧
Re: Patch needed to activate ROCEV2 for Connect 3X 10G card
Yes, it's possible.
The RoCE V2 support accepted at kernel v4.5. commit 3f723f42d9d625bb9ecfe923d19d1d42da775797
check the gid's that generated, should be there RoCE V2 gids.
please make sure that your user space libraries support.
In order to work with RDMA_CM use the configfs
# mount -t configfs none /sys/kernel/config
# cd /sys/kernel/config/rdma_cm
# mkdir mlx4_0
# cd mlx4_0
# echo RoCE V2 > default_roce_mode
# cd ..
# rmdir mlx4_0
↧
Re: Patch needed to activate ROCEV2 for Connect 3X 10G card
Hi Talat,
I have used kernel 4.8, after that if i give
ls /sys/modules/mlx4_core/parameters I did not find any entry related to RoCE mode.
Please tell how we can view gid's. what is the location. Also, please let me know the command to check the support in userspace libraries.
↧
↧
Re: Patch needed to activate ROCEV2 for Connect 3X 10G card
There is no roce_mode parameter in upstream, RoCE V2 is enabled by default.
GID Table in sysfs
GID table is exposed to user space via sysfs
1. GID values can be read from:
/sys/class/infiniband/{device}/ports/{port}/gids/{index}
2. GID type can be read from:
/sys/class/infiniband/{device}/ports/{port}/gid_attrs/types/{index}
3. GID net_device can be read from:
/sys/class/infiniband/{device}/ports/{port}/gid_attrs/ndevs/{index}
Yours,
Talat
↧
Re: Patch needed to activate ROCEV2 for Connect 3X 10G card
Hi Talat,
The above entries are only visible with MLNX_OFED stack. we are not getting with kernel 4.8 with INBOX OFED.
Even the below step giving error.
cd /sys/kernel/config/rdma_cm
# mkdir mlx4_0
Thanks
Rama
↧
Re: Patch needed to activate ROCEV2 for Connect 3X 10G card
i don't think so, is all the modules up ?
please make sure that you have the updated user space and all the modules is up like.
ib_cm,rdma_cm,ib_umad,ib_uverbs,ib_ucm,rdma_ucm,mlx4_ib
could you please tell me the operation system. libibverbs and libmlx4 version ?
Yours,
Talat
↧