Quantcast
Channel: Mellanox Interconnect Community: Message List
Viewing all 6275 articles
Browse latest View live

Abaqus MPI with Connect-IB: rdma error

$
0
0

Hello

, the connect-ib runs with MLNX_OFED_LINUX-2.3-1.0.1-rhel6.5-x86_64,

abaqus (FEM Software) comes with it's own mpi and shows the following error :

 

mpirun: IBM Platform MPI Community Edition:

  09.01.02.00 [11418] Linux x86-64

 

pre:
Rank 0:0: MPI_Init: Could not pin pre-pinned rdma region 0

pre:
Rank 0:0: MPI_Init: hpmp_rdmaregion_alloc() failed

pre:
Rank 0:0: MPI_Init: make_world_rdmaenvelope() failed

pre:
Rank 0:0: MPI_Init: Internal Error: Processes cannot connect to rdma device

 

Q: are there any hints how to manage mpi with the OFED-Driver

 

Thank you

Best Regards

M.Bentele


Expected bandwidth performance from ib_send_bw

$
0
0

Hi,

 

I'm trying to measure my ib bandwidth by using ib_send_bw. Im using the Connect X-3 with 56GbE. I connect two machine point to point with infiniband.

 

[mimos@wnode4 ~]$ ibstatus

Infiniband device 'mlx4_0' port 1 status:

        default gid:     fe80:0000:0000:0000:f452:1403:0033:b4c1

        base lid:        0x0

        sm lid:          0x0

        state:           1: DOWN

        phys state:      2: Polling

        rate:            10 Gb/sec (4X)

        link_layer:      InfiniBand

 

 

Infiniband device 'mlx4_0' port 2 status:

        default gid:     fe80:0000:0000:0000:f452:1403:0033:b4c2

        base lid:        0x1

        sm lid:          0x1

        state:           4: ACTIVE

        phys state:      5: LinkUp

        rate:            56 Gb/sec (4X FDR)

        link_layer:      InfiniBand

 

[mimos@wnode4 ~]$  ib_send_bw -a -R -i 2 if2

---------------------------------------------------------------------------------------

                    Send BW Test

Dual-port       : OFF          Device         : mlx4_0

Number of qps   : 1            Transport type : IB

Connection type : RC           Using SRQ      : OFF

TX depth        : 128

CQ Moderation   : 100

Mtu             : 2048[B]

Link type       : IB

Max inline data : 0[B]

rdma_cm QPs     : ON

Data ex. method : rdma_cm

---------------------------------------------------------------------------------------

local address: LID 0x01 QPN 0x008a PSN 0xe66ab2

remote address: LID 0x02 QPN 0x008a PSN 0x6d5c92

---------------------------------------------------------------------------------------

#bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]

2          1000           10.72              6.87                 3.599459

4          1000           22.02              21.96                5.756531

8          1000           44.12              44.02                5.770259

16         1000           88.24              88.04                5.769635

32         1000           175.85             175.60               5.753926

64         1000           351.69             350.98               5.750456

128        1000           674.28             643.71               5.273302

256        1000           1341.63            1252.38              5.129763

512        1000           2324.42            2131.24              4.364776

1024       1000           2469.36            2442.37              2.500986

2048       1000           2678.66            2672.90              1.368524

4096       1000           2790.35            2790.23              0.714299

8192       1000           2878.93            2871.59              0.367564

16384      1000           2908.87            2905.95              0.185981

32768      1000           2934.02            2932.92              0.093853

65536      1000           2948.41            2948.18              0.047171

131072     1000           2953.17            2952.56              0.023620

262144     1000           2955.65            2955.38              0.011822

524288     1000           2956.79            2956.77              0.005914

1048576    1000           2957.37            2957.31              0.002957

2097152    1000           2957.75            2957.73              0.001479

4194304    1000           2957.68            2957.67              0.000739

8388608    1000           2957.71            2957.66              0.000370

---------------------------------------------------------------------------------------

 

QUESTION:

  1. why are the ib_send_bw showing my peak BW around half the speed as it suppose to be?
  2. Any available benchmark i could refer to?

 

Please advice, thanks.

Amirul.

Beginners guide to Infiniband

Re: Trying to "rate limit" a VF in SR-IOV enabled Mellanox ConnectX-3 HCA

Re: Troubleshooting the iSER enable on Mellanox ConnectX-2 EN and running iSCSI traffic.

$
0
0

Hi Anil,

Do you have MLNX_OFED installed? did you use OFED inbox driver?

Note that the example is from RHEL7 and over there the interface name is different (not eth1,eth2 ..) but enp5s0 or similar.

try

 

#yum -y groupinstall "InfiniBand Support"

#yum -y install perftest infiniband-diags

- for the inbox driver, it may be the easiest.

 

Please send me the output of

#ibdev2netdev

#ofed_info | head

you didn't reach the level of testing ISER, first you need to make sure that RoCE is running, and for that you need RDMA support.


Ophir.

Troubleshooting the iSER enable on Mellanox ConnectX-2 EN and running iSCSI traffic.

$
0
0

Hi,

 

I have folowing card on my system.

lspci | grep -i mellanox

02:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)

 

I am running CentOs 6.5.

uname -a

Linux RV-189 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/issue

CentOS release 6.5 (Final)

Kernel \r on an \m

 

I am following the link .

HowTo setup RoCE connection using Inbox driver (RHEL, Ubuntu)

 

I do not see the enp5s0 or any other interface which can describe the Mellanox card on my system.

#ifconfig enp5s0 12.12.12.1/24 up

 

Can someone pl let me know what am I missing?

 

Thanks in advance,

Anil

Re: Troubleshooting the iSER enable on Mellanox ConnectX-2 EN and running iSCSI traffic.

$
0
0

Thanks Ophir for the reply.

 

Hi Anil,

 

Do you have MLNX_OFED installed? did you use OFED inbox driver?

 

Anil> Whether MLNX_OFED should be available as inbox? I tried your commands

and they did not run, please see the output given below.

 

# yum install "InfiniBand Group"

Loaded plugins: aliases, changelog, downloadonly, fastestmirror, kabi,

presto, refresh-packagekit,

              : security, tmprepo, verify, versionlock

Loading support for CentOS kernel ABI

Loading mirror speeds from cached hostfile

 

  • base: mirrors.unifiedlayer.com

 

  • extras: mirror.anl.gov

 

  • updates: mirrors.easynews.com

Setting up Install Process

No package InfiniBand Group available.

Error: Nothing to do

#

#

# ibdev2netdev

bash: ibdev2netdev: command not found

# ibdev2netdev

bash: ibdev2netdev: command not found

# ofed_info | head

bash: ofed_info: command not found

#

 

 

 

you didn't reach the level of testing ISER, first you need to make sure

that RoCE is running, and for that you need RDMA support.

 

Anil>> I knew, I am missing something. yes, I want to make sure that RoCE

is running first

 

Thanks,

 

Anil

Re: Troubleshooting the iSER enable on Mellanox ConnectX-2 EN and running iSCSI traffic.


Re: Troubleshooting the iSER enable on Mellanox ConnectX-2 EN and running iSCSI traffic.

$
0
0

Ophir,

 

This <http://community.mellanox.com/docs/DOC-1465> is the exact link what I

was following. I installed packages from there and got stuck up at

following section.

 

Configure port parameters:

Configure IP Address and enable the port.

*It can be done via console scripts such as nmtui, ifconfig (not permanent)

or other method*

For example:

 

#ifconfig enp5s0 12.12.12.1/24 <http://12.12.12.1/24> up

 

 

 

I tried to install these packages again and it came up with following

response:

 

# yum -y groupinstall "InfiniBand Support"

Loaded plugins: aliases, changelog, downloadonly, fastestmirror, kabi,

presto, refresh-packagekit, security, tmprepo, verify,

              : versionlock

Loading support for CentOS kernel ABI

Loading mirror speeds from cached hostfile

 

  • base: centos-distro.cavecreek.net

 

  • extras: mirror.nwresd.org

 

  • updates: mirrors.xmission.com

Setting up Group Process

Checking for new repos for mirrors

Package libibverbs-utils-1.1.7-1.el6.x86_64 already installed and latest

version

Package libibverbs-1.1.7-1.el6.x86_64 already installed and latest version

Package librdmacm-utils-1.0.17-1.el6.x86_64 already installed and latest

version

Package libibcm-1.0.5-3.el6.x86_64 already installed and latest version

Package rdma-3.10-3.el6.noarch already installed and latest version

Package librdmacm-1.0.17-1.el6.x86_64 already installed and latest version

Package libmlx4-1.0.5-4.el6.1.x86_64 already installed and latest version

Package libipathverbs-1.2-4.el6.x86_64 already installed and latest version

Package libmthca-1.0.6-3.el6.x86_64 already installed and latest version

Package libnes-1.1.3-1.el6.x86_64 already installed and latest version

Package libcxgb3-1.3.1-1.el6.x86_64 already installed and latest version

Package ibsim-0.5-7.el6.x86_64 already installed and latest version

Package dapl-2.0.34-1.el6.x86_64 already installed and latest version

Package libibmad-1.3.9-1.el6.x86_64 already installed and latest version

Package libibumad-1.3.8-1.el6.x86_64 already installed and latest version

Package rds-tools-2.0.6-3.el6.x86_64 already installed and latest version

Package ibutils-1.5.7-8.el6.x86_64 already installed and latest version

Package ibacm-1.0.8-0.git7a3adb7.el6.x86_64 already installed and latest

version

Warning: Group infiniband does not have any packages.

No packages in any requested group available to install or update

#

#

# yum -y install perftest infiniband-diags

Loaded plugins: aliases, changelog, downloadonly, fastestmirror, kabi,

presto, refresh-packagekit, security, tmprepo, verify,

              : versionlock

Loading support for CentOS kernel ABI

Loading mirror speeds from cached hostfile

 

  • base: mirror.tocici.com

 

  • extras: mirror.nwresd.org

 

  • updates: mirrors.xmission.com

Setting up Install Process

Package perftest-2.0-2.el6.x86_64 already installed and latest version

Package infiniband-diags-1.5.12-5.el6.x86_64 already installed and latest

version

Nothing to do

#

 

 

So it seems like theses packages are there on my system.

 

If you think, I need to do following extra, I can do that.

 

*You can also install MLNX_OFED, see Mellanox website for the driver

package.*

 

Click here, and follow the links.

 

*Mellanox Products: Mellanox OpenFabrics Enterprise Distribution for Linux

(MLNX_OFED)*

<http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers>

 

 

 

 

On Tue, Sep 16, 2014 at 12:02 PM, ophirmaor <socialmedia@mellanox.com>

Re: Troubleshooting the iSER enable on Mellanox ConnectX-2 EN and running iSCSI traffic.

$
0
0

Ophir,

 

I downloaded driver from your web site and probably it seems to behave much

better now. I am giving you output of the commands what you asked earlier.

 

# ibdev2netdev

mlx4_0 port 2 ==> eth4 (Down)

mlx4_0 port 1 ==> eth5 (Down)

# ofed_info | head

MLNX_OFED_LINUX-2.3-1.0.1 (OFED-2.3-1.0.0):

 

ar_mgr:

ar_mgr/e338fbc05dd6b480f8505791e50700b7/ar_mgr-1.0-0.23.g20d8d45.tar.gz

 

bupc:

bupc/6b0f0c8b7d7ff2e258e600d65f6da291/bupc-2.18.0-421.src.rpm

 

cc_mgr:

cc_mgr/4a29eb085479e51b6bb278efcddfe379/cc_mgr-1.0-0.22.g20d8d45.tar.gz

#

 

 

 

So interface eth4 and eth5 can be used for iSER, correct?

 

I have two cards and want to set one card in one system as iSER Initiator

and other card as iSER target.

 

Can I use these two cards in the same system as Initiator and target?

 

Thanks,

Anil

Re: Troubleshooting the iSER enable on Mellanox ConnectX-2 EN and running iSCSI traffic.

$
0
0

eth4 and eth5 will be the interfaces you need to configure. correct.

Do you have two hosts? if so, you can configure one host (with one Mellanox adapter) as target (you can use TGT for example) and another server equipped with Mellanox adapter as the initiator.

you can follow HowTo Configure TGT enabled with iSER Transport for RHEL for that.

 

Ophir.

Re: Windows 2008 R2 Drivers Mellanox ConnectX-2 IPoIB adaptor

$
0
0

Erez,

 

1) 8gb

2) i did so after your response

3) no, not specific

4) i did after your response

 

I works now

i uninstalled all drivers and software

after installing the latest drivers and rebooting the machine the network cards become available

 

thanks for the help

 

Kris.

Re: Why LRO is disabled by default in mlx4_en driver?

$
0
0

LRO settings set to 'OFF' by default as it done by other vendors. This is like old good know behavior and user can change it according to his requirement.

Some applications are bandwidth depended and other are latency and network device parameters should  be set according to the benchmark/application results.

Re: How to create IB partitions without enabling IPOIB

$
0
0

In order to let to the nodes communicate over configured partition child interface should be created for IPoIB interface, or application should specify the P_Key.

Did you try to use ib_send_lat (or other) tools with --pkey_index argument?

Re: InfiniBand RDMA latency test on Xen's dom0 crashes

$
0
0

Something cause to out-of-memory on your box. It is not clean why did you set log_num_mtt to be so high- 4194304. What was the initial failure?

Another point, that you 'dmesg' says that used FW it is probably good to consider firmware upgrade.


Re: Can't Achieve 40Gbps

$
0
0

Your information is little bit confusing. You are talking about Ethernet but runngin 'ethtoool' with ib0 device. You have two Mellanox adapter in the system. Please, verify that the correct one is used.

ibv_devinfo and ibdev2netdev commands will help you.

Windows: Network cable unplugged

$
0
0

Hi there,

 

I have a Mellanox MIS5025Q-1SFC , and 2 x MELLANOX MCX354A-FCBT card.

When I installed them in my windows 2012 r2 and win 7 client machine, and installed the drive pack from Mellanox website, it recognize it, but it showed the cable is not plugged in.  (Via direct-connect between two computer or thru the switch)  the link green light is up, but within Windows, it is still show up as no network cable is connected.  I am using Mellanox MC2207128-003 passive cable, am I doing something wrong ? thanks!

Configuring iSER target using Mellanox MT26448 (ConnectX2) card

$
0
0

Can you please help me with following?

 

I am trying to use Mellanox ConnectX2 card as target and I am getting following error when running tgtd there.

 

tgtd

(null): iser_ib_init(3397) rdma_bind_addr -1: Cannot assign requested address

(null): iscsi_tcp_init_portal(227) unable to bind server socket, Address already in use

(null): iscsi_tcp_init_portal(227) unable to bind server socket, Address already in use

(null): iscsi_add_portal(276) failed to create/bind to portal (null):3260

Re: Configuring iSER target using Mellanox MT26448 (ConnectX2) card

Re: Windows: Network cable unplugged

$
0
0

What is ibv_devinfo command output?

Viewing all 6275 articles
Browse latest View live


<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>