Hello,
i do not use the dpdk_nic_bind script because we do not use UIO (IGB UIO is Intel's uio driver for IGB). For MLNX NICs, DPDK is yet another user space application written over the Raw Eth VERBs interface. control path is going through the mlx4/5 kernel modules and data path directly to HW from user space (somewhat similar to the RDMA mechanism). It has some nice advantages like security and that the NIC can be managed by the kernel like one would do in none DPDK environment (i.e all standard tools like ethtool etc.).
Lee, if you managed to make this work with the script using MLNX NIC, would be great if you can share the details.
I'm currently writing a post for OVS-DPDK and MLNX NICs, so until then, here is a very dirty guide that might help (Sorry for the format, will be much nicer in the post ). this was done on ConnectX-4 but should work the same for ConnectX-3.
Some typos, bad phrasing and minor errors can be expected , i will correct them in the post.
Find the NIC numa node:
# mst start
# mst status -v
MST modules:
------------
MST PCI module loaded
MST PCI configuration module loaded
PCI devices:
------------
DEVICE_TYPE MST PCI RDMA NET NUMA
ConnectX4(rev:0) /dev/mst/mt4115_pciconf0.1 11:00.1 mlx5_1 net-enp17s0f1 0
ConnectX4(rev:0) /dev/mst/mt4115_pciconf0 11:00.0 mlx5_0 net-enp17s0f0 0
# mst stop
Configure Hugepages
OVS needs a system with 1GB hugepages support, which can only be allocated during boot. Note that for NUMA machine the pages will be divided between the NUMA nodes.
For best performance you might want to have two separate hugepages mount points, one for QEMU (1G pages) and one for DPDK (2M pages). see here - 29. Vhost Sample Application — Data Plane Development Kit 2.2.0 documentation
2M pages can be allocated after the machine booted. Here i used only 1G pages (and no performance tuning done)
Adding boot parameters to enable 8 x 1GB HugePages (using grubby here, can be done in many ways)
Need to add "default_hugepagesz=1GB hugepagesz=1GB hugepages=<number of pages>" to the kernel boot parameters.
# yum install grub2-tools
# grubby -c /boot/grub2/grub.cfg --default-kernel
/boot/vmlinuz-3.10.0-229.el7.x86_64
# grubby -c /boot/grub2/grub.cfg --args="default_hugepagesz=1GB hugepagesz=1GB hugepages=8" --update-kernel /boot/vmlinuz-3.10.0-229.el7.x86_64
Verify
# grubby -c /boot/grub2/grub.cfg --info /boot/vmlinuz-3.10.0-229.el7.x86_64
index=0
kernel=/boot/vmlinuz-3.10.0-229.el7.x86_64
args="ro crashkernel=auto rhgb quiet LANG=en_US.UTF-8 default_hugepagesz=1GB hugepagesz=1GB hugepages=8"
root=UUID=c4d1bf80-880c-459e-a996-57cb41de2544
initrd=/boot/initramfs-3.10.0-229.el7.x86_64.img
title=Red Hat Enterprise Linux Server 7.1 (Maipo), with Linux 3.10.0-229.el7.x86_64
Reboot the machine
Configure 4 pages on the right NUMA node (note that this should be done by default, i just like to make sure)
echo 4 > /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages
By default the hugepages should be mounted on /dev/hugepages
Reboot the machine
Build DPDK
Download DPDK
Edit vim config/common_linuxapp:
CONFIG_RTE_BUILD_COMBINE_LIBS=y
CONFIG_RTE_LIBRTE_MLX5_PMD=y
Make sure CONFIG_RTE_LIBRTE_VHOST_USER=y
# make install T=x86_64-native-linuxapp-gcc
Install OVS
# wget https://github.com/openvswitch/ovs/tarball/master
# tar -zxvf master
# cd openvswitch-ovs-39cc5c4/
# ./boot.sh
# export LIBS="-libverbs"
# ./configure --with-dpdk=/var/soft/dpdk/dpdk-2.2.0/x86_64-native-linuxapp-gcc --disable-ssl
# make CFLAGS='-O3 -march=native'
# make install
Start OVS
# mkdir -p /usr/local/etc/openvswitch
# mkdir -p /usr/local/var/run/openvswitch
# rm /usr/local/etc/openvswitch/conf.db ## If not first time run
# ovsdb-tool create /usr/local/etc/openvswitch/conf.db /usr/local/share/openvswitch/vswitch.ovsschema
Start ovsdb-server
# export DB_SOCK=/usr/local/var/run/openvswitch/db.sock
# ovsdb-server --remote=punix:$DB_SOCK --remote=db:Open_vSwitch,Open_vSwitch,manager_options --pidfile --detach
Start OVS
# ovs-vsctl --no-wait init
# ovs-vswitchd --dpdk -c 0xf -n 4 --socket-mem 1024 -- unix:$DB_SOCK --pidfile --detach --log-file=/var/log/openvswitch/ovs-vswitchd.log
Create OVS bridge, add DPDK port and vhost-user port
# ovs-vsctl add-br ovsbr0 -- set bridge ovsbr0 datapath_type=netdev
# ovs-vsctl add-port ovsbr0 dpdk0 -- set Interface dpdk0 type=dpdk
# ovs-vsctl add-port ovsbr0 vhost-user1 -- set Interface vhost-user1 type=dpdkvhostuser
vhost-user device created here -
/usr/local/var/run/openvswitch/vhost-user1
Run VM with vhost-user back-end device-
qemu-system-x86_64 -enable-kvm -m 1024 -smp 2 -chardev socket,id=char0,path=/usr/local/var/run/openvswitch/vhost-user1 -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce -device virtio-net-pci,netdev=mynet1,mac=12:34:00:00:50:2c -object memory-backend-file,id=mem,size=1024M,mem-path=/dev/hugepages,share=on -numa node,memdev=mem -mem-prealloc /data1/vms/rhel6.7-master.qcow2
Hope this helps and i did not missed anything
Anyway a full community post will be available soon