Hi,
We are using Mellanox Connect-X3 PRO card for our solution.
We are facing some problem with dual port configuration.
Below are the configuration done for dual port.
sdn@ubuntu:~$ cat /etc/modprobe.d/mlx4.conf
options mlx4_core port_type_array=2,2 num_vfs=2,2,0 probe_vf=2,2,0 enable_64b_cqe_eqe=0 log_num_mgm_entry_size=-1
sdn@ubuntu:~$ lspci -nn | grep Eth
02:00.0 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01)
02:00.1 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01)
02:00.2 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01)
02:00.3 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01)
04:00.0 Ethernet controller [0200]: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] [15b3:1007]
04:00.1 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
04:00.2 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
04:00.3 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
04:00.4 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
However if we do ip link show on the interfaces.
sudo ip link show eth0
14: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 300000
link/ether 48:0f:cf:f4:0d:00 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 1 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 2 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 3 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
sudo ip link show eth1
15: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 300000
link/ether 48:0f:cf:f4:0d:01 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 1 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 2 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
vf 3 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto
and virtual functions of both the ports points to the same address.
sudo ls -l /sys/class/net/eth0/device/virtfn*
lrwxrwxrwx 1 root root 0 May 27 05:46 /sys/class/net/eth5/device/virtfn0 -> ../0000:04:00.1
lrwxrwxrwx 1 root root 0 May 27 05:46 /sys/class/net/eth5/device/virtfn1 -> ../0000:04:00.2
lrwxrwxrwx 1 root root 0 May 27 05:46 /sys/class/net/eth5/device/virtfn2 -> ../0000:04:00.3
lrwxrwxrwx 1 root root 0 May 27 05:46 /sys/class/net/eth5/device/virtfn3 -> ../0000:04:00.4
sudo ls -l /sys/class/net/eth1/device/virtfn*
lrwxrwxrwx 1 root root 0 May 27 05:46 /sys/class/net/eth5/device/virtfn0 -> ../0000:04:00.1
lrwxrwxrwx 1 root root 0 May 27 05:46 /sys/class/net/eth5/device/virtfn1 -> ../0000:04:00.2
lrwxrwxrwx 1 root root 0 May 27 05:46 /sys/class/net/eth5/device/virtfn2 -> ../0000:04:00.3
lrwxrwxrwx 1 root root 0 May 27 05:46 /sys/class/net/eth5/device/virtfn3 -> ../0000:04:00.4
Are we missing any configuration here?.
PCI-Passthrough:
1. OS is showing both the ports as single PCI device and both the ports have same bus id.
Problem: There is no way to differentiate the ports
2. Guest OS connected to the single PCI device is getting two interfaces created.
Problem: Additional interface is causing problem in identifying the actual interface.
SR-IOV:
1. Created VF’s are showing in both the adapters.
Problem: PF-VF mapping
2. Guest OS connected to the single VF is getting two interfaces created
Problem: Additional interface is causing problem in rearranging interfaces.
The real problem here is, NOVA or LIBVIRT is unable to distinguish between the 2 ports (eth0 and eth1).
we are unable to use both the ports simultaneously for SRIOV or PCI-passthrough.
The reason is both the ports come up with the same PCI address (eg:- 04:00:0--- for eth0 and eth1).
even for VF's created under the PF's (eth0 and eth1). It is showing the same PCI bus address (0000:04:00.1 -- 0000:04:00.4) --- these are VF's.
When we create a VM with direct-physical port (SRIOV) in Openstack. As the OS shows same PCI address for the VF's created under both ports (eth0 and eth1), we are unable to utilize the second port for SRIOV or PCI-passthrough.
Even the libvirt can see only 4 VF's when we do
virsh nodedev-list | grep pci | grep 04_00
pci_0000_04_00_0
pci_0000_04_00_1
pci_0000_04_00_2
pci_0000_04_00_3
pci_0000_04_00_4
Kindly shed some light here and let us know your suggestions.
Thanks,
Manjunath