Hi Rian:
I'm happy to say that upgrading to a newer RH kernel, the VM is now working and I do see the Mellanox device from the VM.
However, I have more questions.
I have 4 VFS devices mapped on the hypervisor
01:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
01:00.1 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
01:00.2 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
01:00.3 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
01:00.4 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
However, the hypervisor creates 6 ethernet interfaces...
Nov 13 13:38:38 ATCA-46XX kernel: mlx4_en: eth4: Using 128 TX rings
Nov 13 13:38:38 ATCA-46XX kernel: mlx4_en: eth4: Using 4 RX rings
Nov 13 13:38:38 ATCA-46XX kernel: mlx4_en: eth4: Initializing port
Nov 13 13:38:38 ATCA-46XX kernel: mlx4_en: eth5: Using 128 TX rings
Nov 13 13:38:38 ATCA-46XX kernel: mlx4_en: eth5: Using 4 RX rings
Nov 13 13:38:38 ATCA-46XX kernel: mlx4_en: eth5: Initializing port
Nov 13 13:38:38 ATCA-46XX kernel: mlx4_en: eth6: Using 128 TX rings
Nov 13 13:38:38 ATCA-46XX kernel: mlx4_en: eth6: Using 4 RX rings
Nov 13 13:38:38 ATCA-46XX kernel: mlx4_en: eth6: Initializing port
Nov 13 13:38:38 ATCA-46XX kernel: mlx4_en: eth5: Link Up
Nov 13 13:38:38 ATCA-46XX kernel: mlx4_en: eth4: Link Up
Nov 13 13:38:38 ATCA-46XX kernel: mlx4_en: eth7: Using 128 TX rings
Nov 13 13:38:38 ATCA-46XX kernel: mlx4_en: eth7: Using 4 RX rings
Nov 13 13:38:38 ATCA-46XX kernel: mlx4_en: eth7: Initializing port
Nov 13 13:38:38 ATCA-46XX kernel: mlx4_en: eth8: Using 128 TX rings
Nov 13 13:38:38 ATCA-46XX kernel: mlx4_en: eth8: Using 4 RX rings
Nov 13 13:38:38 ATCA-46XX kernel: mlx4_en: eth8: Initializing port
Nov 13 13:38:38 ATCA-46XX kernel: mlx4_en: eth9: Using 128 TX rings
Nov 13 13:38:38 ATCA-46XX kernel: mlx4_en: eth9: Using 4 RX rings
Nov 13 13:38:38 ATCA-46XX kernel: mlx4_en: eth9: Initializing port
On my VM, I only map 1 VNIC "PCI 01:00.1"
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</hostdev>
On my VM, I see 1 mellanox device via lspci
[root@rh6_4vm1 ~]# lspci | grep -i mel
00:09.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
But....I see two Ethernet devices on VM, eth4 and eth5
Nov 13 17:37:34 rh6_4vm1 kernel: mlx4_core: Mellanox ConnectX core driver v1.1 (Aug 27 2013)
Nov 13 17:37:34 rh6_4vm1 kernel: mlx4_core: Initializing 0000:00:09.0
Nov 13 17:37:34 rh6_4vm1 kernel: mlx4_core 0000:00:09.0: Detected virtual function - running in slave mode
Nov 13 17:37:34 rh6_4vm1 kernel: mlx4_core 0000:00:09.0: Sending reset
Nov 13 17:37:34 rh6_4vm1 kernel: mlx4_core 0000:00:09.0: Sending vhcr0
Nov 13 17:37:34 rh6_4vm1 kernel: mlx4_core 0000:00:09.0: HCA minimum page size:512
Nov 13 17:37:34 rh6_4vm1 kernel: <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (Aug 27 2013)
Nov 13 17:37:34 rh6_4vm1 kernel: mlx4_core 0000:00:09.0: mlx4_ib: multi-function enabled
Nov 13 17:37:34 rh6_4vm1 kernel: mlx4_core 0000:00:09.0: mlx4_ib: operating in qp1 tunnel mode
Nov 13 17:37:34 rh6_4vm1 kernel: mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.1.6 (Aug 27 2013)
Nov 13 17:37:34 rh6_4vm1 kernel: mlx4_en 0000:00:09.0: Activating port:1
Nov 13 17:37:34 rh6_4vm1 kernel: mlx4_en: 0000:00:09.0: Port 1: Assigned random MAC address a6:a3:14:05:bf:21
Nov 13 17:37:34 rh6_4vm1 kernel: mlx4_en: eth4: Using 16 TX rings
Nov 13 17:37:34 rh6_4vm1 kernel: mlx4_en: eth4: Using 4 RX rings
Nov 13 17:37:34 rh6_4vm1 kernel: mlx4_en: eth4: Initializing port
Nov 13 17:37:34 rh6_4vm1 kernel: mlx4_en 0000:00:09.0: Activating port:2
Nov 13 17:37:34 rh6_4vm1 kernel: mlx4_en: 0000:00:09.0: Port 2: Assigned random MAC address f2:9c:31:e1:04:25
Nov 13 17:37:34 rh6_4vm1 kernel: mlx4_en: eth5: Using 16 TX rings
Nov 13 17:37:34 rh6_4vm1 kernel: mlx4_en: eth5: Using 4 RX rings
Nov 13 17:37:34 rh6_4vm1 kernel: mlx4_en: eth5: Initializing port
Nov 13 17:37:34 rh6_4vm1 kernel: mlx4_en 0000:00:09.0: Fail to create debugfs for eth4
Nov 13 17:37:34 rh6_4vm1 kernel: mlx4_en 0000:00:09.0: Fail to create debugfs for eth5
My questions are:
1. Why are 6 interfaces created for 4 VFS?
2. Why are 2 interfaces mapped against 1 VFS?
Thanks for your help