It's been a while and I had given up on these cards but last week I received two new servers and saw the cards laying on the shelf and figured I'd give it another go.
Figured I'd post my results here if anyone ever tries these cards too.
After trying four or five different drivers (and having to disable driver signature checking every time + rebooting) so about 2 hours later... The card showed up under Device Manager with the proper name but still a yellow triangle.
(Again code 10...)
I installed WinOFED 2.3 (not the latest version) after extracting the MSI installer to XML and removing the check for 2008 R2 and repackaging it and 2 new NIC's showed up (OpenFabrics IPoIB Adapter 1 and 2).
After configuring the IP's for the ports and running opensm the first port came up and now handles traffic from one server to the other (and the other way).
So at least it is possible and I couldn't be more happy.
But as I've come this far, I want that second port to start working too...
I've figured out I need to configure opensm to run for both port GUIDs, but so far I've only found commands for Linux which as it is on Server 2012 R2 don't really help.
So if somebody here knows how to set opensm up please let me know.
Btw, the plan with ESXi was to pass the cards to a Solaris VM using Vt-D as Solaris would work with the cards and then bridge them to a virtual switch back to the other VM's, which was just a desperate attempt to get something out of these cards, but I never got to do it as ESXi refused to let me play with PCI passthrough So I don't know if ESXi will run these cards, Windows Server 2012 R2 will, with some effort and so far only 1 of 2 ports.