I ended up dumping my Infiniband experience to a blog post to hopefully try and help other people not make the same mistakes I did. Let me know if I missed anything or incorrect.
Introduction to Infiniband Pt.2 (Linux and vmware platforms) | Monster Network!