Did you check the MLNX_OFED User Manaul?
http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers
Topology example is depends what do you want to check,
you can start with two servers equipped with Mellanox adapters, connected via a switch or back to back.
Ophir.