Hi Bent,
ConnecX-s Pro EN or VPI support RoCE v2 (routable).
Could you point me to the documentations you saw about that.
SX1036 supports 1GbE (configurable). You will need to use QSA (40->10G adapter) and 10G->1G module.
About security, it is too big subject. it is very hard to recommend here.
I assume that some companies would use special hardware for that (e.g. CheckPoint)
Mellanox switches supports ACL (Access control) so you can limit the access to specific MACs/IPs if that helps.
In some cases I assume that the ISP supply some security services as well.
I don't see a problem with running FCoE and RoCE (v1 or v2) on the same switch.
you will need to enable PFC on the right priority. it could be the same priority or different priority for each of them.
Besides that the RoCE/FCoE layer is transparent to the switch.
Do you plan to use the switch as L3 router? how many subnets do you have in your planed network?
How many ports do you want to use in each server connected to the switches ? two? one for each switch?
It will be helpful if you could send/attach a network diagram.
Ophir.