This issue is solved finally. There were two problems that are needed to be fixed:
1) Openstack icehouse documentation (Configure network node - OpenStack Installation Guide for Red Hat Enterprise Linux, CentOS, and Fedora …) asks the users to do following on network node:
Due to a packaging bug, the Open vSwitch agent initialization script explicitly looks for the Open vSwitch plug-in configuration file rather than a symbolic link /etc/neutron/plugin.ini pointing to the ML2 plug-in configuration file. Run the following commands to resolve this issue:
# cp /etc/init.d/neutron-openvswitch-agent /etc/init.d/neutron-openvswitch-agent.orig
# sed -i 's,plugins/openvswitch/ovs_neutron_plugin.ini,plugin.ini,g' /etc/init.d/neutron-openvswitch-agent
But, this is not required as it causes the neutron-openvswitch-agent to not load the correct config file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). neutron-openvswitch-agent must load config from this file. Also, it looks like they have wrongly fixed it in the icehouse repository. So now, the neutron-openvswitch-agent service script always points to the wrong file on Linux.
2) Mellanox documentation (Mellanox-Neutron-Icehouse-Redhat - OpenStack) asks users to edit /etc/neutron/dhcp_agent.ini to set "interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver". But this should be set to "neutron.agent.linux.interface.OVSInterfaceDriver" otherwise a tap port is not created under br-int for the DHCP process
After making the above changes, my VM instances are getting IPs assigned