ESXi host with the queue problems has been in Maintenance Mode for 4h now and after I issue a "esxcli storage core adapter rescan --all" command, this pops out in the vmkernel.log file:
2014-01-03T14:10:11.389Z cpu56:14437)WARNING: LinScsi: SCSILinuxQueueCommand:1175:queuecommand failed with status = 0x1056 Unknown status vmhba_mlx4_0.1.1:0:1:6 (driver name: ib_srp) - Message repeated 561 times
2014-01-03T14:11:31.334Z cpu42:8234)ScsiDeviceIO: 2311: Cmd(0x4125c0c0b300) 0x28, CmdSN 0x2b from world 1067687 to dev "eui.3731346538376162" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0.
2014-01-03T14:11:31.366Z cpu42:8234)ScsiDeviceIO: 2311: Cmd(0x4125c084bf40) 0x28, CmdSN 0x2f from world 1067687 to dev "eui.3731346538376162" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x5 0x25 0x0.
2014-01-03T14:11:31.388Z cpu42:8234)ScsiDeviceIO: 2311: Cmd(0x4125c00e4fc0) 0x28, CmdSN 0x30 from world 1067687 to dev "eui.3731346538376162" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0.
2014-01-03T14:11:31.755Z cpu42:8234)ScsiDeviceIO: 2311: Cmd(0x4125c0d69000) 0x28, CmdSN 0x32 from world 1067687 to dev "eui.3632656331666463" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x5 0x25 0x0.
It looks like the ib_srp driver is stuck with some SCSI commands, which are retried even after the VM world isn't there.