Re: How to tune mlx4_core various parameters found in /sys/module/mlx4_core/parameters for optimal performance

Yari,

Thanks for the pointer. I looked at the the tuning guide, it is good, but targeted for ethernet over IB performance and doesn't delve much into details of IB driver settings, although there is a section about autotuning. 

But you may have some more, see  the list of params here for kernel modules:

We see that tuning/increasing the default params in /sys/module/mlx4_core/parameters resolve issues, we even see some performance boost for our use case. (when we increase log_num_mtt, log_mtts_per_seg).

We really need to understand how to obtain max performance tuning these and other driver params. I doubt the auto-tuners would resolve it for us..

ll /sys/module/mlx4_core/parameters/

-rw-r--r-- 1 root root 4096 Mar 11 17:23 block_loopback

-rw-r--r-- 1 root root 4096 Mar 11 17:23 debug_level

-rw-r--r-- 1 root root 4096 Mar 11 17:23 enable_pre_t11_mode

-r--r--r-- 1 root root 4096 Mar 11 17:23 enable_qinq

-r--r--r-- 1 root root 4096 Mar 11 17:23 enable_qos

-r--r--r-- 1 root root 4096 Mar 11 17:23 fast_drop

-r--r--r-- 1 root root 4096 Mar 11 17:23 high_rate_steer

-rw-r--r-- 1 root root 4096 Mar 11 17:23 internal_err_reset

-r--r--r-- 1 root root 4096 Mar 11 17:23 log_mtts_per_seg

-r--r--r-- 1 root root 4096 Mar 11 17:23 log_num_cq

-r--r--r-- 1 root root 4096 Mar 11 17:23 log_num_mac

-r--r--r-- 1 root root 4096 Mar 11 17:23 log_num_mcg

-r--r--r-- 1 root root 4096 Mar 11 17:23 log_num_mgm_entry_size

-r--r--r-- 1 root root 4096 Mar 11 17:23 log_num_mpt

-r--r--r-- 1 root root 4096 Mar 11 17:23 log_num_mtt

-r--r--r-- 1 root root 4096 Mar 11 17:23 log_num_qp

-r--r--r-- 1 root root 4096 Mar 11 17:23 log_num_srq

-r--r--r-- 1 root root 4096 Mar 11 17:23 log_rdmarc_per_qp

-r--r--r-- 1 root root 4096 Mar 11 17:23 msi_x

-r--r--r-- 1 root root 4096 Mar 11 17:23 pfcrx

-r--r--r-- 1 root root 4096 Mar 11 17:23 pfctx

-r--r--r-- 1 root root 4096 Mar 11 17:23 probe_vf

-r--r--r-- 1 root root 4096 Mar 11 17:23 set_4k_mtu

-r--r--r-- 1 root root 4096 Mar 11 17:23 sr_iov

-r--r--r-- 1 root root 4096 Mar 11 17:23 use_prio

ll /sys/module/mlx4_en/parameters

total 0

-r--r--r-- 1 root root 4096 Mar 11 17:40 enable_sys_tune

-r--r--r-- 1 root root 4096 Mar 11 17:40 inline_thold

-r--r--r-- 1 root root 4096 Mar 11 17:40 mem_node

-r--r--r-- 1 root root 4096 Mar 11 17:40 num_rx_rings

-r--r--r-- 1 root root 4096 Mar 11 17:40 udp_rss

Thanks,
Mous.