Nowadays, applications of turbulent fluid flow in removing high heat flux in rib-roughened narrow channels are drawing much interest. In this work, an improved version of the κ-ε turbulence model is proposed for better prediction of thermal–hydraulic characteristics of flow inside rib-roughened (pitch-to-rib height (p/k) ratio = 10 and 20) narrow channels (channel height, H = 1.2 mm and 3.2 mm). For this, the four turbulence model parameters, Cμ, Cε1, Cε2, and σk, are calibrated. These parameters are adjustable empirical constants provided for controlling the accuracy of the turbulence model results when needed. The simulated data are used to develop correlations between the relative errors in predicting the friction factor (f), Nusselt number (Nu), and the model parameters using a multivariate nonlinear regression method. These correlations are used to optimize the errors using genetic algorithm. Results reveal that the calibrated parameters are not the same for all the narrow channel configurations. After calibration, the overall predictive improvements are up to 35.83% and 27.30% for p/k = 10 and p/k = 20 respectively when H = 1.2 mm. Also, up to 15.48% and 18.05% improvements are obtained for p/k = 10 and p/k = 20 respectively when H = 3.2 mm. The role of the two parameters Cε1 and Cε2 are found to be of primary importance. Furthermore, three types of nanofluids i.e. Al2O3-water, CuO-water, and TiO2-water are studied using the calibrated model to check the potentiality of heat transfer enhancement. Among them, CuO-water nanofluid is predicted to have around 1.32 times higher value of Nu than pure water for the same narrow channel configuration. Article Highlights κ-ε turbulence model is calibrated for rib-roughened narrow rectangular channels using genetic algorithm. Cε1 and Cε2 are the most influential parameters on the performance of the model inside rib-roughened narrow channel. Suggested calibration process is more effective for channel height of 1.2 mm than 3.2 mm.