We study the modulation instability in a three-component Bose-Einstein condensate and the impact of higher-order interactions brought on by shape-dependent confinement. Using the modified Gross-Pitaevskii equation that accounts for the higher-order residual nonlinearity and the linear stability method, we analyze the dispersion relations and the gain spectrum for instability. In particular, we investigate the role of higher-order interactions over the MI in miscible and immiscible condensates. The observed MI spectrum shows a new regime for soliton generation in a ternary mixture of Bose-Einstein condensates. Our results show that the residual nonlinearity can effectively control the MI gain profile in miscible and immiscible condensates. This allows one to manipulate the MI in a ternary
a mixture of Bose-Einstein condensates to control the dynamics.