This study aims to investigate the impact of various factors, such as wall shape, Rayleigh number, and volume fraction of nanoparticles, on natural convection in a square cavity that is filled with a mixture of Al2O3 solid particles and liquid water. The research employs numerical simulations based on the radial basis function meshless method and the artificial compressibility technique. The results of the study showed that the temperature distribution in the cavity was mostly uniform, except in the vicinity of the hot wall, while the flow was primarily dominated by convection as the Rayleigh number increased. Furthermore, the heat transfer rate increased with the volume fraction of nanoparticles, indicating the significance of nanoparticles in improving the thermal performance of the system. Additionally, the study found that the average Nusselt number, which characterizes the heat transfer efficiency, was highest when the cavity had a wavy wall. For single and double wavy walls, there were respective enhancements of 32% and 6% compared to a regular wall. Additionally, the Nusselt number increased as the volume fraction of nanoparticles, indicating a significant influence of nanoparticle concentration and wall geometry on the fluid flow and heat transfer characteristics in the square cavity. Consequently, this study's outcomes provide crucial insights into designing and optimizing thermal management systems, particularly those utilizing nanofluids.