Recently, there has been a trend among pavement engineers and scientists to utilize natural mineral resources found in different parts of the world to develop and enhance sustainable infrastructure. One of such materials is calcined clay. However, the fatigue performance of asphalt mixtures made with these alternative materials needs to be properly studied. Due to its complex composition, asphalt concrete exhibits diverse non-linear characteristics when subjected to different conditions. As such, the impact of rest time, considering the effect of hardening relaxation and healing on the creep recovery of asphalt mixtures modified with calcined marl filler (CMF), has been evaluated in this present study. Thus, the locally sourced marl was pulverized and calcined to produce CMF. Different amounts of CMF were added to asphalt mixtures as a mineral filler, ranging from 0% to 100% by weight. Afterwards, the rheological properties of CMF mastic using a dynamic shear rheometer (DSR) were investigated. Notably, the outcomes of the experiments revealed compelling insights. Specifically, under the influence of 50% CMF modification, the asphalt mixtures exhibited a remarkable rutting resistance, with values reaching 12.7 kPa for unaged conditions and 16.1 kPa for aged conditions. Additionally, the results underscored an enhancement in the low-temperature characteristics of the bitumen mastic, which consequently contributed to heightened resistance against fatigue-induced damage. Furthermore, the statistical analysis, such as the student t-test, deployed to compare the creep recovery with and without rest time indicated that the creep recovery changes with the application of rest time. Hence, at long rest times, the hardening relaxation behavior reduces and the chances of healing increase, leading to a decrease in the amount of deformation in the samples.