The binder–aggregate adhesiveness plays an important role in preventing asphalt concrete adhesive fracture and it is greatly influenced by the chemical, physical, morphological, and mineralogical properties of aggregates. Many studies have attempted to assess fracture resistance of asphalt mixtures by evaluating parameters found from different laboratory fracture tests. Those parameters are related to the mixture components, and they can change when moisture conditions are induced. This study aims to investigate the effects of aggregate/filler characteristics on mixture fracture resistance including the effect of moisture damage. Two aggregates and four fillers with varied chemical compositions were selected and combined with one unmodified asphalt binder to design five different asphalt mixtures. Each aggregate and filler was tested to characterize physical, chemical, and morphological properties. Asphalt concrete mixtures samples, with and without being subjected to moisture conditioning, were submitted to the semi-circular bend (SCB) fracture test. Statistical analyses were performed to determine the level of significance on the parameter change resulting from the properties of particles and moisture conditioning. The results indicated that the type of aggregates did not present significant influence on the SCB fracture results when moisture conditioning was not involved. Also, the type of filler presented different trends on the parameter’s change when the samples were subjected to moisture. However, it was observed that the effects of aggregates and fillers on the fracture resistance of asphalt mixtures were pronounced when the SCB samples were subjected to moisture conditioning, because of the material-specific binder–aggregate adhesion in the presence of moisture.