High percentage reclaimed asphalt pavement (RAP) is prevailing in pavement engineering for its advantages in sustainability and environmental friendliness, however, its fatigue resistance remains a major concern. Fine aggregate matrix (FAM) is a crucial part in the fatigue resistance of asphalt mixtures with high RAP content. Hence, the linear amplitude sweep (LAS) test of FAM has been developed to study the fatigue resistance of asphalt mixtures. However, the torsional loading mode of the LAS test with a dynamic shear rheometer (DSR) is a limitation to simulate traffic load. In this paper, an alternative LAS test for FAM with high RAP content was proposed. Beam FAM specimens were tested using a dual-cantilever flexural loading fixture in a dynamic mechanical analyzer (DMA). To investigate the influence of RAP content and the rejuvenating agent (RA), four kinds of FAM mixes were tested with this method to study their fatigue resistance. The test results suggested that the repeatability of this alternative approach was reliable. A fatigue failure criterion based on maximum C × N was defined. Then, fatigue life prediction models based on viscoelastic continuum damage (VECD) analysis were established according to the LAS test results and validated by a strain-controlled time sweep (TS) test. It turned out that as RAP content increased, the modulus of FAM would be significantly raised, accompanied with a drop in the phase angle. The fatigue life of FAM would be greatly shortened when the RAP binder replacement rate reached 50%. Adding RA could considerably improve the dynamic properties of FAM mixes with high RAP content, resulting in a decrease in modulus, increase in phase angle and elongating fatigue life, but could not recover to the level of virgin binder.