Alluvial-diluvial soil-rock mixture (ADSRM) is a special geological material, mainly composed of pebbles, gravel, and soils. Gradation design has a remarkable effect on improving the shear strength of ADSRM. Therefore, the N-method of gradation theory was applied to the gradation design of ADSRM fillers, and the influence of gradation on macro-meso shear properties of ADSRM samples was analyzed. Macro-mechanical test results show that the shear strength, the equivalent cohesion, and the friction angle of the graded sample are much higher than that of the ungraded sample, and the shear dilatancy of the graded sample is more obvious. Meso-shear properties indicate that the percentage of pore area of the graded sample is less than that of the ungraded sample, while the percentage of the number of the small pore to the total pore and the crushed rock blocks under the same normal stress are more than that of the ungraded sample. Pore abundance shows the shape of most pores on the shear plane is oblate, secondly, near-equiaxed, and the least long-strip. Finally, the percentage of pore area decreases gradually with the increase of the shear displacement, while the percentage of crushed rock block area increases first and then decreases, which can reflect the change of shear strength at different shear stages. The number of pores and broken rock blocks mainly affect the strength and equivalent cohesion of the filler, while the abundance, roundness, shape coefficient, and fractal dimension of rock block contour mainly affect the internal friction angle.