To maintain a reasonable sediment regulation system in the middle reaches of the Yellow River, it is critical to determine the variation in sediment deposition behind check‐dams for different soil erosion conditions. Sediment samples were collected by using a drilling machine in the Fangta watershed of the loess hilly–gully region and the Manhonggou watershed of the weathered sandstone hilly–gully (pisha) region. On the basis of the check‐dam capacity curves, the soil bulk densities and the couplet thickness in these two small watersheds, the sediment yields were deduced at the watershed scale. The annual average sediment deposition rate in the Manhonggou watershed (702.0 mm/(km2·a)) from 1976 to 2009 was much higher than that in the Fangta watershed (171.6 mm/(km2·a)) from 1975 to 2013. The soil particle size distributions in these two small watersheds were generally centred on the silt and sand fractions, which were 42.4% and 50.7% in the Fangta watershed and 60.6% and 32.9% in the Manhonggou watershed, respectively. The annual sediment deposition yield exhibited a decreasing trend; the transition years were 1991 in the Fangta watershed and 1996 in the Manhonggou watershed (P < 0.05). In contrast, the annual average sediment deposition yield was much higher in the Manhonggou watershed (14011.1 t/(km2·a)) than in the Fangta watershed (3149.6 t/(km2·a)). In addition, the rainfalls that induced sediment deposition at the check‐dams were greater than 30 mm in the Fangta watershed and 20 mm in the Manhonggou watershed. The rainfall was not the main reason for the difference in the sediment yield between the two small watersheds. The conversion of farmland to forestland or grassland was the main reason for the decrease in the soil erosion in the Fangta watershed, while the weathered sandstone and bare land were the main factors driving the high sediment yield in the Manhonggou watershed. Knowledge of the sediment deposition process of check‐dams and the variation in the catchment sediment yield under different soil erosion conditions can serve as a basis for the implementation of improved soil erosion and sediment control strategies, particularly in semi‐arid hilly–gully regions. Copyright © 2018 John Wiley & Sons, Ltd.