Strengthening by grain refinement via the Hall–Petch mechanism and softening by nanograin formation via the inverse Hall–Petch mechanism have been the subject of argument for decades, particularly for ultrafine-grained materials. In this study, the Hall–Petch relationship is examined for ultrafine-grained magnesium, aluminum, copper, and iron produced by severe plastic deformation in the literature. Magnesium, aluminum, copper, and their alloys follow the Hall–Petch relationship with a low slope, but an up-break appears when the grain sizes are reduced below 500–1000 nm. This extra strengthening, which is mainly due to the enhanced contribution of dislocations, is followed by a down-break for grain sizes smaller than 70–150 nm due to the diminution of the dislocation contribution and an enhancement of thermally-activated phenomena. For pure iron with a lower dislocation mobility, the Hall–Petch breaks are not evident, but the strength at the nanometer grain size range is lower than the expected Hall–Petch trend in the submicrometer range. The strength of nanograined iron can be increased to the expected trend by stabilizing grain boundaries via impurity atoms. Detailed analyses of the data confirm that grain refinement to the nanometer level is not necessarily a solution to achieve extra strengthening, but other strategies such as microstructural stabilization by segregation or precipitation are required.