Traditional boring dynamic model ignores the effect of the uneven radial stiffness of the shaft hole of the thin-walled box on the boring vibration characteristic, which will lead to the deviation in the prediction for both the cutting force and the surface topography of the machining hole. To address this shortcoming, a new calculation model of the equivalent radial stiffness of the shaft hole is proposed and then validated by the finite element (FE) method. On this basis, a boring dynamic model of the shaft hole is developed, and an experimental study on the dynamic characteristics of the shaft hole boring is carried out to verify the correctness and effectiveness of the proposed dynamic model. Moreover, a prediction method for the surface topography error of the machined shaft hole is presented based on the proposed dynamic model. Finally, the comparison investigation on the hole machined surface topography predicted by the proposed method and measured in practice is carried out. The results show that the proposed theoretical models and method can achieve accurate predictions for both the boring cutting force and the machined surface topography error of the shaft hole on the thin-walled box. This study can provide a theoretical basis for machining error control and process optimization of the fine boring of the thin-walled transmission box.INDEX TERMS Thin-walled box, shaft hole machining, boring dynamic model, surface topography prediction.