Stress relaxation tests under confining pressures of 1 and 5 MPa were performed on saturated silty mudstone obtained from Badong Formation strata in the Three Gorges Reservoir area of China. The effects of confining pressure on the relaxation stage, magnitude, degree, time and average relaxation rate of rocks were analyzed quantitatively according to these tests. The results show the relaxation stage of rocks without change, the relaxation magnitude and average relaxation rate of rocks increase and the relaxation degree and time of rocks decrease as the confining pressure increases. Then, a damage variable was proposed to consider the weakening of rock parameters. A new nonlinear damage model was established by introducing the damage variable into the Hooke-Kelvin model. The Levenberg-Marquardt algorithm was adopted to identify the model parameters of rocks. Based on the identification results, the influence of confining pressure on the viscosity of rocks is the largest, followed by the viscoelasticity; also, that on the instantaneous elasticity of rocks is the smallest. Increase in confining pressure can effectively decrease the damage degree of rocks induced by stress relaxation. Therefore, the confining pressure has significant influences on the stress relaxation behavior of silty mudstone, which should not be ignored in design and construction of major projects.