We present a thermodynamically based approach to the design of models for viscoelastic fluids with stress diffusion effect. In particular, we show how to add a stress diffusion term to some standard viscoelastic rate-type models (Giesekus, FENE-P, Johnson-Segalman, Phan-Thien-Tanner and Bautista-Manero-Puig) so that the resulting models with the added stress diffusion term are thermodynamically consistent in the sense that they obey the first and the second law of thermodynamics. We point out the potential applications of the provided thermodynamical background in the study of flows of fluids described by the proposed models.The manuscript has been published as Dostalík, M. and Průša, V. and Skřrivan, T.: On diffusive variants of some classical viscoelastic rate-type models, . Unfortunately, the published version of this manuscript contains several minor misprints regarding the additive constant in the Helmholtz free energy ansatz for the FENE-P model. We are fully responsible for the misprints. The correct treatment of the FENE-P is discussed at the end of the manuscript.