This paper is dedicated to the study of the influence of surface roughness on local stress and strain fields in polycrystalline aggregates. Finite element computations are performed with a crystal plasticity model on a 316L stainless steel polycrystalline material element with different roughness states on its free surface. The subsequent analysis of the plastic strain localization patterns shows that surface roughness strongly affects the plastic strain localization induced by crystallography. Nevertheless, this effect mainly takes place at the surface and vanishes under the first layer of grains, which implies the existence of a critical perturbed depth. A statistical analysis based on the plastic strain distribution obtained for different roughness levels provides a simple rule to define the size of the affected zone depending on the rough surface parameters.