In this paper, we present a detailed Raman study of the non-multiferroic compounds PrMnO
3 and NdMnO
3 and the multiferroic compounds TbMnO
3 and DyMnO
3 as a function of temperature and magnetic field. All studied systems show anomalous phonon shifts close to the Néel transition T
N. In PrMnO
3 and NdMnO
3, the frequency softenings are partly attributed to an orbital-spin-phonon coupling whereas in TbMnO
3 and DyMnO
3, the relatively weak frequency shifts are rather attributed to an expansion of the Mn−O bond lengths. On the other hand, the frequencies of TbMnO
3 phonons are shifted as a function of magnetic field, while those of PrMnO
3 remain unaffected. These frequency shifts are interpreted in terms of local oxygen rearrangements under magnetic field that could play an important role in the multiferroicity of TbMnO
3 and DyMnO
3.