Optical spectroscopy techniques represent a powerful tool to study the vibrational modes of nano-objects and address both their intrinsic properties (related to nanoparticle morphology, crystallinity, and local environment) and their interaction with light, which plays a key role for nanophotonics and optomechanics applications. In this paper, we report lowfrequency inelastic light scattering measurements on single gold nanocrystals, combined with detailed characterizations of their morphology and environment using transmission electron microscopy and tomography, and numerical simulations of their optical and vibrational responses. Our studies confirm the conclusions of previous ensemble measurements regarding the large impact of the nanoparticle crystallinity on inelastic scattering spectra. Moreover, their single-particle character allows us to address subtle shape and environment effects. We show that slight nanoparticle shape anisotropies induce further splitting of Raman-active modes. We also demonstrate that local environment anisotropies lead to richer inelastic light scattering spectra and allow the detection of vibrational modes predicted to be Ramaninactive by usual Raman selection rules, which are based on the (here invalid) assumption of a symmetric internal electric field.