Accessing acoustic phonons at high frequencies in nanostructures becomes more and more essential in nanoelectronics, nano-and opto-mechanics and quantum technologies, as phonons can strongly interact with electrons and photons at the nanoscale. In spontaneous Brillouin scattering processes, the scattered photons energy, direction and polarisation are constrained by selection rules for a given input state. These selection rules are usually considered as intrinsic material properties in crystalline solids and the polarisation of the scattered photons depends on the polarisation of the excitation. In this work, we use elliptical optophononic micropillar resonators to control these optical polarisation selection rules. The degeneracy of the optical cavity modes of circular micropillars is lifted due to the elliptical cross-section of the micropillars, leading to two cavity modes orthogonally polarised and split in energy. The optical field polarisation state will depend on both orthogonal cavity modes and their associated polarisation states. Therefore, an incident laser beam linearly polarised along the diagonal axis of the elliptical pillar undergoes a wavelength dependent polarisation rotation. By choosing the polarisation and wavelength of the incident laser, we demonstrate that the polarisation state of the incident and reflected laser and the Brillouin scattering signal are different. In this way, background-free spontaneous Brillouin scattering spectra can be efficiently measured in a cross-polarisation scheme down to 18 GHz. Here, we theoretically and experimentally explore the optimal conditions for the polarisation and wavelength of the incident laser, and the ellipticity of the micropillars, to improve the polarisation-based filtering applied to Brillouin spectroscopy.