We present a detailed analysis of the orbital stability of the Pais-Uhlenbeck oscillator, using Lie-Deprit series and Hamiltonian normal form theories. In particular, we explicitly describe the reduced phase space for this Hamiltonian system and give a proof for the existence of stable orbits for a certain class of self-interaction, found numerically in previous works, by using singular symplectic reduction.