With an increasing coupling between light and mechanics, nonlinearities begin to play an important role in optomechanics. We solve the quantum dynamics of an optomechanical system in the multi-photon strong coupling regime retaining nonlinear terms. This is achieved by performing a Schrieffer-Wolff transformation on the Hamiltonian including driving terms. The approach is valid away from the red-and blue-sideband drive. We show that the mechanical resonator displays self-sustained oscillations in regimes where the linearized model predicts instabilities, and that the amplitude of these oscillations is limited by the nonlinear terms. Related oscillations of the photon number are present due to frequency mixing of the shifted mechanical and cavity frequencies. This leads to additional peaks in the cavity's power spectral density. Furthermore, we show that it is possible to create phonon states with sub-Poissonian statistics when the system is red-detuned. This result is valid even with strong driving and with initial coherent states.By coupling light and mechanics, optomechanical systems enable control of light by mechanical motion and vice versa. This coupling of light to modes of a mechanical resonator is often achieved via radiation pressure. Its nature is intrinsically nonlinear but its strength is typically smaller than all other physical parameters [1], undermining the significance of nonlinear effects. The majority of the observed physical phenomena can be thus understood using a linear description.The achievement of cooling a mechanical mode to its ground state [2,3] paves the way to state preparation and it increased the interest in quantum effects in mechanical motion, specifically creation of non-classical mechanical states. Such states can only be created in a nonlinear system, shifting the focus of attention to the single-photon strong coupling (SPSC) regime, where quantum dynamics and properties have been analyzed theoretically. The exact solution for the isolated system was discovered [4,5]. Dissipation, noise and the coherent drive have also been investigated by including them in the equations of motion and treating them as perturbations to the exact solution [6,7]. For this weak driving regime, it was found that mechanical states with sub-Poissonian statistics can be created in the SPSC regime. This conclusion was corroborated by numerical simulations [8][9][10]. Other works reported similar quantum states [11-13] over some parameter regions in the SPSC regime, and signalled a connection to the self-sustained oscillations present in the nonlinear regime. Experimentally, the SPSC regime remains a challenge, though considerable progress has been made.Higher single-photon coupling strengths were already obtained in a variety of setups where the single-photon coupling surpasses the mechanical frequency Ω [14] (or is close to it [15]). However, in these setups, the cavity linewidth exceeds the single-photon coupling. Alternatively, the interaction can be enhanced by coherently driving the system. The d...