A dynamical analysis of a Mathieu-van der Pol-Duffing nonlinear system with fractional-order derivative under combined parametric and forcing excitation is studied in this paper. The approximate analytical solution is researched for 1/2 sub harmonic resonance coupled with primary parametric resonance based on the improved averaging approach. The steady-state periodic solution including its stability condition is established. The equivalent linear stiffness coefficient (ELDC) and equivalent linear damping coefficient (ELSC) for this nonlinear fractional-order oscillator are defined. Then, the numerical simulations are presented in three typical cases by iterative algorithms. The time history, phase portrait, FFT spectrum and Poincare maps are shown to explain the system response. Some different responses, such as quasi-periodic, multi-periodic and single periodic behaviors are observed and investigated. The results of comparisons between the numerical solutions and the approximate analytical solutions in three typical cases show the correctness of the analytical solutions. The influences of the fractional-order parameters on the system dynamical response are researched based on the ELDC and ELSC. Through analysis, it could be found that the increase of the fractional-order coefficient would result in the rightward and downward movements of the amplitude-frequency curves. The increase of the fractional-order coefficient will also move the bifurcation point rightwards and will make the existing range of steady-state solution larger. It could also be found that the ELSC will become larger and ELDC smaller when the fractional order is closer to zero, so that the decrease of the fractional order would make the response amplitude larger. At last, the detailed conclusions are summarized, which is beneficial to design and control this kind of fractional-order nonlinear system.