Gear disengaging, back-side tooth contact or poor dynamic behavior during operating leads to dynamic instability in planetary gear trains (PGTs). A novel nonlinear dynamic model of PGTs with internal and external gear pairs considering multi-state engagement induced by backlash and contact ratio is established. An improved time-varying meshing stiffness model including temperature stiffness is analytically derived. The time-varying meshing stiffness with temperature effect, friction, backlash, time-varying pressure angle, and time-varying friction arm are integrated into the dynamic model of PGTs. Multi-state engaging behavior is efficiently identified by constructing different Poincaré mappings. A method to calculate dynamic instability is proposed in the time-domain trace. The intrinsic relationship between multi-state engaging and dynamic instability is investigated via multi-section bifurcation plots and phase trajectory topology. The global dynamic instability is revealed based on the bifurcation and evolution of coexistence behavior under the parameter-state synergy. The results show that the multistate engagement is heavily depending on bifurcation and phase trajectory topology, which whereby affects the dynamic instability. Two special phenomena, complete and incomplete bifurcations, are discovered under parameter-state synergy. Complete bifurcation causes global instability and incomplete bifurcation results in local instability and yields coexistence responses. Incomplete bifurcation brings about new bifurcation branches.