The ground state and excitation gap are studied for the anisotropic quantum Rabi model (QRM) which connects the fundamental QRM and the Jaynes-Cummings model (JCM). While conventionally the ground state has a second-order quantum phase transition in the low frequency limit, turning on finite frequencies sheds a novel light on the phase diagram to illuminate a fine structure of first-order transition series. It is found that the conventional quantum phase transition is accompanied with a hidden symmetry breaking, whereas the emerging series transitions are topological transitions without symmetry breaking. The topological structure of the wave function provides a novel universality classification in bridging the QRM and the JCM among the diversity that arises from finite frequencies. The aspect of topological transitions provides a renewed insight for the role of the counter-rotating interaction. Moreover, it is shown that the conventionally established tricritical point is actually a pentacritical or hexacritical point and following this multicritical point emerges a series of quadruple points. Besides the emerging multicriticality and reformed universality, the result demonstrates that a single-qubit system can even exhibit analogs of topological phase transitions which traditionally occur in condensed matter.