We consider a generalization of the quantum Rabi model where the twolevel system and the single-mode cavity oscillator are coupled by an additional Starklike term. By adapting a method recently introduced by Braak [Phys. Rev. Lett. 107, 100401 (2011)], we solve the model exactly. The low-lying spectrum in the experimentally relevant ultrastrong and deep strong regimes of the Rabi coupling is found to exhibit two striking features absent from the original quantum Rabi model: avoided level crossings for states of the same parity and an anomalously rapid onset of two-fold near-degenerate levels as the Rabi coupling increases.