This paper presents the anomalous dispersion characteristic of a periodic substrate integrated waveguide (SIW) with results obtained by a rigorous formulation using the hybrid generalized scattering matrix (GSM) and finite-element method (FEM), showing two types of anomalous modal behaviors, namely: 1) surface wave leakage in connection to the near cutoff transverse resonance of a rectangular waveguide section alternating with either microstrip or parallel sections and 2) complex modes inside the lowest -like propagating zone. A rigorous limiting case study of the SIW via spacing approaching less than 10% of via diameter validated the proposed hybrid GSM/FEM formulation and enabled the detailed description of various types of modes, namely, propagating -like, a pair of complex modes, and above the stopband regimes, respectively. Convergence analysis and comparisons to the existing literature were carried out to validate the proposed method in addition to the limiting case study. Measured results of the CMOS prototypes validate the theoretical prediction from 325 to 450 GHz and confirm the accuracy of the proposed method.