This article introduces effective techniques to improve the stopband frequency response of substrate integrated waveguide filters. First, a filter is constructed by using different cavity resonators which resonate at the same fundamental mode but at different high-order modes. A cascade connection of circular and rectangular cavities is used to separate high-order modes, because the second resonant mode of the circular cavities is more than rectangular ones. Second, inserting transverse radiating slots at the proper positions of cavities, their second longitudinal resonant modes are shifted down and effectively rejected. Third, four cells of slotted plane structure (SPS) are adopted to the top plane of the both input and output feed lines of the filter to reject the undesired passbands or improve steepness of the filter. Also, a combination of the radiation slots with SPS is used to reject the spurious responses accompanied by sharp steepness of the filter. Comparing between the measurement values and simulation results verifies the effectiveness of the proposed configurations and the hybrid technique.