A novel cascade microstrip filter with mixed electric and magnetic coupling is proposed and examined. For the first time, we designed a novel multilayer configuration resonator with a compact size. Compared with conventional open-loop filter, the size decreased 75%. First, using novel multilayer configuration resonator instead of monolayer configuration resonator, the size is decreased about 45%. Meanwhile, two microstrip patches serve as inner coupling capacitor, which create and strengthen electric field. Owing to this inner coupling, the size of filter can be further reduced. Moreover, with a section of high characteristic impedance transmission line connected to the ground though a via-hole, it can serve as coupling inductor and create main magnetic field. Based on this structure, mixed EM coupling is realized and can generate additional transmission zero, which means that adopting low-order filter can achieve the same characteristic as high-order filter. Based on the above solutions, the filter size can be sharply decreased to 12.5 mm × 7.7 mm (i.e., 0.05λ g × 0.033λ g). Furthermore, advantages of using this type filter are not only low insert loss, but also increased attenuation out-of-band with controllable transmission zeros.