In this paper, we present the formalism of the transverse operator method (TOM). A rigorous study of propagation in anisotropic and multi-layer medium using the tensor character of the permeability and permittivity is presented. With the application of the Galerkin method, the propagating modes in metallic rectangular waveguides filled with anisotropic metamaterial are exploited. The results are compared with those previously published and show a good agreement. The complex modes have been obtained.The advantages of the techniques used in this paper lie in the proper analytical formulation of the problem studied, on the one hand, and the speed of convergence, on the other hand. TOM offers a fast convergence of the propagation constant. This shows the effectiveness of our numerical model. As such, the formulation of the transverse operator could be a useful tool for microwave engineers. This type of materials known as metamaterial is widely used and needed by industries and information technology, especially in microwave and radiofrequency devices such as patch antennas, antennas waveguides, resonators, circulators, isolators, phase shifters, and filters.