In order to overcome the disadvantages of heat transfer performance in the shell side of the common circular cross section rod baffle heat exchanger with a low Reynolds number, a numerical simulation on fluid flow and heat transfer in the shell side with different types of rod baffles is carried out. The rod baffles include the circular cross section, trigonal cross section, and rhombic cross section. The influence of heat transfer enhancement and flow resistance reduction affected by baffles is summarized. It is indicated that the trigonal and rhombic cross section rod baffles present the better performance of heat transfer enhancement and flow resistance reduction. With the rhombic cross section rod baffles in the shell side, the higher heat transfer coefficient and overall property in the shell side are achieved when Re is lower, and the heat transfer coefficient in the shell side is 10% higher than that of a circular cross section rod baffle at the same Reynolds number. The trigonal and rhombic cross section rod baffles in the shell side give more optional structure forms for expanding the application scope of rod baffle heat exchangers.