In order to calculate photonic devices with slowly varying waveguide structure along propagation direction, we develop finite element beam propagation method (FE-BPM) with coordinate transformation. In this approach, converting a longitudinally varying waveguide into the equivalent straight waveguide, cumbersome processes in FE-BPM, such as mesh updating and field interpolation processes at each propagation step, can be avoided. We employ this simulation technique in shape optimization of photonic devices and show design examples of mode converter. To show the validity of this approach, the calculated results of designed devices are compared with the finite element method (FEM) or the standard FE-BPM.