A transfer matrix technique is used to model phase coherent spin transport in the weakly disordered quasi one-dimensional channel of a gate-controlled electron spin interferometer [Datta and Das, Appl. Phys. Lett., 56, 665 (1990)]. It includes the effects of an axial magnetic field in the channel of the interferometer (caused by the ferromagnetic contacts), a Rashba spin-orbit interaction, and elastic (non-magnetic) impurity scattering. We show that in the presence of the axial magnetic field, non-magnetic impurities can cause spin relaxation in a manner similar to the Elliott-Yafet mechanism. The amplitudes and phases of the conductance oscillations of the interferometer, and the degree of spin-conductance polarization, are found to be quite sensitive to the height of the interface barrier at the contact, as well as the strength, locations and nature (attractive or repulsive) of just a few elastic non-magnetic impurities in the channel. This can seriously hinder practical applications of spin interferometers.