A self-consistent nonlinear theory of a relativistic electron beam propagating through a waveguide loaded with a sheath helix is developed by making use of Maxwell equations. A closed integrodifferential equation is obtained for the normalized beam current described in terms of the normalized time θ and propagation distance ζ. An analytical investigation of the partial integrodifferential equation of the current modulation is carried out by Fourier decomposing the current modulation with harmonic number ℓ. The mode strength cl(ζ) is obtained in terms of the spatial oscillation frequency ηl and growth rate ξl of the mode amplitude, which are, in turn, determined in terms of the electrostatic and helix effects. The spatial oscillation frequency increases as the electrostatic effect (gl) increases. On the other hand, the growth rate increases with the strength of the helix effect. It is found that the mode strength cl is oscillatory and that it grows during propagation when the spatial frequency ηl (the electrostatic effect) is considerably larger than the growth rate ξl (the helix effect). Otherwise, the exponential growth dominates. Investigation of the helix mode must include the electrostatic effect for an intense beam.