The paper presents a mathematical model of the helical surface roughness Rz, obtained by the form-generating method with a to-size tool with a standard profile of cutting tool inserts. This machining method has maximum productivity with large pitches of the helix. The surface roughness of the auger flight is an important parameter since it works in a friction pair with an elastic element. The resulting dependence makes it possible to predict surface roughness in any of its sections, considering the helix curvature, depending on the feed and the cutting tool tip radius. The obtained dependence allows comparing the numerical solution with the linearized one obtained by finding the minimum distance from the intersection points of the tool edge circles at helical surface adjacent cutting points to the straight line connecting these adjacent points.