Rotary forming is an exciting route for forming flanges of different angles from seamless tubes of high-strength materials. True to its incremental nature, the process offers great flexibility, but the issues encountered are atypical and complex. One such issue observed in experimental trials is the internal buckling of tubes during specific instances of flange formation. The origin of this instability is non-trivial, and ordinarily, finite-element (FE) models fail to capture this instability. To analyse and understand the problem, systematic experimental trials were carried out using different tube thicknesses, tube materials and tool kinematics. This paper summarises the results from a critical analysis to establish (1) a criteria for quantifying the instability and identifying the instances of its occurrence, (2) a validation methodology to fine-tune FE models for the process, and (3) use of FE models to understand the influence of tool path in the flange forming stage.