Combined with the unique processing technology of oolong tea, oolong tea with stem processing has a better flavor compared to oolong tea without stem processing. However, there is currently no available evidence to support the contribution of stems to the taste quality of oolong tea. In this study, the electronic tongue, sensory evaluation method combined with liquid chromatography, and gas chromatography–mass spectrometry were used to explore the influence of the presence of stems on the flavor substances and aroma of oolong tea during processing. The results showed that the presence of stems significantly increased the umami taste of oolong tea (p < 0.05), and the content of seven free amino acids (p < 0.05), including theanine (53.165 μg/mL) and aspartic acid (3.190 μg/mL), two umami-related amino acids, significantly increased. Moreover, the content of nerolidol (2.598 μg/g) in aroma components was significantly increased. This study identifies the contribution of stems to oolong tea quality during processing.