During the period from sowing and planting to harvesting, outdoor crops are directly affected by the natural environment, including wind, rain, frost, and sunlight. Under such circumstances, vegetables change their growth conditions, shape, and flexibility daily. We aimed to develop an agricultural work-support robot that automates monitoring, cultivation, disease detection, and treatment. In recent years, many researchers and venture companies have developed agricultural harvesting robots. In this study, instead of focusing on intensive harvesting operations, we focused on daily farm operations from the beginning of cultivation to immediately before harvest. Therefore, gripping and cutting are considered basic functions that are common to several routine agricultural tasks. To find the assumed objects from a camera image with a low computational load, this study focuses on branch points to detect and identify even if the stems, lateral branches, and axillary buds are swaying in the wind. A branch point is a characteristic part close to the working position, even when the wind blows. Therefore, we propose a method to detect the assumed branch points simultaneously and divide each branch point into the main stem, lateral branch, and axillary bud. The effectiveness of this method is demonstrated through experimental evaluations using three types of vegetables, regardless of whether their stems are swaying.