There is a problem that it is always difficult for picking and spraying robots to identify the objects for such factors such as leaf occlusion, similar color and others. In this paper, based on the citrus image data set collected from the actual orchard scene, image features are extracted and a mathematical model is constructed to identify and count the number of citruses in each image, and the distribution of apples in the whole data set is displayed. A model is constructed to be used to identify and count citrus via these images with diversity of citrus colors and the complexity of the background. Firstly, the yellow mature citruses are focused on using the HSV (Hue, Saturation, Value) method for the data sets. Secondly, the empirical HSV range for two citrus fruits, cyan and yellow, was defined, and contour processing is introduced to optimize the recognition model to process more citrus image datasets. Finally, such three types of objects as immature citrus, leaves with similar colors, and citrus covered by leaves are distinguish. The research results indicate that we adopt more detailed color threshold settings and morphological operations with the eccentricity of the contour is less than 0.85, and can distinguish citrus and leaves through image segmentation,and the recognition effect on citrus objects is improved.