This work demonstrates the fruit sorting improvement system architecture using the Faster-RCNN algorithm to identify fruits of varying size, colour selection, and segregation. The requirement is for a fruit sorting mechanism with accurate and precise detection with accurate placement capabilities. For this purpose, a 5 DOF robotic arm was built in the laboratory. For the control, an algorithm was developed using Denavit-Hartenberg (D-H) kinematic analysis model. The Image processing was implemented in OpenCV by adopting the Faster-RCNN image processing using NumPy and panda packages. Color clustering was employed to improve fruit detection, and segmentation techniques were used to focus more on the fruit image. The algorithm used was based on baseline architecture within relevant architecture, and with the results that are derived, it shows a great improvement from typical fruit detection systems using non-deep learning techniques, especially for position identification and the edge-to-edge estimation of the fruits. However, it was discovered that the time to detection was the same for all colours and weights, while the sorting duration varies with weight.