This investigation delineates an innovative approach to fortify the secure key exchange process by integrating the robustness of the RSA algorithm with the unpredictability of a chaotic system, thereby advancing the security framework for color image encryption. Within this scheme, encryption keys are derived from a chaotic system, the initial conditions of which are dynamically modulated by the delta feature extracted from the source image. Such a design ensures that the system's behavior inherently adapts to the input image. The initial values and parameters governing the five-dimensional chaotic system are securely transmitted from sender to recipient via the RSA algorithm. Subsequently, diffusion and confusion processes are orchestrated through the application of two uniquely computed key matrices, which operate on the image at the column and row levels, respectively. This mechanism is instrumental in altering pixel values throughout the image. Performance evaluation of the proposed algorithm is quantified by several metrics: a high Number of Pixels Change Rate (NPCR) value of 99.621% illustrates its efficacy in pixel value modification, while a Peak Signal-to-Noise Ratio (PSNR) value of 8.898 implies the retention of image quality post-encryption. Furthermore, an Unified Average Changing Intensity (UACI) value of 33.823% signifies the algorithm's proficiency in introducing substantial variations in pixel intensities. The results corroborate the algorithm's competency in encrypting color images, underpinning its utility in diverse applications that necessitate stringent data and image protection measures against unauthorized access.