The transition from internal combustion engine vehicles to electric vehicles (EVs) is gaining momentum due to their significant environmental and economic benefits. This study addresses the challenges of integrating renewable energy sources, particularly solar power, into EV charging infrastructures by using deep learning models to predict photovoltaic (PV) power generation and EV charging demand. The study determines the optimal battery energy storage capacity and charging schedule based on the prediction result and actual data. A dataset of a 15 kWp rooftop PV system and simulated EV charging data are used. The results show that simple RNNs are most effective at predicting PV power due to their adept handling of simple patterns, while bidirectional LSTMs excel at predicting EV charging demand by capturing complex dynamics. The study also identifies an optimal battery storage capacity that will balance the use of the grid and surplus solar power through strategic charging scheduling, thereby improving the sustainability and efficiency of solar energy in EV charging infrastructures. This research highlights the potential for integrating renewable energy sources with advanced energy storage solutions to support the growing electric vehicle infrastructure.