Handwritten recognition model characters have been reported by various researchers in past years and it becomes the most challenging day by day in this age of the digital world. Many machine learning algorithms have a great influence over the handwritten recognition system. In this segment, deep learning is one of the enhanced techniques to solve pattern recognition problems. Such complex problem we have imposed with convolutional neural network (CNN) for the handwritten characters. In this paper, we have made an attempt to build a recognition system for multi-scripts such as Odia and Bangla scripts. Here we have defined the datadriven learning mechanism of CNN along with deriving the discriminate features of handwritten images. This light-weighted CNN provides a feasible solution and reports a high recognition rate. Such deep learning-based approach is a new state of art method for developing an automatic recognition model and to meet real-time challenges.