In diagnosing sleep disorders, sleep stage classification is a very essential yet time-consuming process. Most of the existing state-of-the-art approaches rely on hand-crafted features and multimodality polysomnography (PSG) data, where prior knowledge is compulsory and high computation cost can be expected. Besides, few studies are able to obtain high accuracy sleep staging using raw single-channel electroencephalogram (EEG). To overcome these shortcomings, this paper proposes an end-to-end framework with a deep neural network, namely SingleChannelNet, for automatic sleep stage classification based on raw single-channel EEG. The proposed model utilizes a 90s epoch as the textual input and employs two multi-convolution blocks and several max-average pooling layers to learn different scales of feature representations. To demonstrate the efficiency of the proposed model, we evaluate our model using different raw single-channel EEGs (C4/A1 and Fpz-Cz) on two different datasets (SHHS and Sleep-EDF datasets). Experimental results show that the proposed architecture can achieve better overall accuracy and Cohens kappa (SHHS: 89.2%-84.8%, Sleep-EDF: 89.4%-85%) compared with state-of-the-art approaches. Additionally, the proposed model can learn features automatically for sleep stage classification using different single-channel EEGs with distinct sampling rates from different datasets without using any hand-engineered features.