Automatic and accurate esophageal lesion classification and segmentation is of great significance to clinically estimate the lesion status of esophageal disease and make suitable diagnostic schemes. Due to individual variations and visual similarities of lesions in shapes, colors and textures, current clinical methods remain subject to potential high-risk and time-consumption issues. In this paper, we propose an Esophageal Lesion Network (ELNet) for automatic esophageal lesion classification and segmentation using deep convolutional neural networks (DCNNs). The underlying method automatically integrates dual-view contextual lesion information to extract global features and local features for esophageal lesion classification of four esophageal image types (Normal, Inflammation, Barrett, and Cancer) and proposes lesion-specific segmentation network for automatic esophageal lesion annotation of three esophageal lesion types at pixel level. For established clinical large-scale database of 1051 white-light endoscopic images, ten-fold cross-validation is used in method validation. Experiment results show that the proposed framework achieves classification with sensitivity of 0.9034, specificity of 0.9718 and accuracy of 0.9628, and the segmentation with sensitivity of 0.8018, specificity of 0.9655 and accuracy of 0.9462. All of these indicate that our method enables an efficient, accurate and reliable esophageal lesion diagnosis in clinical.The main contributions of our work can be generalized as follows: 1 For the first time, proposed ELNet enables an automatically and reliably comprehensive esophageal lesions classification of four esophageal lesion types (Normal, Inflammation, Barrett, and Cancer) and lesion-specific segmentation from clinically white-light esophageal images to make suitable and repaid diagnostic schemes for clinicians. 2 A novel Dual-Stream network (DSN) is proposed for esophageal lesion classification. DSN automatically integrates dual-view contextual lesion information using two CNN streams to complementarily extract the global features from the holistic esophageal images and the local features from the lesion patches. 3 Lesion-specific esophageal lesion annotation with Segmentation Network with Classification (SNC) strategy is proposed to automatically annotate three lesion types (Inflammation, Barrett, Cancer) at pixel level to reduce the intra-class differences of esophageal lesions. 4 A clinically large-scale database esophageal database is established for esophageal lesions classification and segmentation. This database includes 1051 white-light esophageal images, which consists of endoscopic images in four different lesion types. Each image in this database has a classification label and its corresponding segmentation annotation.