Difference exists widely in cognition, behavior and psychopathology between males and females, while the underlying neurobiology is still unclear. As brain structure is the fundament of its function, getting insight into structural brain may help us to better understand the functional mechanism of gender difference. Previous structural studies of gender difference in Magnetic Resonance Imaging (MRI) usually focused on gray matter (GM) concentration and structural connectivity (SC), leaving cortical morphology not characterized properly. In this study a large dataset is used to explore whether cortical three-dimensional (3-D) morphology can offer enough discriminative morphological features to effectively identify gender. Data of all available healthy controls (N = 1113) from the Human Connectome Project (HCP) were utilized. We suggested a multivariate pattern analysis method called Hierarchical Sparse Representation Classifier (HSRC) and got an accuracy of 96.77% for gender identification. Permutation tests were used to testify the reliability of gender discrimination (p < 0.001). Cortical 3-D morphological features within the frontal lobe were found the most important contributors to gender difference of human brain morphology. Moreover, we investigated gender discriminative ability of cortical 3-D morphology in predefined Anatomical Automatic Labeling (AAL) and Resting-State Networks (RSN) templates, and found the superior frontal gyrus the most discriminative in AAL and the default mode network the most discriminative in RSN. Gender difference of surface-based morphology was also discussed. The frontal lobe, as well as the default mode network, was widely reported of gender difference in previous structural and functional MRI studies, which suggested that morphology indeed affect human brain function. Our study indicates that gender can be identified on individual level by using cortical 3-D morphology and offers a new approach for structural MRI research, as well as highlights the importance of gender balance in brain imaging studies.