How the complexity or irregularity of heart rate variability (HRV) changes across different sleep stages and the importance of these features in sleep staging are not fully understood. This study aimed to investigate the complexity or irregularity of the RR interval time series in different sleep stages and explore their values in sleep staging. We performed approximate entropy (ApEn), sample entropy (SampEn), fuzzy entropy (FuzzyEn), distribution entropy (DistEn), conditional entropy (CE), and permutation entropy (PermEn) analyses on RR interval time series extracted from epochs that were constructed based on two methods: (1) 270-s epoch length and (2) 300-s epoch length. To test whether adding the entropy measures can improve the accuracy of sleep staging using linear HRV indices, XGBoost was used to examine the abilities to differentiate among: (i) 5 classes [Wake (W), non-rapid-eye-movement (NREM), which can be divide into 3 sub-stages: stage N1, stage N2, and stage N3, and rapid-eye-movement (REM)]; (ii) 4 classes [W, light sleep (combined N1 and N2), deep sleep (N3), and REM]; and (iii) 3 classes: (W, NREM, and REM). SampEn, FuzzyEn, and CE significantly increased from W to N3 and decreased in REM. DistEn increased from W to N1, decreased in N2, and further decreased in N3; it increased in REM. The average accuracy of the three tasks using linear and entropy features were 42.1%, 59.1%, and 60.8%, respectively, based on 270-s epoch length; all were significantly lower than the performance based on 300-s epoch length (i.e., 54.3%, 63.1%, and 67.5%, respectively). Adding entropy measures to the XGBoost model of linear parameters did not significantly improve the classification performance. However, entropy measures, especially PermEn, DistEn, and FuzzyEn, demonstrated greater importance than most of the linear parameters in the XGBoost model.300-s270-s.