Pilot mental fatigue is a growing concern in the aviation field due to its significant contributions to human errors and aviation accidents. Long work hours, sleep loss, circadian rhythm disruption, and workload are well‐known reasons, but there is a need to accurately detect pilot mental fatigue to improve aviation safety. However, due to the highly restricted cockpit environment and the complex nature of mental fatigue, feasible in‐flight detection remains under‐investigated. The objective of this study is to define a promising approach for mental fatigue detection based on psychophysiological measurements in flying‐relevant environments. Eleven participants engaged in a simulated flight experiment, where several conventional heart rate variability (HRV) and ocular indices were examined to study their relevance to mental fatigue. Additionally, a Toeplitz Inverse Covariance‐Based Clustering (TICC) method was performed to determine the ground truth, after which supervised machine learning was adopted to enable automated mental fatigue detection using HRV and eye metrics. Results showed that HRV and eye metrics were sensitive to the mental fatigue induced by prolonged flight‐relevant tasks. The TICC method helped determine the ground truth for mental fatigue and identify its three distinct levels. Furthermore, a supervised learning‐based detection of mental fatigue was achieved using a support vector machine with the greatest detection accuracy of 91.8%. The findings and methodology of this study provide new insights into the fatigue countermeasures in restricted cockpit environment and lay the groundwork for further explorations into the mental fatigue induced by prolonged flight missions.