Identifying built environment barriers to walkability is the first step toward monitoring and improving our walking environment. Although conventional approaches (i.e., surveys by experts or pedestrians, walking interviews, etc.) to identify built environment barriers have contributed to improving the walking environment, these approaches may require time and effort. To address the limitations of conventional approaches, wearable sensing technologies and data analysis techniques have recently been adopted in the investigation of the built environment. Among various wearable sensors, an inertial measurement unit (IMU) can continuously capture gait-related data, which can be used to identify built environment barriers to walkability. To propose a more efficient method, the author adopts a cascaded bidirectional and unidirectional long short-term memory (LSTM)-based deep recurrent neural network (DRNN) model for classifying human gait activities (normal and abnormal walking) according to walking environmental conditions (i.e., normal and abnormal conditions). This study uses 101,607 gait data collected from the author’s previous study for training and testing a DRNN model. In addition, 31,142 gait data (20 participants) have been newly collected to validate whether the DRNN model is feasible for newly added gait data. The gait activity classification results show that the proposed method can classify normal gaits and abnormal gaits with an accuracy of about 95%. The results also indicate that the proposed method can be used to monitor environmental barriers and improve the walking environment.