Pavement management, which is vital in road transportation and maintenance, is facing some troubles, such as high costs of labors and machineries, low detecting efficiency, and low update rate of pavement conditions by means of traditional detection ways. Benefiting from the development of mobile communication, mobile computing, and mobile sensing techniques, the intelligence of mobile crowd sensing (MCS), which mainly relies on ubiquitous mobile smart devices in people’s daily lives, has overcome the above drawbacks to a large extent as one new effective and simple measure for pavement management. As a platform for data collection, processing, and visualization, a common smart device can utilize inertial sensor data, photos, videos, subjective reports, and location information to involve the public in pavement anomalies detection. This paper systematically reviewed the studies in this field from 2008 to 2018 to establish an overall knowledge. Through literature collection and screening, a database of studies was set up for analysis. As a result, the year profile of publications and distribution of research areas indicate that there has been a constant attention from researchers in various disciplines. Meanwhile, the distribution of research topic shows that inertial sensors embedded in smartphones have been the most popular data source. Therefore, the process of pavement anomalies detection based on inertial data was reviewed in detail, including preparatory, data collection, and processing phases of the previous experiments. However, some of the key issues in the experimental phases were investigated by previous studies, while some other challenges were not tackled or noticed. Hence, the challenges in both experiment and implementation stages were discussed to improve the studies and practice. Furthermore, several directions for future research are summarized from the main issues and challenges to offer potential opportunities for more relevant research studies and applications in pavement management.