When Vehicles <italic>See</italic> Pedestrians With Phones: A Multicue Framework for Recognizing Phone-Based Activities of Pedestrians

Abstract

The intelligent vehicle community has devoted considerable efforts to model driver behavior, and in particular to detect and overcome driver distraction in an effort to reduce accidents caused by driver negligence. However, as the domain increasingly shifts towards autonomous and semi-autonomous solutions, the driver is no longer integral to the decision making process, indicating a need to refocus efforts elsewhere. To this end, we propose to study pedestrian distraction instead. In particular, we focus on detecting pedestrians who are engaged in secondary activities involving their cellphones and similar handheld multimedia devices from a purely vision-based standpoint. To achieve this objective, we propose a pipeline incorporating articulated human pose estimation, followed by a soft object label transfer from an ensemble of exemplar SVMs trained on the nearest neighbors in pose feature space. We additionally incorporate head gaze features and prior pose information to carry out cellphone related pedestrian activity recognition. Finally, we offer a method to reliably track the articulated pose of a pedestrian through a sequence of images using a particle filter with a Gaussian Process Dynamical Model (GPDM), which can then be used to estimate sequentially varying activity scores at a very low computational cost. The entire framework is fast (especially for sequential data) and accurate, and easily extensible to include other secondary activities and sources of distraction.

Autonomous Vehicles That Interact With Pedestrians: A Survey of Theory and Practice

“…Rangesh and Trivedi [147] estimate the pose of pedestrians in the scene, and identify whether they are holding cell phones. The combination of the pedestrians' pose and the presence of a cellphone is used to estimate the level of pedestrians engagement in their devices.…”

“…Classification approaches on the other hand report the probability of an event occurring. For instance, accuracy of 80% for TTE between 3 to 6 frames [128], 60% at 16 frames before TTE [146], 90% for 4s to TTE or average precision of 80% [142], 62% [14] for the probability of crossing, and 95% for detecting pedestrian distraction [147]. Some robotic applications report performance in terms of risk and arrival time, e.g.…”

“…Very few works report on the actual processing time of the algorithm on a given hardware, e.g. 2.5 fps in [143], some claim real-time performance [145]- [147], [151] whereas some indicate that the current implementation is not real-time [132], [142].…”

Hybrid Verification Technique for Decision-Making of Self-Driving Vehicles

“…If this is the case, the pedestrian continues to walk at the average speed. The last case could be included in the generated model of the traffic participants using methods explained in [80]. While there are some similar probabilities for the driver with limits to the dynamic movements of the vehicle, the driver may decide to continue driving the same speed, reduce the speed, or to stop in order to give a chance for the AV to pass easily.…”

V2X Communications Applied to Safety of Pedestrians and Vehicles

“…Thus, the protection of pedestrians, cyclists, and other non-vehicle occupants is part of the research areas regarding connected and automated vehicles [4]. In most cases, pedestrians collide with vehicles due to the distraction caused by their interactions with smartphones [5] or other Internet of things (IoT) devices [6][7][8]. This circumstance determines a decrease in the attention threshold and causes hazardous neglects during the crossing of roads [9][10][11].…”