Multimodal Future Localization and Emergence Prediction for Objects in Egocentric View With a Reachability Prior

Makansi, Osama; Çiçek, Özgün; Buchicchio, Kevin; Brox, Thomas

doi:10.1109/cvpr42600.2020.00441

Cited by 32 publications

(28 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1) Vehicle-Mounted Camera Scenario. : In the field of autonomous driving, pedestrian trajectory prediction has been actively researched in order to ensure that an autonomous vehicle can react promptly to avoid collision when encountering pedestrians on road [25], [26], [27], [28], [29], [30]. Some research has also studied the prediction of a pedestrian's intent of crossing or not crossing the road [31], [32], and the detection of traffic accident by predicting pedestrians' future locations [33].…”

Section: B Egocentric Human Trajectory Predictionmentioning

confidence: 99%

Egocentric Human Trajectory Forecasting with a Wearable Camera and Multi-Modal Fusion

Qiu,

Chen,

et al. 2021

Preprint

View full text Add to dashboard Cite

In this paper, we address the problem of forecasting the trajectory of an egocentric camera wearer (egoperson) in crowded spaces. The trajectory forecasting ability learned from the data of different camera wearers walking around in the real world can be transferred to assist visually impaired people in navigation, as well as to instill human navigation behaviours in mobile robots, enabling better humanrobot interactions. To this end, a novel egocentric human trajectory forecasting dataset was constructed, containing real trajectories of people navigating in crowded spaces wearing a camera, as well as extracted rich contextual data. We extract and utilize three different modalities to forecast the trajectory of the camera wearer, i.e., his/her past trajectory, the past trajectories of nearby people, and the environment such as the scene semantics or the depth of the scene. A Transformerbased encoder-decoder neural network model, integrated with a novel cascaded cross-attention mechanism that fuses multiple modalities, has been designed to predict the future trajectory of the camera wearer. Extensive experiments have been conducted, and the results have shown that our model outperforms the state-of-the-art methods in egocentric human trajectory forecasting.

show abstract

Section: B Egocentric Human Trajectory Predictionmentioning

confidence: 99%

Egocentric Human Trajectory Forecasting with a Wearable Camera and Multi-Modal Fusion

Qiu,

Chen,

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Only adopting the pedestrian trajectories to make predictions, existing methods might fail in some complex scenes. In the development of the paper, we focus on fusing extra information like vehicles [25] and scenes [26] through transformer network. Moreover, future work will also pay attention to the variable length history trajectories because missing values [24] often exist in real world.…”

Section: Eth and Ucy Datasetsmentioning

confidence: 99%

Pedestrian Trajectory Prediction via Spatial Interaction Transformer Network

Su¹,

Meng²,

Xu³

2021

Preprint

View full text Add to dashboard Cite

As a core technology of the autonomous driving system, pedestrian trajectory prediction can significantly enhance the function of active vehicle safety and reduce road traffic injuries. In traffic scenes, when encountering with oncoming people, pedestrians may make sudden turns or stop immediately, which often leads to complicated trajectories. To predict such unpredictable trajectories, we can gain insights into the interaction between pedestrians. In this paper, we present a novel generative method named Spatial Interaction Transformer (SIT), which learns the spatio-temporal correlation of pedestrian trajectories through attention mechanisms. Furthermore, we introduce the conditional variational autoencoder (CVAE) [1] framework to model the future latent motion states of pedestrians. In particular, the experiments based on large-scale traffic dataset nuScenes [2] show that SIT has an outstanding performance than state-of-the-art (SOTA) methods. Experimental evaluation on the challenging ETH [3] and UCY [4] datasets confirms the robustness of our proposed model.

show abstract

“…Vehicle-mounted cameras. With the use of visual data captured by vehicle-mounted cameras, predicting pedestrian trajectory has been an active research area in autonomous driving [18], [19], [20], [21], [22]. Studies have also been carried out in predicting pedestrian's intent of crossing or not crossing the street [28], [29], forecasting pedestrian locomotion [23], and detecting traffic accident with pedestrian future location prediction [30].…”

Section: Related Workmentioning

confidence: 99%

“…Most works on future person location and trajectory prediction, however, have only been focused on scenes captured by birds-eye or oblique views [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], or from vehicle-mounted cameras [18], [19], [20], [21], [22], [23]. Little research has been carried out in egocentric scenarios captured by wearable cameras [24], [25], [26].…”

Section: Introductionmentioning

confidence: 99%

Indoor Future Person Localization from an Egocentric Wearable Camera

Qiu¹,

Lo²,

Gu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Accurate prediction of future person location and movement trajectory from an egocentric wearable camera can benefit a wide range of applications, such as assisting visually impaired people in navigation, and the development of mobility assistance for people with disability. In this work, a new egocentric dataset was constructed using a wearable camera, with 8,250 short clips of a targeted person either walking 1) toward, 2) away, or 3) across the camera wearer in indoor environments, or 4) staying still in the scene, and 13,817 person bounding boxes were manually labelled. Apart from the bounding boxes, the dataset also contains the estimated pose of the targeted person as well as the IMU signal of the wearable camera at each time point. An LSTM-based encoder-decoder framework was designed to predict the future location and movement trajectory of the targeted person in this egocentric setting. Extensive experiments have been conducted on the new dataset, and have shown that the proposed method is able to reliably and better predict future person location and trajectory in egocentric videos captured by the wearable camera compared to three baselines.

show abstract

Multimodal Future Localization and Emergence Prediction for Objects in Egocentric View With a Reachability Prior

Cited by 32 publications

References 37 publications

Egocentric Human Trajectory Forecasting with a Wearable Camera and Multi-Modal Fusion

Egocentric Human Trajectory Forecasting with a Wearable Camera and Multi-Modal Fusion

Pedestrian Trajectory Prediction via Spatial Interaction Transformer Network

Indoor Future Person Localization from an Egocentric Wearable Camera

Contact Info

Product

Resources

About