Short-Term Prediction and Multi-Camera Fusion on Semantic Grids

Hoyer, Lukas; Kesper, Patrick; Khoreva, Anna; Fischer, Volker

doi:10.1109/iccvw.2019.00109

Cited by 12 publications

(8 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent methods [50,62,69,25] estimate future frames by reasoning about shape, egomotion, and foreground motion separately. However, none of these methods reason explicitly about individual instances, while our method yields a full future panoptic segmentation forecast.…”

Section: Methods That Anticipatementioning

confidence: 99%

Panoptic Segmentation Forecasting

Graber¹,

Tsai²,

Firman³

et al. 2021

Preprint

View full text Add to dashboard Cite

Our goal is to forecast the near future given a set of recent observations. We think this ability to forecast, i.e., to anticipate, is integral for the success of autonomous agents which need not only passively analyze an observation but also must react to it in real-time. Importantly, accurate forecasting hinges upon the chosen scene decomposition. We think that superior forecasting can be achieved by decomposing a dynamic scene into individual 'things' and background 'stuff'. Background 'stuff' largely moves because of camera motion, while foreground 'things' move because of both camera and individual object motion. Following this decomposition, we introduce panoptic segmentation forecasting. Panoptic segmentation forecasting opens up a middle-ground between existing extremes, which either forecast instance trajectories or predict the appearance of future image frames. To address this task we develop a twocomponent model: one component learns the dynamics of the background stuff by anticipating odometry, the other one anticipates the dynamics of detected things. We establish a leaderboard for this novel task, and validate a state-of-theart model that outperforms available baselines.

show abstract

Section: Methods That Anticipatementioning

confidence: 99%

Panoptic Segmentation Forecasting

Graber¹,

Tsai²,

Firman³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…As a result, the pixel-level semantics from the semantic segmentation can represent the uncountable or difficult to count street elements. In this study, we adopted DeepLab (version 3) pre-trained on the Cityscapes Dataset [53] because it is one of the best models evaluated on the open benchmarking dataset, Cityscapes [59,60].…”

Section: Semantic Feature Detectionmentioning

confidence: 99%

Big Data-Driven Pedestrian Analytics: Unsupervised Clustering and Relational Query Based on Tencent Street View Photographs

Xue

et al. 2021

IJGI

View full text Add to dashboard Cite

Recent technological advancements in geomatics and mobile sensing have led to various urban big data, such as Tencent street view (TSV) photographs; yet, the urban objects in the big dataset have hitherto been inadequately exploited. This paper aims to propose a pedestrian analytics approach named vectors of uncountable and countable objects for clustering and analysis (VUCCA) for processing 530,000 TSV photographs of Hong Kong Island. First, VUCCA transductively adopts two pre-trained deep models to TSV photographs for extracting pedestrians and surrounding pixels into generalizable semantic vectors of features, including uncountable objects such as vegetation, sky, paved pedestrian path, and guardrail and countable objects such as cars, trucks, pedestrians, city animals, and traffic lights. Then, the extracted pedestrians are semantically clustered using the vectors, e.g., for understanding where they usually stand. Third, pedestrians are semantically indexed using relations and activities (e.g., walking behind a guardrail, road-crossing, carrying a backpack, or walking a pet) for queries of unstructured photographic instances or natural language clauses. The experiment results showed that the pedestrians detected in the TSV photographs were successfully clustered into meaningful groups and indexed by the semantic vectors. The presented VUCCA can enrich eye-level urban features into computational semantic vectors for pedestrians to enable smart city research in urban geography, urban planning, real estate, transportation, conservation, and other disciplines.

show abstract

“…Erkent et al [4] predict a semantic grid from camera images but perform an early fusion of occupancy grid computed using lidar data and evaluate in a single camera setting. Hoyer et al [6] predict a top-down semantic grid representation and use images from cameras situated at different angles. However, their pipeline first runs semantic segmentation on the images and then uses stereo-depth in order to map each pixels semantic label to a top-down grid, whereas we operate directly on raw camera images which reduces pipeline complexity.…”

Section: Related Workmentioning

confidence: 99%

FISHING Net: Future Inference of Semantic Heatmaps In Grids

Hendy,

Sloan,

Tian

et al. 2020

Preprint

View full text Add to dashboard Cite

For autonomous robots to navigate a complex environment, it is crucial to understand the surrounding scene both geometrically and semantically. Modern autonomous robots employ multiple sets of sensors, including lidars, radars, and cameras. Managing the different reference frames and characteristics of the sensors, and merging their observations into a single representation complicates perception. Choosing a single unified representation for all sensors simplifies the task of perception and fusion. In this work, we present an end-to-end pipeline that performs semantic segmentation and short term prediction using a topdown representation. Our approach consists of an ensemble of neural networks which take in sensor data from different sensor modalities and transform them into a single common top-down semantic grid representation. We find this representation favorable as it is agnostic to sensor-specific reference frames and captures both the semantic and geometric information for the surrounding scene. Because the modalities share a single output representation, they can be easily aggregated to produce a fused output. In this work we predict short-term semantic grids but the framework can be extended to other tasks. This approach offers a simple, extensible, end-to-end approach for multi-modal perception and prediction.

show abstract

Short-Term Prediction and Multi-Camera Fusion on Semantic Grids

Cited by 12 publications

References 32 publications

Panoptic Segmentation Forecasting

Panoptic Segmentation Forecasting

Big Data-Driven Pedestrian Analytics: Unsupervised Clustering and Relational Query Based on Tencent Street View Photographs

FISHING Net: Future Inference of Semantic Heatmaps In Grids

Contact Info

Product

Resources

About