FlowNet: Learning Optical Flow with Convolutional Networks

Fischer, Philipp; Dosovitskiy, Alexey; Ilg, Eddy; Häusser, Philip; Hazırbaş, Caner; Golkov, Vladimir; Smagt, Patrick van der; Cremers, Daniel; Brox, Thomas

doi:10.48550/arxiv.1504.06852

Cited by 78 publications

(136 citation statements)

References 0 publications

Supporting

Mentioning

135

Contrasting

Order By: Relevance

“…An end-to-end deep learning system has also been suggested [8]. It uses established networks such as DepthNet [29] and FlowNet [30] to get depth estimates of the object. The features are combined and passed through a dense network to obtain velocity.…”

Section: Related Workmentioning

confidence: 99%

A Velocity Estimation Technique for a Monocular Camera Using mmWave FMCW Radars

2021

View full text Add to dashboard Cite

Perception in terms of object detection, classification, and dynamic estimation (position and velocity) are fundamental functionalities that autonomous agents (unmanned ground vehicles, unmanned aerial vehicles, or robots) have to navigate safely and autonomously. To date, various sensors have been used individually or in combination to achieve this goal. In this paper, we present a novel method for leveraging millimeter wave radar’s (mmW radar’s) ability to accurately measure position and velocity in order to improve and optimize velocity estimation using a monocular camera (using optical flow) and machine learning techniques. The proposed method eliminates ambiguity in optical flow velocity estimation when the object of interest is at the edge of the frame or far away from the camera without requiring camera–radar calibration. Moreover, algorithms of various complexity were implemented using custom dataset, and each of them successfully detected the object and estimated its velocity accurately and independently of the object’s distance and location in frame. Here, we present a complete implementation of camera–mmW radar late feature fusion to improve the camera’s velocity estimation performance. It includes setup design, data acquisition, dataset development, and finally, implementing a lightweight ML model that successfully maps the mmW radar features to the camera, allowing it to perceive and estimate the dynamics of a target object without any calibration.

show abstract

Section: Related Workmentioning

confidence: 99%

A Velocity Estimation Technique for a Monocular Camera Using mmWave FMCW Radars

2021

View full text Add to dashboard Cite

show abstract

“…In order to achieve such an alignment, we first estimate the optical flow [29] between the consecutive LDR frames having alternating exposures, which is then used to warp the previous frame to the current frame. Convolution neural network-based optical flow estimation was originally proposed by Dosovitskiy et al [7], which directly generates a flow field from a pair of images. After this, many works have been proposed on neural network-based optical flow estimation [15,37,41].…”

Section: Flow Networkmentioning

confidence: 99%

HDRVideo-GAN: Deep Generative HDR Video Reconstruction

Anand¹,

Harilal²,

Kumar‐Sinha³

et al. 2021

Preprint

View full text Add to dashboard Cite

High dynamic range (HDR) videos provide a more visually realistic experience than the standard low dynamic range (LDR) videos. Despite having significant progress in HDR imaging, it is still a challenging task to capture high-quality HDR video with a conventional off-the-shelf camera. Existing approaches rely entirely on using dense optical flow between the neighboring LDR sequences to reconstruct an HDR frame. However, they lead to inconsistencies in color and exposure over time when applied to alternating exposures with noisy frames. In this paper, we propose an end-to-end GAN-based framework for HDR video reconstruction from LDR sequences with alternating exposures. We first extract clean LDR frames from noisy LDR video with alternating exposures with a denoising network trained in a self-supervised setting. Using optical flow, we then align the neighboring alternating-exposure frames to a reference frame and then reconstruct high-quality HDR frames in a complete adversarial setting. To further improve the robustness and quality of generated frames, we incorporate temporal stabilitybased regularization term along with content and style-based losses in the cost function during the training procedure. Experimental results demonstrate that our framework achieves state-of-the-art performance and generates superior quality HDR frames of a video over the existing methods. CCS CONCEPTS• Computing methodologies → Computational photography; Image-based rendering.

show abstract

“…In this paper, we address these challenges with a learned architecture that identifies and exchanges information across several robots. Our problem formulation and solution draw inspiration from the learning-based works of several domains, including learned cost volume image correspondence [12,7,8,21,20], learned communication [11,10], and learned multi-view fusion [16,17,9]. Based on these prior works, we develop a novel architecture that effectively combines information from multiple robots in a distributed, Fig.…”

Section: Introductionmentioning

confidence: 99%

Overcoming Obstructions via Bandwidth-Limited Multi-Agent Spatial Handshaking

Glaser

Liu

Tian

et al. 2021

2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

In this paper, we address bandwidth-limited and obstruction-prone collaborative perception, specifically in the context of multi-agent semantic segmentation. This setting presents several key challenges, including processing and exchanging unregistered robotic swarm imagery. To be successful, solutions must effectively leverage multiple non-static and intermittently-overlapping RGB perspectives, while heeding bandwidth constraints and overcoming unwanted foreground obstructions. As such, we propose an end-to-end learn-able Multi-Agent Spatial Handshaking network (MASH) to process, compress, and propagate visual information across a robotic swarm. Our distributed communication module operates directly (and exclusively) on raw image data, without additional input requirements such as pose, depth, or warping data. We demonstrate superior performance of our model compared against several baselines in a photo-realistic multi-robot AirSim environment, especially in the presence of image occlusions. Our method achieves an absolute 11% IoU improvement over strong baselines.

show abstract

FlowNet: Learning Optical Flow with Convolutional Networks

Cited by 78 publications

References 0 publications

A Velocity Estimation Technique for a Monocular Camera Using mmWave FMCW Radars

A Velocity Estimation Technique for a Monocular Camera Using mmWave FMCW Radars

HDRVideo-GAN: Deep Generative HDR Video Reconstruction

Overcoming Obstructions via Bandwidth-Limited Multi-Agent Spatial Handshaking

Contact Info

Product

Resources

About