Correlation Image Sensor-Assisted Single-Frame Optical Flow Estimation in Motion-Blurred Scenes

Abstract

Single-frame optical flow estimation is a more challenging task than predicting the optical flow between adjacent frames in a video. This paper presents a two-stream network that combines motion information from correlation images with appearance information from intensity images to estimate optical flow in a single-frame manner. The correlation image generated by a three-phase correlation image sensor (3PCIS) records changes in incident intensity during the exposure time and conveys motion information about moving objects. This is crucial to assist motion-blurred images in estimating the motion state of moving objects. Due to the lack of directly available datasets to train our network, we generate a synthetic dataset. Importantly, we introduce a definition that describes optical flow on a single motion-blurred image, which is essential for creating reasonable ground truth. Our experimental results demonstrate that (i) the network trained on our synthetic dataset achieves an average End Point Error (EPE) of 0.357 and generalizes well to real-world scenes, (ii) our proposed single-frame method outperforms conventional two-frame-based methods in motion-blurred scenes.INDEX TERMS Correlation image sensor, deep learning, synthetic dataset, two-stream network, optical flow.