Panoramic Robust PCA for Foreground–Background Separation on Noisy, Free-Motion Camera Video

Moore, Brian E.; Chen, Gao; Nadakuditi, Raj Rao

doi:10.1109/tci.2019.2891389

Cited by 30 publications

(15 citation statements)

References 80 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition to choosing the noise model, defining background and camera motions, their representation and the RPCA loss function used for modelling and updating the low-rank subspace are particularly impactful. Traditionally, modelling the background/camera motions with rigid or affine transformations [6], [41], [49] is apparently impracticable for modelling the large local deformations of dynamic backgrounds in XCA imaging.…”

Section: Rpca-based Foreground/background Separationmentioning

confidence: 99%

Working memory inspired hierarchical video decomposition with transformative representations

Qin¹,

Mao²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Video decomposition is very important to extract moving foreground objects from complex backgrounds in computer vision, machine learning, and medical imaging, e.g., extracting moving contrast-filled vessels from the complex and noisy backgrounds of X-ray coronary angiography (XCA). However, the challenges caused by dynamic backgrounds, overlapping heterogeneous environments and complex noises still exist in video decomposition. To solve these challenges, this study is the first to introduce a flexible visual working memory model in video decomposition to provide interpretable and high-performance hierarchical deep learning architecture, integrating the transformative representations between sensory and control layers from the perspective of visual and cognitive neuroscience. Specifically, robust PCA unrolling networks acting as a structure-regularized sensor layer decompose XCA into sparse/low-rank structured representations to separate moving contrast-filled vessels from noisy and complex backgrounds. Then, patch recurrent convolutional LSTM networks with a backprojection superresolution module embody unstructured random representations of the control layer in working memory, recurrently projecting spatiotemporally decomposed nonlocal patches into orthogonal subspaces for heterogeneous vessel retrieval and interference suppression. This video decomposition architecture effectively restores the heterogeneous profiles of intensity and geometry of moving objects against the complex background interferences. Experiments show that the proposed method significantly outperforms state-of-the-art methods in accurate moving contrast-filled vessel extraction with excellent flexibility and computational efficiency.

show abstract

Section: Rpca-based Foreground/background Separationmentioning

confidence: 99%

Working memory inspired hierarchical video decomposition with transformative representations

Qin¹,

Mao²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Furthermore, in cases where the background illumination changes, decomposition into more than two components has shown promising results [14], [21], [22]. Other extensions such as considering camera motion with RPCA model have also been studied in the literature [23]. Further discussion and survey of other improvements are provided in [9], [10], [24]- [26].…”

Section: A Motivation and Prior Workmentioning

confidence: 99%

Masked-RPCA: Moving Object Detection With an Overlaying Model

Khalilian-Gourtani

Minaee

Wang

2020

IEEE Open J. Signal Process.

View full text Add to dashboard Cite

Moving object detection in a given video sequence is a pivotal step in many computer vision applications such as video surveillance. Robust Principal Component Analysis (RPCA) performs low-rank and sparse decomposition to accomplish such a task when the background is stationary and the foreground is dynamic and relatively small. A fundamental issue with the RPCA is the assumption that the low-rank and sparse components are added at each pixel, whereas in reality, the moving foreground is overlaid on the background. We propose the masked decomposition (i.e. an overlaying model) where each element either belongs to the low-rank or the sparse component, decided by a mask. We introduce the Masked-RPCA (MRPCA) algorithm to recover the mask (hence the sparse object) and the low-rank components simultaneously, via a non-convex formulation. An adapted version of the Douglas-Rachford splitting algorithm is utilized to solve the proposed formulation. Our experiments using real-world video sequences show consistently better performance for both cases of static and dynamic background videos compared to RPCA and its variants based on the additive model. Additionally, we show that utilizing non-convex priors in our formulation leads to improved results without any added complexity compared to a relaxed formulation using convex surrogates and methods based on the additive model.

show abstract

“…This issue is actually less investigated than the static case. Unlike strategies [71], [119], [219], [236], [252] for small camera jitter which used affine transformation model that describes the motion of the frames in the quasi-static cameras case, Gao et al [88], [198] produced a panoramic low-rank component that spans the entire field of view, automatically stitching together corrupted data from partially overlapping scenes. Practically, the algorithm proceeds by registering the frames of the raw video to a common reference perspective and then it minimizes a modified RPCA cost function that accounts for the partially overlapping views of registered frames and includes TV regularization to decouple the foreground from noise and sparse corruption.…”

Section: ) Extension To Moving Camerasmentioning

confidence: 99%

On the Applications of Robust PCA in Image and Video Processing

et al. 2018

View full text Add to dashboard Cite

Robust PCA (RPCA) via decomposition into lowrank plus sparse matrices offers a powerful framework for a large variety of applications such as image processing, video processing and 3D computer vision. Indeed, most of the time these applications require to detect sparse outliers from the observed imagery data that can be approximated by a lowrank matrix. Moreover, most of the time experiments show that RPCA with additional spatial and/or temporal constraints often outperforms the state-of-the-art algorithms in these applications. Thus, the aim of this paper is to survey the applications of RPCA in computer vision. In the first part of this paper, we review representative image processing applications as follows: (1) lowlevel imaging such as image recovery and denoising, image composition, image colorization, image alignment and rectification, multi-focus image and face recognition, (2) medical imaging like dynamic Magnetic Resonance Imaging (MRI) for acceleration of data acquisition, background suppression and learning of inter-frame motion fields, and (3) imaging for 3D computer vision with additional depth information like in Structure from Motion (SfM) and 3D motion recovery. In the second part, we present the applications of RPCA in video processing which utilize additional spatial and temporal information compared to image processing. Specifically, we investigate video denoising and restoration, hyperspectral video and background/foreground separation. Finally, we provide perspectives on possible future research directions and algorithmic frameworks that are suitable for these applications.

show abstract

Panoramic Robust PCA for Foreground–Background Separation on Noisy, Free-Motion Camera Video

Cited by 30 publications

References 80 publications

Working memory inspired hierarchical video decomposition with transformative representations

Working memory inspired hierarchical video decomposition with transformative representations

Masked-RPCA: Moving Object Detection With an Overlaying Model

On the Applications of Robust PCA in Image and Video Processing

Contact Info

Product

Resources

About