Calipso: physics-based image and video editing through CAD model proxies

Abstract

Figure 1: Our method allows a user to interact in a 3D manner with objects in images and videos by producing rigid and deformable transforms, topological changes and physical attribute editing (e.g., mass, stiffness, gravity). Using an intuitive mesh refinement and 3D/2D alignment approach and estimating dynamics from image flow, our system produces a final composition without cumbersome user input, while preserving visual consistency. Left: editing Dali's painting by interactively pulling the watch or decreasing its stiffness leading to melting it. Right: bending a car bridge by pulling it from a side or by progressively increasing its mass. AbstractWe present Calipso, an interactive method for editing images and videos in a physically-coherent manner. Our main idea is to realize physics-based manipulations by running a full physics simulation on proxy geometries given by non-rigidly aligned CAD models. Running these simulations allows us to apply new, unseen forces to move or deform selected objects, change physical parameters such as mass or elasticity, or even add entire new objects that interact with the rest of the underlying scene. In Calipso, the user makes edits directly in 3D; these edits are processed by the simulation and then transfered to the target 2D content using shape-to-image correspondences in a photo-realistic rendering process. To align the CAD models, we introduce an efficient CAD-to-image alignment procedure that jointly minimizes for rigid and non-rigid alignment while preserving the high-level structure of the input shape. Moreover, the user can choose to exploit image flow to estimate scene motion, producing coherent physical behavior with ambient dynamics. We demonstrate Calipso's physics-based editing on a wide range of examples producing myriad physical behavior while preserving geometric and visual consistency.

Generalizing Interactive Backpropagating Refinement for Dense Prediction

“…In this work, we investigate whether a pretrained automatic dense prediction method can be effectively converted into an efficient interactive method without any additional retraining. This is a significant task as deep networks are commonly applied in interactive ways for photography [11,32,34,37,38], videography [22,23,31], special effects [6,8,28], etc. Two prior works, both focused primarily on interactive segmentation, have inspired our method.…”

Copy-Move Forgery Detection and Localization Using a Generative Adversarial Network and Convolutional Neural-Network

“…Given the increasing difficulty to distinguish between photographic and computer-generated forgeries, the need to develop equally sophisticated detection modes is becoming more urgent [17]. The latest incarnations of computer-vision-based image forgeries show a significantly higher degree of photorealism than was previously exhibited [18][19][20][21][22][23]. Especially compelling is image copy-move forgery (CMF), which changes the features of one image by digitally translating one scene as another [24].…”