We present the first end-to-end solution to create high-quality freeviewpoint video encoded as a compact data stream. Our system records performances using a dense set of RGB and IR video cameras, generates dynamic textured surfaces, and compresses these to a streamable 3D video format. Four technical advances contribute to high fidelity and robustness: multimodal multi-view stereo fusing RGB, IR, and silhouette information; adaptive meshing guided by automatic detection of perceptually salient areas; mesh tracking to create temporally coherent subsequences; and encoding of tracked textured meshes as an MPEG video stream. Quantitative experiments demonstrate geometric accuracy, texture fidelity, and encoding efficiency. We release several datasets with calibrated inputs and processed results to foster future research.

This method provides an effective quality control mechanism for evaluating the DVHs of the OARs. Adoption of such a method will advance the quality of current IMRT planning, providing better treatment plan consistency.

Abstract. In this paper we address the challenge of matching patient geometry to facilitate the design of patient treatment plans in radiotherapy. To this end we propose a novel shape descriptor, the Overlap Volume Histogram, which provides a rotation and translation invariant representation of a patient's organs at risk relative to the tumor volume. Using our descriptor, it is possible to accurately identify database patients with similar constellations of organ and tumor geometries, enabling the transfer of treatment plans between patients with similar geometries. We demonstrate the utility of our method for such tasks by outperforming state of the art shape descriptors in the retrieval of patients with similar treatment plans. We also preliminarily show its potential as a quality control tool by demonstrating how it is used to identify an organ at risk whose dose can be significantly reduced.

Abstract

Input range scans Oriented points and depth hull SPR SPR w/ Dirichlet envelope RMS = 7.8 × 10 −3 RMS = 3.8 × 10 −3 Figure 1: Given a collection of range scans, one can obtain a set of oriented points and a depth hull volume. (Points are colored by normal orientation.) Applying traditional Screened Poisson Reconstruction (SPR) to the oriented points yields a surface with unwanted artifacts in regions with missing data. By incorporating the depth hull as a Dirichlet constraint within the global Poisson formulation, we prevent the emergence of extraneous surfaces, resulting in a more accurate model.

Scanned performances are commonly represented in virtual environments as sequences of textured triangle meshes. Detailed shapes deforming over time benefit from meshes with dynamically evolving connectivity. We analyze these unstructured mesh sequences to automatically synthesize motion graphs with new smooth transitions between compatible poses and actions. Such motion graphs enable natural periodic motions, stochastic playback, and user-directed animations. The main challenge of unstructured sequences is that the meshes differ not only in connectivity but also in alignment, shape, and texture. We introduce new geometry processing techniques to address these problems and demonstrate visually seamless transitions on high-quality captures.

We convert a sequence of unstructured textured meshes into a mesh with incrementally changing connectivity and atlas parameterization. Like prior work on surface tracking, we seek temporally coherent mesh connectivity to enable efficient representation of surface geometry and texture. Like recent work on evolving meshes, we pursue local remeshing to permit tracking over long sequences containing significant deformations or topological changes. Our main contribution is to show that both goals are realizable within a common framework that simultaneously evolves both the set of mesh triangles and the parametric map. Sparsifying the remeshing operations allows the formation of large spatiotemporal texture charts. These charts are packed as prisms into a 3D atlas for a texture video. Reducing tracking drift using mesh-based optical flow helps improve compression of the resulting video stream.

In this paper, we present a novel approach for efficiently evolving meshes using mean‐curvature flow. We use a finite‐elements hierarchy that supports an efficient multigrid solver for performing the semi‐implicit time‐stepping. Although expensive to compute, we show that it is possible to track this hierarchy through the process of surface evolution. As a result, we provide a way to efficiently flow the surface through the evolution, without requiring a costly initialization at the beginning of each time‐step. Using our approach, we demonstrate a factor of nearly seven‐fold improvement over the non‐tracking implementation, supporting the evolution of surfaces consisting of 1M triangles at a rate of just a few seconds per update.