Volumetric Isosurface Rendering with Deep Learning-Based Super-Resolution

Abstract

a) (b) (c) Fig. 1: Our super-resolution network can upscale (a) an input sampling of isosurface normals and depths at low resolution (i.e., 320x240), to (b) a high-resolution normal and depth map (i.e., 1280x960) with ambient occlusion. For ease of interpretation, only the shaded output is shown. (c) The ground truth is rendered at 1280x960. Samples are from a 1024 3 grid, ground truth renders at 0.16 and 18.6 secs w/ and w/o ambient occlusion, super-resolution takes 0.07 sec Abstract-Rendering an accurate image of an isosurface in a volumetric field typically requires large numbers of data samples. Reducing the number of required samples lies at the core of research in volume rendering. With the advent of deep learning networks, a number of architectures have been proposed recently to infer missing samples in multi-dimensional fields, for applications such as image super-resolution and scan completion.In this paper, we investigate the use of such architectures for learning the upscaling of a low-resolution sampling of an isosurface to a higher resolution, with high fidelity reconstruction of spatial detail and shading. We introduce a fully convolutional neural network, to learn a latent representation generating a smooth, edge-aware normal field and ambient occlusions from a low-resolution normal and depth field. By adding a frame-to-frame motion loss into the learning stage, the upscaling can consider temporal variations and achieves improved frame-to-frame coherence. We demonstrate the quality of the network for isosurfaces which were never seen during training, and discuss remote and in-situ visualization as well as focus+context visualization as potential applications.

Deep Direct Volume Rendering: Learning Visual Feature Mappings From Exemplary Images

“…Deep learning has also been used in the context of volume rendering to synthesize novel views or views with different parameters [12,14], for super-resolution of volume isosurface renderings [45], for compressed rendering of time-varying data sets [15] and for prediction of ambient occlusion volumes [8]. However, contrary to our work, none of these approaches explicitly take the DVR method into account directly in the model architecture.…”

IsoExplorer: an isosurface-driven framework for 3D shape analysis of biomedical volume data

“…Han et al (2018) utilized a 3D CNN-based autoencoder model to learn dense representations of stream surfaces and lines, and then used projection to assist with selection and clustering analysis. Weiss et al (2019) leveraged a frame-recurrent neural network to introduce super-resolution techniques to IVR, reducing the amount of data samples required to render isosurfaces.…”

A comparative study of convolutional neural network models for wind field downscaling

“…They demonstrate the downsampling of vector field data at simulation time and upsample the reduced data back to the original resolution. Weiss et al (2019) extend image upscaling to geometry images of isosurfaces in 3D scalar fields by including depth and normal information.…”

“…Weiss et al . (2019) extend image upscaling to geometry images of isosurfaces in 3D scalar fields by including depth and normal information.…”