Real-time Image Enhancer via Learnable Spatial-aware 3D Lookup Tables

Wang, Tao; Li, Yong; Peng, Jingyang; Ma, Yipeng; Wang, Xian; Song, Fenglong; Yan, Youliang

doi:10.1109/iccv48922.2021.00247

Cited by 47 publications

(34 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In contrast, the proposed V4D does not need the additional supervision signal apart from the collected sequence of posed images, and it achieves the superior 4D scene representation ability with a much lower computational resource requirement. At last, the proposed pixel-level refinement module is related to the works in the image enhancement task [63,50]. [63] proposes image-adaptive 3D LUTs for real-time image enhancement and [50] considers the adaptive 3D LUTs with the global scenario and local spatial information, which could get better results.…”

Section: D Representationmentioning

confidence: 99%

“…At last, the proposed pixel-level refinement module is related to the works in the image enhancement task [63,50]. [63] proposes image-adaptive 3D LUTs for real-time image enhancement and [50] considers the adaptive 3D LUTs with the global scenario and local spatial information, which could get better results. Different from them, we treat the 3D LUTs as the refinement module in our 4D novel view synthesis task with dedicated design.…”

Section: D Representationmentioning

confidence: 99%

“…An important step is to combine the interpolated RGB value from the basic LUT as the final output. In the image enhancement task [63,50], the low-resolution image is used to predict the weight to combine the interpolated RGB value. However, it is unpractical in the novel view synthesis task, where such operation is losing the 3D spatial awareness.…”

Section: Luts Refinementmentioning

confidence: 99%

See 2 more Smart Citations

V4D: Voxel for 4D Novel View Synthesis

Wanshui¹,

Xu²,

Huang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Neural radiance fields have made a remarkable breakthrough in the novel view synthesis task at the 3D static scene. However, for the 4D circumstance (e.g., dynamic scene), the performance of the existing method is still limited by the capacity of the neural network, typically in a multilayer perceptron network (MLP). In this paper, we present the method to model the 4D neural radiance field by the 3D voxel, short as V4D, where the 3D voxel has two formats. The first one is to regularly model the bounded 3D space and then use the sampled local 3D feature with the time index to model the density field and the texture field. The second one is in look-up tables (LUTs) format that is for the pixel-level refinement, where the pseudo-surface produced by the volume rendering is utilized as the guidance information to learn a 2D pixel-level refinement mapping. The proposed LUTsbased refinement module achieves the performance gain with a little computational cost and could serve as the plug-and-play module in the novel view synthesis task. Moreover, we propose a more effective conditional positional encoding toward the 4D data that achieves performance gain with negligible computational burdens. Extensive experiments demonstrate that the proposed method achieves state-of-the-art performance by a large margin. At last, the proposed V4D is also a computational-friendly method in both the training and testing phase, where we achieve 2 times faster in the training phase and 10 times faster in the inference phase compared with the state-of-the-art method. The relevant code will be available in https://github.com/GANWANSHUI/V4D.Preprint. Under review.

show abstract

Section: D Representationmentioning

confidence: 99%

Section: D Representationmentioning

confidence: 99%

Section: Luts Refinementmentioning

confidence: 99%

See 1 more Smart Citation

V4D: Voxel for 4D Novel View Synthesis

Wanshui¹,

Xu²,

Huang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…These methods usually use a lowresolution image to extract features and predict the parameters of some predefined global or local color transformation, and later apply the predicted color transformation to the original high-resolution image. Different color transformations have been used in existing works, including quadratic transforms [46,7,32,41], local affine transforms [11], curve based transforms [6,13,29,23,36], filters [10,35], lookup tables [49,43], and customized transforms [5]. Compared with image-toimage translation based methods, these methods usually use smaller models and are more efficient.…”

Section: Sequential Imagementioning

confidence: 99%

Neural Color Operators for Sequential Image Retouching

Wang¹,

Li²,

Xu³

et al. 2022

Preprint

View full text Add to dashboard Cite

We propose a novel image retouching method by modeling the retouching process as performing a sequence of newly introduced trainable neural color operators. The neural color operator mimics the behavior of traditional color operators and learns pixelwise color transformation while its strength is controlled by a scalar. To reflect the homomorphism property of color operators, we employ equivariant mapping and adopt an encoder-decoder structure which maps the non-linear color transformation to a much simpler transformation (i.e., translation) in a high dimensional space. The scalar strength of each neural color operator is predicted using CNN based strength predictors by analyzing global image statistics. Overall, our method is rather lightweight and offers flexible controls. Experiments and user studies on public datasets show that our method consistently achieves the best results compared with SOTA methods in both quantitative measures and visual qualities.Code is available at https://github.com/amberwangyili/neurop.

show abstract

“…The high effectiveness and efficiency of LUTs motivate recent advances in deep learning to propose learnable, image-adaptive LUTs for enhanced real-time image enhancement [25,31,18,17,21,11,14,26,1,36,33]. However, these methods encode a complete color transform to only a single type of LUTs, either 1D or 3D, but neglecting the limited capability of a single module to model componentindependent and component-correlated transformations simultaneously.…”

Section: Introductionmentioning

confidence: 99%

SepLUT: Separable Image-adaptive Lookup Tables for Real-time Image Enhancement

Yang¹,

Jin²,

Xu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Image-adaptive lookup tables (LUTs) have achieved great success in real-time image enhancement tasks due to their high efficiency for modeling color transforms. However, they embed the complete transform, including the color component-independent and the component-correlated parts, into only a single type of LUTs, either 1D or 3D, in a coupled manner. This scheme raises a dilemma of improving model expressiveness or efficiency due to two factors. On the one hand, the 1D LUTs provide high computational efficiency but lack the critical capability of color components interaction. On the other, the 3D LUTs present enhanced component-correlated transform capability but suffer from heavy memory footprint, high training difficulty, and limited cell utilization. Inspired by the conventional divide-and-conquer practice in the image signal processor, we present SepLUT (separable image-adaptive lookup table) to tackle the above limitations. Specifically, we separate a single color transform into a cascade of componentindependent and component-correlated sub-transforms instantiated as 1D and 3D LUTs, respectively. In this way, the capabilities of two subtransforms can facilitate each other, where the 3D LUT complements the ability to mix up color components, and the 1D LUT redistributes the input colors to increase the cell utilization of the 3D LUT and thus enable the use of a more lightweight 3D LUT. Experiments demonstrate that the proposed method presents enhanced performance on photo retouching benchmark datasets than the current state-of-the-art and achieves real-time processing on both GPUs and CPUs.

show abstract

Real-time Image Enhancer via Learnable Spatial-aware 3D Lookup Tables

Cited by 47 publications

References 16 publications

V4D: Voxel for 4D Novel View Synthesis

V4D: Voxel for 4D Novel View Synthesis

Neural Color Operators for Sequential Image Retouching

SepLUT: Separable Image-adaptive Lookup Tables for Real-time Image Enhancement

Contact Info

Product

Resources

About