The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

Zhang, Richard; Isola, Phillip; Efros, Alexei A.; Shechtman, Eli; Wang, Oliver

doi:10.1109/cvpr.2018.00068

Cited by 7,382 publications

(4,246 citation statements)

References 53 publications

Supporting

Mentioning

3,782

Contrasting

Unclassified

Order By: Relevance

“…warping, changes in saturation and hue, and blurring) and asked study participants to select the most similar images among sets of three. The results of this experiment confirmed that Perceptual Loss correlates strongly with human judgement, outperforming many other commonly used image similarity measures (Zhang et al 2018). Zhang et al (2018) further demonstrated that the correlation between Perceptual Loss and human judgment could be improved by re-weighting the deep features based on their correlation with human perceptual judgment in their experiment.…”

Section: Introductionsupporting

confidence: 73%

“…The results of this experiment confirmed that Perceptual Loss correlates strongly with human judgement, outperforming many other commonly used image similarity measures (Zhang et al 2018). Zhang et al (2018) further demonstrated that the correlation between Perceptual Loss and human judgment could be improved by re-weighting the deep features based on their correlation with human perceptual judgment in their experiment. This re-weighted Perceptual Loss metric was described as "Learned Perceptual Image Patch Similarity" (LPIPS) (Zhang et al 2018).…”

Section: Introductionsupporting

confidence: 73%

“…Zhang et al (2018) further demonstrated that the correlation between Perceptual Loss and human judgment could be improved by re-weighting the deep features based on their correlation with human perceptual judgment in their experiment. This re-weighted Perceptual Loss metric was described as "Learned Perceptual Image Patch Similarity" (LPIPS) (Zhang et al 2018).…”

Section: Introductionmentioning

confidence: 91%

See 2 more Smart Citations

Measuring Perceptual Distance of Organismal Color Pattern using the Features of Deep Neural Networks

Wham

Ezray

Hines

2019

Preprint

View full text Add to dashboard Cite

A wide range of research relies upon the accurate and repeatable measurement of the degree to which organisms resemble one another. Here, we present an unsupervised workflow for analyzing the relationships between organismal color patterns. This workflow utilizes several recent advancements in deep learning based computer vision techniques to calculate perceptual distance. We validate this approach using previously published datasets surrounding diverse applications of color pattern analysis including mimicry, population differentiation, heritability, and development. We demonstrate that our approach is able to reproduce the biologically relevant color pattern relationships originally reported in these studies. Importantly, these results are achieved without any task-specific training. In many cases, we were able to reproduce findings directly from original photographs or plates with minimum standardization, avoiding the need for intermediate representations such as a cartoonized images or trait matrices. We then present two artificial datasets designed to highlight how this approach handles aspects of color patterns, such as changes in pattern location and the perception of color contrast. These results suggest that this approach will generalize well to support the study of a wide range of biological processes in a diverse set of taxa while also accommodating a variety of data formats, preprocessing techniques, and study designs.

show abstract

Section: Introductionsupporting

confidence: 73%

Section: Introductionsupporting

confidence: 73%

Section: Introductionmentioning

confidence: 91%

See 1 more Smart Citation

Measuring Perceptual Distance of Organismal Color Pattern using the Features of Deep Neural Networks

Wham

Ezray

Hines

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…Available quality metrics are not aimed at considering environmental or individual impacts on the image, therefore there is a strong need in the community for perceptual image quality metrics. Some studies have previously been done in this area 28, 29 …”

Section: Evaluating the Resultsmentioning

confidence: 99%

“…Some studies have previously been done in this area. 28,29 As the ultimate goal of PDP is to improve the viewer's experience by means of minimizing visible differences between intended and perceived content under given viewing conditions, we ran experiments to quantify the perceptual improvement created by the applied processing. Our explanations in this section will be more focused on ambient illumination conditions as the source of image degradation and evaluation of DRIVEvue to compensate for this degradation.…”

Section: Evaluating the Resultsmentioning

confidence: 99%

Solving Challenges and Improving the Performance of Automotive Displays

Akhavan¹,

Yoo²,

Chubarau³

2019

Information Display

View full text Add to dashboard Cite

As the number, size, and role of in‐vehicle screens expand, more dynamic content rendering is needed to ensure that drivers accurately perceive display information, regardless of lighting conditions or individual vision issues, according to IRYStec, a Canadian‐based start‐up that specializes in perceptual display processing technology. Here, Tara Akhavan, the company’s founder and chief technology officer, and her team present a software solution that they say can improve user experience exponentially.

show abstract

Organic, Inorganic, Hybrid, and Bio Materials Research at Sungkyunkwan University

Lee

2019

Advanced Materials

View full text Add to dashboard Cite

Recent deep-learning-based single image super-resolution (SISR) methods have shown impressive performance whereas typical methods train their networks by minimizing the pixelwise distance with respect to a given high-resolution (HR) image. However, despite the basic training scheme being the predominant choice, its use in the context of ill-posed inverse problems has not been thoroughly investigated. In this work, we aim to provide a better comprehension of the underlying constituent by decomposing target HR images into two subcomponents: (1) the optimal centroid which is the expectation over multiple potential HR images, and (2) the inherent noise defined as the residual between the HR image and the centroid. Our findings show that the current training scheme cannot capture the ill-posed nature of SISR and becomes vulnerable to the inherent noise term, especially during early training steps. To tackle this issue, we propose a novel optimization method that can effectively remove the inherent noise term in the early steps of vanilla training by estimating the optimal centroid and directly optimizing toward the estimation. Experimental results show that the proposed method can effectively enhance the stability of vanilla training, leading to overall performance gain. Codes are available at github.com/2minkyulee/ECO.

show abstract

The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

Cited by 7,382 publications

References 53 publications

Measuring Perceptual Distance of Organismal Color Pattern using the Features of Deep Neural Networks

Measuring Perceptual Distance of Organismal Color Pattern using the Features of Deep Neural Networks

Solving Challenges and Improving the Performance of Automotive Displays

Organic, Inorganic, Hybrid, and Bio Materials Research at Sungkyunkwan University

Contact Info

Product

Resources

About