PIE-Net: Photometric Invariant Edge Guided Network for Intrinsic Image Decomposition

Das, Partha; Karaoğlu, Sezer; Gevers, Theo

doi:10.1109/cvpr52688.2022.01917

Cited by 19 publications

(26 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most prior data-driven models utilize architectures that estimate shading and albedo separately [Baslamisli et al 2018b;Cheng et al 2018;Das et al 2022;Li and Snavely 2018a,b;Luo et al 2020;Shi et al 2017;Takuya Narihira and Yu 2015;Zhou et al 2019]. These methods enforce constraints on each intrinsic component and incorporate a reconstruction loss that favors outputs that reproduce the input image when multiplied.…”

Section: Related Workmentioning

confidence: 99%

“…When the input image is resized to the training resolution for inference, as many data-driven setups do by default [Das et al 2022;Liu et al 2020;Luo et al 2020], we can generate a consistent shading structure for the entire scene. In this scenario, since the entire image fits in the receptive field size of the network, we see a consistent shading structure in the estimation.…”

Section: Multi-resolution Behaviormentioning

confidence: 99%

“…where 𝐼 and 𝐷 represent the input image and the estimated inverse shading, respectively. Most intrinsic decomposition methods [Baslamisli et al 2018b;Cheng et al 2018;Das et al 2022;Li and Snavely 2018a,b;Luo et al 2020;Shi et al 2017;Takuya Narihira and Yu 2015;Zhou et al 2019] estimate the shading and albedo as two different variables and formulate a reconstruction loss using Equation 2. This often results in the estimated 𝑆 and 𝐴 not being able to reconstruct the image 𝐼 correctly.…”

Section: Output Formulationmentioning

confidence: 99%

“…We focus our comparison on recent learning-based methods with open-source implementations. We compare our method to PIE-Net [Das et al 2022], the joint semantic segmentation and intrinsic decomposition method (JSI) by Baslamisli et al [2018a], the unsupervised approach (USI) by Liu et al [2020], NIID-Net [Luo et al 2020], and CGIntrinsics [Li and Snavely 2018a]. For all of these methods, we use the publicly available implementations provided by the authors.…”

Section: Qualitative Evaluationmentioning

confidence: 99%

See 3 more Smart Citations

Intrinsic Image Decomposition via Ordinal Shading

Careaga,

Aksoy

2023

ACM Trans. Graph.

View full text Add to dashboard Cite

Intrinsic decomposition is a fundamental mid-level vision problem that plays a crucial role in various inverse rendering and computational photography pipelines. Generating highly accurate intrinsic decompositions is an inherently under-constrained task that requires precisely estimating continuous-valued shading and albedo. In this work, we achieve high-resolution intrinsic decomposition by breaking the problem into two parts. First, we present a dense ordinal shading formulation using a shift- and scale-invariant loss in order to estimate ordinal shading cues without restricting the predictions to obey the intrinsic model. We then combine low- and high-resolution ordinal estimations using a second network to generate a shading estimate with both global coherency and local details. We encourage the model to learn an accurate decomposition by computing losses on the estimated shading as well as the albedo implied by the intrinsic model. We develop a straightforward method for generating dense pseudo ground truth using our model’s predictions and multi-illumination data, enabling generalization to in-the-wild imagery. We present exhaustive qualitative and quantitative analysis of our predicted intrinsic components against state-of-the-art methods. Finally, we demonstrate the real-world applicability of our estimations by performing otherwise difficult editing tasks such as recoloring and relighting.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Multi-resolution Behaviormentioning

confidence: 99%

Section: Output Formulationmentioning

confidence: 99%

Section: Qualitative Evaluationmentioning

confidence: 99%

See 2 more Smart Citations

Intrinsic Image Decomposition via Ordinal Shading

Careaga,

Aksoy

2023

ACM Trans. Graph.

View full text Add to dashboard Cite

show abstract

“…Following previous works [18], we also introduce an extra guidance image G. We believe the content of the guidance image can benefit the learning of interpolation parameters (described in Sec 3.3) and make the DPF outputs better aligned with the high-resolution guidance image. We directly use the input image of different resolutions as the guidance image instead of introducing a task-specific guidance map (e.g., the edge guidance in [15,18]) that requires domain-specific pre-processing.…”

Section: Network Architecturementioning

confidence: 99%

Drug design and repurposing with a sequence-to-drug paradigm

Chen

Fan

Chang

et al. 2022

Preprint

View full text Add to dashboard Cite

Drug development based on target proteins has been a successful approach in recent decades. A conventional structure-based drug design pipeline is a complex, human-engineered pipeline with multiple independently optimized steps. Advances in end-to-end differentiable learning suggest the potential benefits of similarly reformulating drug design. Here, we proposed a new sequence-to-drug paradigm that discovers drug-like small-molecule modulators directly from protein sequences and validated this concept for the first time in three stages. First, we designed TransformerCPI2.0 as a core tool for the sequence-to-drug paradigm, which exhibited competitive performance with conventional structure-based drug design approaches. Second, we validated the binding knowledge that TransformerCPI2.0 has learned. Third, we applied a sequence-to-drug paradigm to discover new hits for E3 ubiquitin-protein ligases: speckle-type POZ protein (SPOP) and ring finger protein 130 (RNF130) which does not have a 3D structure. This first proof of concept shows that the sequence-to-drug paradigm is a promising direction for drug development.

show abstract