A Generalist Framework for Panoptic Segmentation of Images and Videos

Chen, Ting; Li, Lala; Saxena, Saurabh; Hinton, Geoffrey E.; Fleet, David J.

doi:10.48550/arxiv.2210.06366

Cited by 10 publications

(19 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this work, we empirically study noise scheduling strategies for diffusion models and show their importance. The noise scheduling not only plays an important role in image generation but also for other tasks such as panoptic segmentation [1]. A simple strategy of adjusting input scaling factor [1] works well across different image resolutions.…”

Section: Discussionmentioning

confidence: 99%

“…The noise scheduling not only plays an important role in image generation but also for other tasks such as panoptic segmentation [1]. A simple strategy of adjusting input scaling factor [1] works well across different image resolutions. When combined with recently proposed RIN architecture [9], our noise scheduling strategy enables single-stage generation of high resolution images.…”

Section: Discussionmentioning

confidence: 99%

“…Built on top of existing work related to noise scheduling [6,12,11,1,9], we systematically study two different noise scheduling strategies for diffusion models.…”

Section: Strategies To Adjust Noise Schedulingmentioning

confidence: 99%

“…Another way to indirectly adjust noise scheduling, proposed in [1], is to scale the input x 0 by a constant factor b, which results in the following noising processing.…”

Section: Strategy 2: Adjusting Input Scaling Factormentioning

confidence: 99%

See 3 more Smart Citations

On the Importance of Noise Scheduling for Diffusion Models

Chen¹

2023

Preprint

View full text Add to dashboard Cite

We empirically study the effect of noise scheduling strategies for denoising diffusion generative models. There are three findings: (1) the noise scheduling is crucial for the performance, and the optimal one depends on the task (e.g., image sizes), ( 2) when increasing the image size, the optimal noise scheduling shifts towards a noisier one (due to increased redundancy in pixels), and (3) simply scaling the input data [1] by a factor of b while keeping the noise schedule function fixed (equivalent to shifting the logSNR by log b) is a good strategy across image sizes. This simple recipe, when combined with recently proposed Recurrent Interface Network (RIN) [9], yields state-of-the-art pixel-based diffusion models for high-resolution images on ImageNet, enabling single-stage, end-to-end generation of diverse and high-fidelity images at 1024×1024 resolution (without upsampling/cascades).

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

“…Built on top of existing work related to noise scheduling [6,12,11,1,9], we systematically study two different noise scheduling strategies for diffusion models.…”

Section: Strategies To Adjust Noise Schedulingmentioning

confidence: 99%

“…Another way to indirectly adjust noise scheduling, proposed in [1], is to scale the input x 0 by a constant factor b, which results in the following noising processing.…”

Section: Strategy 2: Adjusting Input Scaling Factormentioning

confidence: 99%

See 2 more Smart Citations

On the Importance of Noise Scheduling for Diffusion Models

Chen¹

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…We advocate that using DPMs for the graph-structured prediction can exploit their extraordinary capability in learning highly structured data (Li et al, 2022;Chen et al, 2022b;Hoogeboom et al, 2022b). In particular, our DPM makes a prediction using a GNN-based reverse diffusion process.…”

Section: Introductionmentioning

confidence: 99%

Diffusion Probabilistic Models for Graph-Structured Prediction

Jang¹,

Mo²,

Ahn³

2023

Preprint

View full text Add to dashboard Cite

This paper studies graph-structured prediction for supervised learning on graphs with node-wise or edge-wise target dependencies. To solve this problem, recent works investigated combining graph neural networks (GNNs) with conventional structured prediction algorithms like conditional random fields. However, in this work, we pursue an alternative direction building on the recent successes of diffusion probabilistic models (DPMs). That is, we propose a new framework using DPMs to make graph-structured predictions. In the fully supervised setting, our DPM captures the target dependencies by iteratively updating each target estimate based on the estimates of nearby targets. We also propose a variational expectation maximization algorithm to train our DPM in the semisupervised setting. Extensive experiments verify that our framework consistently outperforms existing neural structured prediction models on inductive and transductive node classification. We also demonstrate the competitive performance of our framework for algorithmic reasoning tasks. 1

show abstract

DiffModeler: Large Macromolecular Structure Modeling in Low-Resolution Cryo-EM Maps Using Diffusion Model

Wang,

Zhu,

Terashi

et al. 2024

Preprint

View full text Add to dashboard Cite

Cryogenic electron microscopy (cryo-EM) has now been widely used for determining multi-chain protein complexes. However, modeling a complex structure is challenging particularly when the map resolution is low, typically in the intermediate resolution range of 5 to 10 Å. Within this resolution range, even accurate structure fitting is difficult, let alone de novo modeling. To address this challenge, here we present DiffModeler, a fully automated method for modeling protein complex structures. DiffModeler employs a diffusion model for backbone tracing and integrates AlphaFold2-predicted single-chain structures for structure fitting. Extensive testing on cryo-EM maps at intermediate resolutions demonstrates the exceptional accuracy of DiffModeler in structure modeling, achieving an average TM-Score of 0.92, surpassing existing methodologies significantly. Notably, DiffModeler successfully modeled a protein complex composed of 47 chains and 13,462 residues, achieving a high TM-Score of 0.94. Further benchmarking at low resolutions (10-20 Å) confirms its versatility, demonstrating plausible performances. Moreover, when coupled with CryoREAD, DiffModeler excels in constructing protein-DNA/RNA complex structures for near-atomic resolution maps (0-5 Å), showcasing state-of-the-art performance with average TM-Scores of 0.88 and 0.91 across two datasets.

show abstract

A Generalist Framework for Panoptic Segmentation of Images and Videos

Cited by 10 publications

References 46 publications

On the Importance of Noise Scheduling for Diffusion Models

On the Importance of Noise Scheduling for Diffusion Models

Diffusion Probabilistic Models for Graph-Structured Prediction

DiffModeler: Large Macromolecular Structure Modeling in Low-Resolution Cryo-EM Maps Using Diffusion Model

Contact Info

Product

Resources

About