PPMN: Pixel-Phrase Matching Network for One-Stage Panoptic Narrative Grounding

Semi-Supervised Panoptic Narrative Grounding

“…Specifically, the PNG task seeks to segment objects and regions in an image corresponding to nouns in its long text description. Numerous studies have been conducted on this task [10,13,53]. González et al [13] first introduced this new task, establishing a benchmark that includes new standard data and evaluation methods, and proposed a robust baseline method as the foundation for future work.…”

“…González et al [13] first introduced this new task, establishing a benchmark that includes new standard data and evaluation methods, and proposed a robust baseline method as the foundation for future work. To address the limitations of the previous twostage approach, such as low-quality proposals and spatial details loss, Ding et al [10] proposed a one-stage Pixel-Phrase Matching Network that directly matches each phrase to its corresponding pixels and outputs panoptic segmentation. Concurrently, Wang et al [53] proposed a similar one-stage network for real-time PNG, but with a greater focus on the real-time performance of the model.…”

“…A proper initialization is essential for both student and teacher models in SSL learning [31,32], as the teacher model generates pseudo-labels to train the student model in later stages. To achieve this, we initially use PPMN [10] as our Burn-In model for fully supervised training to obtain the prediction 𝑌 𝑙 𝑖 ∈ R 𝑁 𝑙 𝑖 ×𝐻 𝑙 𝑖 ×𝑊 𝑙 𝑖 of the 𝑖-th image I 𝑙 𝑖 :…”

“…In this paper, we approach panoptic narrative grounding as a segmentation task. Consequently, we adopt Binary Cross-Entropy (BCE) loss and Dice loss [38] as our segmentation loss functions, following the precedent set by previous research [10,13,53]. BCE loss, a widely-used binary classification loss function, quantifies the distance between two probability distributions in binary classification problems.…”

“…We adopt PPMN [10] as the baseline of the panoptic narrative grounding network. To maintain consistency with PPMN, we adopt the same version of the backbone as PPMN for the feature extraction stage of both the visual modality and the linguistic modality.…”