SeqTR: A Simple Yet Universal Network for Visual Grounding

Zhu, Chuhong; Zhou, Yiyi; Shen, Yunhang; Luo, Gen; Pan, Xingjia; Lin, Mingbao; Chen, Chao; Cao, Liujuan; Sun, Xiaoshuai; Ji, Rongrong

doi:10.1007/978-3-031-19833-5_35

Cited by 50 publications

(26 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The efforts on feature fusion have explored feature concatenation [28,48], attention mechanisms [7,29,72,86], and multi-modal Transformers [19,37,42,77]. The method most related to ours is Se-qTR [93], which also adopts a transformer model for generating the polygon vertices sequentially. However, SeqTR can only produce a single polygon of 18 vertices with coarse segmentation mask, failing to outline objects with complex shapes and occlusion.…”

Section: Related Workmentioning

confidence: 99%

“…Referring Expression Comprehension (REC) predicts a bounding box that tightly encompasses the target object in an image corresponding to a referring expression. Existing works include two-staged methods [26,27,90,94] that are based on region proposal ranking, and one-stage methods [4,34,42,46,84,93] that directly predict the target bounding box. Several papers [42,57,93] explore multitask learning of REC and RIS since they are two closely related tasks.…”

Section: Related Workmentioning

confidence: 99%

“…Existing works include two-staged methods [26,27,90,94] that are based on region proposal ranking, and one-stage methods [4,34,42,46,84,93] that directly predict the target bounding box. Several papers [42,57,93] explore multitask learning of REC and RIS since they are two closely related tasks. However, MCN [57] and RefTR [42] require task-specific heads.…”

Section: Related Workmentioning

confidence: 99%

“…[3] further shows that scaling up language models greatly improves few-shot performance. Inspired by these successes in NLP, recent endeavors in computer vision and visionlanguage also start to explore seq2seq modeling for various tasks [9,10,54,76,83,93]. However, they cast geometric localization tasks as a classification problem, i.e., quantizing coordinates into discrete bins and predicting the coordinates as one of the bins.…”

Section: Related Workmentioning

confidence: 99%

“…Vertex and Special Tokens. For each vertex coordinate x or y, previous works [9,10,54,76,83,93] uniformly quantize it into an integer between [1, B], where B ∈ N is the number of bins of the coordinate codebook. In contrast, we maintain the continuous floating-point value of the original x or y coordinate without any quantization.…”

Section: Target Sequence Constructionmentioning

confidence: 99%

See 4 more Smart Citations

PolyFormer: Referring Image Segmentation as Sequential Polygon Generation

Liu¹,

Cai²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

In this work, instead of directly predicting the pixel-level segmentation masks, the problem of referring image segmentation is formulated as sequential polygon generation, and the predicted polygons can be later converted into segmentation masks. This is enabled by a new sequence-tosequence framework, Polygon Transformer (PolyFormer), which takes a sequence of image patches and text query tokens as input, and outputs a sequence of polygon vertices autoregressively. For more accurate geometric localization, we propose a regression-based decoder, which predicts the precise floating-point coordinates directly, without any coordinate quantization error. In the experiments, PolyFormer outperforms the prior art by a clear margin, e.g., 5.40% and 4.52% absolute improvements on the challenging Re-fCOCO+ and RefCOCOg datasets. It also shows strong generalization ability when evaluated on the referring video segmentation task without fine-tuning, e.g., achieving competitive 61.5% J &F on the Ref-DAVIS17 dataset.

show abstract