Masked Non-Autoregressive Image Captioning

Gao, Junlong; Meng, Xiaofeng; Wang, Shiqi; Li, Xia; Wang, Shiqi; Ma, Siwei; Gao, Wen

doi:10.48550/arxiv.1906.00717

Cited by 13 publications

(21 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The other is that beam search has been found to output low-quality when applied to large search spaces [34]. Non-autoregressive method is first proposed by [12,14] to address the above issues, allowing the image captioning model to generate all target words simultaneously. NAIC replaces 𝑤 <𝑡 with independent latent variable 𝑧 to remove the sequential dependencies and rewrite Equation 1 as:…”

Section: Non-autoregressive Image Captionmentioning

confidence: 99%

“…The previous one-pass NAIC can be further boosted by introducing the multi-pass refinement mechanism [10,12,13]. Specifically, IR-NAIC applies a fusion function 𝑓 to deal with the sentence 𝑆 ′ produced in the preceding stage and comprehensively predict the new sentence 𝑆 by:…”

Section: Iterative Refinement Based Non-autoregressive Image Captionmentioning

confidence: 99%

“…In this way, the sentence generation process of IR-NAIC can be concluded as: first, the NAIC model produces a coarse caption as the initial hypothesis and then iteratively refines it until reaches a pre-set criteria with constant steps. In this work, we utilize masked prediction [12,19] as the representation of IR-NAIC due to its excellent performance and simplicity, where tokens are randomly masked in training but selected with low confidences during inference.…”

Section: Iterative Refinement Based Non-autoregressive Image Captionmentioning

confidence: 99%

“…(3) Most previous NAIC methods need to train the captioning model with an extra independent length predictor [11,12,19]. However, such a length predictor is implicitly modeled in our Outliner-Filler module because the length of captions is a by-product of autoregressive Outliner, i.e., 𝑁 𝑓 𝑖𝑙𝑙 = 𝑘 × 𝑁 𝑜𝑢𝑡 , where 𝑁 𝑜𝑢𝑡 is the length of sequence produced by Outliner and 𝑁 𝑓 𝑖𝑙𝑙 is the final sequence length from Filler.…”

Section: Model Architecturementioning

confidence: 99%

“…Recent studies show that extending one-pass NAIC to constant multi-pass, called iterative refinement (IR-NAIC), is promising to break the captioning performance dilemma [10,12,19,40]. Unlike one-pass NAIC, which outputs the total descriptions immediately, IR-NAIC takes the caption hypothesis from the previous iteration as a reference and regularly modifies the new sentence, with a masking strategy or fusion network.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Semi-Autoregressive Image Captioning

Xu¹,

Fei²,

Li³

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Current state-of-the-art approaches for image captioning typically adopt an autoregressive manner, i.e., generating descriptions word by word, which suffers from slow decoding issue and becomes a bottleneck in real-time applications. Non-autoregressive image captioning with continuous iterative refinement, which eliminates the sequential dependence in a sentence generation, can achieve comparable performance to the autoregressive counterparts with a considerable acceleration. Nevertheless, based on a well-designed experiment, we empirically proved that iteration times can be effectively reduced when providing sufficient prior knowledge for the language decoder. Towards that end, we propose a novel two-stage framework, referred to as Semi-Autoregressive Image Captioning (SAIC), to make a better trade-off between performance and speed. The proposed SAIC model maintains autoregressive property in global but relieves it in local. Specifically, SAIC model first jumpily generates an intermittent sequence in an autoregressive manner, that is, it predicts the first word in every word group in order. Then, with the help of the partially deterministic prior information and image features, SAIC model non-autoregressively fills all the skipped words with one iteration. Experimental results on the MS COCO benchmark demonstrate that our SAIC model outperforms the preceding non-autoregressive image captioning models while obtaining a competitive inference speedup. Code is available at https://github.com/feizc/SAIC.

show abstract

Section: Non-autoregressive Image Captionmentioning

confidence: 99%

Section: Iterative Refinement Based Non-autoregressive Image Captionmentioning

confidence: 99%

Section: Iterative Refinement Based Non-autoregressive Image Captionmentioning

confidence: 99%

Section: Model Architecturementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Semi-Autoregressive Image Captioning

Xu¹,

Fei²,

Li³

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

show abstract

Efficient Modeling of Future Context for Image Captioning

Fei

2022

Proceedings of the 30th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Existing approaches to image captioning usually generate the sentence word-by-word from left to right, with the constraint of conditioned on local context including the given image and history generated words. There have been many studies target to make use of global information during decoding, e.g., iterative refinement. However, it is still under-explored how to effectively and efficiently incorporate the future context. To respond to this issue, inspired by that Non-Autoregressive Image Captioning (NAIC) can leverage two-side relation with modified mask operation, we aim to graft this advance to the conventional Autoregressive Image Captioning (AIC) model while maintaining the inference efficiency without extra time cost. Specifically, AIC and NAIC models are first trained combined with shared visual encoders, forcing the visual encoder to contain sufficient and valid future context; then the AIC model is encouraged to capture the causal dynamics of cross-layer interchanging from NAIC model on its unconfident words, which follows a teacher-student paradigm and optimized with the distribution calibration training objective. Empirical evidences demonstrate that our proposed approach clearly surpass the state-of-the-art baselines in both automatic metrics and human evaluations on the MS COCO benchmark. The source code is available at: https://github.com/feizc/Future-Caption. CCS CONCEPTS• Computing methodologies → Computer vision; Natural language processing.

show abstract

Partially Non-Autoregressive Image Captioning

Fei

2021

AAAI

View full text Add to dashboard Cite

Current state-of-the-art image captioning systems usually generated descriptions autoregressively, i.e., every forward step conditions on the given image and previously produced words. The sequential attribution causes a unavoidable decoding latency. Non-autoregressive image captioning, on the other hand, predicts the entire sentence simultaneously and accelerates the inference process significantly. However, it removes the dependence in a caption and commonly suffers from repetition or missing issues. To make a better trade-off between speed and quality, we introduce a partially non-autoregressive model, named PNAIC, which considers a caption as a series of concatenated word groups. The groups are generated parallelly in global while each word in group is predicted from left to right, and thus the captioner can create multiple discontinuous words concurrently at each time step. More importantly, by incorporating curriculum learning-based training tasks of group length prediction and invalid group deletion, our model is capable of generating accurate captions as well as preventing common incoherent errors. Extensive experiments on MS COCO benchmark demonstrate that our proposed method achieves more than 3.5× speedup while maintaining competitive performance.

show abstract

Masked Non-Autoregressive Image Captioning

Cited by 13 publications

References 15 publications

Semi-Autoregressive Image Captioning

Semi-Autoregressive Image Captioning

Efficient Modeling of Future Context for Image Captioning

Partially Non-Autoregressive Image Captioning

Contact Info

Product

Resources

About