Look back and predict forward in image captioning
Proceedings of the IEEE/CVF conference on computer vision and …, 2019•openaccess.thecvf.com
Most existing attention-based methods on image captioning focus on the current word and
visual information in one time step and generate the next word, without considering the
visual and linguistic coherence. We propose Look Back (LB) method to embed visual
information from the past and Predict Forward (PF) approach to look into future. LB method
introduces attention value from the previous time step into the current attention generation to
suit visual coherence of human. PF model predicts the next two words in one time step and …
visual information in one time step and generate the next word, without considering the
visual and linguistic coherence. We propose Look Back (LB) method to embed visual
information from the past and Predict Forward (PF) approach to look into future. LB method
introduces attention value from the previous time step into the current attention generation to
suit visual coherence of human. PF model predicts the next two words in one time step and …
Abstract
Most existing attention-based methods on image captioning focus on the current word and visual information in one time step and generate the next word, without considering the visual and linguistic coherence. We propose Look Back (LB) method to embed visual information from the past and Predict Forward (PF) approach to look into future. LB method introduces attention value from the previous time step into the current attention generation to suit visual coherence of human. PF model predicts the next two words in one time step and jointly employs their probabilities for inference. Then the two approaches are combined together as LBPF to further integrate visual information from the past and linguistic information in the future to improve image captioning performance. All the three methods are applied on a classic base decoder, and show remarkable improvements on MSCOCO dataset with small increments on parameter counts. Our LBPF model achieves BLEU-4/CIDEr/SPICE scores of 37.4/116.4/21.2 with cross-entropy loss and 38.3/127.6/22.0 with CIDEr optimization. Our three proposed methods can be easily applied on most attention-based encoder-decoder models for image captioning.
openaccess.thecvf.com
Showing the best result for this search. See all results