No abstract

Many vision and language models suffer from poor visual grounding -often falling back on easy-to-learn language priors rather than basing their decisions on visual concepts in the image. In this work, we propose a generic approach called Human Importance-aware Network Tuning (HINT) that effectively leverages human demonstrations to improve visual grounding. HINT encourages deep networks to be sensitive to the same input regions as humans. Our approach optimizes the alignment between human attention maps and gradient-based network importances -ensuring that models learn not just to look at but rather rely on visual concepts that humans found relevant for a task when making predictions. We apply HINT to Visual Question Answering and Image Captioning tasks, outperforming top approaches on splits that penalize over-reliance on language priors (VQA-CP and robust captioning) using human attention demonstrations for just 6% of the training data.

This study was funded by Novartis Pharmaceuticals, which provided the concept, general oversight, and research collaboration on the project. Covvey and Kamal received research funding from Novartis Pharmaceuticals and the College of Psychiatric and Neurologic Pharmacists. Zacker is employed by, and owns stock in, Novartis Pharmaceuticals. A related poster abstract was presented at the Academy of Managed Care Pharmacy April 2016 Annual Meeting and published as Kamal KM, Covvey JR, Dashputre A, Ghosh S, Zacker C. A conceptual framework for valuebased oncology treatment: a societal perspective. J Manag Care Spec Pharm. 2016;22(4 Suppl A):S28. A publication-only abstract was presented at the American Society of Clinical Oncology 2016 Annual Meeting and published as Covvey JR, Kamal KM, Dashputre A, Ghosh S, Zacker C. The impact of cancer treatment on work productivity of patients and caregivers: a systematic review of the evidence. J Clin Oncol. 2016;34(Suppl):e18249. Study concept and design were contributed by Zacker, Kamal, and Covvey. Dashputre and Ghosh took the lead in data collection, along with Kamal and Covvey, and data interpretation was performed primarily by Shah and Bhosle, along with Ghosh, Dashputre, Covvey, and Kamal. The manuscript was written by Kamal, Covvey, Shah, and Bhosle and revised primarily by Zacker, along with Shah, Bhosle, Kamal, and Covvey.