Towards Task Understanding in Visual Settings

Abstract

We consider the problem of understanding real world tasks depicted in visual images. While most existing image captioning methods excel in producing natural language descriptions of visual scenes involving human tasks, there is often the need for an understanding of the exact task being undertaken rather than a literal description of the scene. We leverage insights from real world task understanding systems, and propose a framework composed of convolutional neural networks, and an external hierarchical task ontology to produce task descriptions from input images. Detailed experiments highlight the efficacy of the extracted descriptions, which could potentially find their way in many applications, including image alt text generation.

Special issue on learning from user interactions

“…Understanding and learning from user interactions involves a number of different aspects (Mehrotra et al 2018;Mehrotra 2018)-from understanding user intent and tasks (Mehrotra et al 2016b, c;Mehrotra and Yilmaz 2017a;Santy et al 2019;White et al 2015), to developing user models and personalization services (Mehrotra andYilmaz 2015, 2017;Liu et al 2019). Beyond understanding user needs, learning from user interactions involves developing the right metrics for evaluation and experimentation systems , c, Verma et al 2016), understanding user interaction processes (Liu et al 2014), their usage context (Mehrotra and Yilmaz 2017b) and designing interfaces capable of helping users (Hassan Awadallah et al 2014).…”