Combining Visual and Textual Systems within the Context of User Feedback

Abstract

Abstract. It has been proven experimentally, that a combination of textual and visual representations can improve the retrieval performance ([20], [23]). It is due to the fact, that the textual and visual feature spaces often represent complementary yet correlated aspects of the same image, thus forming a composite system.In this paper, we present a model for the combination of visual and textual sub-systems within the user feedback context. The model was inspired by the measurement utilized in quantum mechanics (QM) and the tensor product of cooccurrence (density) matrices, which represents a density matrix of the composite system in QM. It provides a sound and natural framework to seamlessly integrate multiple feature spaces by considering them as a composite system, as well as a new way of measuring the relevance of an image with respect to a context. The proposed approach takes into account both intra (via co-occurrence matrices) and inter (via tensor operator) relationships between features' dimensions. It is also computationally cheap and scalable to large data collections. We test our approach on ImageCLEF2007photo data collection and present interesting findings.

Strategies & digital technologies for Autism integration

“…In addition, the use of technology has motivated students and at the same time teachers and students can more easily achieve their goals. The portability and accessibility that smartphones and tablets provide make them interesting tools for use in learning environments for students with special needs (Kaliciak et al, 2013).…”

A Survey of Quantum Theory Inspired Approaches to Information Retrieval

“…Later on, Kaliciak et al [Kaliciak et al 2013] proposed a quantum-inspired framework for a cross-modal retrieval task. That is, given a text query, to retrieve the most relevant images.…”

“…Projection Models [van Rijsbergen 2004], [Melucci 2005a], [Melucci 2005b], [Piwowarski and Lalmas 2009a], [Piwowarski et al 2008], [Piwowarski et al 2010a], [Piwowarski et al 2010c], [Piwowarski et al 2010b], [Caputo et al 2011], ] Quantum Language Models [Sordoni et al 2013b], [Xie et al 2015], [Zhang et al 2018d] [Hou and Song 2009], [Hou et al 2013], [Zhang et al 2018b], [Zhang et al 2018a], [Zhang et al 2018d] , [Blacoe et al 2013], [Blacoe 2015], [Basile and Tamburini 2017], [Li et al 2018a] Quantum-inspired Ranking [Zuccon et al 2009], , , Multimodal IR [Wang et al 2010a], [Kaliciak et al 2011], [Kaliciak et al 2013], [Gkoumas et al 2018], [Zhang et al 2018c], ] Quantum-inspired Representation Learning [Aerts and Czachor 2004], [Bruza and Cole 2005], [Sordoni et al 2013a], [Zuccon et al 2011], [Wittek et al 2014], ], [Jaiswal et al 2018, , , [Buccio et al 2018], User Interactions Projection Models [Melucci and White 2007], [Melucci 2008], [Piwowarski and Lalmas 2009a], [Frommholz et al 2011], , , [Wang et al 2017…”

A Survey of Quantum Theory Inspired Approaches to Information Retrieval

“…Projection Models [van Rijsbergen 2004], [Melucci 2005a], [Melucci 2005b], [Piwowarski and Lalmas 2009a], [Piwowarski et al 2008], [Piwowarski et al 2010a], [Piwowarski et al 2010c], [Piwowarski et al 2010b], [Caputo et al 2011], ] Quantum Language Models [Sordoni et al 2013b], [Xie et al 2015], [Zhang et al 2018d] [Hou andSong 2009], [Hou et al 2013], [Zhang et al 2018b], [Zhang et al 2018a], [Zhang et al 2018d] , [Blacoe et al 2013], [Blacoe 2015], [Basile and Tamburini 2017], [Li et al 2018a] Quantum-inspired Ranking [Zuccon et al 2009], , ], Multimodal IR [Wang et al 2010a], [Kaliciak et al 2011], [Kaliciak et al 2013], [Gkoumas et al 2018], [Zhang et al 2018c], ] Quantum-inspired Representation Learning [Aerts and Czachor 2004], [Bruza and Cole 2005], [Sordoni et al 2013a], [Zuccon et al 2011], [Wittek et al 2014], ], [Jaiswal et al 2018], ], , [Buccio et al 2018], User Interactions Projection Models [Melucci and White 2007], [Melucci 2008], [Piwowarski and Lalmas 2009a], [Frommholz et al 2011], ], ], [Wang et al 2017] Feedback...…”