Multimodal alignment of scholarly documents and their presentations

Abstract

We present a multimodal system for aligning scholarly documents to corresponding presentations in a fine-grained manner (i.e., per presentation slide and per paper section). Our method improves upon a state-of-the-art baseline that employs only textual similarity. Based on an analysis of baseline errors, we propose a three-pronged alignment system that combines textual, image, and ordering information to establish alignment. Our results show a statistically significant improvement of 25%, confirming the importance of visual content in improving alignment accuracy.

TalkSumm: A Dataset and Scalable Annotation Method for Scientific Paper Summarization Based on Conference Talks

“…Finally, alignment between different modalities (e.g., presentation, videos) and text was studied in different domains. Both Kan (2007) and Bahrani and Kan (2013) studied the problem of document to presentation alignment for scholarly documents. Kan (2007) focused on the the discovery and crawling of document-presentation pairs, and a model to align between documents to corresponding presentations.…”

“…Kan (2007) focused on the the discovery and crawling of document-presentation pairs, and a model to align between documents to corresponding presentations. In Bahrani and Kan (2013) they extended previous model to include also visual components of the slides. Aligning video and text was studied mainly in the setting of enriching videos with textual information (Bojanowski et al, 2015;Malmaud et al, 2015;Zhu et al, 2015).…”

Automatic Explanation Spot Estimation Method Targeted at Text and Figures in Lecture Slides

“…Based on these research results, the effectiveness of indicating explanation spots can be considered. For lecture retrieval, researches for aligning lecture slides with speech recognition results or articles are relatively common [4,5,6,7]. However, rarely are studies undertaken to align individual contents in the slide with utterances, as dealt with in this paper.…”