“…Instructional video understanding. Beyond image semantics (Yatskar et al, 2016), unlike existing tasks for learning from instructional video (Zhou et al, 2018c;Tang et al, 2019;Alayrac et al, 2016;Song et al, 2015;Sener et al, 2015;Huang et al, 2016;Sun et al, 2019b,a;Plummer et al, 2017;Palaskar et al, 2019), combining video & text information in procedures (Yagcioglu et al, 2018;Fried et al, 2020), visual-linguistic reference resolution (Huang et al, 2018(Huang et al, , 2017, visual planning (Chang et al, 2019), joint learning of object and actions Richard et al, 2018;Gao et al, 2017;Damen et al, 2018b), pretraining joint embedding of high level sentence with video clips (Sun et al, 2019b;Miech et al, 2019), our task proposal requires explicit structured knowledge tuple extraction.…”