Learning and Knowledge Transfer with Memory Networks for Machine Comprehension

Abstract

Enabling machines to read and comprehend unstructured text remains an unfulfilled goal for NLP research. Recent research efforts on the "machine comprehension" task have managed to achieve close to ideal performance on simulated data. However, achieving similar levels of performance on small real world datasets has proved difficult; major challenges stem from the large vocabulary size, complex grammar, and the frequent ambiguities in linguistic structure. On the other hand, the requirement of human generated annotations for training, in order to ensure a sufficiently diverse set of questions is prohibitively expensive. Motivated by these practical issues, we propose a novel curriculum inspired training procedure for Memory Networks to improve the performance for machine comprehension with relatively small volumes of training data. Additionally, we explore various training regimes for Memory Networks to allow knowledge transfer from a closely related domain having larger volumes of labelled data. We also suggest the use of a loss function to incorporate the asymmetric nature of knowledge transfer. Our experiments demonstrate improvements on Dailymail, CNN, and MCTest datasets.

A Survey on Machine Reading Comprehension Systems

“…One direction (Chen et al 2017), (Yang et al 2017b), (Weissenborn et al 2017), (Shen et al 2017), (Wang et al 2017b), (Huang et al 2018), (Tan et al 2018a), (Wang et al 2018b), (Hewlett et al 2017), (Xie and Xing 2017), (Salant and Berant 2018), (Wang et al 2018c), (Yadav, Vig and Shroff 2017), (Dhingra et al 2017), (Wang et al 2018a), (Kundu and Ng 2018b), (Prakash et al 2018), (Lin et al 2018), (Liu et al 2018d), (Liu et al 2018a), (Wang and Jiang 2017), (Chen et al 2016), (Kobayashi et al 2016), (Cui et al 2016), (Yin et al 2016), (Wang et al 2016), (Kadlec et al 2016a), (Kadlec et al 2016b), ), (Chaturvedi et al 2018), (Ghaeini et al 2018), (Sheng et al 2018, (Sugawara et al 2018), (Gupta et al 2018), (Ke et al 2018), (Tan et al 2018b), (Wang et al 2018f), (Mihaylov and Frank 2018), (Song et al 2018), (Sachan and Xing 2018), (Htut et al 2018), (Liu and Perez 2017),…”

“…. .. Cloze (Yadav et al 2017), (Cui et al 2017b), (Dhingra et al 2017), (Hoang et al 2018), (Dhingra et al 2018), (Yang et al 2017a), (Seo et al 2017), (Chen et al 2016), (Kobayashi et al 2016), (Cui et al 2016), (Kadlec et al 2016a), (Kadlec et al 2016b), (Liu and Perez 2017), (Munkhdalai and Yu 2017), (Ma et al 2018), (Ghaeini et al 2018), (Sugawara et al 2018), (Mihaylov and Frank 2018), (Song et al 2018), (Zhang et al 2019b), (Wang et al 2019a), (Wang et al 2020a), (Niu et al 2020), (Pappas et al 2020) ..…”

“…. ..Candidate ranking(Min et al 2017),),(Shen et al 2017),,(Duan et al 2017),(Yadav et al 2017),(Prakash et al 2018),,(Chen et al 2016),(Kobayashi et al 2016),(Yin et al 2016),(Wang et al 2016),(Liu and Perez 2017),(Ma et al 2018),(Chaturvedi et al 2018),,(Tay et al 2018),(Du and Cardie 2018),(Wang et al 2018f),(Song et al 2018),(Sachan and Xing 2018),(Angelidis et al 2019),(Jiang et al 2019),(Tu et al 2019),(Guo et al 2020),(Wang et al 2020a),(Ren et al 2020),(Pappas et al 2020), (Song et al 2020) .. . .…”