Fear the REAPER: A System for Automatic Multi-Document Summarization with Reinforcement Learning

Abstract

This paper explores alternate algorithms, reward functions and feature sets for performing multi-document summarization using reinforcement learning with a high focus on reproducibility. We show that ROUGE results can be improved using a unigram and bigram similarity metric when training a learner to select sentences for summarization. Learners are trained to summarize document clusters based on various algorithms and reward functions and then evaluated using ROUGE. Our experiments show a statistically significant improvement of 1.33%, 1.58%, and 2.25% for ROUGE-1, ROUGE-2 and ROUGE-L scores, respectively, when compared with the performance of the state of the art in automatic summarization with reinforcement learning on the DUC2004 dataset. Furthermore query focused extensions of our approach show an improvement of 1.37% and 2.31% for ROUGE-2 and ROUGE-SU4 respectively over query focused extensions of the state of the art with reinforcement learning on the DUC2006 dataset.

BanditSum: Extractive Summarization as a Contextual Bandit

“…The use of reinforcement learning (RL) in extractive summarization was first explored by Ryang and Abekawa (2012), who proposed to use the TD(λ) algorithm to learn a value function for sentence selection. Rioux et al (2014) improved this framework by replacing the learning agent with another TD(λ) algorithm. However, the performance of their methods was limited by the use of shallow function approximators, which required performing a fresh round of reinforcement learning for every new document to be summarized.…”

Preference-based interactive multi-document summarisation

“…(8)) α = 10 −3 learning rate for preference learning φ(y, x) vectorised representation of summary y for document cluster x (see Eq. (8)); we use the same vector representation as Rioux et al (2014)…”

“…[0,10]). For the vector representation φ, we use the same 200-dimensional bag-of-bigram representation as Rioux et al (2014) (w d = 1) .288 * .297 * .319 * den (we = 1)…”

“…The reward function R returns an evaluation score of the summary once the action terminate is performed; otherwise it returns 0 because the summary is still under construction and thus not ready to be evaluated (so-called delayed rewards). Providing non-zero rewards before the action terminate can lead to even worse result, as reported by Rioux et al (2014). The terminal states set T includes all states corresponding to summaries exceeding the given length requirement and an absorbing state s T .…”

Scientific Article Summarization Using Citation-Context and Article's Discourse Structure

“…Many works have viewed the summarization problem as a supervised classification problem in which several features are used to predict the inclusion of document sentences in the summary. Variations of supervised models have been utilized for summary generation, such as: maximum entropy (Osborne, 2002), HMM (Conroy et al, 2011), CRF (Galley, 2006;Shen et al, 2007;Chali and Hasan, 2012), SVM (Xie and Liu, 2010), logistic regression (Louis et al, 2010) and reinforcement learning (Rioux et al, 2014). Problems with supervised models in context of summarization include the need for large amount of annotated data and domain dependency.…”