Generating Subjective Responses to Opinionated Articles in Social Media: An Agenda-Driven Architecture and a Turing-Like Test

Abstract

Natural language traffic in social media (blogs, microblogs, talkbacks) enjoys vast monitoring and analysis efforts. However, the question whether computer systems can generate such content in order to effectively interact with humans has been only sparsely attended to. This paper presents an architecture for generating subjective responses to opinionated articles based on users' agenda, documents' topics, sentiments and a knowledge graph. We present an empirical evaluation method for quantifying the humanlikeness and relevance of the generated responses. We show that responses generated using world knowledge in the input are regarded as more human-like than those that rely on topic, sentiment and agenda only, whereas the use of world knowledge does not affect perceived relevance.

Data-Driven Broad-Coverage Grammars for Opinionated Natural Language Generation (ONLG)

“…Set in an open domain, it is not trivial to find a "gold-standard" for this task, or even a method to obtain one. Our evaluation thus follows two tracks: an automated assessment track, where we quantitatively assess the responses, and a Turing-like test similar to that of Cagan et al (2014), where we aim to gauge human-likeness and response relevance. 1 Here we use Microsoft's WebLM API which is part of the Microsoft Oxford Project (Microsoft, 2011). Materials.…”

“…Such a discrepancy need not be very surprising, as noted by others before (Belz and Reiter, 2006). Cagan et al (2014) show that there are extra-grammatical factors affecting human-likeness, e.g. world knowledge.…”

“…Let d be a document containing a single article, and let a be a user agenda as in Cagan et al (2014). Specifically, a user agenda a can consist of one or more pairs of a topic (represented by a weighted bag-of-words) and an associated sentiment.…”

“…Cagan et al (2014) and Li et al (2017)). Our evaluation shows that while relational realizational (RR) inspired grammars (Tsarfaty and Sima'an, 2008) get good language model scores, simple head-driven lexicalized grammars a la Collins (2003) get better human rating and are more sensitive to sentiment.…”

“…In previous work we addressed the ONLG challenge using a template-based approach (Cagan et al, 2014). The proposed system generated subjective responses to articles, driven by user agendas.…”