DSTC7 Task 1: Noetic End-to-End Response Selection

Gunasekara, Chulaka; Kummerfeld, Jonathan K.; Polymenakos, Lazaros; Lasecki, Walter S.

doi:10.18653/v1/w19-4107

Cited by 36 publications

(30 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is Recall@1 using 99 responses sampled from the test dataset as negatives. This 1-of-100 accuracy metric has been used in previous studies: ( Al-Rfou et al, 2016;Henderson et al, 2017;Kumar et al, 2018;Gunasekara et al, 2019). While there is no guarantee that the 99 randomly selected negatives will all be bad responses, the metric nevertheless provides a simple summary of model performance that has been shown to correlate with user-driven quality metrics (Henderson et al, 2017).…”

Section: Response Selection Taskmentioning

confidence: 99%

A Repository of Conversational Datasets

Henderson¹,

Budzianowski²,

Casanueva³

et al. 2019

Proceedings of the First Workshop on NLP for Conversational AI

View full text Add to dashboard Cite

Progress in Machine Learning is often driven by the availability of large datasets, and consistent evaluation metrics for comparing modeling approaches. To this end, we present a repository of conversational datasets consisting of hundreds of millions of examples, and a standardised evaluation procedure for conversational response selection models using 1-of-100 accuracy. The repository contains scripts that allow researchers to reproduce the standard datasets, or to adapt the pre-processing and data filtering steps to their needs. We introduce and evaluate several competitive baselines for conversational response selection, whose implementations are shared in the repository, as well as a neural encoder model that is trained on the entire training set.

show abstract

Section: Response Selection Taskmentioning

confidence: 99%

A Repository of Conversational Datasets

Henderson¹,

Budzianowski²,

Casanueva³

et al. 2019

Proceedings of the First Workshop on NLP for Conversational AI

View full text Add to dashboard Cite

show abstract

“…To compare with other systems, Table 4 presents the official scores for each team which submitted results for all 8 subtasks of the DSTC7 response selection track. More details can be found in (Gunasekara et al, 2019a). Among 8 subtasks in total, our results (Team 3) rank top 1 on 7 subtasks, rank the second best on subtask 2 of Ubuntu, and overall rank top 1 on both datasets of the response se-lection challenge 4 .…”

Section: Dstc7 Resultsmentioning

confidence: 92%

“…Recall@N is used by the challenge organizers following Lowe et al (2015), which counts how often the correct answer is within the top N specified by a system. For DSTC7 results in this paper, N is set to 1, 10, 50, due to the large candidate set (100 candidates) (Gunasekara et al, 2019b). Mean Reciprocal Rank (MRR) as a widely used metric from the ranking literature is also used by the challenge organizers (Gunasekara et al, 2019b).…”

Section: Dstc7 Resultsmentioning

confidence: 99%

“…For DSTC7 results in this paper, N is set to 1, 10, 50, due to the large candidate set (100 candidates) (Gunasekara et al, 2019b). Mean Reciprocal Rank (MRR) as a widely used metric from the ranking literature is also used by the challenge organizers (Gunasekara et al, 2019b).…”

Section: Dstc7 Resultsmentioning

confidence: 99%

“…The official DSTC7 noetic end-to-end response selection track results cited from(Gunasekara et al, 2019a). Teams which submitted results for all subtasks are shown here.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Sequential neural networks for noetic end-to-end response selection

Chen

Wang

2020

Computer Speech & Language

View full text Add to dashboard Cite

The noetic end-to-end response selection challenge as one track in the 7th Dialog System Technology Challenges (DSTC7) aims to push the state of the art of utterance classification for real world goal-oriented dialog systems, for which participants need to select the correct next utterances from a set of candidates for the multi-turn context. This paper presents our systems that are ranked top 1 on both datasets under this challenge, one focused and small (Advising) and the other more diverse and large (Ubuntu). Previous state-of-the-art models use hierarchybased (utterance-level and token-level) neural networks to explicitly model the interactions among different turns' utterances for context modeling. In this paper, we investigate a sequential matching model based only on chain sequence for multi-turn response selection. Our results demonstrate that the potentials of sequential matching approaches have not yet been fully exploited in the past for multi-turn response selection. In addition to ranking top 1 in the challenge, the proposed model outperforms all previous models, including state-of-the-art hierarchy-based models, on two large-scale public multi-turn response selection benchmark datasets. Keywords:DSTC7, response selection, ESIM, BERT, end-to-end, sequential matching approaches 1. We develop an Enhanced Sequential Inference Model (ESIM) based system for the DSTC7 noetic end-to-end response selection track. On top of the ESIM model, we explore methods for exploiting multiple word embeddings, heuristic data augmentation, tuning the ratio between positive and negative samples, and emphasizing the importance of the most recent context utterances. 2. We propose a two-step approach for selecting the next utterance from a large amount of candidates (i.e., for subtask 2 on the Ubuntu dataset, we need to select the next utterance from a candidate pool of 120,000 sentences), by first using a sentence-encoding based method to select the top N candidates from the large set of candidates and then reranking them using ESIM, achieving a high performance with an acceptable overall computational cost. 3. We conduct systematic ablation analysis of the above-mentioned methods for enhancing the ESIM model performance. In particular, we develop effective and efficient model ensemble by averaging the output from models

show abstract

Dialog Modelling Experiments with Finnish One-to-One Chat Data

Kauttonen

Aunimo

2020

Communications in Computer and Information Science

View full text Add to dashboard Cite

DSTC7 Task 1: Noetic End-to-End Response Selection

Cited by 36 publications

References 8 publications

A Repository of Conversational Datasets

A Repository of Conversational Datasets

Sequential neural networks for noetic end-to-end response selection

Dialog Modelling Experiments with Finnish One-to-One Chat Data

Contact Info

Product

Resources

About