Evaluating Learning Language Representations

Abstract

Abstract. Machine learning offers significant benefits for systems that process and understand natural language: a) lower maintenance and upkeep costs than when using manually-constructed resources, b) easier portability to new domains, tasks, or languages, and c) robust and timely adaptation to situation-specific settings. However, the behaviour of an adaptive system is less predictable than when using an edited, stable resource, which makes quality control a continuous issue. This paper proposes an evaluation benchmark for measuring the quality, coverage, and stability of a natural language system as it learns word meaning. Inspired by existing tests for human vocabulary learning, we outline measures for the quality of semantic word representations, such as when learning word embeddings or other distributed representations. These measures highlight differences between the types of underlying learning processes as systems ingest progressively more data.

Plausibility Testing for Lexical Resources

“…Evaluating learning lexical resources is challenging for several reasons [1]. Firstly, operationalizable intrinsic measures for knowledge based models risk being irrelevant for system performance or measure outcome, rather than learning process.…”

“…To generate this task, we create a set of sentences containing a number of sentences which occurred naturally and have remained unaltered, as well as a number of sentences which occurred naturally but have had one or more words within them swapped for some other word. We call this set a coconut, as per examples given by Karlgren et al [1]. The task is then to sort these sentences in order of likelihood of having occurred naturally.…”