Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision

Tan, Hao; Bansal, Mohit

doi:10.18653/v1/2020.emnlp-main.162

Cited by 61 publications

(90 citation statements)

References 47 publications

(47 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, while predicting words across multiple timescales may be an effective learning objective for language acquisition, it is by no means the only feasible objective. Thus, it is likely that the brain relies on additional simple objectives at different timescales to facilitate learning 59, 63 .…”

Section: Discussionmentioning

confidence: 99%

“…Future studies, however, should assess whether these cognitively plausible, prediction-based, feedback signals are indeed available at a young age as we learn language, and whether the brain can use such predictive signals to guide language acquisition. Further, while next-word prediction may be an effective learning objective for language acquisition, it is not the only feasible objective-the brain may optimize additional simple objectives, at different timescales, to facilitate learning 18,62 .…”

Section: Next-word Predictionmentioning

confidence: 99%

See 1 more Smart Citation

Thinking ahead: spontaneous prediction in context as a keystone of language in humans and machines

Goldstein

Zada

Buchnik

et al. 2020

Preprint

View full text Add to dashboard Cite

Departing from rule-based linguistic models, advances in deep learning resulted in a new type of autoregressive deep language models (DLMs). These models are trained using a self-supervised next word prediction task. We provide empirical evidence for the connection between autoregressive DLMs and the human language faculty using spoken narrative and electrocorticographic recordings. Behaviorally, we demonstrate that humans have a remarkable capacity for word prediction in natural contexts, and that, given a sufficient context window, DLMs can attain human-level prediction performance. Leveraging on DLM embeddings we demonstrate that the brain constantly and spontaneously predicts the identity of the next word in natural speech, hundreds of milliseconds before they are perceived. Finally, we demonstrate that contextual embeddings derived from autoregressive DLMs capture neural representations of the unique, context-specific meaning of words in the narrative. Our findings suggest that DLMs provides a novel biologically feasible computational framework for studying the neural basis of language.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Next-word Predictionmentioning

confidence: 99%

Thinking ahead: spontaneous prediction in context as a keystone of language in humans and machines

Goldstein

Zada

Buchnik

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Distributional evidence used in extrapolations presented and quoted in this paper belongs to the World Scope of the written world. Bringing in perceptual grounding has-to some extent-been done in the visually supervised language model (Tan & Bansal, 2020), with work on interactive and interpersonal grounding largely still to follow (Bisk et al, 2020, p. 8721-8725).…”

Section: Discussionmentioning

confidence: 99%

The Croatian psycholinguistic database: Estimates for 6000 nouns, verbs, adjectives and adverbs

et al. 2021

View full text Add to dashboard Cite

Psycholinguistic databases containing ratings of concreteness, imageability, age of acquisition, and subjective frequency are used in psycholinguistic and neurolinguistic studies which require words as stimuli. Linguistic characteristics (e.g. word length, corpus frequency) are frequently coded, but word class is seldom systematically treated, although there are indications of its significance for imageability and concreteness. This paper presents the Croatian Psycholinguistic Database (CPD; available at: 10.17234/megahr.2019.hpb), containing 6000 Croatian nouns, verbs, adjectives and adverbs, rated for concreteness, imageability, age of acquisition, and subjective frequency. Moreover, we present computationally obtained extrapolations of concreteness and imageability to the remainder of the Croatian lexicon (available at: https://github.com/megahr/lexicon/blob/master/predictions/hr_c_i.predictions.txt). In the two studies presented here, we explore the significance of word class for concreteness and imageability in human and computationally obtained ratings. The observed correlations in the CPD indicate correspondences between psycholinguistic measures expected from the literature. Word classes exhibit differences in subjective frequency, age of acquisition, concreteness and imageability, with significant differences between nouns, verbs, adjectives and adverbs. In the computational study which focused on concreteness and imageability, concreteness obtained higher correlations with human ratings than imageability, and the system underpredicted the concreteness of nouns, and overpredicted the concreteness of adjectives and adverbs. Overall, this suggests that word class contains schematic conceptual and distributional information. Schematic conceptual content seems to be more significant in human ratings of concreteness and less significant in computationally obtained ratings, where distributional information seems to play a more significant role. This suggests that word class differences should be theoretically explored.

show abstract

“…We refer interested readers in this research direction to Brisk et al [156]. Vision-language learning has recently improved GLUE benchmark performance [157]. ConVIRT [97] is an interesting direction to bring a language "grounding" to medical image processing.…”

Section: Discussionmentioning

confidence: 99%

Deep Learning applications for COVID-19

2021

View full text Add to dashboard Cite

This survey explores how Deep Learning has battled the COVID-19 pandemic and provides directions for future research on COVID-19. We cover Deep Learning applications in Natural Language Processing, Computer Vision, Life Sciences, and Epidemiology. We describe how each of these applications vary with the availability of big data and how learning tasks are constructed. We begin by evaluating the current state of Deep Learning and conclude with key limitations of Deep Learning for COVID-19 applications. These limitations include Interpretability, Generalization Metrics, Learning from Limited Labeled Data, and Data Privacy. Natural Language Processing applications include mining COVID-19 research for Information Retrieval and Question Answering, as well as Misinformation Detection, and Public Sentiment Analysis. Computer Vision applications cover Medical Image Analysis, Ambient Intelligence, and Vision-based Robotics. Within Life Sciences, our survey looks at how Deep Learning can be applied to Precision Diagnostics, Protein Structure Prediction, and Drug Repurposing. Deep Learning has additionally been utilized in Spread Forecasting for Epidemiology. Our literature review has found many examples of Deep Learning systems to fight COVID-19. We hope that this survey will help accelerate the use of Deep Learning for COVID-19 research.

show abstract

Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision

Cited by 61 publications

References 47 publications

Thinking ahead: spontaneous prediction in context as a keystone of language in humans and machines

Thinking ahead: spontaneous prediction in context as a keystone of language in humans and machines

The Croatian psycholinguistic database: Estimates for 6000 nouns, verbs, adjectives and adverbs

Deep Learning applications for COVID-19

Contact Info

Product

Resources

About