Transformer language models have shown remarkable ability in detecting when a word is anomalous in context, but likelihood scores offer no information about the cause of the anomaly. In this work, we use Gaussian models for density estimation at intermediate layers of three language models (BERT, RoBERTa, and XLNet), and evaluate our method on BLiMP, a grammaticality judgement benchmark. In lower layers, surprisal is highly correlated to low token frequency, but this correlation diminishes in upper layers. Next, we gather datasets of morphosyntactic, semantic, and commonsense anomalies from psycholinguistic studies; we find that the best performing model RoBERTa exhibits surprisal in earlier layers when the anomaly is morphosyntactic than when it is semantic, while commonsense anomalies do not exhibit surprisal at any intermediate layer. These results suggest that language models employ separate mechanisms to detect different types of linguistic anomalies.
Abstract. This paper presents the potential of nonlinear and linear versions of an observation operator for simulating polarimetric variables observed by weather radars. These variables, deduced from the horizontally and vertically polarized backscattered radiations, give information about the shape, the phase and the distributions of hydrometeors. Different studies in observation space are presented as a first step toward their inclusion in a variational data assimilation context, which is not treated here. Input variables are prognostic variables forecasted by the AROME-France numerical weather prediction (NWP) model at convective scale, including liquid and solid hydrometeor contents. A nonlinear observation operator, based on the T-matrix method, allows us to simulate the horizontal and the vertical reflectivities (ZHH and ZVV), the differential reflectivity ZDR, the specific differential phase KDP and the co-polar correlation coefficient ρHV. To assess the uncertainty of such simulations, perturbations have been applied to input parameters of the operator, such as dielectric constant, shape and orientation of the scatterers. Statistics of innovations, defined by the difference between simulated and observed values, are then performed. After some specific filtering procedures, shapes close to a Gaussian distribution have been found for both reflectivities and for ZDR, contrary to KDP and ρHV. A linearized version of this observation operator has been obtained by its Jacobian matrix estimated with the finite difference method. This step allows us to study the sensitivity of polarimetric variables to hydrometeor content perturbations, in the model geometry as well as in the radar one. The polarimetric variables ZHH and ZDR appear to be good candidates for hydrometeor initialization, while KDP seems to be useful only for rain contents. Due to the weak sensitivity of ρHV, its use in data assimilation is expected to be very challenging.
This article presents a counter-argument to Sauerland's (2002) claim that the present tense is vacuous. Sauerland's conclusion is based on the premise that one cannot account for the felicity conditions of sentences like Every Tuesday this month, I fast if one assumes that the present tense denotes the time of utterance, or presupposes that its reference overlaps with the time of utterance. In a first step of the counter-argument, I show that Sauerland's anti-presuppositional analysis of the present tense makes incorrect predictions with simple present sentences in certain contexts of utterance. In a second step of the counter-argument, I show that this premise is false: a non-vacuous analysis of the present tense in Sauerland's examples is possible, once we acknowledge that they are interpreted either as futurates or as habituals. I conclude that the nonvacuous analysis of the present has the upper-hand. Incidentally, I propose a modal analysis of futurates that derives their temporal orientation from Condoravdi's (2002) Diversity Condition on metaphysical modality and that is argued to be superior to existing analyses of futurity.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.