Sentence encoders based on the transformer architecture have shown promising results on various natural language tasks. The main impetus lies in the pre-trained neural language models that capture long-range dependencies among words, owing to multi-head attention that is unique in the architecture. However, little is known for how linguistic properties are processed, represented, and utilized for downstream tasks among hundreds of attention heads inside the pre-trained transformerbased model. For the initial goal of examining the roles of attention heads in handling a set of linguistic features, we conducted a set of experiments with ten probing tasks and three downstream tasks on four pre-trained transformer families (GPT, GPT2, BERT, and ELECTRA). Meaningful insights are shown through the lens of heat map visualization and utilized to propose a relatively simple sentence representation method that takes advantage of most influential attention heads, resulting in additional performance improvements on the downstream tasks.
Recent reports have suggested an association between rotavirus infection and a distinctive pattern of white matter injury (WMI) in neonates with seizures; however, the connection between the two is not fully understood. To evaluate the underlying mechanism, we profiled and compared eight cytokines (IL [interleukin]-1β, IL-6, IL-8, IL-10, IFN-γ [interferon-γ ], MCP-1 [monocyte chemoattractant protein-1], MIP-1β [macrophage inflammatory protein-1β], and TNF-α [tumor necrosis factor-α]) in the cerebrospinal fluid (CSF) of 33 neonates with seizures who had no other well-known causes of seizures and 13 control patients (rotavirus-induced gastroenteritis but without seizures). Among the 33 neonates with seizures, 9 showed WMI and all were infected with rotavirus (R + W + ). Among the 24 patients without WMI, 11 were infected with rotavirus (R + W − ) and 13 were not (R − W − ).Only MCP-1 and MIP-1β were different between the groups. MCP-1 was increased in R+ W+ compared with R + W− (p < 0.01), R − W− (p < 0.01), and control (p = 0.03) patients. MIP-1β was decreased in R + W+ compared with R − W− (p < 0.01) and control (p < 0.01), but not R + W− (p = 0.23) patients. MCP-1 and MIP-1β are C-C chemokines that recruit immune cells to the site of inflammation. Our pilot study suggests MCP-1-mediated monocyte recruitment may be linked with this complication caused by rotavirus.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.